DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
i. Claims 1, 5, 8, 10, 11, 15, 18, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Mullins (2018/0005440) in view of Yun (2015/0269783).
Regarding claim 1, Mullins discloses a first electronic device (Figures 1, 11: Comprising 106) comprising:
a sensor ([0019]: Comprising sensors);
a display ([0019]: Comprising transparent display);
a transceiver (Comprising 1124; [0060]);
memory (Comprising at least one of 1110, 1122), comprising one or more storage media, storing instructions [0058]; and
at least one processor (Comprising 1104) communicatively coupled to the sensor, the display [0019], the transceiver (Comprising 1124), and the memory (Comprising at least one 1110, 1122), wherein the instructions, when executed by the at least one processor individually or collectively, cause the first electronic device [0021] to:
receive, from a second electronic device (Comprising 108), first application information associated with a first application executed on the second electronic device (e.g. non-AR application on desktop or tablet [0030] communicates with AR application [0042] of the AR device [0023]),
in response to receiving the first application information, identify first user interface (UI) information associated with the first application information among UI information (Identification, discerned status [0036] encompassing user interface data generated by a non-AR application [0042]) stored in the memory ([0037]: Among datasets included in storage device), and
based on the first UI information, control the display to display at least one first UI for controlling the first application (Communicated user interface data of non-AR application [0042] forming basis for commands, controls rendered within context of the AR application [0044]) in a specified spatial area (Overlay with physical object [0023] including example of table [0035]).
Mullins does not explicitly disclose the device wherein the specified spatial area includes a spatial area set by a user within a real environment image obtained in real-time by the sensor.
In the same field of endeavor, Yun discloses a virtual input interface [0003] wherein the specified spatial area includes a spatial area set by a user within a real environment image obtained in real-time by the sensor (Size [0027] and shape [0128] among parameters of input region defined by recognized motion of a user’s drawing [0104]). This is among measures implemented to increase the accuracy of input in a virtual interface [0124].
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the specified spatial area includes a spatial area set by a user within a real environment image obtained in real-time by the sensor, in view of the teaching of Yun, to increase the accuracy of input in a virtual interface.
Regarding claim 5, Mullins in view of Yun discloses the first electronic device of claim 1.
Mullins does not explicitly disclose the device wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a drawing trajectory generated by the user in the real environment image obtained in real-time by the sensor, and determine an area corresponding to the identified drawing trajectory as a spatial area set by the user.
In the same field of endeavor, Yun discloses virtual input interface [0003] wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a drawing trajectory generated by the user in the real environment image obtained in real-time by the sensor, and determine an area corresponding to the identified drawing trajectory as a spatial area set by the user ([0126] – [0129]). This is among measures implemented to increase the accuracy of input in a virtual interface [0124].
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a drawing trajectory generated by the user in the real environment image obtained in real-time by the sensor, and determine an area corresponding to the identified drawing trajectory as a spatial area set by the user, in view of the teaching of Yun, to increase the accuracy of input in a virtual interface.
Regarding claim 8, Mullins in view of Yun discloses the first electronic device of claim 1. Mullins discloses the device wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: receive, from the second electronic device, second application information associated with a plurality of second applications executed on the second electronic device ([0024]: Second, differing one among 110, 112), in response to receiving the second application information, identify second UI information associated with the second application information among the pre-stored UI information ([0025]: AR API allows second, differing one among 110, 112 to make use of functionality of AR application), and control the display to display at least one second UI in the specified spatial area based on the second UI information, and wherein the at least one second UI includes at least one of at least one UI for controlling functions provided by each of the plurality of second applications at once, or a plurality of UIs for separately controlling functions provided by each of the plurality of second applications (Availability of AR application functionality to multiple applications [0025] encompassing the mapping of controls for each application [0044] for AI device display [0041]).
Regarding claim 10, Mullins in view of Yun discloses the first electronic device of claim 1.
Mullins does not explicitly disclose the device wherein the at least one first UI includes at least one UI for controlling at least one function provided by the first application using an input device of a fourth electronic device linked with the first electronic device.
In the same field of endeavor, Yun discloses a virtual input interface [0003] wherein the at least one first UI includes at least one UI for controlling at least one function provided by the first application using an input device of a fourth electronic device linked with the first electronic device ([0344] – [0351]). This is among measures implemented to increase accuracy of input in a virtual interface [0124].
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the at least one first UI includes at least one UI for controlling at least one function provided by the first application using an input device of a fourth electronic device linked with the first electronic device, in view of the teaching of Yun, to increase the accuracy of input in a virtual interface.
Regarding claim 11, Mullins discloses a method for providing a user interface (UI) [0001] by a first electronic device (Figures 1, 11: Comprising 106), the method comprising:
receiving, by the first electronic device (Comprising 106), from a second electronic device (Comprising 108), first application information associated with a first application executed on the second electronic device (e.g. non-AR application on desktop or tablet [0030] communicates with AR application [0042] of the AR device [0023]);
in response to receiving the first application information, identifying, by the first electronic device, first UI information associated with the first application information (Identification, discerned status [0036] encompassing user interface data generated by a non-AR application [0042]) among stored UI information ([0037]: Among datasets included in storage device); and
based on the first UI information, displaying, by the first electronic device, at least one first UI for controlling the first application (Communicated user interface data of non-AR application [0042] forming basis for commands, controls rendered within context of the AR application [0044]) in a specified spatial area (Overlay with physical object [0023] including example of table [0035]).
Mullins does not explicitly disclose the method wherein the specified spatial area includes a spatial area set by a user within a real environment image obtained in real-time by the first electronic device.
In the same field of endeavor, Yun discloses a virtual input interface [0003] wherein the specified spatial area includes a spatial area set by a user within a real environment image obtained in real-time by the first electronic device. This is among measures implemented to increase the accuracy of input in a virtual interface [0124].
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the specified spatial area includes a spatial area set by a user within a real environment image obtained in real-time by the first electronic device, in view of the teaching of Yun, to increase the accuracy of input in a virtual interface.
Method claims 15, 18 are rejected as reciting limitations similar to those recited in device claims 5, 8, respectively.
Media claim 19 is rejected as reciting limitations similar to those recited in method claim 11.
ii. Claims {a} 2, 3, {b} 12, 13, {c} 20 are rejected under 35 U.S.C. 103 as being unpatentable over Mullins in view of Yun, as respectively applied to claim 1, 11, 19 above, and further in view of Keenan et al. (2024/0019980; hereinafter Keenan; cited in Applicant’s 7 April 2026 IDS).
Regarding claim 2, Mullins in view of Yun discloses the first electronic device of claim 1. Mullins discloses the device wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: control the display to display a first screen associated with the first application (Figure 7: Comprising 708, 714, 716) based on screen information included in the first application information (Corresponding to 704, 706, 708), determine whether any one of the at least one first UI has been selected by the user ([0042], Figure 8: Comprising 810).
Mullins in view of Yun does not explicitly disclose the device wherein, based on any one of the at least one first UI being selected by the user, control the transceiver to transmit control information corresponding to the selected UI to the second electronic device, in response to transmitting the control information, receive updated screen information from the second electronic device through the transceiver, and based on receiving the updated screen information, control the display to change the first screen to a second screen corresponding to the updated screen information.
In the same field of endeavor, Keenan discloses mixed reality application control [0001] wherein based on any one of the at least one first UI being selected by the user, control the transceiver to transmit control information corresponding to the selected UI to the second electronic device, in response to transmitting the control information, receive updated screen information from the second electronic device through the transceiver, and based on receiving the updated screen information, control the display to change the first screen to a second screen corresponding to the updated screen information (Augmented reality interactions [0111], [0112] generate updates in the 2D application [0119]). This is among measures implemented to simplify device operation by medical personnel [0003].
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein, based on any one of the at least one first UI being selected by the user, control the transceiver to transmit control information corresponding to the selected UI to the second electronic device, in response to transmitting the control information, receive updated screen information from the second electronic device through the transceiver, and based on receiving the updated screen information, control the display to change the first screen to a second screen corresponding to the updated screen information, in view of the teaching of Keenan, to simplify device operation by medical personnel.
Regarding claim 3, Mullins in view of Yun discloses the first electronic device of claim 1.
Mullins in view of Yun does not explicitly disclose the device wherein the at least one first UI is generated based on user setting information provided from the second electronic device, and wherein the user setting information includes information indicating at least one control command selected by the user among one or more control commands for controlling the first application, and information indicating at least one of a name, an icon, a color, a font, or a visual effect set by the user corresponding to the at least one first UI.
In the same field of endeavor, Keenan discloses mixed reality application control [0001] wherein the at least one first UI is generated based on user setting information provided from the second electronic device, and wherein the user setting information includes information indicating at least one control command selected by the user among one or more control commands for controlling the first application, and information indicating at least one of a name, an icon, a color, a font, or a visual effect set by the user corresponding to the at least one first UI ([0096]: User labeling of respective ones among augmented reality user interaction buttons). This is among measures implemented to simplify device operation by medical personnel [0003].
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the at least one first UI is generated based on user setting information provided from the second electronic device, and wherein the user setting information includes information indicating at least one control command selected by the user among one or more control commands for controlling the first application, and information indicating at least one of a name, an icon, a color, a font, or a visual effect set by the user corresponding to the at least one first UI, in view of the teaching of Keenan, to simplify device operation by medical personnel.
Method claims 12, 13 are rejected as reciting limitations similar to those recited in device claims 2, 3, respectively.
Media claim 20 is rejected as reciting limitations similar to those recited in device claim 2.
iii. Claims 4, 14 are rejected under 35 U.S.C. 103 as being unpatentable over Mullins in view of Yun, as respectively applied to claim 1, 11 above, and further in view of Pla I Conesa et al. (2021/0303107; hereinafter Conesa).
Regarding claim 4, Mullins in view of Yun discloses the first electronic device of claim 1. Mullins discloses the device wherein the sensor is further configured to detect a gaze of the user [0037].
Mullins in view of Yun does not explicitly disclose the device wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a UI corresponding to the gaze of the user among the at least one first UI, and control the display to display a preview image associated with the identified UI.
In the same field of endeavor, Conesa discloses gaze based interface navigation [0002] wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a UI corresponding to the gaze of the user among the at least one first UI, and control the display to display a preview image associated with the identified UI [0132]. This is among measures implemented to make device interactions more efficient and intuitive [0005].
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a UI corresponding to the gaze of the user among the at least one first UI, and control the display to display a preview image associated with the identified UI, in view of the teaching of Conesa, to make device interactions more efficient and intuitive.
Method claim 14 is rejected as reciting limitations similar to those recited in device claim 4.
iv. Claims 6, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Mullins in view of Yun, as respectively applied to claim 1, 11 above, and further in view of Berliner et al. (2022/0253139; hereinafter Berliner).
Regarding claim 6, Mullins in view of Yun discloses the first electronic device of claim 1.
Mullins in view of Yun does not explicitly disclose the device wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: determine whether the at least one first UI is displayed overlapping at least one object within the real environment image, and based on the at least one first UI being displayed overlapping the at least one object, control the display to relocate and display the at least one first UI at a position not overlapping the at least one object.
In the same field of endeavor, Berliner discloses providing productivity applications in an extended reality environment [0002] wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: determine whether the at least one first UI is displayed overlapping at least one object within the real environment image, and based on the at least one first UI being displayed overlapping the at least one object, control the display to relocate and display the at least one first UI at a position not overlapping the at least one object [0815]. This is among measures implemented to avoid various obstruction forms ([0596], [0597]).
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: determine whether the at least one first UI is displayed overlapping at least one object within the real environment image, and based on the at least one first UI being displayed overlapping the at least one object, control the display to relocate and display the at least one first UI at a position not overlapping the at least one object, in view of the teaching of Berliner, to avoid various forms of interface obstructions.
Method claim 16 is rejected as reciting limitations similar to those recited in device claim 6.
v. Claims 7, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Mullins in view of Yun, as respectively applied to claim 1, 11 above, and further in view of Gupta et al. (6,553,461; hereinafter Gupta).
Regarding claim 7, Mullins in view of Yun discloses the first electronic device of claim 1.
Mullins in view of Yun does not explicitly disclose the device wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a UI selected by the user among the at least one first UI, based on the selected UI being associated with a large-capacity object having a capacity equal to or greater than a set capacity, obtain the large-capacity object stored in the memory, and control the display to display a screen based on the obtained large-capacity object, and wherein the large-capacity object includes at least one of an image, audio, or video provided from the second electronic device.
In the same field of endeavor, Gupta discloses resource fetching (col. 3, ll. 38 – 39) wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a UI selected by the user among the at least one first UI, based on the selected UI being associated with a large-capacity object having a capacity equal to or greater than a set capacity, obtain the large-capacity object stored in the memory (col. 3, ll. 20 – 37), and control the display to display a screen based on the obtained large-capacity object, and wherein the large-capacity object includes at least one of an image, audio, or video provided from the second electronic device (col. 3, ll. 39 – 45). This is among measures implemented to overcome previous systems’ waste of cache space (col. 1, ll. 39 – 41).
It would be obvious to one having ordinary skill in the art before the filing date of the claimed invention for the device of Mullins to be modified wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify a UI selected by the user among the at least one first UI, based on the selected UI being associated with a large-capacity object having a capacity equal to or greater than a set capacity, obtain the large-capacity object stored in the memory, and control the display to display a screen based on the obtained large-capacity object, and wherein the large-capacity object includes at least one of an image, audio, or video provided from the second electronic device, in view of the teaching of Gupta, to avoid the waste of cache space.
Method claim 17 is rejected as reciting limitations similar to those recited in device claim 7.
Allowable Subject Matter
Claim 9 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 9, Mullins in view of Yun discloses the first electronic device of claim 1.
The cited prior art fails to singularly or collectively disclose the device wherein the instructions, when executed by the at least one processor individually or collectively, further cause the first electronic device to: identify that at least one third UI associated with a third application executed on the second electronic device is displayed on a third electronic device linked with the second electronic device, based on the at least one third UI being displayed on the third electronic device, identify third UI information associated with the at least one third UI among the pre-stored UI information, and control the display to display at least one fourth UI in the specified spatial area based on the third UI information, and wherein the at least one fourth UI includes at least one UI for changing at least one of a size, a color, an icon, a font, or a visual effect of the at least one third UI displayed on the third electronic device.
Thus, claim 9 is objected to.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure follows:
Mullins (2015/0185825; [0001]) and Harvey et al. (2021/0103331; [0002]) provide relevant examples of input device implementation within virtual and augmented reality.
Inquirires
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaron Midkiff whose telephone number is (571)270-5875. The examiner can normally be reached Monday - Friday, 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr Awad can be reached at (571)272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON MIDKIFF/
Examiner, Art Unit 2621
/AMR A AWAD/Supervisory Patent Examiner, Art Unit 2621
