DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-9, 13-21, 28 and 30 are pending.
Title
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed: EXTENDED REALITY (XR) DEVICE MANAGEMENT USING EYE TRACKING SENSORS AND UI TRIGGER ZONE.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 6-8, 13-14, 16, 18-21 and 28 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Abbott, III et al. (US 2002/0044152 A1, IDS).
As to claim 1, Abbott discloses an apparatus for imaging (Abbott, FIG. 1, [0023], “computer 100” with “eyeglass-mounted display 118”), the apparatus comprising:
at least one memory (Abbott, FIG. 1, [0028], “memory 142”); and
at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) coupled to the at least one memory (Abbott, FIG. 1, [0028], “memory 142”) and configured to (Abbott, FIG. 1, [0028], “application programs 146”):
determine a direction of gaze of a user (Abbott, FIG. 1, [0030], “CDOS system 150 may be used to generate data indicating where the user is looking”) toward one or more displays of the apparatus (Abbott, FIGS. 2-3, [0034], “eyeglass mounted display 118”), wherein the direction of gaze is based on image data obtained using one or more cameras (Abbott, FIGS. 1-3, [0022], “user input devices 114 … pupil tracking devices”; [0056], “eye movement sensors, blink detection sensors”) included in the apparatus (Abbott, FIG. 1, [0023], “computer 100” with “eyeglass-mounted display 118”);
determine a region (Abbott, FIGS. 2-3, [0035], e.g., “menu 204”) of the one or more displays corresponding to the direction of gaze of the user (Abbott, FIGS. 2-3, [0077], “through eye tracking techniques, for example, the system can detect whether the user is looking at a specific virtual object”);
generate one or more graphical user interface (GUI) control actions indicative of a respective configuration of a GUI associated with the one or more displays (Abbott, FIG. 8, [0085], “the application program 146, and namely the transparent UI 148, determines how to best present the virtual information (block 804”), wherein the one or more GUI control actions are based on the determined region (Abbott, FIGS. 2-3, [0035], e.g., “menu 204 is placed along the side of the display to minimize any distraction from the user's vision of the real world”); and
output, using the one or more displays and based on the one or more GUI control actions, the respective configuration of the GUI (Abbott, FIG. 8, [0085], “based on this information, the transparent UI 148 might initially assign a degree of transparency and a location on the display (block 806)”).
As to claim 2, Abbott discloses the apparatus of claim 1, wherein, to determine the region of the one or more displays corresponding to the direction of gaze of the user (Abbott, FIG. 1, [0030], “CDOS system 150 may be used to generate data indicating where the user is looking”), the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
generate an indication of a UI trigger zone (Abbott, FIGS. 2-3, [0035], e.g., “menu 204”) based on the direction of gaze of the user (Abbott, FIGS. 2-3, [0044], e.g., “on the other hand, when the user returns their focus to the virtual information, the objects become visibly opaque”) corresponding to a first sub-area of a larger area of the one or more displays (Abbott, FIGS. 2-3, [0035], e.g., “along the side of the display”); or
generate an indication of a UI defocus zone (Abbott, e.g., FIGS. 2-6, [0078], “center position 602”; i.e., inside “the side of the display”) based on the direction of gaze of the user (Abbott, FIGS. 2-3, [0044], “when the user is focused on the real world, the virtual object's transparency increases as the user no longer focuses on the object”) corresponding to a second sub-area of the larger area of the one or more displays (Abbott, e.g., FIGS. 2-6, [0078], “center position 602”; i.e., inside “the side of the display”), wherein the second sub-area is non-overlapping with the first sub-area (Abbott, FIGS. 2-3, [0035], e.g., “the side of the display” is non-overlapping with the “inside of the side of the display” by definition).
As to claim 3, Abbott discloses the apparatus of claim 2, wherein the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is further configured to:
generate the indication of the UI trigger zone (Abbott, FIGS. 2-3, [0035], e.g., “menu 204”) based on the direction of gaze of the user (Abbot, FIGS. 2-3, [0045], “the transparency may further be configured to change over time”) corresponding to the first sub-area (Abbott, FIGS. 2-3, [0035], e.g., “along the side of the display”) for at least a first configured time duration (Abbot, FIGS. 2-3, [0045], “the window may then fade back into view when the user attention is returned to it”); or
generate the indication of the UI defocus zone (Abbott, e.g., FIGS. 2-6, [0078], “center position 602”; i.e., inside “the side of the display”) based on the direction of gaze of the user corresponding to the second sub-area (Abbott, e.g., FIGS. 2-6, [0078], “center position 602”; i.e., inside “the side of the display”) for at least a second configured time duration (Abbott, FIGS. 2-3, [0045], “an unused window can fade from view, becoming very transparent or perhaps eventually fully transparent, when the user maintains their focus elsewhere”).
As to claim 4, Abbott discloses the apparatus of claim 2, wherein, to output the respective configuration of the GUI, the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
display, based on the indication of the UI trigger zone (Abbott, FIGS. 2-3, [0035], e.g., “menu 204”), one or more GUI elements of a plurality of GUI elements included in the GUI (Abbott, FIGS. 2-3, [0035], e.g., “the menu items include mapping, email, communication, body parameters, and geographical location”).
As to claim 6, Abbott discloses the apparatus of claim 4, wherein the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
display, based on the indication of the UI trigger zone, one or more GUI events or notifications (Abbott, FIGS. 2-3, [0053], e.g., “The watermark notification also functions as an active control that may be selected by the user to control an underlying application. When the user looks at the watermark image, or in some other way selects the image, it becomes visibly opaque”);
wherein the one or more GUI events or notifications are output based on the direction of gaze of the user corresponding to a particular location within the first sub-area corresponding to the UI trigger zone (Abbott, FIGS. 2-3, [0053], e.g., “The user's method for selecting the image includes any of the various ways a user of a wearable personal computer can perform selections of graphical objects (e.g., blinking, voice selection, etc.)”).
As to claim 7, Abbott discloses the apparatus of claim 2, wherein, to output the respective configuration of the GUI, the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
display, based on the indication of the UI defocus zone, one or more GUI elements of a plurality of GUI elements (Abbott, FIGS. 2-3, [0035], e.g., “the menu items include mapping, email, communication, body parameters, and geographical location”) included in the GUI, wherein each GUI element of the one or more GUI elements is dimmed or defocused (Abbott, FIGS. 2-3, [0050], “The envelope icon is illustrated in dashed lines around the edge of the full display to demonstrate that the icon is faintly visible (or highly transparent) to avoid obscuring the view of the mountain range”).
As to claim 8, Abbott discloses the apparatus of claim 7, wherein, to output the respective configuration of the GUI, the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is further configured to disable one or more GUI elements of the plurality of GUI elements (Abbott, FIG. 6, [0078], “Here, the user is focusing on the compass to get a bearing before scaling the mountain. When the user returns their attention to the climbing task and focuses once again on the real world 202, the eye tracking feedback is given to the application program, which slowly migrates the compass 600 from its center position to a peripheral location 604 as illustrated by the direction arrow 606”).
As to claim 13, Abbott discloses the apparatus of claim 1, wherein, to generate the one or more GUI control actions, the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
determine gaze direction information using an eye tracking framework associated with the one or more cameras (Abbott, FIG. 1, [0030], “CDOS system 150 may be used to generate data indicating where the user is looking”; ); and
generate the one or more GUI control actions using a GUI management heuristic (Abbott, [0076], “This behavior can be configured by the user, or alternatively, the system can track eye focus to dynamically and automatically adjust the visibility of virtual information without occluding too much of the real world”; FIG. 8, [0086], “The system then monitors the user behavior and conditions that gave rise to presentation of the virtual information (block 808)”).
As to claim 14, Abbott discloses the apparatus of claim 1, wherein:
the apparatus (Abbott, FIG. 1, [0023], “computer 100” with “eyeglass-mounted display 118”) is an extended reality (XR) glasses device (Abbott, see FIG. 1);
the one or more displays (Abbott, FIGS. 2-3, [0034], “eyeglass mounted display 118”) comprise one or more transparent panes of the XR glasses device (Abbott, e.g., see FIGS. 1-3, [0035], “transparent screen presentation 200”); and
the GUI comprises a respective overlay rendered on each transparent pane of the one or more transparent panes (Abbott, FIGS. 2-3, [0023], “is implemented as a display type that allows the user to view real world images from their surroundings while simultaneously overlaying or otherwise presenting computer-generated information to the user in an unobtrusive manner”).
As to claim 16, Abbott discloses the apparatus of claim 1, wherein the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is further configured to:
obtain multimodal sensor data associated with one or more sensors included in the apparatus (Abbott, e.g., [0056], “When the wearable computer 100 is equipped with context aware components (e.g., eye movement sensors, blink detection sensors, head movement sensors, GPS systems, and the like), the application program 146 may be provided with context data that influences how the virtual information is presented to the user via the transparent UI”; [0076], “Then, an opposite eye-blink would give prominence to the real-world view, instead of the virtual-world view. Alternatively, the user can direct the system to give prominence to a specific view by issuing a voice command. The user can tell the system to increase or decrease transparency of the virtual world or virtual objects”);
determine, based on the multimodal sensor data, one or more of a current activity or a current state associated with the user (Abbott, e.g., FIG. 8, [0086], “monitor user behavior, context, and conditions 808” →”change? 810” → “Yes”); and
generate one or more updated GUI control actions based on one or more of the current activity or the current state, wherein each respective updated GUI control action of the one or more updated GUI control actions is indicative of a corresponding updated configuration for the GUI (Abbott, e.g., FIG. 8, [0086], “change transparency and/or modify prominence 812”).
As to claim 18, it differs from claim 1 only in that it is the method performed by the apparatus of claim 1. It recites substantially the same limitations as in claim 1, and Abbott discloses them. Please see claim 1 for detailed analysis.
As to claims 19-21 and 28, they recite substantially the same limitations as in claims 2-4 and 16, respectively, and Abbott disclose them. Please see claims 2-4 and 16 for detailed analysis.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Abbott, III et al. (US 2002/0044152 A1, IDS) in view of Beans et al. (US 2016/0155267 A1, IDS).
As to claim 5, Abbott does not teach the apparatus of claim 4, wherein the one or more GUI elements comprise a subset of the plurality of GUI elements corresponding to a particular sub-area of a larger area of the one or more displays, and wherein the direction of gaze of the user is detected within the particular sub-area.
However, Beans teaches the concept of a subset of the plurality of GUI elements (Bean, FIG. 5A, [0061], “visual element(s) 600”) corresponding to a particular sub-area of a larger area of the one or more displays, and wherein the direction of gaze of the user is detected within the particular sub-area (Bean, FIG. 5A, [0061], “Based on such a conclusion, the processing unit 540 generates the control signal so as to modify the transparency of the visual element(s) 600 to substantially 0%. In other words, the processing unit 540 controls the visual element(s) to be entirely visible (i.e. not see-through or invisible) so that they are shown to the user in the transparent display area 520”; FIG. 6, “determine offset distance 770”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the “menu 204” taught by Abbott to further comprise the “visual element(s) 600” to be detected within, e.g., the “offset distance 770”, as taught by Bean, in order to provide more “user-friendly augmented reality (AR)” (Beans, [0003]).
As to claim 15, Abbott in view of Beans teaches the apparatus of claim 14, wherein the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
determine a first direction of gaze corresponding to a left eye of the user, based on image data associated with a left eye tracking camera of the XR glasses device; determine a second direction of gaze corresponding to a right eye of the user, based on image data associated with a right eye tracking camera of the XR glasses device; intersect the first direction of gaze with a left transparent pane of the XR glasses device to determine a region corresponding to the first direction of gaze; and intersect the second direction of gaze with a right transparent pane of the XR glasses device to determine a region corresponding to the second direction of gaze (Bean, see FIGS. 2 and 5, [0053], “This may provide, for example, a direction that each eye of user is looking in which can then be employed in 3D model to determine a point or location where rays tracing the directions intersect. Put another way, imaginary rays 580 can be projected in a direction substantially perpendicular to the pupil of the respective eye so as to indicate a viewing direction of the eye. A location of intersection of the imaginary rays projected from the driver's left and right eyes, respectively, may then be determined to be the point of focus P of the driver. Thus, by determining where such intersection occurs, the visual focus (e.g. focal point) of the user in 3D space can be obtained”). Examiner renders the same motivation as in claim 5.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Abbott, III et al. (US 2002/0044152 A1, IDS) in view of Hwang (US 2014/0333521 A1).
As to claim 9, Abbott teaches the apparatus of claim 7, wherein, to output the respective configuration of the GUI, the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
dim or defocus the one or more GUI elements based on the direction of gaze of the user corresponding to the UI defocus zone (Abbott, FIGS. 2-3, [0045], e.g., “when the user maintains their focus elsewhere”, e.g., “center position 602”) for at least a first configured time duration (Abbott, FIGS. 2-3, [0045], “an unused window, i.e., menu 600, can fade from view, becoming very transparent or perhaps eventually fully transparent”); and
disable the one or more GUI elements based on the direction of gaze of the user corresponding to the UI defocus zone (Abbott, FIG. 6, [0078], “Here, the user is focusing on the compass to get a bearing before scaling the mountain. When the user returns their attention to the climbing task and focuses once again on the real world 202, the eye tracking feedback is given to the application program, which slowly migrates the compass 600 from its center position to a peripheral location 604 as illustrated by the direction arrow 606”) for at least a certain configured time duration (Abbott, FIG. 8, [0083], “The system then monitors the user behavior and conditions that gave rise to presentation of the virtual information (block 808). Based on this monitoring or in response to express user commands, the system determines whether a change in transparency or prominence is justified (block 810)”).
Abbott does not teach the certain configured time duration to be a “second configured time duration”; and “wherein the second configured time duration is greater than the first configured time duration”.
However, Hwang teaches the concept that the second configured time duration is greater than the first configured time duration (Hwang, FIGS. 5A-5B, [0067], “By way of example, but not limitation, as depicted in FIG. 5A, a transparency of information area 130 is changed, if the gaze point exists at the position for more than a predetermined threshold time duration. Further, by way of another example, as depicted in FIG. 5B, a transparency of information area 130 is changed in an S-shaped curve, if the gaze point exists at the position for more than the predetermined threshold time duration”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the steps of “dimming” and “disabling” to be comprise the “predetermined threshold time duration”, respectively, as taught by Hwang, in order to provide “changing a display property of the information element based at least in part on the activity of the at least one pupil” (Hwang, [0004]).
Claims 17 and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Abbott, III et al. (US 2002/0044152 A1, IDS) in view of Mullins et al. (US 2016/0231573 A1).
As to claim 17, Abbott teaches the apparatus of claim 16, wherein the at least one processor (Abbott, FIG. 1, [0028], “CPU 140”) is configured to:
display one or more GUI elements (Abbott, FIGS. 2-3, [0035], e.g., “the menu items include mapping, email, communication, body parameters, and geographical location”) corresponding to the current activity or current state associated with the user (Abbott, e.g., see FIGS. 2-5); and
adjust a brightness or transparency level of virtual content rendered on the one or more displays based on multimodal sensor data (Abbott, e.g., FIG. 8, [0086], “change transparency and/or modify prominence 812”).
Abbott does not teach the multimodal sensor data to be “associated with one or more of an ambient sensor or an ambient temperature sensor: or apply one or more color tone transformations for a corresponding one or more GUI elements based on environmental conditions determined from the multimodal sensor data”.
However, Mullins teaches the concept of the sensor data to be associated with one or more of an ambient light sensor (Mullins, FIG. 3, [0039], “ambient light sensor 302”; FIG. 6, [0054-0056], “determine ambient light 602” → “adjust lighting to computed light output 608”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the steps of “changing transparency and/or modify prominence 812” to be further associated with the “determined ambient light 602”, as taught by Mullins, in order to provide “changing a display property of the information element based at least in part on the activity of the at least one pupil” (Hwang, [0004]).
As to claim 30, it recites substantially the same limitations as in claim 17, and Abbott in view of Mullins teaches them. Examiner renders the same motivation as in claim 17. Please see claim 17 for detailed analysis.
Conclusion
The prior arts made of record and not relied upon are considered pertinent to applicant’s disclosure:
Krasadakis (US 2017/0212583 A1, IDS) teaches the concept of “enabling a user to navigate between content using gaze tracking” (Abs.); and
Hamada (US 2024/0196065 A1) teaches the concept of “setting a region of interest and a region of non-interest in a display region … extracting a gaze object at which the user gazes” (Abs.).
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICHARD J HONG whose telephone number is (571) 270-7765. The examiner can normally be reached on 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chanh Nguyen can be reached on (571) 272-7772. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Jul. 22, 2026
/RICHARD J HONG/Primary Examiner, Art Unit 2623
***
Prosecution Insights