FINAL REJECTION, SECOND DETAILED ACTION
Status of Prosecution
The present application 18/110,323, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The Office mailed a first detailed action on January 16, 2026.
Applicant initiated an interview with Examiner on April 9, 2026. Applicant filed amendments with accompanying remarks and arguments on April 16, 2026.
Claims 137-180 are pending and are all rejected. Claims 137, 179 and 180 are independent. Claims 1-136 and 181-329 are cancelled by preliminary amendment.
Claims Status
Claims 137-145, 150, 152, 155-157, 160, 162-165, 172, 175 and 179-180 are rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003.
Claims 146, 177 and 178 are rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay in view of Lopez in further view of Gu et al. (“Gu”), United States Patent Application Publication 2018/0136716 published on May 17, 2018.
Claim 147-149 is rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay in view of Lopez in further view of Eledath et al. (“Eledath”), United States Patent Application Publication 2016/0378861 published on Dec. 29, 2016.
Claim 151 is rejected under 35 U.S.C. § 103 as being unpatentable over Lemelson in view of Lopez in further view of Kuehne et al. (“Kuehne”), United States Patent Application Publication 2018/0136716 published on May 17, 2018.
Claim 153 is rejected under 35 U.S.C. § 103 as being unpatentable over Lemelson in view of Lopez in further view of Shinohara et al. (“Shinohara”), United States Patent Application Publication 2018/0325483 published on Nov. 15, 2018.
Claims 144, 154, 158-159, 161, 166-171 and 173-174 are rejected under 35 U.S.C. § 103 as being unpatentable over Lemelson in view of Lopez in further view of George-Svahn et al. (“George-Svahn”), United States Patent Application Publication 2017/0235360 published on Aug. 17, 2017.
Claim 176 is rejected under 35 U.S.C. § 103 as being unpatentable over Lemelson in view of Lopez in further view of Clements, United States Patent 10,802,582, published on Oct. 13, 2020.
Response to Remarks and Arguments
Examiner thanks Applicant for the courtesies extended during the April 9, 2026 interview. Additionally, Examiner appreciates the remarks and arguments presented with the amended claims filed.
First, regarding the § 101 rejection to claim 180, Examiner withdraws based on the amended language.
Second, regarding the § 112(b) rejection to claims 177-178, Examiner has considered the arguments and reconsidered the claim language. Examiner is persuaded to withdraw, and also chooses to give an interpretation that the word “first” is merely a label that allows it to be distinguished from an (unclaimed) second level of proximity if needed.
Finally, regarding the prior art rejections, Examiner has newly adjusted the rejections as noted below.
The claims stand rejected.
Claim Rejections -- 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
A.
Claims 137-145, 150, 152, 155-157, 160, 162-165, 172, 175 and 179-180 are rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003.
As to Claim 137, Lindsay teaches: A method, including:
at a first computer system that is in communication with a first display generation component and one or more input devices (Lindsay: pars. 0002, 0019, the head mounted device [104] is a display device that has a display, a logic subsystem and is able to receive user input):
while a first view of an three-dimensional extended reality environment is visible via the first display generation component within a viewport that represents a portion of the three-dimensional extended reality environment that is visible to a user (Lindsay: Figs. 5A-5B, par. 0017, an augmented reality environment (i.e. three-dimensional extended reality environment) is visible within a field of view [106]), wherein the viewport is fixed with respect to the user’s head, detecting (Lindsay: pars. 0017-18, the HMD is fixed to the user’s head and the viewport is fixed), via the one or more input devices, a first user input, including detecting a first gaze input that is directed to a first position in the extended reality environment (Lindsay: pars. 0032-33, a user input may be detected to activate a menu [504\]; par. 0041, eye-gaze detection may be also detected); and
in response to detecting the first user input (Lindsay: par. 0032, in response to detecting a hand [506] (i.e. first user input);
in accordance with a determination that the first position in the extended reality environment has a first spatial relationship to a viewport through which the is visible, displaying a first user interface object in the first view of the extended reality environment (Lindsay: par. 0032, the displayed menu [504] may be device-locked, or stays positionally fixed relative to the HMD field of view (i.e. a first spatial relationship to a viewport), wherein the first user interface object includes one or more affordances for accessing a first set of functions of the first computer system (Lindsay: par. 0032, the menu may have user selectable elements (i.e. affordances) to control the display, etc.)
in accordance with a determination that the [first user input is ceased] forgoing displaying the first user interface object in the first view of the three-dimensional extended reality environment (Lindsay: par. 0033).
PNG
media_image1.png
832
645
media_image1.png
Greyscale
Lindsay may not explicitly teach: in response to detecting the first user input including detecting the first gaze input: …
wherein the first user interface object is displayed at a second position in the extended reality environment that has a second spatial relationship, different from the first spatial relationship, to the viewport;
in accordance with a determination that the first position in the three-dimensional extended reality environment does not have the first spatial relationship to the viewport through which the three- dimensional extended reality environment is visible, forgoing displaying the first user interface object in the first view of the three-dimensional extended reality environment.
Lopez teaches in general concepts related to an augmented reality extended reality environment that allows a user to look at and utilize its gaze information to control devices (Lopez: Abstract). Specifically, Lopez teaches that once a user’s gaze is detected upon a certain device, a control user interface may be displayed next to it (i.e. in a second spatial relationship) (Lopez: par. 0049, Fig. 8, the UI may be presented as an overlay next to or on top of the physical device). Once the user’s gaze has been moved away from the device, the control user interface is removed from the display (Lopez: par. 0059).
PNG
media_image2.png
914
635
media_image2.png
Greyscale
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay disclosures and teachings by implementing the system in an augmented (i.e. three-dimensional extended reality environment) with the offset menu and removal of menu as taught and suggested by Lopez. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the ease of control of items and preservation of augmented real estate in an augmented reality setting (Lopez: par. 0021).
As to Claim 138, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez as combined further teaches: while the first user interface object is not visible in a currently displayed view of the three-dimensional extended reality environment, detecting a first change of a viewpoint of a user from a first viewpoint associated with the first view of the three-dimensional extended reality environment to a second viewpoint associated with a second view of the three-dimensional extended reality environment;
in response to detecting the first change in the viewpoint of the user, updating the currently displayed view of the three-dimensional extended reality environment in accordance with the first change in the viewpoint of the user, to display the second view of the three-dimensional extended reality environment;
while the second view of the three-dimensional extended reality environment is visible via the first display generation component, detecting, via the one or more input devices, a second user input, including detecting a second gaze input that is directed to a third position, different from the first position, in the three-dimensional extended reality environment; and
in response to detecting the second user input including detecting the second gaze input:
in accordance with a determination that the third position in the three-dimensional extended reality environment has the first spatial relationship to the viewport through which the three- dimensional extended reality environment is visible, displaying the first user interface object in the second view of the three-dimensional extended reality environment, at a fourth position in the three-dimensional extended reality environment that has the second spatial relationship to the second view of the three-dimensional extended reality environment; and
in accordance with a determination that the third position in the three-dimensional extended reality environment does not have the first spatial relationship to the viewport through which the three- dimensional extended reality environment is visible, forgoing displaying the first user interface object in the second view of the three-dimensional extended reality environment (Examiner asserts that the general movement and reactivation of the menu per the combination would be contemplated within the disclosed head mounted device use of moving one’s head and the field of view accordingly).
As to Claim 139, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez as combined further teaches: in response to detecting the first user input including detecting the first gaze input:
in accordance with a determination that the first position in the three-dimensional extended reality environment does not have the first spatial relationship to viewport through which the three- dimensional extended reality environment is visible and that a second user interface object, different from the first user interface object, occupies the first position in the three-dimensional extended reality environment, performing a respective operation that corresponds to the second user interface object (Lopez: Fig. 8, different menus may be activated as attached to different areas in a world-anchored view as well).
As to Claim 140, Lindsay and Lopez teach the limitations of claim 138.
Lindsay further teaches: while the first view of the three-dimensional extended reality environment is visible and the first user interface object is not displayed in the first view of the three-dimensional extended reality environment, detecting a third user input that includes a third gaze input that is directed to a fifth position in the three- dimensional extended reality environment;
in response to detecting the third user input that includes the third gaze input:
in accordance with a determination that the fifth position in the three-dimensional extended reality environment is within a first region that includes a respective position having the first spatial relationship to the viewport through which the three-dimensional extended reality environment is visible, displaying a third user interface object at the respective position in the three-dimensional extended reality environment (Lopez: par. 0061, individual buttons on a thermostat (also a type of menu and a third gaze input)); and
in accordance with a determination that the fifth position in the three- dimensional extended reality environment is not within the first region that includes the respective position having the first spatial relationship to the viewport through which the three-dimensional extended reality environment is visible, forgoing displaying the third user interface object at the respective position in the three-dimensional extended reality environment (Examiner asserts that per the disclosure, when the gaze is not on the first region, the hypertext is not highlighted).
As to Claim 141, Lindsay and Lopez teach the limitations of claim 140.
Lindsay further teaches: the first region includes a first subregion including the respective position that has the first spatial relationship to the viewport through which the three-dimensional extended reality environment is visible and a second subregion that does not include the respective position (Examiner asserts that the highlighting effect is a second region that does not include the respective position, per a z-order).
As to Claim 142, Lindsay and Lopez teach the limitations of claim 141.
Lopez further teaches: wherein displaying the first user interface object at the second position in response to detecting the first user input including the first gaze input is further in accordance with a determination that the first gaze input is maintained within the first subregion for at least a first threshold amount of time (Lopez: par. 0061, activation occurs after a predetermined of time when a user’s gaze has lingered).
As to Claim 143, Lindsay and Lopez teach the limitations of claim 142.
Lopez as combined further teaches: while the first user interface object is not visible in the first view of the three-dimensional extended reality environment, detecting, via the one or more input devices, a fourth user input, including detecting a fourth gaze input that is directed to the first subregion and that has not been maintained within the first subregion for at least the first threshold amount of time; and
in response to detecting the fourth user input including the fourth gaze input:
in accordance with a determination that a respective gesture meeting first criteria has been detected while the fourth gaze input is maintained in the first subregion, displaying the first user interface object at the second position in the three-dimensional extended reality environment (Lopez: par. 0060, a user may use hand gestures in combination with the presented user interface items).
As to Claim 145, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez further teach: while displaying the first user interface object, in the first view of the three-dimensional extended reality environment, at the second position in the three-dimensional extended reality environment that has the second spatial relationship to the first view of the three-dimensional extended reality environment, detecting that user attention is no longer directed to the object position in the three-dimensional extended reality environment; and
in response to detecting that the user attention is no longer directed to the first object position in the three-dimensional extended reality environment, ceasing to display the first user interface object in the first view of the three-dimensional extended reality environment (Examiner asserts that the combination would allow for the removal of the user interface object, even in the second position, as contemplated by Lopez’s combination thereof. Operatively, the interface object would cease regardless of the positioning).
As to Claim 150, Lindsay and Lopez teach the limitations of claim 137.
Lindsay further teaches: while displaying the first view of the three-dimensional extended reality environment via the first display generation component, determining a current spatial relationship between the first display generation component and a user (Lindsay: par. 0043, the location of the user’s eye, pupils are used to determine a direction in which o project gaze lines from the user eyes); and
adjusting criteria for determining whether the respective position has the first spatial relationship to the viewport through which the three-dimensional extended reality environment is visible in accordance with the current spatial relationship between the first display generation component and the user (Lindsay: par. 0043, Examiner asserts that the field of view is determined per the measurement and detection of the eyes’ relationship to the device).
As to Claim 152, Lindsay and Lopez teach the limitations of claim 150.
Lindsay further teaches: wherein displaying the first user interface object at the second position that has the second spatial relationship to the viewport through which the three-dimensional extended reality environment is visible includes:
adjusting criteria for establishing the second spatial relationship between the first user interface object and the viewport through which the three-dimensional extended reality environment is visible in accordance with the current spatial relationship between the first display generation component and the user (Lindsay: par. 0043, Examiner asserts that the field of view is determined per the measurement and detection of the eyes’ relationship to the device; Examiner asserts that the same adjusted criteria may be applied in the second spatial relationship).
As to Claim 155, Lindsay and Lopez teach the limitations of claim 140.
Lindsay and Lopez as combined further teaches: while displaying the third user interface object at the respective position in the three- dimensional extended reality environment that has the first spatial relationship to the viewport through which the three-dimensional extended reality environment is visible, detecting, via the one or more input devices, a second change of the viewpoint of the user from the first viewpoint a third viewpoint (Examiner asserts that the scrolling of the document will change the viewpoints); and
in response to detecting the second change in the viewpoint of the user, displaying the viewport through which the three-dimensional extended reality environment is visible and displaying the third user interface object at an updated position in the view of the three-dimensional extended reality environment that has the first spatial relationship to the viewport through which the three-dimensional extended reality environment is visible (Examiner asserts the change in viewpoint would update the view accordingly).
As to Claim 156, Lindsay and Lopez teach the limitations of claim 155.
Lopez further teaches: wherein the third user interface object is translucent and has an appearance that is based on at least a portion of the three-dimensional extended reality environment over which the third user interface object is displayed (Lindsay: par. 0061, the button may be highlighted on selection (i.e. change in translucency)).
As to Claim 157, Lindsay and Lopez teach the limitations of claim 155.
Lopez further teaches: while the three-dimensional extended reality environment is visible through the viewport, displaying the third user interface object with a first appearance at a first indicator position in the three- dimensional extended reality environment, wherein the first appearance of the third user interface object at the first indicator position is based at least in part on a characteristic of the three-dimensional extended reality environment at the first indicator position in the viewport through which the three-dimensional extended reality environment is visible (Lopez: par. 113, the hypertext link may remain selected event if the document is scrolled) ; and
in response to detecting a movement of the viewpoint of the user from the first viewpoint to the third viewpoint in the three-dimensional extended reality environment, displaying the third user interface object with a respective appearance at a respective indicator position in the three-dimensional extended reality environment that has the first spatial relationship to the viewport through which the three- dimensional extended reality environment is visible, wherein the respective appearance of the first user interface object at the respective indicator position is based at least in part on a characteristic of the three- dimensional extended reality environment at the respective indicator position (Examiner asserts the scrolling is a movement of the viewpoint that will change the indicator of the position (the text of the document) accordingly and the appearance is maintained).
As to Claim 160, Lindsay and Lopez teach the limitations of claim 155.
Lindsay and Lopez as combined further teaches: in response to detecting the first user input that includes the first gaze input:
in accordance with a determination that the first position in the three-dimensional extended reality environment has the first spatial relationship to the viewport through which the three- dimensional extended reality environment is visible:
displaying an indication of the first user interface object before displaying the first user interface object at the second position (Lopez: par. 0061, upon detection of the gaze on a button, the button may be highlighted (i.e. an indication); and
after displaying the indication of the first user interface object;
in accordance with a determination that criteria for displaying the first user interface object is met by the first user input, replacing the indication of the first user interface object with the first user interface object (Lopez: par. 0061, in response to a determination that the button is activated, it will be selected and activated and shown as such); and
in accordance with a determination that criteria for displaying the first user interface object is not met by the first user input and that the first gaze input has moved away from the first position that has the first spatial relationship with the viewport through which the three-dimensional extended reality environment is visible, ceasing to display the indication of the third user interface object and forgoing display the third user interface object at the second position in the three-dimensional extended reality environment (Examiner asserts that the teaching of Lopez of gaze detection of the user turning away attention from the first position will result in the view being reset or changing accordingly to focus on other elements).
As to Claim 162, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez further teaches: while displaying the first user interface object including the one or more affordances for accessing the first set of functions of the first computer system, detecting a fifth user input including detecting gaze input directed to a respective affordance of the one or more affordances; and
in response to detecting the fifth user input:
in accordance with a determination that the respective affordance is a first affordance corresponding to a first function of the first computer system and that the fifth user input includes a gesture input that meets gesture criteria, performing the first function (Lopez: par. 0060, a hand gesture (i.e. fifth user input) may be associated with a particular menu option);
and in accordance with a determination that the respective affordance is the first affordance corresponding to the first unction function of the first computer system and that the fifth user input does not include a gesture input that meets the gesture criteria, forgoing performing the first function (Examiner asserts that the triggering condition of the gesture input is needed for the first function to be presented, as taught and disclosed by Lopez); and
in accordance with a determination that the respective affordance is a second affordance corresponding to a second function of the first computer system and that the fifth user input does not include a gesture input that meets the gesture criteria, performing the second function (Examiner asserts that any of the already discussed functions that do not require a gesture input, and simply the gaze input would satisfy the “second function”).
As to Claim 163, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez as combined further teaches: while displaying the first user interface object including the one or more affordances for accessing the first set of functions of the first computer system, detecting a change in pose of a first portion of the user; and
in response to detecting the change in pose of the first portion of the user (As per the combination, if the user changes her gaze, and thus a change in pose):
in accordance with a determination that the change in pose of the first portion of the user results in a first type of pose, changing an appearance of a respective affordance of the one or more affordances (The result is the removal of the affordance, a change in appearance); and
in accordance with a determination that the change in pose of the first portion of the user does not result in the first type of pose, forgoing changing the appearance of the respective affordance (if there is no change, then the affordance remains).
As to Claim 164, Lindsay and Lopez teach the limitations of claim 163.
Lindsay and Lopez as combined further teaches: in response to detecting the change in pose of the first portion of the user:
in accordance with a determination that the change in pose of the first portion of the user results in the first type of pose, forgoing changing an appearance of at least one affordance of the one or more affordances different from the respective affordance (if there is no change, then the affordance remains as noted earlier).
As to Claim 165, Lindsay and Lopez teach the limitations of claim 137.
Lopez further teaches: while displaying the first user interface object including the one or more affordances for accessing the first set of functions of the first computer system, detecting, via the one or more input devices, a sixth user input including gaze input directed to a respective affordance of the one or more affordances; and in response to detecting the sixth user input directed to the respective affordance, displaying additional content associated with the respective affordance (Lopez: par. 0061, the additional settings may be scrolled through based on the gaze input being directed to the top or bottom (i.e. sixth user input).
As to Claim 172, Lindsay and Lopez teach the limitations of claim 137.
Lopez further teaches: while displaying, via the first display generation component, the first user interface object, detecting, via the one or more input devices, a thirteenth user input that activates a fifth affordance of the one or more of affordances for accessing the first set of functions of the first computer system; and
in response to detecting the thirteenth user input that activates the fifth affordance:
performing a respective operation that corresponds to activation of the fifth affordance (Lopez: par. 0060, a hand gesture (i.e. thirteenth user input) may be associated with a particular menu option).
As to Claim 175, Lindsay and Lopez teach the limitations of claim 138.
Lopez further teaches: wherein displaying the first user interface object in the first view of the three-dimensional extended reality environment includes displaying the first user interface object at a first simulated distance from the first viewpoint of the user (Lopez: Fig. 7, par. 0053, the UI for the thermostat is shown at a simulated distance from the user) , wherein the first simulated distance is less than respective simulated distances of one or more other user interface objects displayed in the first view of the three- dimensional extended reality environment from the first viewpoint of the user (Lopez: par. 0053, the larger UI may be shown, when selected, which Examiner asserts may be design choice of being simulated as closer).
PNG
media_image3.png
618
933
media_image3.png
Greyscale
As to Claim 179, it is rejected for similar reasons as claim 137.
As to Claim 180, it is rejected for similar reasons as claim 137 and 179.
B.
Claims 146, 177 and 178 are rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003 in further view of Gu et al. (“Gu”), United States Patent Application Publication 2018/0136716 published on May 17, 2018.
As to Claim 146, Lindsay and Lopez teach the limitations of claim 145.
Lindsay and Lopez may not explicitly teach: the determination that the first position in the three-dimensional extended reality environment has the first spatial relationship to the viewport through which the three-dimensional extended reality environment is visible includes a determination that the first position is within a first response region of a first size; and
detecting that the user attention is no longer directed to the first position in the three-dimensional extended reality environment includes detecting that the user attention has moved from within the first response region to outside of a second response region of a second size that is different from the first size.
Gu teaches in general concepts related to a vision-based control apparatus utilizing a rear-view mirror of a vehicle (Gu: Title). Specifically, Gu teaches that selectable objects are displayed in the rear-view mirror of the vehicle (Gu: par. 00008). If the detected user gaze position is within a threshold distance of a selectable object in the displayed viewport area, then it is selected (Gu: par. 0023). Lopez teaches the removal of the activated user interface object when the gaze is no longer directed at it.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings utilizing the threshold distance for activation as taught by Gu. Such a person would have been motivated to do so with a reasonable expectation of success to allow for a margin of error in precision of interacting with objects.
As to Claim 177 it is rejected for similar reasons as claim 146.
As to Claim 178 it is rejected for similar reasons as claim 146.
C.
Claim 147-149 is rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003 in further view of Eledath et al. (“Eledath”), United States Patent Application Publication 2016/0378861 published on Dec. 29, 2016.
As to Claim 147, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez may not explicitly teach: while displaying the first user interface object in the first view of the three-dimensional extended reality environment, detecting a fourth user input including detecting gaze input directed to a respective affordance of the one or more affordances for accessing the first set of functions of the first computer system in conjunction with detecting a first speech input from a user; and
in response to detecting the fourth user input, performing a respective operation corresponding to the respective affordance in accordance with the first speech input.
Eledath teaches in general concepts related to a vision based user interface platform that analyzes multi-modal user interactions (Eledath: Abstract). Specifically, Eledath teaches the use of display device that allows for an augmented reality environment (Eledath: par. 0094). Graphical pop-up menus may be displayed based on gaze and or voice interaction(Eledath: par. 0112). The user is able to use multi-modal input controls to interact with the system may include gaze and speech (Eledath: par. 0153).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay-Lopez disclosures and teachings by allowing for the use of speech user input as taught by Eledath. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the expansion of control gestures.
As to Claim 148, Lemelson, Lopez and Eledath teach the limitations of claim 147.
Eledath further teaches: wherein performing the respective operation corresponding to the respective affordance in accordance with the first speech input includes:
in accordance with a determination that the respective affordance is an affordance for accessing a virtual assistant function of the first computer system, performing an operation corresponding to instructions contained in the first speech input (Eledath: par. 0111, a virtual personal assistant may be used in combination with the image processing system).
As to Claim 149, Lemelson, Lopez and Eledath teach the limitations of claim 147.
Eledath further teaches: wherein performing the respective operation corresponding to the respective affordance in accordance with the first speech input includes:
in accordance with a determination that the respective affordance is an affordance for accessing a text entry function of the first computer system that accepts text input, providing text converted from the first speech input as input to the text entry function (Eledath: pars. 0071-72, the speech recognition is a selection protocol that may be used to understand the text equivalent command).
D.
Claim 151 is rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003 in further view of Kuehne et al. (“Kuehne”), United States Patent Application Publication 2018/0136716 published on May 17, 2018.
As to Claim 151, Lindsay and Lopez teach the limitations of claim 150.
Lindsay further teaches: in accordance with a determination that the current spatial relationship between the first display generation component and the user no longer meets alignment criteria, displaying a second visual indication that the current spatial relationship between the first display generation component and the user no longer meets the alignment criteria.
Kuehne teaches in general concepts related to how position and head alignment of a user wearing virtual reality glasses are detected and used in a virtual extended reality environment (Kuehne: Abstract). Specifically, Kuehne teaches that a warning (i.e. a visual indication) when the user is about to move out of a specific region for the use and detection of the virtual extended reality environment device (Kuehne: par. 0010, when the user is about to move out of the pre-specified distance, a visual indication is given).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings to provide for the warning as taught by Kuehne. Such a person would have been motivated to do so with a reasonable expectation of success to allow for a reliable manner of warning the user of the loss of being able to be detected in the specific region (Kuehne: par. 0010).
E.
Claim 153 is rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003 in further view of Shinohara et al. (“Shinohara”), United States Patent Application Publication 2018/0325483 published on Nov. 15, 2018.
As to Claim 153, Lindsay and Lopez teach the limitations of claim 150.
Lindsay and Lopez may not explicitly teach: displaying one or more user interface objects in the first view of the three-dimensional extended reality environment, wherein the one or more user interface objects are different from the first user interface object, wherein respective positions of the one or more user interface objects in the first view of the three-dimensional extended reality environment do not change in accordance with a change to the current spatial relationship between the first display generation component and the user.
Shinohara teaches in general concepts related to displaying tomographic images in a overlayed fashion (Shinohara: Abstract). Specifically, Shinohara teaches that elements of a view may have fixed elements such as the title is fixed even if a gaze-directed scrolling takes place (Shinohara: par. 0113).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings to provide for fixed user interface elements as taught by Shinohara. Such a person would have been motivated to do so with a reasonable expectation of success to allow for certain information to remain fixed in the virtual extended reality environment that would need to be referred to as other information is dynamically changed.
F.
Claims 144, 154, 158-159, 161, 166-171 and 173-174 are rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003 in further view of George-Svahn et al. (“George-Svahn”), United States Patent Application Publication 2017/0235360 published on Aug. 17, 2017.
As to Claim 144, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez may not explicitly teach: wherein the first user interface object includes a respective system user interface for accessing one or more system functions of the first computer system.
George-Svahn teaches in general concepts related to gaze-based input to interact with a graphical user interface (George-Svahn: Abstract). Specifically, George-Svahn teaches that the volume of a media (i.e. a system function) being played on the system may be controlled via gaze-detection (George-Svahn: par. 0052, a volume slider control may be controlled via a joint gaze and touch gesture).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings by allowing for the system functions to be controlled via the menu interface object as taught by George-Svahn. Such a person would have been motivated to do so with a reasonable expectation of success to allow for ease of control of the system.
As to Claim 154, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez may not explicitly teach: at a first time, the one or more affordances for accessing the first set of functions of the first computer system include a first affordance for adjusting an audio level of the first computer system; and
at a second time, different from the first time, the one or more affordances for accessing the first set of functions of the first computer system include a second affordance for adjusting an audio level of a first type of audio provided by the first computer system and a third affordance for adjusting an audio level of a second type of audio provided by the first computer system, wherein the second affordance and the third affordance are different from the first affordance.
George-Svahn teaches in general concepts related to gaze-based input to interact with a graphical user interface (George-Svahn: Abstract). Specifically, George-Svahn teaches that the volume of a media being played on the system may be controlled via gaze-detection (George-Svahn: par. 0052, a volume slider control may be controlled via a joint gaze and touch gesture). Examiner notes that the disclosure is not limited to only one slider for one audio source.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings by allowing for the volume of several sources to be controlled via the menu interface with multiple scrolling objects as taught and suggested by George-Svahn. Such a person would have been motivated to do so with a reasonable expectation of success to allow for ease of control of the system.
As to Claim 158, Lindsay, Lopez and George-Svahn teach the limitations of claim 155.
Lindsay further teaches: wherein displaying the first user interface object in response to detecting the first user input including the first gaze input, includes displaying an animated transition of the one or more affordances for accessing the first set of functions of the first computer system emerging from the third user interface object in a first direction (George-Svahn: par. 0201, animation may be used as a transition when the object is gazed upon. Examiner asserts that the manner of the appearance of the emerging is a design choice).
As to Claim 159, Lindsay, Lopez and George-Svahn teach the limitations of claim 155.
George-Svahn further teaches: wherein displaying the first user interface object in response to detecting the first user input including the first gaze input, includes displaying an animated transition of the one or more affordances for accessing the first set of functions of the first computer system gradually appearing (George-Svahn: par. 0201, animation may be used as a transition when the object is gazed upon).
As to Claim 161, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez may not explicitly teach: wherein the first position in the three-dimensional extended reality environment is in a periphery region of the viewport through which the three-dimensional extended reality environment is visible.
George-Svahn teaches in general concepts related to gaze-based input to interact with a graphical user interface (George-Svahn: Abstract). Specifically, George-Svahn teaches that the menu may be displayed in a combination of a gaze on an edge area and a slide input (Georges-Svahn: par. 0051, the edge of an information area is gazed upon (i.e. a periphery region)
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings by allowing the position of the first position to be on the periphery as taught and suggested by George-Svahn. Such a person would have been motivated to do so with a reasonable expectation of success to allow for ease of control of the system with an expected region of the view to be used for the activation of a menu system.
As to Claim 166, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez may not explicitly teach: while displaying the first user interface object, detecting, via the one or more input devices, a seventh user input that activates a first affordance of the one or more affordances for accessing the first set of functions of the first computer system; and
in response to detecting the seventh user input that activates the first affordance, displaying a first system user interface for a first system function of the first computer system in the three-dimensional extended reality environment.
George-Svahn teaches in general concepts related to gaze-based input to interact with a graphical user interface (George-Svahn: Abstract). Specifically, George-Svahn teaches that the volume of a media (i.e. a system function) being played on the system may be controlled via gaze-detection (George-Svahn: par. 0052, a volume slider control may be controlled via a joint gaze and touch gesture).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings by allowing for the system functions to be controlled via the menu interface object as taught by George-Svahn. Such a person would have been motivated to do so with a reasonable expectation of success to allow for ease of control of the system.
As to Claim 167, Lindsay, Lopez and George-Svahn teach the limitations of claim 166.
Lindsay, Lopez and George-Svahn as combined further teaches: while displaying the first user interface object and the first system user interface, detecting, via the one or more input devices, an eighth user input that activates a second affordance, different from the first affordance, of the one or more of affordances for accessing the first set of functions of the first computer system; and
in response to detecting the eighth user input that activates the second affordance:
displaying a second system user interface, different from the first system user interface, for a second system function of the first computer system; and ceasing to display the first system user interface.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have further modified the Lindsay -Lopez-George-Svahn disclosures and teachings by allowing for multiple affordances as taught and suggested by George-Svahn for controlling the functions (e.g. George-Svahn- par. 0053, check boxes). Such a person would have been motivated to do so with a reasonable expectation of success to allow for ease of control of the system.
As to Claim 168, Lindsay, Lopez and George-Svahn teach the limitations of claim 166.
Lopez further teaches: while displaying the first system user interface and the first user interface object, detecting, via the one or more input devices, a ninth user input that includes a gaze input directed to the first user interface object; and
in response to detecting the ninth user input, changing one or more visual properties of the first system user interface to reducing visual prominence of the first system user interface relative to the first user interface object (Lopez: par. 0061, the button may be highlighted. Examiner asserts that shading of the highlighting may be interpreted to be of reduced visual prominence broadly).
As to Claim 169, Lindsay, Lopez and George-Svahn teach the limitations of claim 166.
Lopez further teaches: while displaying the first system user interface and the first user interface object, detecting, via the one or more input devices, a tenth user input that includes gaze input directed to the first system user interface; and
in response to detecting the tenth user input, changing one or more visual properties of the first user interface object to reduce visual prominence of the first user interface object relative to the first system user interface (Lopez: par. 0061, the button may be highlighted. Examiner asserts that shading of the highlighting may be interpreted to be of reduced visual prominence broadly).
As to Claim 170, Lindsay, Lopez and George-Svahn teach the limitations of claim 166.
Lindsay, Lopez and George-Svahn further teaches: while displaying, via the first display generation component, an application launching user interface in the three-dimensional extended reality environment and the first user interface object, detecting, via the one or more input devices, an eleventh user input that activates a third affordance of the one or more of affordances for accessing the first set of functions of the first computer system; and
in response to detecting the eleventh user input that activates the respective affordance:
displaying a third system user interface for a third system function of the first computer system that corresponds to the third affordance; and
ceasing to display the application launching user interface.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have further modified the Lindsay -Lopez-George-Svahn disclosures and teachings by allowing for multiple affordances as taught and suggested by George-Svahn for controlling the functions (e.g. George-Svahn- par. 0053, check boxes). Such a person would have been motivated to do so with a reasonable expectation of success to allow for ease of control of the system.
As to Claim 171, Lindsay, Lopez and George-Svahn teach the limitations of claim 166.
Lindsay further teaches: while displaying, via the first display generation component, an application user interface in the three-dimensional extended reality environment and the first user interface object, detecting, via the one or more input devices, a twelfth user input that activates a fourth affordance of the one or more of affordances for accessing the first set of functions of the first computer system; and
in response to detecting the twelfth user input that activates the respective affordance:
displaying a fourth system user interface for a fourth system function of the first computer system that corresponds to the fourth affordance, concurrently with the application user interface.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have further modified the Lindsay -Lopez-George-Svahn disclosures and teachings by allowing for multiple affordances as taught and suggested by George-Svahn for controlling the functions (e.g. George-Svahn- par. 0053, check boxes). Such a person would have been motivated to do so with a reasonable expectation of success to allow for ease of control of the system.
As to Claim 173, Lindsay, Lopez and George-Svahn teach the limitations of claim 172.
Lopez further teaches: wherein performing the respective operation includes displaying one or more controls for adjusting one or more settings of the first computer system, wherein the one or more controls are displayed overlaying at least a portion of the first user interface object (Lopez: par. 0049, Fig. 8, the UI may be presented as an overlay next to or on top of the physical device.)
As to Claim 174, Lindsay, Lopez and George-Svahn teach the limitations of claim 173.
George-Svanh further teaches: wherein detecting the thirteenth user input that activates the fifth affordance of the one or more of affordances for accessing the first set of functions of the first computer system includes detecting a pinch and release gesture that is directed to the fifth affordance (Georges-Svahn: par. 0049, the pinching of two of his or fer fingers can allow for the zooming of the object part, in combination with drag), and the method includes:
while displaying the one or more controls for adjusting one or more settings of the first computer system, detecting a pinch and drag gesture that is directed to a first control of the one or more controls; and
in response to detecting the pinch and drag gesture that is directed to the first control of the one or more controls, adjusting a first setting that corresponds to the first control in accordance with one or more characteristics of the pinch and drag gesture (Georges-Svahn: par. 0049, the pinching of two of his or fer fingers can allow for the zooming of the object part, in combination with drag).
G.
Claim 176 is rejected under 35 U.S.C. § 103 as being unpatentable over Lindsay et al. (“Lindsay”), United States Patent Application Publication 2017/0270715 published on Sept. 21, 2017 in view of Lopez et al. (“Lopez”), United States Patent Application Publication 2016/0274762 published on Jan. 30, 2003 in further view of Clements, United States Patent 10,802,582, published on Oct. 13, 2020.
As to Claim 176, Lindsay and Lopez teach the limitations of claim 137.
Lindsay and Lopez may not explicitly teach: displaying a plurality of system status indicators that include information about a status of the first computer system, concurrently with displaying the first user interface object.
Clements teaches in general eye tracking devices as part of an augmented reality headset(Clements: Abstract). Specifically, Clements teaches that the user is able to direct her eye gaze to a position to trigger display of a window including system information (Clements: col. 7, lines 39-44, the icon is activated by eye gaze and shows an icon window including information about the device).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the Lindsay -Lopez disclosures and teachings by allowing for the displayed information to be about multiple system indicators as taught by Clements. Such a person would have been motivated to do so with a reasonable expectation of success to reduce the cognitive burden on the user.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ballard et al., US Patent Application Publication 2015/0156803 (June 4, 2015) (describing gaze-based selection of menu and sub menus);
Ziraknejad et al., US Patent Application Publication 2020/0038120 (Feb. 6, 2020) (describing gaze-based surgical system).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES T TSAI whose telephone number is (571)270-3916. The examiner can normally be reached M-F 8-5 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached on 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES T TSAI/Primary Examiner, Art Unit 2174
Prosecution Insights