DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Streja et al. (US 20230401795 A1, hereinafter Streja), in view of Dasher et al. (US 20250251787 A1, hereinafter Dasher).
Regarding Claim 1, Streja teaches A head-wearable device, comprising: one or more displays; and one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, the one or more programs including instructions for: while the head-wearable device is worn by a user: (Streja, Paragraph [0040], "In some examples, system 100 is a head-mounted display (HMD) device."; [0006], "An example electronic device comprises a display; one or more sensors; one or more processors; a memory; and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors"; [0004], "while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device"), [[ in response to a detection of an object within a field-of-view of the user, presenting, via the one or more displays, a first extended reality (XR) augment overlaid over a first portion of the field-of-view of the user that is associated with the object; ]] in accordance with a determination that a first user eye movement is focused on the first XR augment for a first predetermined time, replacing the first XR augment with a second XR augment, (Streja, Paragraph [0076], "detecting the user gaze at first object 304 includes determining that the user gaze is directed at first object 304 for a predetermined duration, e.g., 0.1 seconds, 0.25 seconds, 0.5 seconds, or 1 second."; [0077], "in response to detecting the user gaze at first object 304, device 302 expands first object 304 into list 306 (a virtual object) of objects…expanding first object 304 into list 306 includes ceasing to display first object 304 and displaying list 306."), wherein the second XR augment: appears to overlie a second portion of the field-of-view of the user, (Streja, Paragraph [0077], "device 302 displays list 306 within a predetermined distance from first object 304."), and includes one or more focus-action selectable elements, each focus-action selectable element associated with an object-specific action; (Streja, Paragraph [0078], "list 306 includes one or more user selectable objects, e.g., music icon 308, web browser icon 310, and second object 312 representing a digital assistant"), and while the second XR augment is presented, in accordance with a determination that a second user eye movement is focused outside a perimeter of the second XR augment for a second predetermined time, replacing the second XR augment with a third XR augment, (Streja, Paragraph [0109], "device 302 ceases to display second object 312 in accordance with a determination that one or more DA dismissal criteria are satisfied. In some examples, satisfying the DA dismissal criteria includes determining that the user's attention is not directed to virtual object 314 for at least a predetermined duration"; [0083], "second object 312 increases in size, moves away from list 306, and list 306 ceases to be displayed."), wherein the third XR augment: appears to overlie a third portion of the field-of-view of the user, (Streja, Paragraph [0090], "device 302 displays second object 312 to appear a predetermined distance (e.g., 0.25 meters, 0.5 meters, 1 meter) from the user"), and includes one or more object-specific selectable elements, each object-specific selectable element associated with a respective object-specific action. (Streja, Paragraph [0120], "suggested command(s) 420 include a command to move object 404 and a command to provide more information about object 404"), (Streja, Paragraph [0125], "displays suggested commands 422, e.g., to copy the street address, to save the street address, and to navigate to the street address.").
Streja does not explicitly disclose but Dasher teaches in response to a detection of an object within a field-of-view of the user, presenting, via the one or more displays, a first extended reality (XR) augment overlaid over a first portion of the field-of-view of the user that is associated with the object (Dasher, Paragraph [0057], "the system determines that the TV 103 is in the field of view of the XR device. For example, after determining that the TV 103 is in the field of view of the XR device for a minimum threshold period of time … the TV 103 may be set as the object of focus."; [0055], "the user wearing XR device 101 may see the ESPN and Netflix application icons 105 <read on first XR augment> in response to a number of different events."; [0076], "the visual element may be automatically, or by default, be positioned on or near the TV 103.").
Dasher and Streja are analogous since both address gaze-driven presentation of virtual UI elements in an XR head-mounted device FOV. Streja provided a way of progressively transitioning between XR augments based on user gaze dwell and lack-of-attention criteria. Dasher provided a way of detecting a physical object in the XR device's FOV and anchoring a visual element (icon) at a fixed location on/near that object. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Dasher's object-detection-triggered anchored visual element into the modified invention of Streja such that Streja's first object 304 is presented in response to detecting an object in the FOV and overlaid on a portion of the FOV associated with that object. The motivation is to avoid cluttering the user's FOV with irrelevant virtual elements and to surface only visual elements relevant to the current object of interest, as discussed by Dasher in Paragraph [0003].
Regarding Claim 2, the combination of Streja and Dasher teaches the invention in Claim 1.
The combination further teaches wherein the one or more programs further include instructions for: while the head-wearable device is worn by the user [[in response to a detection of another object within the field-of-view of the user, presenting, via the one or more displays, another XR augment, wherein the other XR augment appears to overlie over another portion of the field-of-view of the user that is associated with the other object]] (Streja, Paragraph "first object 304 is persistent in the current field of view of environment 300" and expansion into list 306).
But Strejadoes does not explicitly disclose detection of another distinct object and presenting another XR augment over another portion associated with the other object.
However, Dasher teaches in response to a detection of another object within the field-of-view of the user, presenting, via the one or more displays, another XR augment, wherein the other XR augment appears to overlie over another portion of the field-of-view of the user that is associated with the other object (Dasher, Paragraph [0005], "User focus for an object may be determined based on physical proximity of the XR device to the physical object, a field of view of the XR device encompassing the physical device or other object"; [0006], "The system may keep track of the location of physical devices or other objects in a physical space and of the XR device so as to be able to determine proximity of the XR device to objects and to determine when the field of view and/or gaze is trained on the physical device or other object"; [0004], "Based on the prediction of the object of focus, icons or other virtual visual elements may be positioned to be on or near the object of focus with which they are associated"; [0057], "the system determines that the TV 103 is in the field of view of the XR device. For example, after determining that the TV 103 is in the field of view of the XR device for a minimum threshold period of time"; [0076], "The visual element may be automatically, or by default, be positioned on or near the TV”).
Dasher and Streja are analogous since both deal with XR head-wearable devices displaying gaze-contingent visual elements anchored to physical objects. Streja provides gaze detection and list expansion. Dasher provides tracking multiple physical devices/objects in space and anchoring visual elements on/near each object of focus.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate Streja's gaze-based system to handle multiple objects in FOV without clutter into modified invention of Dasher such that tracking of other objects and anchoring visual elements on/near them. Motivation is to display only relevant visual elements on or near predicted object of focus as discussed by Dasher in Paragraph.[0004]
Regarding Claim 3, the combination of Streja and Dasher teaches the invention in Claim 1. The combination further teaches [[the object and the other object are ranked based on proximity to]] a location of the first user eye movement (Streja, Paragraph [0076], "Device 302 detects a user gaze at first object 304. For example, device 302 detects the user gaze using sensor(s), e.g., image sensor(s) 108 configured to detect a user's gaze. In some examples, detecting the user gaze at first object 304 includes determining that the user gaze is directed at first object 304 for a predetermined duration"; it is noted location of first eye movement determined).
But Streja does not teach ranking based on proximity.
However, Dasher teaches the object and the other object are ranked based on proximity (Dasher, Paragraph [0139], , "The system utilizes depth information to calculate the distance between the XR device 101 and the detected object of focus … In this embodiment, closer objects of interest may trigger display of a visual element more prominently, while icons or other visual elements related to distant objects of interest might be minimized or hidden" ; [0005], "User focus for an object may be determined based on physical proximity of the XR device to the physical object, a field of view of the XR device encompassing the physical device or other object, and/or a user gaze detected by the XR device to be on the physical device or other object").
Dasher and Streja are analogous since both deal with XR head-wearable devices displaying gaze-contingent visual elements anchored to physical objects. Streja teaches gaze location for XR device location. Dasher already ranks prominence by proximity to decide which icon to surface to avoid clutter. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate ranks prominence taught by Dasher into modified invention of Strejasuch that XR device location as proximity origin is obvious design choice to surface object nearest to where user is already looking.
Regarding Claim 4, the combination of Streja and Dasher teaches the invention in Claim 1. The combination further teaches wherein the one or more programs further include instructions for: [[in accordance with a determination that a third user eye movement is focused on the first XR augment and the other XR augment, presenting, via the one or more displays, a zoomed-in portion of the field-of-view of the user]]
wherein the zoomed-in portion includes the first portion of the field-of-view of the user, the other portion of the field-of-view of the user, the object, and the other object (Streja, Paragraph [0077], "In FIG. 3B, in response to detecting the user gaze at first object 304, device 302 expands first object 304 into list 306 (a virtual object) of objects. For example, device 302 displays list 306 within a predetermined distance from first object 304" "expanding first object 304 into list 306 includes concurrently displaying first object 304 and list 306"; it is noted list display within predetermined distance containing first object and other objects reads on zoomed-in portion containing first portion, other portion, object, other object).
Streja does not explicitly disclose but Dasher teaches in accordance with a determination that a third user eye movement is focused on the first XR augment and the other XR augment [[presenting, via the one or more displays, a zoomed-in portion of the field-of-view of the user, wherein the zoomed-in portion includes the first portion of the field-of-view of the user, the other portion of the field-of-view of the user, the object, and the other object (Dasher, Paragraph [0005], "User focus for an object may be determined based on... a user gaze detected by the XR device to be on the physical device or other object"; [0076], "The visual element may be automatically, or by default, be positioned on or near the TV 103"; [0080], "Visual elements may be adapted in their visual or audible representations. The XR system may dynamically alter their size, orientation, or transparency depending on the user's perspective and distance from the spatial anchor that is the object of focus"; [0086], "a size of the visual element 305 it is displayed may be determined according to the distance from the XR device").
Dasher and Streja are analogous since both deal with XR head-wearable devices displaying gaze-contingent visual elements anchored to physical objects. Streja teaches list-within-distance for XR device location. Dasher teaches size-adaptation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate size-adaptation taught by Dasher into modified invention on Streja such that system will be able to designed the way to avoid obscuring and allow selection when augments overlap.
Regarding Claim 5, the combination of Streja and Dasher teaches the invention in Claim 1. The combination further teaches [[ wherein presenting the zoomed-in portion of the field-of-view of the user includes: ]] replacing the XR augment and the other XR augment with a first object-tag augment, associated with the object, and a second object-tag augment, associated with the other object, respectively (Streja, Paragraph [0078], "list 306 includes one or more user selectable objects, e.g., music icon 308, web browser icon 310, and second object 312" - icons as object-tag augments).
Dasher further teaches the presenting the zoomed-in portion of the field-of-view of the user includes replacing the XR augment and the other XR augment with a first object-tag augment, associated with the object, and a second object-tag augment, associated with the other object, respectively (Dasher, Paragraph, "Based on the prediction of the object of focus, icons or other virtual visual elements may be positioned to be on or near the object of focus with which they are associated"; Paragraph, "The visual element may be automatically, or by default, be positioned on or near the TV 103").
As explained in rejection of claim 4, the obviousness for combining of Dasher into Streja is provided above.
Regarding Claim 6, the combination of Streja and Dasher teaches the invention in Claim 1. The combination further teaches wherein presenting the zoomed-in portion of the field-of-view of the user further includes: indicating, to the user, that the first object-tag augment is currently selected (Streja, Paragraph [0125], "Device 302 thus modifies the first display state of object 414 to the second display state, as indicated by the displayed border around object 414."), [[ in response to the user performing a scroll gesture, indicating, to the user, that the second object-tag augment is currently selected,]] in response to the user performing a select gesture, replacing the zoomed-in portion of the field-of-view with an additional XR augment (Streja, Paragraph [0088], "device 302 receives a hand gesture (e.g., a pointing gesture, a pinching gesture, a grabbing gesture, and the like) corresponding to a selection of second object 312." ; [0133], "The user can then select object 410 (e.g., via a pinch gesture) to perform the corresponding command of pausing the playing media."). wherein the additional XR augment: appears to overlie the second portion of the field-of-view of the user, and includes one or more additional object-specific selectable elements (Streja, Paragraph [0120], "modifying the first display state of object 404 to the second display state includes displaying one or more suggested commands 420 to interact with object 404." ; [0125], "displays suggested commands 422, e.g., to copy the street address, to save the street address, and to navigate to the street address."). each additional object-specific selectable element associated with a respective additional object-specific action (Streja, Paragraph [0120], "suggested command(s) 420 include a command to move object 404 and a command to provide more information about object 404"). Streja does not explicitly disclose but Dasher teaches in response to the user performing a scroll gesture, indicating, to the user, that the second object-tag augment is currently selected (Dasher, Paragraph [0094], "the current display 603 in FIG. 6 includes a number of media items that the user may scroll through and the complementary visual element 605 for controlling the current display 603. The visual element 605 displayed by the XR device 101 is a controller that may select, using the XR device, a media item displayed as part of the current display 603 on the TV 103." ; [0117], "a command may be received for controlling the associated application. The command may be provided as a gesture input to virtual controller shown by the visual element.").
Dasher and Streja are analogous since both of them are dealing with XR head-wearable devices presenting virtual UI elements associated with physical objects and controlled by hand gestures and eye gaze. Streja provided a way of indicating, to the user, that an object is currently selected by modifying display state as indicated by displayed border, and selecting an object via pinching gesture to perform corresponding command and displaying additional object-specific selectable elements as suggested commands. Dasher provided a way of scrolling through a number of media items and controlling via gesture input to virtual controller. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate scrolling through media items and gesture input to virtual controller taught by Dasher into modified invention of Streja such that the presenting the zoomed-in portion of the field-of-view of the user further includes indicating, to the user, that the first object-tag augment is currently selected, and, in response to the user performing a scroll gesture, indicating, to the user, that the second object-tag augment is currently selected, and further includes, in response to the user performing a select gesture, replacing the zoomed-in portion with an additional XR augment that appears to overlie the second portion and includes one or more additional object-specific selectable elements. The motivation is to provide a complementary display to control the current display that includes a number of media items that the user may scroll through discussed by Dasher in Paragraph [0094].
Regarding Claim 7, the combination of Streja and Dasher teaches the invention in Claim 1. The combination further teaches wherein the presenting the zoomed-in portion of the field-of-view of the user is in response to the user performing a zoom gesture (Streja, Paragraph [0088], "device 302 receives a hand gesture (e.g., a pointing gesture, a pinching gesture, a grabbing gesture, and the like) corresponding to a selection”; it is noted the pinching gesture is basis for pinch-and-expand zoom).
Dasher further teaches the same limitation (Dasher, Paragraph [0080], "Visual elements may be adapted in their visual or audible representations. The XR system may dynamically alter their size, orientation, or transparency depending on the user's perspective and distance from the spatial anchor").
Dasher and Streja are analogous are analogous since both of them are dealing with XR head-wearable devices where visual element size and display are adapted based on user perspective, distance and hand gestures. Streja provided a way of receiving a hand gesture including a pinching gesture corresponding to a selection of an object. Dasher provided a way of dynamically altering size, orientation, or transparency depending on the user's perspective and distance from the spatial anchor and determining size according to distance. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate dynamically altering size depending on perspective and distance taught by Dasher into modified invention of Streja such that the presenting the zoomed-in portion of the field-of-view of the user is in response to the user performing a zoom gesture (e.g., a pinch-and-expand gesture). The motivation is to make for a more intuitive user interface relevant to the user's current context and needs and to adjust display based on proximity discussed by Dasher in Paragraph [0139].
Regarding Claim 8, the combination of Streja and Dasher teaches the invention in Claim 1. The combination further teaches wherein the head-wearable device further comprises one or more speakers, and the one or more programs further include instructions for: while the head-wearable device is worn by the user: in accordance with the determination that the first user eye movement is focused on the first XR augment for the first predetermined time, presenting an audio cue to the user, wherein the audio cue includes a description of the object (Streja, Paragraph [0037], "Device 100 a includes various components, such as processor(s) 102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108, orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116, speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122." ; [0120], "modifying the first display state of object 404 to the second display state includes displaying one or more suggested commands 420 to interact with object 404. The DA is capable of satisfying suggested command(s) 420. In FIG. 4C, suggested command(s) 420 include a command to move object 404 and a command to provide more information about object 404, e.g., provide information about the recognized plant species from a knowledge base such as Wikipedia." ; [0130], "Device 302 thus provides an output such as “Bellis perennis, the daisy, is a common European species of the family Asteraceae . . . .” " ; [0090], "device 302 outputs DA generated audio (e.g., responses to user requests) so the audio appears to originate from the location of second object 312).
Regarding Claim 9, the combination of Streja and Dasher teaches the invention in Claim 1.
The combination further teaches each object-specific selectable element includes a representation of the object (Dasher, Paragraph [0119], " The XR interface 1151 may facilitate this process by either allowing direct interaction with the physical device in a “pass-thru” mode, with a virtual representation of the physical device, with a virtual representation of a placed or “pinned” spatial anchor"; [0082], "a frame representation or other visual element related to the show").
As explained in rejection of claim 1, the obviousness for combining of Dasher into Streja is provided above.
Regarding Claim 10, the combination of Streja and Dasher teaches the invention in Claim 1.
The combination further teaches the head-wearable device further comprises at least one of an eye-tracking camera and an inertial measurement unit (IMU), and the first user eye movement and the second user eye movement are detected by the at least one of the eye-tracking camera and the IMU (Streja, Paragraph [0076], "Device 302 detects a user gaze at first object 304. For example, device 302 detects the user gaze using sensor(s), e.g., image sensor(s) 108 configured to detect a user's gaze. In some examples, detecting the user gaze at first object 304 includes determining that the user gaze is directed at first object 304 for a predetermined duration"; [0047], "Orientation sensor(s) 110 optionally include one or more gyroscopes and/or one or more accelerometers").
Dasher further teaches the head-wearable device further comprises at least one of an eye-tracking camera and an IMU, and the first user eye movement and the second user eye movement are detected by the at least one of the eye-tracking camera and the IMU (Dasher, Paragraph [0107], "eye-tracking systems 915 which may control cameras for tracking eye HMD user movements"; [0146], "If provided as a SLAM-enabled device, the XR device may be equipped with an IMU (inertial measurement unit). IMU data may be used for location/orientation/movement determination").
As explained in rejection of claim 1, the obviousness for combining of movement tracking of Dasher into Streja is provided above.
Regarding Claim 11, it recites limitations similar in scope to the limitations of claim 1 and the combination of Streja and Dasher teaches all the limitations as of Claim 1. And Streja discloses these features can be implemented on a computer readable storage medium (Streja, Paragraph [0005], “An example non-transitory computer-readable storage medium stores one or more programs. The one or more programs comprise instructions, which when executed by one or more processors of an electronic device with a display and one or more sensors”).
Regarding Claim 12, it recites limitations similar in scope to the limitations of Claim 2 and therefore is rejected under the same rationale.
Regarding Claim 13, it recites limitations similar in scope to the limitations of Claim 4 and therefore is rejected under the same rationale.
Regarding Claim 14, it recites limitations similar in scope to the limitations of Claim 7 and therefore is rejected under the same rationale.
Regarding Claim 15, it recites limitations similar in scope to the limitations of Claim 8 and therefore is rejected under the same rationale.
Regarding Claim 16, it recites limitations similar in scope to the limitations of Claim 1 but as a method and the combination of Streja and Dasher teaches all the limitations as of Claim 1. Therefore is rejected under the same rationale.
Regarding Claim 17, it recites limitations similar in scope to the limitations of Claim 2 and therefore is rejected under the same rationale.
Regarding Claim 18, it recites limitations similar in scope to the limitations of Claim 4 and therefore is rejected under the same rationale.
Regarding Claim 19, it recites limitations similar in scope to the limitations of Claim 7 and therefore is rejected under the same rationale.
Regarding Claim 20, it recites limitations similar in scope to the limitations of Claim 8 and therefore is rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 12196952 B2 Display systems and methods for determining registration between a display and a user's eyes
US 20240103614 A1 DEVICES, METHODS, FOR INTERACTING WITH GRAPHICAL USER INTERFACES
US 20230144091 A1 DYNAMIC CONTENT PRESENTATION FOR EXTENDED REALITY SYSTEMS
US 11126319 B2 Mixed reality device gaze invocations
US 10855979 B2 Interpreting eye gaze direction as user input to near-eye-display (NED) devices for enabling hands free positioning of virtual items
US 9323325 B2 Enhancing an object of interest in a see-through, mixed reality display device
US 20130241805 A1 Using Convergence Angle to Select Among Different UI Elements
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YUJANG TSWEI whose telephone number is (571)272-6669. The examiner can normally be reached 8:30am-5:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached on (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YuJang Tswei/Primary Examiner, Art Unit 2614
Prosecution Insights