DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The response filed on 3/26/2026 has been entered and made of record. Claims 151, 153, 158-161, 168, 169, 171, and 180-181 are amended. Claims 151-181 are pending.
The previous rejections under 35 USC 102 and 35 USC 103 have been withdrawn as necessitated by amendment. New rejections under 35 USC 103 have been made to claims 151-158 and 160-181 with the same previously cited art, but now additionally in view of Gribetz et al. Claim 159 is still objected.
Information Disclosure Statement
The information disclosure statements (IDS) were filed on 12/31/2025. The submissions are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Drawings
The drawings filed 9/15/2023 were accepted.
Allowable Subject Matter
Claim 159 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 151-153, 155-157, 161-164, 166, and 179-181 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala et al (hereinafter Agarawala; US20210165557A1; filed 12/29/2020) in view of Gribetz et al (hereinafter Gribetz; US20180095541A1; filed 7/19/2017).
With regards to claim 1, Agarawala discloses A computer system configured to communicate with one or more display generation components and one or more input devices, the computer system comprising: one or more processors (Agarawala, paragraph 205: “Computer system 4400 may include one or more processors”); and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions (Agarawala, paragraph 208: “Computer system 4400 may also include a main or primary memory 4408, such as random access memory (RAM). Main memory 4408 may include one or more levels of cache. Main memory 4408 may have stored therein control logic (i.e., computer software) and/or data.”) for:
concurrently displaying, via the one or more display generation components, in a three-dimensional environment, representations of a plurality of augmented reality experiences (Agarawala, paragraph 35: “Using the feed and the user's account information, AR system may extract and display new or other data in a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user”), including:
a first representation of a first augmented reality experience (Agarawala, paragraph 35: “Using the feed and the user's account information, AR system may extract and display new or other data in a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” Fig. 35A: Twitter application preview; The representations are interpreted as the previews of the applications, as shown in Figures 5, 10A-10B, 14B, 36A, and 37); and
a second representation of a second augmented reality experience that is different from the first augmented reality experience, wherein the second representation is different from the first representation (Agarawala, paragraph 4: “with different applications, the preview, new, followed, or most recently accessed, data may vary. For example, on a shopping site, the user may see sale items, or items that were recently accessed/viewed, recently purchased items, or items and/or prices of items in a user's shopping cart;” Fig. 36A-B show multiple representations of applications, such as twitter, which is expanded by the user in the following figures);
while concurrently displaying, in the three-dimensional environment, the representations of the plurality of augmented reality experiences, receiving, via the one or more input devices, a first user input (Agarawala, paragraph 118: “In FIG. 36A a user may select the Twitter® application from the various available applications that are accessible in the VR space”); and
in response to receiving the first user input:
ceasing display of the representations of one or more of the plurality of augmented reality experiences (Agarawala, paragraph 119: “As shown in FIG. 36B, upon selection of Twitter® or the performance of another hand gesture, the elements or data of Twitter may be expanding across one or more meshes or locations within the room. In an embodiment, if a user is reopening the application, it may resume from a previous state with a similar physical arrangement.” The preview is no longer displayed when the application is reopened); and
in accordance with a determination that the first user input corresponds to selection of the first representation of the first augmented reality experience, displaying, via the one or more display generation components, in the three-dimensional environment, the first augmented reality experience (Agarawala, paragraph 119: “As shown in FIG. 36B, upon selection of Twitter® or the performance of another hand gesture, the elements or data of Twitter may be expanding across one or more meshes or locations within the room.”).
However, Agarawala does not disclose the first representation of the first augmented reality experience includes a first border that surrounds representations of a first set of interactive elements; and displaying the first augmented reality experience includes displaying the first set of interactive elements and ceasing display of the first border.
Gribetz teaches the first representation of the first augmented reality experience includes a first border that surrounds representations of a first set of interactive elements (Gribetz, Fig. 8A-8C: the interface elements are shown in “bubbles”, which include borders around elements; paragraph 879: “A bubble is, in the illustrated example, a symbol of an icon, in which the icon appears as a bubble, and behaves like a bubble (to some extent) to the user via the user interface 800 of FIG. 8A”); and displaying the first augmented reality experience includes displaying the first set of interactive elements and ceasing display of the first border (Gribetz, paragraph 890: “each bubble of the bubble set 803, once selected, allows the user to view a setting, application, collection of data, and/or the like associated with the selected bubble;” paragraph 891: “FIG. 8B depicts a schematic example of the user interface 800 using bubbles (as icons) for a second set-up operation. An application folder may be opened, in one implementation, by popping the icon associated with the application folder, such as may be detected via gesture-recognition techniques;” also see paragraph 960 for additional details on popping icon bubbles).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala and Gribetz such that the groupings of augmented reality elements were surrounded by a removable border. This would have provided a visual representation for the elements (Gribetz, paragraph 955: “In one implementation, the icon may be represented by a bubble icon or any shaped icon. In one implementation, a bubble icon is a semi-translucent sphere resembling a bubble.”).
With regards to claim 152, which depends on claim 151, Agarawala discloses the representations of the plurality of augmented reality experiences are displayed on one or more additive light displays; and
the three-dimensional environment is an optical passthrough environment that is visible to a user through the one or more additive light displays (Agarawala, Paragraph 11: "In an embodiment, a user may be in an AR enabled environment, or may be wearing AR enabled glasses or a headset that allows the user to see digital objects that overlay or augment their real-world physical environment;" Figs. 5, 36A: show the real world background behind the app preview/representation).
With regards to claim 153, which depends on claim 151, Agarawala discloses in response to receiving the first user input:
in accordance with a determination that the first user input corresponds to selection of the second representation of the second augmented reality experience, displaying, via the one or more display generation components, in the three-dimensional environment, the second augmented reality experience (Agarawala, paragraph 38: “If the user then activates Instagram®, the set of images may be expanded and/or more data may be displayed or made available for interaction/access;” paragraph 40: “with different applications, the preview, new, followed, or most recently accessed, data may vary.” The second augmented experience is interpreted as a different application from the first one, which also has its own preview and displayed data), wherein:
displaying the second augmented reality experience includes displaying a second set of interactive elements different from the first set of interactive elements (Agarawala, paragraph 40: “with different applications, the preview, new, followed, or most recently accessed, data may vary.”);
the first representation of the first augmented reality experience includes representations of the first set of interactive elements (Agarawala, paragraph 38: “the AR system may generate a preview that may include other data or images, such as most ‘liked’ images, or images associated with particular user accounts the user is following or has indicated to be of particular interest.”); and
the second representation of the second augmented reality experience includes representations of the second set of interactive elements different from the representations of the first set of interactive elements (Agarawala, paragraph 40: “with different applications, the preview, new, followed, or most recently accessed, data may vary.”).
With regards to claim 155, which depends on claim 153, Agarawala discloses the representations of the first set of interactive elements are non-interactive; and the representations of the second set of interactive elements are non-interactive (Agarawala, paragraph 38: “the AR system may generate a preview that may include other data or images… If the user then activates Instagram®, the set of images may be expanded and/or more data may be displayed or made available for interaction/access;” paragraph 38 describes the functions of the previews that are used for each application; the preview displays images or data representative of the application, but the interaction/access with the application is only provided once the application is activated).
With regards to claim 156, which depends on claim 153, Agarawala discloses at least a portion of the first representation of the first augmented reality experience is displayed in a first color corresponding to the first augmented reality experience (Agarawala, paragraph 38: “the AR system may generate a preview that may include other data or images;” the preview/representation includes multiple images which would have a wide variety of colors); and
at least a portion of the second representation of the second augmented reality experience is displayed in a second color corresponding to the second augmented reality experience, wherein the second color is different from the first color (Agarawala, paragraph 38: “the AR system may generate a preview that may include other data or images;” as mentioned in previous rejections, multiple previews are displayed together; the preview/representation includes multiple images which would have a wide variety of colors).
With regards to claim 157, which depends on claim 153, Agarawala discloses the first representation of the first augmented reality experience includes a first identifier corresponding to the first augmented reality experience; the second representation of the second augmented reality experience includes a second identifier different from the first identifier and corresponding to the second augmented reality experience (Agarawala, Fig. 5, 36A: icons representing their respective applications are displayed by the previews – twitter and instagram);
displaying the first augmented reality experience includes displaying the first identifier as part of the first augmented reality experience; and displaying the second augmented reality experience includes displaying the second identifier as part of the second augmented reality experience (Agarawala, Figs. 49A-B: icons representing the applications are shown when the applications are active).
With regards to claim 161, which depends on claim 153, Agarawala discloses in response to receiving the first user input: in accordance with a determination that the first user input corresponds to selection of the first representation of the first augmented reality experience (Agarawala, paragraph 118: “In FIG. 36A a user may select the Twitter® application from the various available applications that are accessible in the VR space”):
ceasing display of the representations of the first set of interactive elements; and displaying, via the one or more display generation components, the first set of interactive elements (Agarawala, paragraph 38: “the AR system may generate a preview that may include other data or images… If the user then activates Instagram®, the set of images may be expanded and/or more data may be displayed or made available for interaction/access;” the expanded view with interactive elements is considered distinct from the preview, thus displaying the expanded view includes ceasing display of the preview elements).
With regards to claim 162, which depends on claim 151, Agarawala discloses prior to receiving the first user input, and while concurrently displaying, in the three-dimensional environment, the representations of the plurality of augmented reality experiences (Agarawala, paragraph 35: “a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” fig. 36A: Shows that multiple preview objects can be shown at the same time):
displaying, via the one or more display generation components, the first representation of the first augmented reality experience at a first display position (Agarawala, paragraph 35: “a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” Fig. 61: multiple digital objects are displayed in a meeting space);
while displaying the first representation of the first augmented reality experience at the first display position, receiving, via the one or more input devices, a second user input corresponding to a request to navigate from the first representation of the first augmented reality experience to the second representation of the second augmented reality experience (Agarawala, paragraph 241: “Scroll 6116 may be a visual or digital indicator that appears on a digital canvas that only displays a portion or subset of the digital objects 6110 that are pinned to that wall… User 6102 may use a hand gesture to select scroll 6116 with their fingers;” Fig. 61: scroll); and
in response to receiving the second user input: ceasing display of the first representation of the first augmented reality experience at the first display position; and displaying, via the one or more display generation components, the second representation of the second augmented reality experience at the first display position (Agarawala, paragraph 240: “AR environment 6104 may configure the digital canvases of Room 2 with a scroll feature or scroll button (e.g., scroll 6116) as illustrated in opened space 6124A.” The user can scroll to move the digital objects (the preview/representations) around the 3D space).
With regards to claim 163, which depends on claim 151, Agarawala discloses wherein concurrently displaying the representations of the plurality of augmented reality experiences includes displaying the representations of the plurality of augmented reality experiences in a stack, wherein the first representation of the first augmented reality experience is stacked on top of the second representation of the second augmented reality experience (Agarawala, paragraph 48: “The user may perform any number of interactions with the data elements within the AR environment… stack/fan-out (stack elements together like playing cards, or fan out the elements of a stack so that one or more of the elements is more or fully visible);” the previews are data elements within the AR environment).
With regards to claim 164, which depends on claim 151, Agarawala discloses prior to receiving the first user input, and while concurrently displaying, in the three-dimensional environment, the representations of the plurality of augmented reality experiences including concurrently displaying the first representation of the first augmented reality experience and the second representation of the second augmented reality experience (Agarawala, paragraph 35: “a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” fig. 36A: Shows that multiple preview objects can be shown at the same time), receiving, via the one or more input devices, a third user input corresponding to a request to navigate through the representations of the plurality of augmented reality experiences (Agarawala, paragraph 48: “The user may perform any number of interactions with the data elements within the AR environment.”); and
in response to receiving the third user input: ceasing display of the first representation of the first augmented reality experience while maintaining display of the second representation of the second augmented reality experience (Agarawala, paragraph 48: “Example interactions include… hide/delete (removing an element from the AR display area)”).
With regards to claim 166, which depends on claim 151, Agarawala discloses the determination that the first user input corresponds to selection of the first representation of the first augmented reality experience comprises a determination that the first user input is a selection input (Agarawala, paragraph 118: “In FIG. 36A a user may select the Twitter® application from the various available applications that are accessible in the VR space”) that includes:
a voice input indicative of a user request to select a selectable object (Agarawala, paragraph 201: “As described herein, the AR/VR system may listen for and process voice commands from the users, and may generate visual displays of keywords or speech that may be acted upon by users in a workspace.” Paragraph 100: “The XR platform may receive data or input from a variety of different devices or methodologies. For example, input may be received through keyboards, voice, hand or other body gestures, touchscreen devices, etc”).
With regards to claim 179, which depends on claim 151, Agarawala discloses wherein the plurality of augmented reality experiences includes one or more of: a camera augmented reality experience; a translate augmented reality experience; a reading augmented reality experience; a music augmented reality experience; a navigation augmented reality experience; a photos augmented reality experience (Agarawala, fig. 5: Instagram is arguably a “photos” application, so the AR application would be a photos AR experience); a video messaging augmented reality experience; and/or a workout augmented reality experience (Agarawala, paragraph 16: “The data may include any 2D data, such as images, word processing documents, HTML, or other web pages, videos, etc;” the “experiences” are interpreted as the AR applications, which can include a wide variety of functionality).
Claim 180 recites substantially similar limitations to claim 151 and is thus rejected along the same rationale.
Claim 181 recites substantially similar limitations to claim 151 and is thus rejected along the same rationale.
Claim(s) 154 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz et al, and further in view of Kramer et al (US20170300122A1; filed 4/28/2017).
With regards to claim 154, which depends on claim 153, Agarawala discloses displaying the first augmented reality experience includes displaying the first set of interactive elements overlaid on a passthrough environment (Agarawala, paragraph 38: “If the user then activates Instagram®, the set of images may be expanded and/or more data may be displayed or made available for interaction/access;” figs. 36A-B show another example of an application (twitter) being expanded to display multiple objects in the AR environment);
displaying the second augmented reality experience includes displaying the second set of interactive elements overlaid on the passthrough environment (The second experience is just interpreted as a different application being opened. E.g. twitter instead of Instagram or vice versa; Agarawala, figs. 5, 36A-B);
the first representation of the first augmented reality experience includes first… background content representative of the passthrough environment; and the second representation of the second augmented reality experience includes second… background content representative of the passthrough environment (Agarawala, Fig. 5: the background of the application preview is transparent, and shows the passthrough environment behind the preview).
However, Agarawala and Gribetz do not teach yet Kramer et al teaches placeholder background content representative of the… environment (Kramer et al, paragraph 1801: “The asset placeholders are rectangles that exhibit behaviors identical to those of their full-resolution counterparts… The background of the asset is filled in with a light, transparent grey”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Kramer et al such that the background of the preview is a placeholder. This would have enabled the preview to be displayed smoothly even when the assets of the preview are loading slowly (Kramer et al, paragraph 1800: “this information is lightweight and allows the creation of placeholders that as nearly as possible represent the state of the dossier, and allow smooth transitions when the assets do finally load.”).
Claim(s) 158 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz et al, and further in view of Bradski et al (US20160026253A1; filed 6/13/2015).
With regards to claim 158, which depends on claim 153, Agarawala discloses in response to receiving the first user input: in accordance with a determination that the first user input corresponds to selection of the first representation of the first augmented reality experience (Agarawala, paragraph 118: “In FIG. 36A a user may select the Twitter® application from the various available applications that are accessible in the VR space”):
prior to displaying the first augmented reality experience, displaying, via the one or more display generation components, … the first representation of the first augmented reality experience moves toward a viewpoint of a user of the computer system (Agarawala, paragraph 35: “a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” paragraph 39: “In an embodiment, the Instagram® application may have been activated and may take on greater visual prominence relative to other inactive or background apps. For example, the image shows a New York Times® app that appears in the background and is less visible or at a greater depth than the Instagram® app.” The selected/expanded application is at a lower depth (closer) than the background application, thus it has been moved closer to the user’s viewpoint).
However, Agarawala and Gribetz do not disclose a first animation in which the… moves toward a viewpoint of a user.
Bradski et al teaches a first animation in which the… moves toward a viewpoint of a user (Bradski et al, paragraph 1314: “In another embodiment, the user interaction with the system may be through virtual “spiderwebs” created in the physical space around the user. For example, as shown in FIG. 131A, the user, may make a first and open it up 13102 such that virtual spider web strings are flung across space (FIG. 131B). To select a particular virtual icon/application/category of application, the user may pull along the spider web string 13104 to pull the virtual icon closer to him/her (FIG. 131C-131D).” Also see paragraph 1309: “the selection of a particular virtual icon may cause the virtual icon or page to move in the z direction by a hand gesture 12912 that makes the virtual icon come toward the user”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Bradski et al such that the selected object is animated as moving towards the user. This would have enabled the user to interact with the virtual object as if it were a real object by animating them moving naturally (Bradski et al, paragraph 1310: “In the AR system, the user is able to touch and hold documents, webpages, etc. as 3D virtual objects that can be flung into space, moved around, and physically manipulated as if they were real objects”).
Claim(s) 160 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz and Bradski et al, and further in view of Kramer et al.
With regards to claim 160, which depends on claim 158, Agarawala discloses in response to receiving the first user input: in accordance with a determination that the first user input corresponds to selection of the first representation of the first augmented reality experience: displaying, via the one or more display generation components,… of the representations of the first set of interactive elements with the first set of interactive elements (Agarawala, paragraph 38: “If the user then activates Instagram®, the set of images may be expanded and/or more data may be displayed or made available for interaction/access;” the activation is via the user input, and the preview expands into the activated version of the application).
However, Agarawala, Gribetz and Bradski et al do not disclose crossfading of the representations of the first set of interactive elements with the first set of interactive elements.
Kramer et al teaches crossfading of the representations of the first set of interactive elements with the first set of interactive elements (Kramer et al, Paragraph 1804: "The transition simply causes the new image to fade in from fully transparent to fully opaque over a short duration. The underlying placeholder or thumbnail does not fade out during this transition, but remains opaque until the transition completes, at which point it may be removed as appropriate. The thumbnail fades in linearly with a duration of 0.4 seconds. The crossfade from thumbnail to full res asset lasts 1.5 seconds.").
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, Bradski et al, and Kramer et al such that the representation crossfaded into the actual experience. This would have provided a smooth transition between the representation and the full user interface (Kramer et al, paragraph 1804: "The system provides a smooth transition between placeholder and thumbnail, and between thumbnail and full resolution image.”).
Claim(s) 165 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz, and further in view of Powderly et al (US20170109936A1; filed 10/18/2016).
With regards to claim 165, which depends on claim 151, Agarawala discloses the determination that the first user input corresponds to selection of the first representation of the first augmented reality experience comprises a determination that the first user input is a selection input… towards the first representation of the first augmented reality experience (Agarawala, paragraph 118: “In FIG. 36A a user may select the Twitter® application from the various available applications that are accessible in the VR space”).
However, Agarawala and Gribetz do not disclose a selection input that includes a gaze input directed toward the first…; and a hardware press input detected while the gaze input is directed toward the first...
Powderly et al teaches a selection input that includes a gaze input directed toward the first…; and a hardware press input detected while the gaze input is directed toward the first... (Powderly et al, paragraph 34: “The object nearest to the center of the user's field of view can be highlighted as a potential target object, and the user can actuate a user input device (e.g., by swiping on a touchscreen) to transport the highlight from one object to another. The user can confirm the selection of the target object by actuating the user input device again (e.g. by touching the touchscreen)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Powderly et al such that the input included a combination of gaze and press input. This would have enabled the user to make selections in mixed reality that are easy and do not cause significant fatigue (Powderly et al, paragraph 33: “For example, a user may select a virtual object using body poses such as physically approaching, grabbing or touching the items. The user may also select the virtual object by pointing at and clicking on the objects with virtual rays or beams. But these techniques can cause fatigue and can be difficult to select the objects with precision because the user may be required to hold his pose still to achieve the selection”).
Claim(s) 167 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz et al, and further in view of Schoen (US20220129082A1; filed 1/10/2022).
With regards to claim 167, which depends on claim 151, Agarawala discloses the determination that the first user input corresponds to selection of the first representation of the first augmented reality experience comprises a determination that the first user input is a selection input that includes: … input directed toward the first representation of the first augmented reality experience… (Agarawala, paragraph 118: “In FIG. 36A a user may select the Twitter® application from the various available applications that are accessible in the VR space”).
However, Agarawala and Gribetz do not disclose a gaze input directed toward… that meets a first set of gaze duration criteria.
Schoen teaches a gaze input directed toward… that meets a first set of gaze duration criteria (Schoen, paragraph 19: “While in the no-hands, 3DoF interaction mode, the interaction mode system can use a dwell timer that begins counting down when the direction vector does not make a threshold change within a time period and is reset when time the threshold change in the direction vector is detected. When the dwell timer counts down to zero, this can indicate a user action. Such an action, in combination with the vector direction, can cause a selection or other actuation of an object indicated by the gaze cursor”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz and Schoen such that the input included a gaze input with a dwell time threshold. This would have enabled a user to make selections without needing to use their hands (Schoen, paragraph 19: “Interaction mode context factors that can trigger a transition to the no-hands, 3DoF interaction mode can include factors such as when position tracking cameras are off or otherwise unavailable (e.g., when in low power mode, privacy mode, public mode, etc.) or when lighting conditions are such that reliable position and hand tracking cannot be accomplished”).
Claim(s) 168, 170 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz et al, and further in view of Faulkner et al (US20210096726A1; filed 9/23/2020).
With regards to claim 168, which depends on claim 151, Agarawala discloses while concurrently displaying the representations of the plurality of augmented reality experiences, displaying… controls;
… while concurrently displaying the representations of the plurality of augmented reality experiences and while the first setting is set to the second value, receiving, via the one or more input devices, a third user input (Agarawala, paragraph 35: “Using the feed and the user's account information, AR system may extract and display new or other data in a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” fig. 36A: Shows that multiple preview objects can be shown at the same time); and
in response to receiving the third user input:
in accordance with a determination that the third user input corresponds to selection of the first representation of the first augmented reality experience, displaying, via the one or more display generation components, in the three-dimensional environment, the first augmented reality experience while maintaining the first setting at the second value (Agarawala, paragraph 119: “As shown in FIG. 36B, upon selection of Twitter® or the performance of another hand gesture, the elements or data of Twitter may be expanding across one or more meshes or locations within the room.” A system setting such as volume is generally unaffected by the activating or selecting of an application); and
in accordance with a determination that the third user input corresponds to selection of the second representation of the second augmented reality experience, displaying, via the one or more display generation components, in the three-dimensional environment, the second augmented reality experience while maintaining the first setting at the second value (Agarawala, paragraph 119: “As shown in FIG. 36B, upon selection of Twitter® or the performance of another hand gesture, the elements or data of Twitter may be expanding across one or more meshes or locations within the room.” There are multiple previews that can be selected in Fig. 36A; A system setting such as volume is generally unaffected by the activating or selecting of an application).
However, Agarawala and Gribetz do not disclose displaying, via the one or more display generation components, one or more setting controls, including a first setting control corresponding to a first setting of the computer system;
while displaying the one or more setting controls, receiving, via the one or more input devices, a first settings input corresponding to the first setting of the computer system;
in response to receiving the first settings input, modifying the first setting from a first value to a second value different from the first value.
Faulkner et al teaches displaying, via the one or more display generation components, one or more setting controls, including a first setting control corresponding to a first setting of the computer system (Faulkner et al, paragraph 36: “a control user interface for adjusting one or more device parameters”);
while displaying the one or more setting controls, receiving, via the one or more input devices, a first settings input corresponding to the first setting of the computer system; in response to receiving the first settings input, modifying the first setting from a first value to a second value different from the first value (Faulkner et al, paragraph 36: “a control user interface for adjusting one or more device parameters of the computer system (e.g., brightness of the display, audio volume, network connection, etc.)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz and Faulkner et al such that the user is able to change system settings that are independent of individual applications. This would have enabled the user to have more control of the device (Faulkner et al, paragraph 12: “Such methods and interfaces produce a more efficient human-machine interface and allow the user more control of the device, allow the user to use the device with better safety, reduced cognitive burden, and improved user experience”).
With regards to claim 170, which depends on claim 168, Agarawala and Gribetz do not disclose the first setting is a volume setting; the first value corresponds to a first volume; and the second value corresponds to a second volume different from the first volume.
However, Faulkner et al teaches the first setting is a volume setting; the first value corresponds to a first volume; and the second value corresponds to a second volume different from the first volume (Faulkner et al, paragraph 36: “a control user interface for adjusting one or more device parameters of the computer system (e.g., brightness of the display, audio volume, network connection, etc.);” adjusting a parameter would mean changing the parameter, which would necessitate a second volume).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Faulkner et al such that the user is able to change system settings that are independent of individual applications. This would have enabled the user to have more control of the device (Faulkner et al, paragraph 12: “Such methods and interfaces produce a more efficient human-machine interface and allow the user more control of the device, allow the user to use the device with better safety, reduced cognitive burden, and improved user experience”).
Claim(s) 169 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz and Faulkner et al, and further in view of Osman et al (US20180311583A1; filed 7/3/2018).
With regards to claim 169, which depends on claim 168, the combination of Agarawala, Gribetz, and Faulkner et al does not teach the first setting is a passthrough tinting setting;
the first value corresponds to a first amount of tinting applied to the three-dimensional environment;
and the second value corresponds to a second amount of tinting applied to the three-dimensional environment different from the first amount of tinting.
However, Osman et al teaches the first setting is a passthrough tinting setting;
the first value corresponds to a first amount of tinting applied to the three-dimensional environment;
and the second value corresponds to a second amount of tinting applied to the three-dimensional environment different from the first amount of tinting (Osman et al, abstract: “transitioning at least a portion of the screen to a transparent mode. The transparent mode provides at least a partial view into the real world space using the camera of the HMD;” paragraph 113: “The game processing module of HMD detects occurrence of a trigger event and allows a HMD view to transition between a “non-transparent” mode, a “semi-transparent” mode, a “simulated transparent” mode and a “fully transparent” mode based on trigger event. In some embodiments, the user may be provided with options to determine which transparent mode transition is desired by the user and the transition is performed in accordance to the option chosen”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, Faulkner et al, and Osman et al such that the user can change settings for the tinting or transparency of the passthrough HMD. This would have enabled the user to select the mode they desire to use (Osman et al, paragraph 113: “the user may be provided with options to determine which transparent mode transition is desired by the user”).
Claim(s) 171 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz and Faulkner et al, and further in view of Scavezze et al (US20140320389A1; filed 4/29/2013).
With regards to claim 171, which depends on claim 168, Agarawala discloses while concurrently displaying the representations of the plurality of augmented reality experiences (Agarawala, paragraph 35: “Using the feed and the user's account information, AR system may extract and display new or other data in a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” fig. 36A: Shows that multiple preview objects can be shown at the same time).
However, Agarawala and Gribetz do not disclose while concurrently displaying… and the one or more setting controls, displaying, via the one or more display generation components, device status information indicative of a status of one or more characteristics of the computer system.
Faulkner et al teaches while concurrently displaying… and the one or more setting controls, (Faulkner et al, paragraph 36: “a control user interface for adjusting one or more device parameters of the computer system (e.g., brightness of the display, audio volume, network connection, etc.);” adjusting a parameter would mean changing the parameter, which would necessitate a second volume).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz and Faulkner et al such that the user is able to change system settings that are independent of individual applications. This would have enabled the user to have more control of the device (Faulkner et al, paragraph 12: “Such methods and interfaces produce a more efficient human-machine interface and allow the user more control of the device, allow the user to use the device with better safety, reduced cognitive burden, and improved user experience”).
Scavezze et al teaches displaying, via the one or more display generation components, device status information indicative of a status of one or more characteristics of the computer system (Scavezze et al, paragraph 66: “The current volume may also be displayed as a digital volume indicator 380.” Fig. 3: volume indicator 380 in AR view).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz Faulkner et al, and Scavezze et al such that the current volume can be displayed to the user. This would have enabled the user to know the current volume of the device when they are adjusting it (Scavezze et al, paragraph 66: “the user 304 may pick up and physically rotate the basketball 332 to correspondingly adjust the volume of the broadcast via the virtual volume control knob”).
Claim(s) 172-175 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz et al, and further in view of Poulos et al (US20140347390A1; filed 5/22/2013).
With regards to claim 172, which depends on claim 151, Agarawala discloses the representations of the plurality of augmented reality experiences (Agarawala, paragraph 35: “Using the feed and the user's account information, AR system may extract and display new or other data in a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user”).
However, Agarawala and Gribetz do not disclose wherein… are viewpoint-locked objects that stay in a respective region of a field of view of a user of the computer system as the viewpoint of the user shifts relative to the three-dimensional environment.
Poulos et al teaches wherein… are viewpoint-locked objects that stay in a respective region of a field of view of a user of the computer system as the viewpoint of the user shifts relative to the three-dimensional environment (Poulos et la, paragraph 13: “In a head-locking approach, a virtual object may be locked to the position and direction of a user's head such that the location of the virtual object in a user's field of view appears fixed regardless of a direction in which the user faces”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Poulos et al such that the representations were head-locked/viewpoint-locked. This would have enabled the invention to persistently display the previews/representations to the user (Poulos et al, paragraph 13: “Head-locking may be used, for example, to display persistent user interface icons”).
With regards to claim 173, which depends on claim 172, Agarawala discloses concurrently displaying the representations of the plurality of augmented reality experiences comprises concurrently displaying the representations of the plurality of augmented reality experiences in a first orientation in which the representations of the plurality of augmented reality experiences are aligned … concurrently displaying the representations of the plurality of augmented reality experiences (Agarawala, paragraph 35: “Using the feed and the user's account information, AR system may extract and display new or other data in a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” fig. 36A: Shows that multiple preview objects can be shown at the same time).
However, Agarawala and Gribetz do not disclose a first orientation… aligned to gravity; and the one or more programs further include instructions for: while … displaying…, detecting a change in orientation of the viewpoint of the user; and in response to detecting the change in orientation of the viewpoint of the user: rotating the representations of the plurality of augmented reality experiences from the first orientation to a second orientation based on the change in orientation of the viewpoint of the user to continue to align the representations of the plurality of augmented reality experiences to gravity.
Poulos et al teaches a first orientation… aligned to gravity (Poulos et al, paragraph 35: “information representative of gravity (e.g., a gravity vector made available by an accelerometer) may be utilized to maintain an upward orientation of a holographic television from the user's perspective”); and the one or more programs further include instructions for: while… displaying…, detecting a change in orientation of the viewpoint of the user; and in response to detecting the change in orientation of the viewpoint of the user: rotating the representations of the plurality of augmented reality experiences from the first orientation to a second orientation based on the change in orientation of the viewpoint of the user (Poulos et al, paragraph 35: “one or more dimensions of a virtual object may be rotated to automatically direct a selected aspect of a virtual object toward the user's body. Permitted rotational and translational motion of objects may be properties or policies associated with selected virtual objects and/or virtual object types;” automatically directing the orientation towards the user requires detecting changes in user orientation) to continue to align the representations of the plurality of augmented reality experiences to gravity (Poulos et al, paragraph 35: “information representative of gravity (e.g., a gravity vector made available by an accelerometer) may be utilized to maintain an upward orientation of a holographic television from the user's perspective.”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Poulos et al such that the representations were head-locked/viewpoint-locked. This would have enabled the invention to persistently display the previews/representations to the user (Poulos et al, paragraph 13: “Head-locking may be used, for example, to display persistent user interface icons”).
With regards to claim 174, which depends on claim 173, Agarawala discloses displaying, via the one or more display generation components, the representations of the plurality of augmented reality experiences in the first orientation… the representations of the plurality of augmented reality experiences… displaying, via the one or more display generation components, the representations of the plurality of augmented reality experiences (Agarawala, paragraph 35: “Using the feed and the user's account information, AR system may extract and display new or other data in a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” fig. 36A: Shows that multiple preview objects can be shown at the same time).
However, Agarawala and Gribetz do not disclose at a first time subsequent to detecting the change in orientation of the viewpoint of the user, displaying… in the first orientation, wherein at the first time… are not aligned to gravity due, at least in part, to the change in orientation of the viewpoint of the user; and at a second time subsequent to the first time, displaying… in the second orientation to align the representations of the plurality of augmented reality experiences to gravity.
Poulos et al teaches at a first time subsequent to detecting the change in orientation of the viewpoint of the user, displaying… in the first orientation, wherein at the first time… are not aligned to gravity due, at least in part, to the change in orientation of the viewpoint of the user; and at a second time subsequent to the first time, displaying… in the second orientation to align the representations of the plurality of augmented reality experiences ((Poulos et al, paragraph 35: “one or more dimensions of a virtual object may be rotated to automatically direct a selected aspect of a virtual object toward the user's body. Permitted rotational and translational motion of objects may be properties or policies associated with selected virtual objects and/or virtual object types;” paragraph 33: “repositioning of virtual objects may be performed in a more smooth and gradual manner, and may be carried out in an object-specific way;” the smooth and gradual manner requires intermediate steps of rotation prior to the correct alignment) to gravity (Poulos et al, paragraph 35: “information representative of gravity (e.g., a gravity vector made available by an accelerometer) may be utilized to maintain an upward orientation of a holographic television from the user's perspective”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Poulos et al such that the representations were head-locked/viewpoint-locked. This would have enabled the invention to persistently display the previews/representations to the user (Poulos et al, paragraph 13: “Head-locking may be used, for example, to display persistent user interface icons”).
With regards to claim 175, which depends on claim 172, Agarawala discloses displaying the first augmented reality experience includes concurrently displaying a first set of objects, including a first object and a second object, and wherein: the first object is a… object; and the second object is an environment-locked object (Agarawala, Figs. 50A-50B, 59B: several figures show the expanded applications, which include a plurality of objects; Fig. 2B, 3: virtual objects are positioned on meshes, which can be anchored on walls (fig 2B) or hover in the air (fig 3-5); paragraph 28: “The various surfaces may include walls, ceilings, floors, screens, table tops, etc. In an embodiment, the AR system may superimpose a mesh, grid, or screen on each of these surfaces. These meshes may indicate a default or initial starting point for where data may be positioned.”).
However, Agarawala and Gribetz do not disclose the first object is a viewpoint-locked object.
Poulos et al teaches the first object is a viewpoint-locked object (Poulos et la, paragraph 13: “In a head-locking approach, a virtual object may be locked to the position and direction of a user's head such that the location of the virtual object in a user's field of view appears fixed regardless of a direction in which the user faces”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Poulos et al such that the representations were head-locked/viewpoint-locked. This would have enabled the invention to persistently display the previews/representations to the user (Poulos et al, paragraph 13: “Head-locking may be used, for example, to display persistent user interface icons”).
Claim(s) 176 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz et al, and further in view of Jacobsen et al (US20120236025A1; filed 9/16/2011).
With regards to claim 176, which depends on claim 151, Agarawala discloses displaying, via the one or more display generation components, in the three-dimensional environment, the first augmented reality experience (Agarawala, paragraph 38: “If the user then activates Instagram®, the set of images may be expanded and/or more data may be displayed or made available for interaction/access;” the expanded view with interactive elements is interpreted as the experience);
while displaying the first augmented reality experience, receiving, via the one or more input devices… input indicative of a user request to change from the first augmented reality experience to the second augmented reality experience (Agarawala, paragraph 38: “If the user then activates Instagram®, the set of images may be expanded and/or more data may be displayed or made available for interaction/access;” paragraph 40: “with different applications, the preview, new, followed, or most recently accessed, data may vary.” The second augmented experience is interpreted as a different application from the first one, which also has its own preview and displayed data, and can be opened similarly to the first application).
However, Agarawala and Gribetz do not disclose receiving, via the one or more input devices, a first voice input indicative of a user request to change from the first… to the second…in response to receiving the first voice input: ceasing display of the first augmented reality experience; and displaying, via the one or more input devices, in the three-dimensional environment, the second augmented reality experience.
Jacobsen et al teaches receiving, via the one or more input devices, a first voice input indicative of a user request to change from the first… to the second… (Jacobsen et al, paragraph 57: “voice inputs can be used to operate multiple desktops and/or running-applications… To watch a movie, the user can issue a ‘new desktop’ spoken command and then select a movie viewing application from the new desktop. This then allows the movie viewing to take place in the current window. However the photo viewing application is still running but not visible. The user can switch to the photo viewing application by issuing a suitable command to switch viewable desktops (“Go to desktop 1”)”); in response to receiving the first voice input: ceasing display of the first augmented reality experience; and displaying, via the one or more input devices, in the three-dimensional environment, the second augmented reality experience (Jacobsen et al, paragraph 57: “FIG. 9 illustrates another implementation for how the head tracking and/or voice inputs can be used to operate multiple desktops and/or running-applications… To watch a movie, the user can issue a ‘new desktop’ spoken command and then select a movie viewing application from the new desktop. This then allows the movie viewing to take place in the current window. However the photo viewing application is still running but not visible. The user can switch to the photo viewing application by issuing a suitable command to switch viewable desktops (“Go to desktop 1”)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and Jacobsen et al such that the user can switch between experiences with voice input. This would have enabled hands-free switching between displayed AR experiences/applications (Jacobsen et al, paragraph 7: “The processor converts the voice, motion or gesture inputs or combination thereof, to a user command. The user command is then interpreted by an operating system or application program to change aspects of the information displayed on the microdisplay”).
Claim(s) 177-178 is/are rejected under 35 U.S.C. 103 as being unpatentable over Agarawala in view of Gribetz et al, and further in view of White (US20030149929A1; filed 2/1/2002)
With regards to claim 177, which depends on claim 151, Agarawala discloses
displaying, via the one or more display generation components, the first augmented reality experience (Agarawala, paragraph 119: “As shown in FIG. 36B, upon selection of Twitter® or the performance of another hand gesture, the elements or data of Twitter may be expanding across one or more meshes or locations within the room.”).
However, Agarawala and Gribetz do not disclose while the computer system is in a sleep state, receiving, via the one or more input devices, a first wake input corresponding to a request to transition the computer system from a sleep state to a wake state; and in response to receiving the first wake input, displaying…
White teaches while the computer system is in a sleep state, receiving, via the one or more input devices, a first wake input corresponding to a request to transition the computer system from a sleep state to a wake state; and in response to receiving the first wake input, displaying (White, paragraph 6: “When the user returns to the laptop computer and enters a command, such as striking any key on the keyboard, the CPU restores power to the sleeping components and the user is able to resume activity right where they left off”)…
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and White such that the device could enter a sleep mode, and wake the device to resume the previous activities where the user left off. This would have enabled a device to conserve power when not in use (White, paragraph 6: “Power conservation is an important consideration in many modern devices”).
With regards to claim 178, which depends on claim 151, Agarawala discloses
displaying, via the one or more display generation components, the representations of the plurality of augmented reality experiences (Agarawala, paragraph 35: “a three dimensional floating preview (also called 3D flow view) in the real-world environment of the user;” fig. 36A: Shows that multiple preview objects can be shown at the same time).
However, Agarawala and Gribetz do not disclose while the computer system is in a sleep state, receiving, via the one or more input devices, a first wake input corresponding to a request to transition the computer system from a sleep state to a wake state; and in response to receiving the first wake input, displaying…
White teaches while the computer system is in a sleep state, receiving, via the one or more input devices, a first wake input corresponding to a request to transition the computer system from a sleep state to a wake state; and in response to receiving the first wake input, displaying (White, paragraph 6: “When the user returns to the laptop computer and enters a command, such as striking any key on the keyboard, the CPU restores power to the sleeping components and the user is able to resume activity right where they left off”)…
It would have been obvious to a person of ordinary skill in the art before the effective filing date to have combined Agarawala, Gribetz, and White such that the device could enter a sleep mode, and wake the device to resume the previous activities where the user left off. This would have enabled a device to conserve power when not in use (White, paragraph 6: “Power conservation is an important consideration in many modern devices”).
Response to Arguments
Applicant’s arguments with respect to the independent claims, and all claims dependent on them, have been considered but are moot because the new ground of rejection does not rely on only the same references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Applicant argues that Agarawala fails to disclose the amendments to the independent claims regarding the border that is ceased to display. Examiner agrees, however upon additional search and consideration has updated the rejections above to be further in view of Gribetz et al, who teaches surrounding elements such as applications and folders with bubbles which can be popped to open the folder/application while removing the border (bubble outline). Thus the amended claims are rejected and the argument is moot.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRODERICK C ANDERSON whose telephone number is (313)446-6566. The examiner can normally be reached Monday-Tuesday, Thursday-Saturday 9-5 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at 5712724124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.C.A/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
Prosecution Insights