Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
1. This action is responsive to applicant’s amendment dated 7/2/2026.
2. Claims 1-20 are pending in the case.
3. Claims 1, 9 and 17 are independent claims.
Applicant’s Response
4. In Applicant’s response dated 7/2/2026, applicant has amended the following:
a) Claims 1, 5-7, 9 and 17
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 4-6 , 9, 10, 12-14, 17, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Poulos et al. (hereinafter “Poulos”), U.S. Published Application No. 20140306993 A1 in view of Mott et al. (hereinafter “Mott”), U.S. Patent No. 9734634 B, in further view of Thiercelin et al. (hereinafter “Thiercelin”), U.S. Published Application No. 20180067631 A1.
Claim 1:
Poulos teaches A method comprising:
rendering, by at least one processor, a semi-transparent or opaque board on a display of an extended reality (XR) headset in communication with the at least one processor, (e.g., rendering a virtual bulletin board (i.e., opaque board) via HMD (i.e., XR headset) par. 17; The snap grid space may be associated with a real-world object (e.g., a portion of a wall within a room or a tabletop surface) or a virtual object (e.g., a moveable virtual bulletin board or virtual tabletop). In some embodiments, a mobile device, such as a head-mounted display device (HMD), may identify one or more grid spaces within an augmented reality environment, Par. 18; In some cases, the snap grid space may be associated with a second virtual object (e.g., a virtual desk or a virtual wall) within an augmented reality environment and the virtual object may be positioned relative to the second virtual object)
receiving, by the at least one processor, a first user input to open a widget library; (e.g., input to access virtual objects via virtual data engine (i.e., widget library) Examiner considers the rendering of the virtual objects by the virtual data engine prior to selection to move onto the grid to be consider the “widget library” par. 18; In some embodiments, a virtual object (e.g., a holographic TV, a holographic web browser, or a holographic painting) may be positioned within an augmented reality environment and then automatically snapped to a grid location corresponding with a snap grid space within the augmented reality environment. Par. 47; Virtual data engine 197 processes virtual objects and registers the position and orientation of virtual objects in relation to various maps of a real-world environment stored in memory unit 192. The virtual data engine may also render images associated with virtual objects for display to an end user of computing system 10.)
rendering, by the at least one processor, a grid structure on the board after receiving the first user input; (e.g., rendering a snap grid structure with virtual bulletin board after receiving a positioning event gesture par. 17; The snap grid space may be associated with a real-world object (e.g., a portion of a wall within a room or a tabletop surface) or a virtual object (e.g., a moveable virtual bulletin board or virtual tabletop). Par. 18; In some cases, the snap grid space may be associated with a second virtual object (e.g., a virtual desk or a virtual wall) within an augmented reality environment and the virtual object may be positioned relative to the second virtual object par. 19; In some embodiments, a virtual object within a snap grid space may be automatically snapped to the closest (or nearest) grid point within the snap grid space upon the detection of a positioning event (e.g., after being positioned and released by an end user of an HMD). A positioning event may be triggered, for example, if an end user of an HMD performs a virtual object copy-and-paste function, a virtual object cut-and-paste function, or a virtual object move function.)
receiving, by the at least one processor, a second user input to select a widget from the widget library (e.g., input to access virtual objects via virtual data engine (i.e., widget library) Examiner considers the rendering of the virtual objects by the virtual data engine prior to selection to move onto the grid to be consider the “widget library” par. 18; In some embodiments, a virtual object (e.g., a holographic TV, a holographic web browser, or a holographic painting) may be positioned within an augmented reality environment and then automatically snapped to a grid location corresponding with a snap grid space within the augmented reality environment. Par. 32; In some embodiments, an end user of a mobile device may position (or reposition) a virtual object within an augmented reality environment by selecting the virtual object and moving the virtual object into a region of the augmented reality environment associated with a snap grid space.)
receiving, by the at least one processor, a third user input to move the selected widget to the grid structure on the board; (e.g., positioning virtual object onto a snap grid virtual bulletin board par. 18; In some embodiments, a virtual object (e.g., a holographic TV, a holographic web browser, or a holographic painting) may be positioned within an augmented reality environment and then automatically snapped to a grid location corresponding with a snap grid space within the augmented reality environment. Par. 32; In some embodiments, an end user of a mobile device may position (or reposition) a virtual object within an augmented reality environment by selecting the virtual object and moving the virtual object into a region of the augmented reality environment associated with a snap grid space.)
placing, by the at least one processor, the selected widget at a position in the grid structure on the board; (e.g., placing virtual object over the snap grid on the virtual bulletin board par. 17; The snap grid space may be associated with a real-world object (e.g., a portion of a wall within a room or a tabletop surface) or a virtual object (e.g., a moveable virtual bulletin board or virtual tabletop). par. 79; In one embodiment, each of the one or more two-dimensional snap grid spaces may be highlighted by placing a highlighting box and/or a highlighting color over each of the one or more two-dimensional snap grid spaces. In some embodiments, a subset of the one or more two-dimensional snap grid spaces may be highlighted if the first virtual object is within a proximity of the subset or overlaps with the subset. The highlighting may comprise projecting a virtual wireframe mesh or virtual point grid of snap points over the subset of the one or more two-dimensional snap grid spaces. In step 575, an overlap of the first virtual object and a first snap grid space of the one or more two-dimensional snap grid spaces is detected. In step 576, the first snap grid space is outputted. The first snap grid may be outputted upon a virtual object release gesture performed by an end user of an HMD.)
and stopping, by the at least one processor, rendering the grid structure while displaying the selected widget at the position on the board with an orientation determined based at least partly on a user's head position. (e.g., stop highlighting the snap grid space after positioning the virtual object and the orientation determined based on the HMD (i.e., user’s head position) par. 17; In some embodiments, a mobile device, such as a head-mounted display device (HMD), may identify one or more grid spaces within an augmented reality environment, detect a positioning (or re-positioning) of a virtual object within the augmented reality environment, determine a target grid space of the one or more grid spaces in which to position the virtual object, determine a grid spacing associated with the target grid space, determine a position and an orientation of the virtual object within the target grid space based on the grid spacing, and display the virtual object within the augmented reality environment based on the position and the orientation of the virtual object within the target grid space. par. 32; The snap grid space may be highlighted (e.g., a virtual wireframe mesh or virtual point grid of snap points may be used to highlight the grid space) to help identify the snap grid space to the end user during positioning of the virtual object within the augmented reality environment. Once the virtual object has been positioned (e.g., by being moved and released by the end user) relative to a snap grid space, the virtual object may be automatically snapped to the closest grid point within the snap grid space.)
Poulos fails to expressly teach such that the board does not overlap with at least one real-world object recognized by the processor;
However, Mott teaches such that the board does not overlap with at least one real-world object recognized by the processor; (e.g., preventing virtual objects from overlapping real world objects col. 8 line 37; In some embodiments, a container 320 may be confined to moving with one of its surfaces on a plane, which may be on or parallel to the plane that a marker 310 is on. Of course, in some embodiments collision detection may be employed to prevent a container 320 from overlapping with an object in the real-world (regardless of whether the computing device utilizes a single camera, stereoscopic vision, etc.). col. 12 line 66; In another example, a computing device 810 may prevent a user from moving an object such that it overlaps with another object in a camera environment (e.g., the container 870 stops moving when it collides with an object that is based on an object in the real-world environment).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the placement of virtual objects as taught by Poulos to avoid real world objects as taught by Mott, with a reasonable expectation of success, to provide the benefit of improving the organization of virtual objects viewed in the augment reality environment.
Poulos/Mott fails to expressly teach rendering, by the at least one processor, at least part of the widget library on the display of the XR headset;
the widget library rendered on the display of the XR headset;
However, Thiercelin teaches
rendering, by the at least one processor, at least part of the widget library on the display of the XR headset; receiving, by the at least one processor, a second user input to select a widget from the widget library rendered on the display of the XR headset; (e.g., collection of selectable widgets displayed in user interface of a wearable computing device of a virtual reality or augmented reality system par. 36; As shown in FIG. 2, UI manager 212 can include various movable object-aware UI elements, including widgets 226, panels 228, collections 238, and workflows 230. Widgets 226 may include small UI elements which provide information (e.g., height, compass, battery, range, video signal strength, remote control signal strength, telemetry data, pitch and roll, etc.) or may be selected to perform a function (e.g., return to home, takeoff, capture a photograph, capture video, etc.). par. 41; In some embodiments, collections 238 can define groupings of widgets. Par. 87; As discussed, various UI elements can register with a data manager interface of a UI manager, which may receive data from a movable object. In various embodiments, the at least one user interface element can be a widget or a panel including a plurality of user interface elements. Par. 89; At step 1406, a first rendering of the at least one user interface element can be caused to be displayed in a user interface, the first rendering determined based on a type or amount of the sensor data associated with the movable object. Par. 92; In some embodiments, the user interface can be displayed on a user device, and the user device may include one of a portable computing device, wearable computing device, virtual reality system, or augmented reality system.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the collection of virtual objects as taught by Poulos/Mott to be displayed as a collection of selectable widgets within in a user interface of a wearable computing device of a virtual reality system as taught by Thiercelin, with a reasonable expectation of success, to provide the benefit of improving the management of organized selectable virtual object widgets viewed in an augment reality environment (see Thiercelin; par. 36, par. 41;).
Claim 2 depends on claim 1:
Poulos teaches wherein the grid structure includes a dot grid structure or a line grid structure. (e.g., snap grid with anchor points (i.e., dot grid structure) par. 19; In some cases, the virtual object may snap its position and orientation relative to a set of grid points (or anchor points) within the snap grid space)
Claim 4 depends on claim 1:
Poulos teaches wherein placing the selected widget at the position comprises moving one or more pre-existing widgets within the grid structure. (e.g., re-positioning virtual object within snap grid virtual bulletin board par. 18; In some embodiments, a virtual object (e.g., a holographic TV, a holographic web browser, or a holographic painting) may be positioned within an augmented reality environment and then automatically snapped to a grid location corresponding with a snap grid space within the augmented reality environment. Par. 32; In some embodiments, an end user of a mobile device may position (or reposition) a virtual object within an augmented reality environment by selecting the virtual object and moving the virtual object into a region of the augmented reality environment associated with a snap grid space.)
Claim 5 depends on claim 1:
Poulos teaches wherein placing the selected widget at the position comprises: highlighting one or more candidate grid positions for the selected widget; and visually indicating one or more locations occupied by one or more virtual or real-world objects recognized by the at least one processor as obstacles. (e.g., placing virtual object over highlighted candidate grid positions within the snap grid on the virtual bulletin board and visually indicating the occupied placement of the released virtual object par. 17; The snap grid space may be associated with a real-world object (e.g., a portion of a wall within a room or a tabletop surface) or a virtual object (e.g., a moveable virtual bulletin board or virtual tabletop). par. 79; In one embodiment, each of the one or more two-dimensional snap grid spaces may be highlighted by placing a highlighting box and/or a highlighting color over each of the one or more two-dimensional snap grid spaces. In some embodiments, a subset of the one or more two-dimensional snap grid spaces may be highlighted if the first virtual object is within a proximity of the subset or overlaps with the subset. The highlighting may comprise projecting a virtual wireframe mesh or virtual point grid of snap points over the subset of the one or more two-dimensional snap grid spaces. In step 575, an overlap of the first virtual object and a first snap grid space of the one or more two-dimensional snap grid spaces is detected. In step 576, the first snap grid space is outputted. The first snap grid may be outputted upon a virtual object release gesture performed by an end user of an HMD.)
Claim 6 depends on claim 1:
Poulos teaches wherein: the second user input and the third user input comprise images of the user's eyes captured using an eye tracking camera coupled to the XR headset; and the method further comprises detecting, based on the images from the eye tracking camera, that the user's eyes are focused on the selected widget. (e.g., eye tracking camera detecting gaze detection input on virtual objects par. 39; In some embodiments, HMD 200 may perform gaze detection for each eye of an end user's eyes using gaze detection elements par. 68; In some cases, the grid spacing may be determined based on the precision of the tracking angular resolution of the mobile device providing the augmented reality environment (e.g., an HMD). For example, if the mobile device can only detect movements greater than one degree movements of the end user's head (or eyes))
Independent Claim 9:
Claim 9 is substantially encompassed in claim 1, therefore, Examiner relies on the same rationale set forth in claim 1 to reject claim 9. (see Poulos, Figure 1; networked computing environment)
Claim 10 depends on claim 9:
Claim 10 is substantially encompassed in claim 2, therefore, Examiner relies on the same rationale set forth in claim 2 to reject claim 10.
Claim 12 depends on claim 9:
Claim 12 is substantially encompassed in claim 4, therefore, Examiner relies on the same rationale set forth in claim 4 to reject claim 12.
Claim 13 depends on claim 9:
Claim 13 is substantially encompassed in claim 5, therefore, Examiner relies on the same rationale set forth in claim 5 to reject claim 13.
Claim 14 depends on claim 9:
Claim 14 is substantially encompassed in claim 6, therefore, Examiner relies on the same rationale set forth in claim 6 to reject claim 14.
Independent Claim 17:
Claim 17 is substantially encompassed in claim 1, therefore, Examiner relies on the same rationale set forth in claim 1 to reject claim 9. (see Poulos, Figure 1; networked computing environment)
Claim 18 depends on claim 17:
Claim 18 is substantially encompassed in claim 2, therefore, Examiner relies on the same rationale set forth in claim 2 to reject claim 9.
Claim 20 depends on claim 17:
Claim 20 is substantially encompassed in claim 4, therefore, Examiner relies on the same rationale set forth in claim 4 to reject claim 20.
Claims 3, 11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Poulos/ Mott/Thiercelin as cited above, in further view of Ravasz et al. (hereinafter “Ravasz”), U.S. Patent No. 11334212 B2.
Claim 3 depends on claim 1:
Poulos/Mott/Thiercelin fail to expressly teach wherein the second user input includes a pinch gesture, and the third user input includes a drag gesture while maintaining the pinch gesture.
However, Ravasz teaches wherein the second user input includes a pinch gesture, and the third user input includes a drag gesture while maintaining the pinch gesture. (e.g., using pinch gesture and pull gesture (i.e., drag gesture) to manipulate UI elements in artificial reality environment col 2. Line 8; In some examples, a user may interact with such a UI pinch element through a gesture or movement that may include two fingers of a hand being brought together and/or forming a pinching configuration in the vicinity of the UI pinch element, followed by a subsequent pulling motion of the hand and/or fingers while the hand is in the pinching configuration. Col. 2 line19; In some examples, the graphical user interface element presented in response to the pinch and pull gesture (or in response to a stylus action) may present options for specifying input, where that input may be specified by a user for the purpose of making an adjustment to one or more aspects of the artificial reality environment. )
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the placement of virtual objects as taught by Poulos/ Mott/Thiercelin to be based on pinch and pull gestures as taught by Ravasz with a reasonable expectation of success, to provide the benefit of flexible intuitive interactions from a user in effort to improve the user experience.
Claim 11 depends on claim 9:
Claim 11 is substantially encompassed in claim 3, therefore, Examiner relies on the same rationale set forth in claim 3 to reject claim 11.
Claim 19 depends on claim 17:
Claim 19 is substantially encompassed in claim 3, therefore, Examiner relies on the same rationale set forth in claim 3 to reject claim 19.
Claims 7, 8, 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Poulos/ Mott/Thiercelin as cited above, in further view of Solichin; Jonathan, U.S. Published Application No. 20230368472 A1.
Claim 7 depends on claim 1:
Poulos/ Mott/Thiercelin teaches wherein: the second user input comprises images of the user's eyes captured using an eye tracking camera coupled to the XR headset; (e.g., eye tracking camera detecting gaze detection input on virtual objects Poulos;par. 39; In some embodiments, HMD 200 may perform gaze detection for each eye of an end user's eyes using gaze detection elements par. 68; In some cases, the grid spacing may be determined based on the precision of the tracking angular resolution of the mobile device providing the augmented reality environment (e.g., an HMD). For example, if the mobile device can only detect movements greater than one degree movements of the end user's head (or eyes))
the third user input comprises a voice input from the user; (e.g., voice recognition techniques; Poulos; par. 48; or example, object recognition may be used to detect particular objects (e.g., a pencil held by an end user of an HMD) and facial recognition may be used to detect the face of a particular person within an environment. Image and audio processing engine 194 may apply audio and voice recognition techniques to audio data. For example, audio recognition may be used to detect a particular sound. The particular faces, voices, sounds, and objects to be detected may be stored in one or more memories contained in memory unit 192.)
Poulos/ Mott/Thiercelin teaches fails to expressly teach the method further comprises providing the voice input to a natural language processing module trained to identify keywords in the voice input including instructions to move the selected widget to the grid structure on the board.
However, Solichin teaches the method further comprises providing the voice input to a natural language processing module trained to identify keywords in the voice input including instructions to move the selected widget to the grid structure on the board. (e.g., par. 114; The voice system 706 captures audio commands from the user and determines keywords of the audio commands. The keywords are used to generate or modify AR effects or AR objects in an AR experience. The voice system 706 uses a machine-learning model that detects keywords from a stream of audio data. The machine-learning model may be a natural language processing (NLP) model. The keywords detected by the machine-learning model may be stored in association with the keyword table 320.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the AR manipulation of virtual objects onto a snap grid virtual bulletin board as taught by Poulos/ Mott/Thiercelin to be based on keywords of voice commands as taught by Solichin, with a reasonable expectation of success, to provide the benefit of flexible intuitive interactions from a user in effort to improve the user experience.
Claim 8 depends on claim 7:
Poulos/ Mott/Thiercelin/Solichin teaches wherein the keywords include a real-world object and a desired position for the widget relative to the real-world object. (e.g., keyword table for voice commands to modify AR objects within AR environment Solichin; par. 79; The database 126 can also store gesture data in the gesture table 322, hints in the hint table 318 and voice command keywords in the keyword table 320. The gesture data may include hand gestures that can be recognized by the AR Experience System 130 to modify one or more aspects of the AR experience. For example, hand gestures may be used to select AR objects generated by the AR Experience System 130 during an interactive AR experience. The hint table 318 stores textual cues associated with the AR experience. The textual cues may be based on commonalities of the real-world. The keyword table 320 stores voice commands keywords that are used to trigger AR effects within the AR Experience System 130.)
Claim 15 depends on claim 9:
Claim 15 is substantially encompassed in claim 7, therefore, Examiner relies on the same rationale set forth in claim 7 to reject claim 15.
Claim 16 depends on claim 15:
Claim 16 is substantially encompassed in claim 8, therefore, Examiner relies on the same rationale set forth in claim 8 to reject claim 16.
Response to Arguments
Applicant’s arguments, with respect to the previously cited prior art failing to disclose the new limitations has been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new grounds of rejection is made in view of newly applied “Thiercelin” reference.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HENRY ORR whose telephone number is (571)270-1308. The examiner can normally be reached 9AM-5PM EST M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Queler can be reached at (571)272-4140. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HENRY ORR/Primary Examiner, Art Unit 2172
Prosecution Insights