DETAILED ACTION
Status
This Office Action is responsive to claims filed on 07/28/2025. Please note Claims 21-40 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 21, 23, 28, 30, 35 and 37 are objected to because of the following informalities:
Claim 21 recites “…based the set of surface properties”, which should be “based on the set of surface properties”. Appropriate correction is required.
Claims 28 and 35 recite similar features of claim 21, and appropriate corrections are required.
Claim 23 recites “…based the contextual factors”, which should be “based on the contextual factors”. Appropriate correction is required.
Claims 30 and 37 recite similar features of claim 23, and appropriate corrections are required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 23, 30 and 37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 23 recites “a second set of objects a threshold distance of the augment”. It’s unclear how the “second set of objects” are related to the “threshold distance”. Clarification is needed.
Claims 30 and 37 recite similar features of claim 23 and are therefore rejected.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21-40 are rejected under 35 U.S.C. 103 as being unpatentable over PETILL (US 20210012113 A1), in view of Forbes (US 20200151965 A1).
Regarding Claim 21, PETILL discloses a non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform ([0093] “Non-volatile storage device 1706 includes one or more physical devices configured to hold instructions executable by the logic processors to implement the methods and processes described herein.”):
generating a surface within an artificial-reality environment for associating one or more augments ([0045] “FIG. 4 illustrates an example surface reconstruction and decomposition pipeline 44. At step (1) of the pipeline, a geometric representation 46 is generated for the scene data 36 received from the camera device 18, such as, for example, a mesh having vertices and indices that represents the physical environment 28. The geometric representation 46 of the scene data 36 may be generated via any suitable surface reconstruction method.”);
determining a set of surface properties for the surface, the set of surface properties including one or more of a surface shape and a surface location within the artificial-reality environment ([0054] “At FIG. 5 at (D), the detected object may be processed by the trained artificial intelligence learning model 40 to recognize the detected object based on different factors or characteristics of the object, such as, for example, geometric shape, size, color, surface texture, etc.” [0058] “The database 54 may also store detected physical locations 58 for the detected physical objects 46. The detected physical locations 58 may be updated in real-time based on scene data 36 captured by the camera device 18. Virtual objects 56 that have been instantiated may also have virtual locations 60 in the physical environment 28 where the virtual object 56 may be displayed via the display device 22 in an augment reality configuration.”);
identifying an augment of the one or more augments associated with the surface ([0079] “At 108, the method 100 may include selecting a target virtual object and a target physical object from the physical objects and virtual objects in the database based on the identified one or more semantic tags.”); and
causing display of the augment on or at the surface based the set of surface properties for the surface ([0081] “At 112, the method 100 may include displaying the target virtual object at a physical location associated with the target physical object.” [0068] “To perform the move operation, the HMD device 24 may be configured to update a virtual location 60 of the virtual plant object 56A from the previous location in the corner of the room to a new location based on the physical location 58 associated with the first table physical object 30E, such as, for example, on top of the table.”).
PETILL does not expressly disclose wherein the surface accommodates one or more of an existing augment and a new augment.
However, in the same field of endeavor, Forbes discloses the surface accommodates one or more of an existing augment and a new augment ([0049] “As part of operation 306 from method 300, the MR condition generator module 124 matches the properties of the table 452 (e.g., received via the sensors 116 and camera 114 as part of operation 304) to conditions associated with the first digital tree 406A, the digital rock 406B, the second digital tree 406C and drop-target digital object 402A from the example shown in FIG. 4C.”), and causing display of the augment on or at the surface based the set of surface properties for the surface ([0049] “For example, based on a shape of the detected real table 452 being rectangular, and detected dimensions of the real table 452 being within the maximum size 403A and minimum size 403B of table dimension range described within the conditions as described with respect to FIG. 4B, and an elevation of the real table 452 being within the range of elevation values as described with respect to FIG. 4B and shown in 402D, and a distance of the table 452 from the wall 451A being within the range of distance values as described with respect to FIG. 4B and shown in 402C, then the table 452 is determined as satisfying the conditions for the first digital tree 406A, the digital rock 4068, the second digital tree 406C and the drop-target digital object 402A.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the medium of PETILL with the feature of accommodating one or more of an existing augment and a new augment on the surface. Doing so could provide a more realistic view to the viewer.
Regarding Claim 22, PETILL-Forbes discloses the non-transitory computer-readable storage medium of claim 21, wherein generating the surface within the artificial-reality environment for associating the one or more augments includes:
determining surface definitions for one or more surfaces relative to an artificial reality system position (PETILL [0045] “The deep neural network processor 38 may be configured to process the scene data 36 using a surface reconstruction and decomposition pipeline 44 to detect one or more physical objects 30 in the physical environment 28, and recognize the one or more physical objects 30 based on the trained artificial intelligence machine learning model 40.” [0046] “for example, a Deep Neural Network (DNN) to identify object boundaries within the scene data. After identifying object boundaries, the surface reconstruction and decomposition pipeline 44 may include object segmentation to detect one or more objects in the scene”);
determining surface geometries for the artificial-reality environment (PETILL [0045] “At step (1) of the pipeline, a geometric representation 46 is generated for the scene data 36 received from the camera device 18, such as, for example, a mesh having vertices and indices that represents the physical environment 28.”);
determining object types for the surface (PETILL [0047] “For example, the FCN may be trained to identify objects including, but limited to, an unrecognized object, a room, a wall, a room furnishing, a table, a chair, a surface, physical location, and a person.”);
receiving user defined surface properties (PETILL [0049] “For example, the processor 12 may be configured to receive a user input directed to the detected physical object 46 in the physical environment 28, the user input including a user specified semantic tag 48A.”); and
wherein the generating the surface is based on i) the surface definitions for one or more surfaces relative to an artificial reality system position, ii) the surface geometries for the artificial-reality environment, iii) the object types for the surface, or iv) the user defined surface properties (PETILL [0032] “As noted above, the processor 12 may execute instructions to generate a 3D model of the physical environment including surface reconstruction information, which may include generating a geometric representation, such as a geometric mesh, of the physical environment that may be used to identify surfaces and boundaries between objects, and recognize those objects in the physical environment based on a trained artificial intelligence machine learning model.”).
Regarding Claim 23, PETILL-Forbes discloses the non-transitory computer-readable storage medium of claim 21, wherein the instructions, when executed by the computing system, further cause the computing system to perform:
determining contextual factors for the surface, the contextual factors including one or more of an operating mode of an artificial reality system, a first set of objects within the artificial-reality environment, a second set of objects a threshold distance of the augment, characteristics of a user, characteristic of the augment, artificial reality environment conditions, and user inputs (PETILL, Fig. 7 and Fig. 8, [0065] As shown, the HMD device 24 has received a natural language input 42 via a voice input of “Move the virtual plant to the table”... However, the semantic tag “table” may match to both the first table physical object 30E associated with the semantic tag “living room table” and the second table physical object 30F associated with the semantic tag “small table”. Due to multiple matches, the HMD device 24 may determine that the selection of the target physical object 74 has a confidence value below a threshold value, and thus enter a disambiguation mode.),
wherein the display of the augment on or at the surface is further based the contextual factors (PETILL [0066] “In one example, to disambiguate the selection of the target physical object 74, the HMD device 24 may be configured to determine a user indicated direction 76 for a user of the head mounted display device 24.” [0068] “After determining the user specified operation 64, the target virtual object 72, and the target physical object 74, the virtual object handling module 70 may be configured to perform the determined user specified operation 64 on the target virtual object 72 based on the target physical object 74.”).
Regarding Claim 24, PETILL-Forbes discloses the non-transitory computer-readable storage medium of claim 21, wherein causing the display of the augment on or at the surface includes one or more of: invoking corresponding logic on the augment based on the set of surface properties; and displaying the augment in a display mode of a plurality of display modes based on the set of surface properties (PETILL [0068] “To perform the move operation, the HMD device 24 may be configured to update a virtual location 60 of the virtual plant object 56A from the previous location in the corner of the room to a new location based on the physical location 58 associated with the first table physical object 30E, such as, for example, on top of the table.” [0073] “As shown, to perform the application start operation for the user specified operation 64, the HMD device 24 may be configured to generate a target virtual object 72 associated with the target application program selected from the plurality of application programs 60 based on the identified semantic tags 66.”).
Regarding Claim 25, PETILL-Forbes discloses the non-transitory computer-readable storage medium of claim 21, wherein identifying the augment of the one or more augments associated with the surface includes one or more of identifying: an augment attached with or within a threshold distance of the surface; an augment placed on or at the surface by a user; and an augment created on or at the surface (PETILL [0068] “To perform the move operation, the HMD device 24 may be configured to update a virtual location 60 of the virtual plant object 56A from the previous location in the corner of the room to a new location based on the physical location 58 associated with the first table physical object 30E, such as, for example, on top of the table.”).
Regarding Claim 26, PETILL-Forbes discloses the non-transitory computer-readable storage medium of claim 21, wherein the set of surface properties further includes a type of augment accommodated by the surface (PETILL [0050] “Additionally, specific object class-based heuristics may be applied to improve robustness of this association.”), a set of augments associated with the surface (PETILL [0062] “For example, the processor 12 may be configured to select the target virtual object 72 and the target physical object 74 based on matches between the associated semantic tags 48 for the target virtual object and target physical object 74 and the identified semantic tags 66 in the natural language input 42.”), surface meta-tags (PETILL [0050] “At step (4), the processor 12 and/or deep neural network processor 38 may be configured to store a reference to the recognized physical object 30 in a database with an associated semantic tag 48.”), and a surface layout (Forbes Figs. 4A, 4B, 6A, 6B, [0046] “The two detected objects including the rectangular table 402 and the circular table 404 described with respect to FIG. 4A.”).
Regarding Claim 27, PETILL-Forbes discloses the non-transitory computer-readable storage medium of claim 21, wherein the surface is synthetic, semantic, or geometric (PETILL [0045] “FIG. 4 illustrates an example surface reconstruction and decomposition pipeline 44. At step (1) of the pipeline, a geometric representation 46 is generated for the scene data 36 received from the camera device 18, such as, for example, a mesh having vertices and indices that represents the physical environment 28. The geometric representation 46 of the scene data 36 may be generated via any suitable surface reconstruction method.”).
Regarding Claim 28, it recites similar limitations of claim 21. The rationale of claim 21 rejection is applied to reject claim 28.
Regarding Claim 29, it recites similar limitations of claim 22. The rationale of claim 22 rejection is applied to reject claim 29.
Regarding Claim 30, it recites similar limitations of claim 23. The rationale of claim 23 rejection is applied to reject claim 30.
Regarding Claim 31, it recites similar limitations of claim 24. The rationale of claim 24 rejection is applied to reject claim 31.
Regarding Claim 32, it recites similar limitations of claim 25. The rationale of claim 25 rejection is applied to reject claim 32.
Regarding Claim 33, it recites similar limitations of claim 26. The rationale of claim 26 rejection is applied to reject claim 33.
Regarding Claim 34, it recites similar limitations of claim 27. The rationale of claim 27 rejection is applied to reject claim 34.
Regarding Claim 35, it recites similar limitations of claim 21. The rationale of claim 21 rejection is applied to reject claim 35.
Regarding Claim 36, it recites similar limitations of claim 22. The rationale of claim 22 rejection is applied to reject claim 36.
Regarding Claim 37, it recites similar limitations of claim 23. The rationale of claim 23 rejection is applied to reject claim 37.
Regarding Claim 38, it recites similar limitations of claim 24. The rationale of claim 24 rejection is applied to reject claim 38.
Regarding Claim 39, it recites similar limitations of claim 25. The rationale of claim 25 rejection is applied to reject claim 39.
Regarding Claim 40, it recites similar limitations of claim 26. The rationale of claim 26 rejection is applied to reject claim 40.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
MAK (US 20190371279 A1) – This reference teaches systems and methods for adjusting display of virtual content when a wearable display device detects a trigger event to change display of content that is relative to a physical or virtual surface. The method includes displaying content relative to a first surface, detecting a trigger that is one of motion or content driven; and adjusting display of the content to a user-centric virtual surface in the user's field of view.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHONG WU whose telephone number is (571)270-5207. The examiner can normally be reached MON-FRI: 9AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHONG WU/Primary Examiner, Art Unit 2613
Prosecution Insights