CTFR 18/430,113 CTFR 84034 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Response to Arguments 07-38-02 AIA Applicant’s arguments, see pages 10-13 , filed 05 February 2026 , with respect to the rejection(s) of claim(s) 1 and similar claims in substance under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Garretson et al. (US 8,190,295 B1), Knuffner, JR. (US 9,665,800 B1), and Grabska-Barwinska et al. (US 2020/0349418 A1) . Claim Rejections - 35 USC § 112 07-30-01 AIA The following is a quotation of the first paragraph of 35 U.S.C. 112(a): (a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention. The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112: The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention. 07-31-01 Claim 3 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Regarding claim 3, the specification does not clearly describe one or more predictions are in an image space associated with the one or more images, and the one or more control operations are based at least on determining one or more 3D space predictions based at least on converting, using spatial correspondence information between the one or more images and the virtual environment the one or more images from the image space to the coordinate space. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1, 2, 4, 5, 9, 10, 15, 16, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garretson et al. (US 8,190,295 B1) in view of Knuffner, JR. (US 9,665,800 B1) in view of Sriram (US 2022/0366186 A1) and further in view of Grabska-Barwinska et al. (US 2020/0349418 A1) . Regarding claim 1, Garretson discloses a method comprising: generating, using sensor data capturing one or more views of an environment, a 3D representation of the environment that includes at least a portion of an object in a virtual environment, the 3D representation in a coordinate space positioned relative to a machine; (Column 3, lines 1-43, mixed reality environment with live and virtual objects created using a robotic vehicle’s GPS and compass where the vehicle occupies GPS coordinates) rendering, using virtual sensors in the virtual environment, one or more images of the 3D representation of the environment, the virtual sensor oriented, using one or more of a location corresponding to the machine or a location corresponding to the object in the coordinates space; (Column 3, lines 34-64, virtual cameras for interacting with objects in virtual space that can be oriented towards virtual entities that can be displayed in the mixed reality environment) . Garretson does not clearly disclose rendering, using virtual sensors in the virtual environment, one or more images of the 3D representation of the environment, the virtual sensors oriented, using one or more of a location corresponding to the machine or a location corresponding to the object in the coordinate space, to capture at least two different sides of the object. Knuffner, Jr. discloses positioning and orienting multiple virtual cameras around a 3D object to create virtual views of the 3D object that displays various sides of the 3D object (Column 8, lines 16-33) . Knuffner, Jr.’s technique of positioning and orienting multiple virtual cameras around a 3D object to create virtual views of the 3D object that displays various sides of the 3D object would have been recognized by one of ordinary skill in the art to be applicable to the virtual cameras for interacting with objects in virtual space that are displayed in a mixed reality environment of Garretson and the results would have been predictable in the positioning and orienting of multiple virtual cameras around an object in virtual space to create views of the object of different sides of the object that are displayed in mixed reality. Therefore, the claimed subject matter would have bene obvious to one of ordinary skill in the art before the effective filing date of the claimed invention. Garretson in view of Knuffner, Jr. does not clearly disclose generating, based at least on applying the one or more images to one or more machine learning models (MLMs), one or more predictions corresponding to the environment; and performing one or more control operations for a machine in the environment based at least on the one or more predictions generated using the one or more machine learning models (MLMs). Sriram discloses using machine learning to take input image data of an environment and output predictions on the location of objects and using the predictions to operate a vehicle or robot (Paragraphs 0012-0013) . Sriram’s technique of using machine learning to take input image data of an environment and output predictions on the location of objects and using the predictions to operate a vehicle or robot would have been recognized by one of ordinary skill in the art to be applicable to the different views of virtual objects from different sides from multiple virtual cameras for interaction with a robotic vehicle of Garretson in view of Knuffner, Jr. and the results would have been predictable in using machine learning to take input images of different views of virtual objects from difference sides from multiple virtual cameras to output predictions that can be used to operate a robotic vehicle. Therefore, the claimed subject matter would have bene obvious to one of ordinary skill in the art before the effective filing date of the claimed invention. Garretson in view of Knuffner, Jr. and further in view of Sriram does not clearly disclose generating one or more predictions corresponding to a physical version of the object in the environment. Grabska-Barwinska discloses a neural network system for predicting an image or video sequence of a physical object in a simulated or real-world environment for planning in a simulated or real-world environment (Paragraph 0016) . Grabska-Barwinska’s technique of using a neural network for predicting an image or video sequence of a physical object in a simulated or real-world environment for planning in a simulated or real-world environment would have been recognized by one of ordinary skill in the art to be applicable to using machine learning for making predictions based on virtual objects in a mixed-reality environment for operating a robotic vehicle of Garretson in view of Knuffner, Jr. and further in view of Sriram and the results would have been recognized by one of ordinary skill in the art to be applicable to using machine learning including a neural network for predicting an image or sequence of physical and virtual objects in a mixed-reality environment for operating a robotic vehicle. Therefore, the claimed subject matter would have bene obvious to one of ordinary skill in the art before the effective filing date of the claimed invention. Regarding claim 2, Knuffner, Jr. discloses wherein the at least two different sides include at least two opposing sides of the object, and the virtual sensors are position on the at least two opposing sides (Column 8, lines 16-32, spacing virtual cameras around the object to obtain opposing views including, front and back) . Regarding claim 4, Knuffner, Jr. discloses wherein at least one of the virtual sensors is oriented in the coordinate space with respect to the location corresponding to the object (Column 8, lines 16-32, spacing virtual cameras around the object) . Regarding claim 5, Garretson in view of Knuffner, Jr. in view of Sriram and further in view of Grabska-Barwinska discloses a portion of the 3D representation depicts one or more portions of the machine in the virtual environment (Garretson, column 3, lines 30-33, representation of the robotic vehicle in the mixed reality environment) , and the virtual sensors are oriented, using one or more of the location corresponding to the machine or the location corresponding to the object in the coordinate space, to capture at least two opposing sides of the portion of the 3D representation depicting the one or more portions of the machine (Knuffner, Jr., column 8, lines 16-32, virtual cameras can be oriented around 3D objects such as the robotic vehicle to capture opposing sides, such as front and back) . Regarding claim 9, Garretson discloses wherein the machine includes a robot, and the one or more control operations correspond to a three-dimensional object manipulation task (Column 3, lines 1-21, robotic vehicle) . Regarding claims 10 and 16, similar reasoning as discussed in claim 1 is applied. Regarding claim 15, Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska discloses wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system for performing one or more generative AI operations; a system implemented using an edge device; a system implemented using a machine; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Garretson, column 3, lines 1-12, and Sriram, paragraph 0013, controlling a vehicle or robot) . Regarding claim 20, similar reasoning as discussed in claim 15 is applied . 07-21-aia AIA Claim (s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garretson et al. (US 8,190,295 B1) in view of Knuffner, JR. (US 9,665,800 B1) in view of Sriram (US 2022/0366186 A1) and further in view of Grabska-Barwinska et al. (US 2020/0349418 A1) and further in view of Ocean et al. (US 2022/0351751 A1) . Regarding claim 7, Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska discloses all limitations as discussed in claim 1. Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska does not clearly disclose tracking, using the coordinate space, a physical location of the physical version of the object over time, wherein the virtual sensors are oriented based at least on the tracking. Ocean discloses tracking of objects in a physical and virtual environment using a virtual camera (Paragraph 0063) . Ocean’s technique of tracking of objects in a physical and virtual environment using a virtual camera would have been recognized by one of ordinary skill in the art to be applicable to the physical and virtual objects in real and mixed reality environments of Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska and the results would have been predictable in the tracking of physical and virtual objects in real and mixed reality environments using virtual cameras. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention . 07-21-aia AIA Claim (s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garretson et al. (US 8,190,295 B1) in view of Knuffner, JR. (US 9,665,800 B1) in view of Sriram (US 2022/0366186 A1) in view of Grabska-Barwinska et al. (US 2020/0349418 A1) and further in view of Main et al. (US 9,704,270 B1) . Regarding claim 8, Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska discloses all limitations as discussed in claim 1. Garretson further discloses robotic vehicles in the mixed reality environment can be represented by CAD models (Column 3, lines 30-33 and lines 56-64) . Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska does not clearly disclose wherein the generating of the virtual environment uses images of the environment and at least one image of the one or more images of the 3D representation of the environment has a higher resolution than each of the images. Main discloses that multiple resolutions available for CAD models (Column 15, lines 25-50) . Main’s technique of providing CAD models at multiple resolutions would have been recognized by one of ordinary skill in the art to be applicable to the CAD models for representing robotic vehicles in a mixed reality environment of Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska and the results would have been predictable in providing CAD models at multiple resolutions for robotic vehicles in a mixed reality environment where one resolution can be higher than the other. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention . 07-21-aia AIA Claim (s) 11 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garretson et al. (US 8,190,295 B1) in view of Knuffner, JR. (US 9,665,800 B1) in view of Sriram (US 2022/0366186 A1) in view of Grabska-Barwinska et al. (US 2020/0349418 A1) and further in view of Kiyota et al. (US 2016/0328833 A1) . Regarding claim 11, Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska discloses all limitations as discussed in claim 10. Garretson in view of Kuffner in view of Sriram and further in view of Grabska- Barwinska does not clearly disclose wherein at least one image of the one or more images is generated using an orthographic projection of the virtual environment. Kiyota discloses outputting images from a virtual camera using various projection methods including orthographic projection (Paragraphs 0120-0121) . Kiyota’s technique of outputting images from a virtual camera using orthographic projection would have been recognized by one of ordinary skill in the art to be applicable to the virtual camera views of a mixed reality environment of Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska and the results would have been predictable in the outputting of images of a mixed reality environment from a virtual camera using orthographic projection. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention. Regarding claim 17, similar reasoning as discussed in claim 11 is applied . 07-21-aia AIA Claim (s) 12 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garretson et al. (US 8,190,295 B1) in view of Knuffner, JR. (US 9,665,800 B1) in view of Sriram (US 2022/0366186 A1) and further in view of Grabska-Barwinska et al. (US 2020/0349418 A1) in view of Fitzgibbon et al. (US 2011/0228976 A1) and further in view of Perkins et al. (US 2021/0256287 A1) . Regarding claim 12, Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska discloses all limitations as discussed in claim 10. Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska does not clearly disclose wherein the operations further include computing depth information for the one or more images. Fitzgibbon discloses rendering a depth image from a view of a virtual camera (Paragraph 0082) . Fitzgibbon’s technique of rendering a depth image from a view of a virtual camera would have been recognized by one of ordinary skill in the art to be applicable to the capturing of a view of a mixed reality environment from a view of a virtual camera of Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska and the results would have been predictable in the rendering a depth image of a virtual environment from a view of a virtual camera. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention. Garretson in view of Kuffner in view of Sriram in view of Grabska-Barwinska and further in view of Fitzgibbon does not clearly disclose the depth information is applied to the one or more MLMs to determine the one or more predictions. Perkins discloses a machine learning model receiving depth images to predict the location of a target box (Claim 10) . Perkins’ technique of receiving depth images at a machine learning model to predict the location of a target box would have been recognized by one of ordinary skill in the art to be applicable to the depth images of a virtual environment having objects with determined locations from a view of a virtual camera of Garretson in view of Kuffner in view of Sriram in view of Grabska-Barwinska and further in view of Fitzgibbon and the results would have been predictable in the prediction of the locations of objects in a virtual environment using a machine learning model based on depth images of the virtual environment from a view of a virtual camera. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention. Regarding claim 18, similar reasoning as discussed in claim 12 is applied . 07-21-aia AIA Claim (s) 13 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garretson et al. (US 8,190,295 B1) in view of Knuffner, JR. (US 9,665,800 B1) in view of Sriram (US 2022/0366186 A1) in view of Grabska-Barwinska et al. (US 2020/0349418 A1) and further in view of Heiniger et al. (US 2024/0370487 A1) . Regarding claim 13, Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska discloses all limitations as discussed in claim 10. Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska does not clearly disclose applying, to the one or more MLMs to determine the one or more predictions, one or more token embeddings corresponding to a structured language command. Heiniger discloses a machine learning model that can receive image query inputs as well as textual queries using token embeddings (Paragraph 0048) . Heiniger’s technique of using a machine learning model that can receive image query inputs as well as textual queries using token embeddings would have been recognized by one of ordinary skill in the art to be applicable to the machine learning model for receiving image inputs for predicting the location of objects in an environment of Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska and the results would have been predictable in a machine learning model that can receive image and textual inputs using token embeddings for predicting the location of objects in an environment. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention. Regarding claim 19, similar reasoning as discussed in claim 13 is applied . 07-21-aia AIA Claim (s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garretson et al. (US 8,190,295 B1) in view of Knuffner, JR. (US 9,665,800 B1) in view of Sriram (US 2022/0366186 A1) in view of Grabska-Barwinska et al. (US 2020/0349418 A1) and further in view of Sibley et al. (US 2022/0118555 A1) . Regarding claim 14, Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska discloses all limitations as discussed in claim 10. Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska does not clearly disclose wherein the one or more images include at least two images, and the operations further include: determining correspondence information indicating a correspondence between at least two two-dimensional points across the at least two images with one or more three-dimensional points in the virtual environment; and applying the correspondence information to the one or more MLMS to generate the one or more predictions. Sibley discloses determining a cluster of pixels in image frame that can be tracked using a machine learning algorithm to perform motion estimation (Paragraph 0177) . Sibley’s technique of determining a cluster of pixels in image frame that can be tracked using a machine learning algorithm to perform motion estimation would have been recognized by one of ordinary skill in the art to be applicable to the prediction of the location of objects using machine learning of Garretson in view of Kuffner in view of Sriram and further in view of Grabska-Barwinska and the results would have been predictable in predicting the location of objects and their motion according to tracked clusters of pixels in images. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention . Allowable Subject Matter 07-43 Claims 3 and 6 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and all other rejections are resolved. Regarding claim 3, the prior art does not clearly disclose one or more predictions are in an image space associated with the one or more images, and the one or more control operations are based at least on determining one or more 3D space predictions based at least on converting, using spatial correspondence information between the one or more images and the virtual environment the one or more images from the image space to the coordinate space. Regarding claim 6, the prior art does not clearly disclose the method of claim 1, wherein the one or more images includes at least a first image and a second image and the applying the one or more images to the one or more MLMs includes: separately evaluating, using one or more first layers of the one or more MLMs, a first set of image patches corresponding to the first image and a second set of image patches corresponding to the second image, to generate self-attention information for the first image and the second image; and jointly evaluating, using one or more second layers of the one or more MLMs and the self-attention information, the first set of image patches and the second set of image patches to generate joint attention information for the first image and the second image, wherein the one or more predictions correspond to the joint attention information. Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Paul et al. (US 2020/0357184 A1) discloses capturing images of multiple sides of a virtual object. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL . See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHI HOANG whose telephone number is (571)270-3417. The examiner can normally be reached Mon-Fri 8:00-5:00. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JASON CHAN can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /PHI HOANG/Primary Examiner, Art Unit 2619 Application/Control Number: 18/430,113 Page 2 Art Unit: 2619 Application/Control Number: 18/430,113 Page 3 Art Unit: 2619 Application/Control Number: 18/430,113 Page 4 Art Unit: 2619 Application/Control Number: 18/430,113 Page 5 Art Unit: 2619 Application/Control Number: 18/430,113 Page 6 Art Unit: 2619 Application/Control Number: 18/430,113 Page 7 Art Unit: 2619 Application/Control Number: 18/430,113 Page 8 Art Unit: 2619 Application/Control Number: 18/430,113 Page 9 Art Unit: 2619 Application/Control Number: 18/430,113 Page 10 Art Unit: 2619 Application/Control Number: 18/430,113 Page 11 Art Unit: 2619 Application/Control Number: 18/430,113 Page 12 Art Unit: 2619 Application/Control Number: 18/430,113 Page 13 Art Unit: 2619 Application/Control Number: 18/430,113 Page 14 Art Unit: 2619 Application/Control Number: 18/430,113 Page 15 Art Unit: 2619 Application/Control Number: 18/430,113 Page 16 Art Unit: 2619 Application/Control Number: 18/430,113 Page 17 Art Unit: 2619 Application/Control Number: 18/430,113 Page 18 Art Unit: 2619
Prosecution Insights