Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 7, 14 are objected to because of the following informalities:
Claim 7 recites "controlling the controlling the robotic system comprises controlling the robotic arm to pick up one or more items from a first location and move the one or more items to a second location”, which instead should recite "controlling the robotic system comprises controlling the robotic arm to pick up one or more items from a first location and move the one or more items to a second location."
Claim 14 should be amended similarly to claim 7.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 8-11, 15-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blukis (US 20240153196 A1) in view of Wrenninge (US 20220237410 A1).
Regarding claim 1, Blukis discloses a method comprising: generating a synthetic dataset using a Neural Radiance Field (NeRF) model (Fig. 6, [0108] generate an image from at least the set of color and density values; [0106] a neural network to generate a set of color and density values, wherein the neural network may be a compositional NeRF; [0110]-[0111] the system may use the representation of the scene to generate any number of images of the one or more objects from any suitable viewpoint), wherein the NeRF model:
receives as input a real image of a real scene, wherein the real image comprises at least a plurality of objects and a background (Fig. 6, [0096] obtain, at block 602, a representation of a scene including one or more objects, the scene may be any suitable environment such as a real world environment);
extracts, from the real image, a feature volume for each object of the plurality of objects ([0098] the system may calculate a bounding volume for each object of the one or more objects, in which the bounding volume may indicate a volume of the object, such as a bounding box volume); and
renders one or more synthetic scenes of the synthetic dataset, wherein each scene of the synthetic scenes comprises at least one rendered synthetic object having a pose that differs in other scenes of the one or more synthetic scenes ([0110] the system may use the representation of the scene to generate any number of images of the one or more objects from any suitable viewpoint such as to generate a second image that the depicts the one or more objects from the second viewpoint).
Blukis fails to disclose training a perception model based at least in part on the synthetic dataset including the one or more synthetic scenes; and controlling a robotic system based at least in part on output from the perception model that has been trained on the synthetic dataset.
Wrenninge, in a related system from the same field of endeavor of synthetic data generation for training a system (Abstract), discloses training a perception model based at least in part on the synthetic dataset including the one or more synthetic scenes (Fig. 1, [0080] the method can include block S600 which includes training a model based on the synthetic image dataset; [0070] the output of block S300 preferably includes a two dimensional synthetic image that realistically depicts a realistic 3D scene; [0076] block S500 includes generating a synthetic image dataset); and controlling a robotic system based at least in part on output from the perception model that has been trained on the synthetic dataset ([0115] an exemplary application is for training or evaluating a model used to control an autonomous vehicle).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Wrenninge with Blukis and train a perception model based on the synthetic dataset and control a robot system based on the output, as disclosed by Wrenninge, as part of a method for generating a synthetic dataset using a Neural Radiance Field (NeRF) model, as disclosed by Blukis, for the purpose of improving performance of a model trained on the data, validating a model, and performing object detections and other real-world tasks based on the generated dataset (See Wrenninge: [0020], [0115]).
Regarding claim 2, Blukis in view of Wrenninge discloses the method of claim 1 as applied above. Blukis further discloses wherein the NeRF model, prior to extracting the feature volume for each object of the plurality of objects, decomposes the real scene into the plurality of objects (Fig. 1, [0067] the system 100 may be associated with a scene which may refer to any suitable environment such as a…real world environment; [0068] the system 100 may generate the scene representation 102 by at least generating a scene graph and one or more tuples, each corresponding to a respective object in the scene; [0063] the system may utilize a NeRF decoder).
Blukis fails to disclose wherein the model decomposes the real scene including the background.
Wrenninge, in a related system from the same field of endeavor of synthetic data generation for training a system (Abstract), discloses wherein the model decomposes the real scene including the background (Fig. 4, [0088] a set of parameter values is determined including computing a ground surface parameter based on a stochastic variable (see also in [0088]: determining a virtual roadway, virtual sky, etc.)).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Wrenninge with Blukis wherein the model decomposes the real scene including the background, as disclosed by Wrenninge, as part of a method for generating a synthetic dataset using a Neural Radiance Field (NeRF) model, as disclosed by Blukis, for the purpose of improving performance of a model trained on the data, validating a model, and performing object detections and other real-world tasks based on the generated dataset (See Wrenninge: [0020], [0115]).
Regarding claim 3, Blukis in view of Wrenninge discloses the method of claim 1 as applied above. Blukis further discloses wherein the NeRF model generates the feature volume for each object of the plurality of objects from learned feature vectors specific to each object of the plurality of objects ([0097] the scene graph may include one or more tuples, a tuple may be data indicating at least a pose, a bounding volume, and/or a latent code associated with an object; [0098] the system may calculate a bounding volume for each object of the one or more objects, in which the bounding volume may indicate a volume of the object, such as a bounding box volume).
Regarding claim 4, Blukis in view of Wrenninge discloses the method of claim 1 as applied above. Blukis further discloses training the NeRF model with one or more real images and one or more synthetic images ([0063] the system may utilize a NeRF decoder pre-trained across a large dataset of objects; [0112] a dataset of scenes, in which each scene may include a ground truth scene graph and ground truth images rendered from random camera poses (see also [0106] wherein the neural network trained as in [0112] may be a compositional NeRF); [0568] ground truth data may be synthetically produced, real produced, machine-automated, human annotated, and/or a combination thereof).
Regarding claim 8, Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 1) wherein Blukis further discloses one or more non-transitory computer-readable storage media having program instructions stored thereon, wherein the program instructions, when executed by a computing system, direct the computing system to perform operations ([0094] some or all of the process 600 is performed under control of one or more computer systems configured with computer-executable instructions and is implemented as code…code is stored on a computer-readable storage medium which is a non-transitory computer-readable medium).
Regarding claim 9, Blukis in view of Wrenninge discloses the one or more non-transitory computer-readable storage media of claim 8 as applied above. Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 2).
Regarding claim 10, Blukis in view of Wrenninge discloses the one or more non-transitory computer-readable storage media of claim 8 as applied above. Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 3).
Regarding claim 11, Blukis in view of Wrenninge discloses the one or more non-transitory computer-readable storage media of claim 8 as applied above. Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 4).
Regarding claim 15, Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 1) wherein Blukis further discloses a system comprising: one or more computer-readable storage media ([0094] some or all of the process 600 is performed under control of one or more computer systems configured with computer-executable instructions and is implemented as code…code is stored on a computer-readable storage medium which is a non-transitory computer-readable medium);
a processing system operatively coupled with the one or more computer-readable storage media ([0094] code is stored on a computer-readable storage medium in the form of a computer program including a plurality of computer-readable instructions executable by one or more processors); and
program instructions stored on the one or more computer-readable storage media, wherein the program instructions, when read and executed by the processing system, direct the processing system ([0094] some or all of the process 600 is performed under control of one or more computer systems configured with computer-executable instructions and is implemented as code…code is stored on a computer-readable storage medium which is a non-transitory computer-readable medium).
Regarding claim 16, Blukis in view of Wrenninge discloses the system of claim 15 as applied above. Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 2).
Regarding claim 17, Blukis in view of Wrenninge discloses the system of claim 15 as applied above. Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 3).
Regarding claim 18, Blukis in view of Wrenninge discloses the system of claim 15 as applied above. Blukis in view of Wrenninge discloses everything claimed as applied above (see rejection of claim 4).
Claim(s) 5, 12, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blukis (US 20240153196 A1) in view of Wrenninge (US 20220237410 A1) in further view of Back (US 20230289971 A1).
Regarding claim 5, Blukis in view of Wrenninge discloses the method of claim 1 as applied above. Blukis fails to disclose wherein the perception model comprises at least one of: a modal instance segmentation model and an amodal instance segmentation model.
Back, in a related system from the same field of endeavor of object identification and segmentation in an image scene (Abstract), discloses wherein the perception model comprises at least one of: a modal segmentation model and an amodal segmentation model (Fig. 7, [0159] in block S708, the computer system segments the object based on the class, the bounding box, the visible mask, the amodal mask (or invisible mask), and the occlusion derived in block S706 (see also [0120]-[0122] wherein the model of the system is trained based on a training dataset)).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Back with Blukis in view of Wrenninge wherein the perception model comprises at least one of: a modal instance segmentation model and an amodal instance segmentation model, as disclosed by Back, as part of a method for generating a synthetic dataset using a Neural Radiance Field (NeRF) model, as disclosed by Blukis in view of Wrenninge, for the purpose of accurately detecting objects in a cluttered scene, including when an object is occluded by other objects, for applications such as robotic manipulation (See Back: [0003]-[0006]).
Regarding claim 12, Blukis in view of Wrenninge discloses the one or more non-transitory computer-readable storage media of claim 8 as applied above. Blukis in view of Wrenninge and Back further discloses everything claimed as applied above (see rejection of claim 5).
Regarding claim 19, Blukis in view of Wrenninge discloses the system of claim 15 as applied above. Blukis in view of Wrenninge and Back further discloses everything claimed as applied above (see rejection of claim 5).
Claim(s) 6, 13, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blukis (US 20240153196 A1) in view of Wrenninge (US 20220237410 A1) in further view of Yucel (US 20230114028 A1).
Regarding claim 6, Blukis in view of Wrenninge discloses the method of claim 1 as applied above. Blukis fails to disclose wherein the perception model is a depth estimation model.
Yucel, in a related system from the same field of endeavor of generating a data for training an ML model to perform depth estimation (Abstract), discloses wherein the perception model is a depth estimation model ([0019] generating a training dataset for training a machine learning model using federated learning to perform depth estimation).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Yucel with Blukis in view of Wrenninge wherein the perception model is a depth estimation model, as disclosed by Yucel, as part of a method for generating a synthetic dataset using a Neural Radiance Field (NeRF) model, as disclosed by Blukis in view of Wrenninge, for the purpose of improved performance in real-world applications of depth estimation such as autonomous driving, robotics, and augmented reality (See Yucel: [0005]-[0006], [0010]).
Regarding claim 13, Blukis in view of Wrenninge discloses the one or more non-transitory computer-readable storage media of claim 8 as applied above. Blukis in view of Wrenninge and Yucel further discloses everything claimed as applied above (see rejection of claim 6).
Regarding claim 20, Blukis in view of Wrenninge discloses the system of claim 15 as applied above. Blukis in view of Wrenninge and Yucel further discloses everything claimed as applied above (see rejection of claim 6).
Claim(s) 7, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blukis (US 20240153196 A1) in view of Wrenninge (US 20220237410 A1) in further view of Shahapurkar (US 20250303558 A1).
Regarding claim 7, Blukis in view of Wrenninge discloses the method of claim 1 as applied above. Blukis fails to disclose wherein the robotic system comprises a robotic arm; and controlling the controlling the robotic system comprises controlling the robotic arm to pick up one or more items from a first location and move the one or more items to a second location.
Shahapurkar, in a related system from the same field of endeavor of training models using synthetic datasets to perform object manipulation tasks (Abstract), discloses wherein the robotic system comprises a robotic arm (Fig. 1, [0017] the autonomous machine 104 can further include a robotic arm or manipulator 110; Fig. 2, [0027] the depth image and the map can define an annotated synthetic dataset that can be used to train a neural network to identify grasps on objects in bins that are arranged in a variety of configurations); and controlling the controlling the robotic system comprises controlling the robotic arm to pick up one or more items from a first location and move the one or more items to a second location (Fig. 1, [0017] the end effector 116 can include one or more tools configured to grasp and/or move objects 106…so as to place or move the objects within the physical environment).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Shahapurkar with Blukis in view of Wrenninge wherein the robotic system comprises a robotic arm and controlling the robotic system comprises controlling the robotic arm to move one or more items from a first location to a second location, as disclosed by Shahapurkar, as part of a method for generating a synthetic dataset using a Neural Radiance Field (NeRF) model, as disclosed by Blukis in view of Wrenninge, for the purpose of improved training and implementation of robotic picking and grasping tasks (See Shahapurkar: [0001]-[0003]).
Regarding claim 14, Blukis in view of Wrenninge discloses the one or more non-transitory computer-readable storage media of claim 8 as applied above. Blukis in view of Wrenninge and Shahapurkar further discloses everything claimed as applied above (see rejection of claim 7).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jain (Ajay Jain, Matthew Tancik, Pieter Abbeel; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 5885-5894) discloses using a NeRF model to extract object images to generate novel views as synthetic images of the objects.
Mildenhall (Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.) discloses a method of using a NeRF model to render new views of input images of objects using feature volume rendering.
Moreau (Moreau, A., Piasco, N., Tsishkou, D., Stanciulescu, B. & Fortelle, A.d.L.. (2022). LENS: Localization enhanced by NeRF synthesis. <i>Proceedings of the 5th Conference on Robot Learning</i>, in <i>Proceedings of Machine Learning Research</i> 164:1347-1356 Available from https://proceedings.mlr.press/v164/moreau22a.html.) discloses using a NeRF model to generate a synthetic dataset based on density volumes and virtual camera locations to then be used by a perception model to perform real-time localization of objects.
Zhong (Zhong, S., Albini, A., Jones, O.P., Maiolino, P. & Posner, I.. (2023). Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data Generation. <i>Proceedings of The 6th Conference on Robot Learning</i>, in <i>Proceedings of Machine Learning Research</i> 205:1618-1628 Available from https://proceedings.mlr.press/v205/zhong23a.html.) discloses using a NeRF model to render images of objects of interest and to then generate tactile sensory data to be used in robotics applications such as manipulation and control.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROLINE DEPALMA whose telephone number is (571)270-0769. The examiner can normally be reached Mon-Thurs 9:00am-4pm Eastern Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at 571-272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROLINE E. DEPALMA/Examiner, Art Unit 2675
/SJ Park/Primary Examiner, Art Unit 2675
Prosecution Insights