Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-4, 6, 8-11, 13-20 are pending. As an initial matter, the 35 USC 112 rejection for claim 10 has been withdrawn in view of applicant’s amendments.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-4, 6, 8-11, 13-20 have been considered but are moot because the arguments are directed towards the newly amended claim limitations that change the scope of the claims as a whole and are open to new grounds of rejection/interpretation.
Claim Objections
Claim 19 is objected to because of the following informalities:
Claim 19 claim wherein the machine-learned model comprises a landmark model…but independent claim 17 claims a landmark estimator model. The specification supports a landmark model, but never clarifies what the landmark model is, so it is unclear if the landmark model is directed towards the landmark estimator model or some other model not further described.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 8, 17, 18, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rematas et al. (“ShaRF: Shape-conditioned Radiance Fields from a Single View”) in view of Sun et al. (“Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”) and Oz et al. (US 20210358195).
Re claim 1, Rematas teaches a computer-implemented method for generative neural radiance field model training, the method comprising:
obtaining a plurality of images, wherein the plurality of images depict a plurality of different objects that belong to a shared class, (see p. 1-2, introduction, In summary, our contributions are: (1) a new model to represent object classes that enables reconstructing objects from a single image, (2) a new representation that combines an intermediate volumetric shape representation to condition a high fidelity radiance field, and (3) optimization and fine-tuning strategies during inference that allow estimating radiance fields from real image), (see p. 4-5, 4.2. Shape Network, Training: “training data uses 3D objects of a ShapeNet class (eg. chairs or cars) and each instance has its own shape latent code”) and (see Fig. 5, wherein a plurality of input images such as different chairs of a shared chair class).
Determine a respective set of one or more camera parameters for each image of the plurality of images (see p. 3, 3. Background, “goal is to synthesize realistic images of an object from any viewpoint and at any resolution given a single view as input (an image with its camera parameters)”).
For each image of the plurality of images: processing a latent code associated with a respective object depicted in the image with a generative neural radiance field model (see p. 5, 4.3. Appearance Network: “Our appearance network F models a radiance field similar to the one of NeRF, but extended to include additional conditioning inputs…appearance of latent code controlling the appearance of the object)
to generate a reconstruction output, wherein the reconstruction output comprises a volume rendering generated based at least in part on the respective set of one or more camera parameters for the image (see p. 5-6, in reference to Fig. 3-4, wherein shapes are reconstructed and outputted based on shape/appearance code, and network), (see p. 3, 3.Background, wherein radiance field can be rendered to an image pixel seen by a camera with parameters…radiance field F can be used to render a new view specified by camera parameters).
evaluating a loss function that evaluates a difference between the image and the reconstruction output (see p. 3, 3.Background, train appearance network F by minimizing the loss, comparing this final color with the ground truth) and (see p. 4-5. Training: wherein loss consists of three parts, first, the weighted binary cross entropy loss between predicted and ground truth voxel grid).
adjusting one or more parameters of the generative neural radiance field model based at least in part on the loss function (see p. 3. 3. Background, refer to this process of generating a new image given the radiance field F and camera parameters) and (see p.4, 4.1. Generative Neural Rendering, estimate the radiance field of an object from a image to render novel views, controlling shape and appearance).
Rematas does not explicitly teach wherein determine respective set of one or more camera parameters for each image of the plurality of images is by processing the plurality of images with a landmark estimator model, comprising determining a plurality of 2d landmarks in each image.
However, Sun teaches processing the plurality of images with a landmark estimator model to determine a respective set of one or more camera parameters for each image of the plurality of images, wherein determining the respective set of one or more camera parameters comprises determining a plurality of two-dimensional landmarks in each image (see p. 2975, Fig. 2, from data source images/pictures, align the shape with 2d silhouettes by minimizing the 2d coordinates of the keypoints, wherein keypoints are 2d landmarks in each image), and (see p. 2974, wherein Pix3D collects a large number of image-shape pairs, and collect 2d keypoint annotations of objects in the images).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Remata’s system of estimating scene representation from an image input to explicitly include processing landmark estimators to determine a plurality of 2d landmarks in each image, as taught by Sun, as the references are in the analogous art of shape modeling from captured input images. An advantage of the modification is that it achieves the result of explicitly determining additional camera parameters including determining a plurality of 2d landmarks from the captured image to evaluate more camera parameters to better estimate shape of objects within an image.
Rematas teaches latent codes for faces (See p. 2, 2. Related Work), but Rematas and Sun do not explicitly teach wherein the shared class comprises a faces class, and aligning the plurality of images based on fitting 2d landmarker outputs to class-specific canonical 3d keypoints for the faces class.
However, Oz teaches wherein the shared class comprises a faces class, and aligning the plurality of images based on fitting 2d landmarker outputs to class-specific canonical 3d keypoints for the faces class ([0164] In another method, many 2D images of real people can be obtained and then a 3D model can be created from these multiple 2D images by using photogrammetric software. In yet another method a depth camera that also includes an RGB camera, such as the Kinnect camera or the Intel RealSense camera can be used to obtain both 3D depth models and corresponding 2D images. After training the network using the methods described above, at run time it may be supplied with a 2D image as an input and the network outputs the 3D model. The 3D model can be output as a point cloud, a mesh or a set of parameters that describe a 3D model in a given parametric space), [0168] Specifically, a Generative Adversarial Network (GAN) may be trained based on many images of a certain person or on many images of multiple people to generate images of people from angles that may be different than the angle at which the camera may be currently seeing the person), ([0366] Approximated deformation parameters of the person in the video and an approximated camera model can be found by standard 3DMM fitting techniques, for example by using a face landmark detection method to detect known face parts parameters and optimize the camera and pre-annotated landmarks in a least-squared sense. The initialization does not need to be precise but only approximated and can be generated via commonly known technique), ([0368] At each frame, the deformed mesh will be referred to as the current 3D face mesh, and its deformation parameters on top of the template may be chosen based on a set of landmarks deduced from the 2D face parts segmentation and the pre-annotated segmentation. To that end, the suggested method may use a 2D face parts segmentation method, in conjunction with a classical 2D rigid registration technique utilizing an ICP (Iterative Closest Point) method to track and deform a model of a 3D face based on an input RGB monocular video).
Rematas, Sun, and Oz teaches claim 1. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Remata and Sun’s system of estimating scene representation from an image input including processing landmarks in images to determine a respective set of one or more camera parameters for each image to explicitly include faces class, wherein processing of the plurality of images include aligning the plurality of images based on fitting 2d landmarker outputs to class-specific canonical 3d keypoints for the faces class, as taught by Oz, as the references are in the analogous art of shape modeling from captured input images. An advantage of the modification is that it achieves the result of explicitly determining additional camera parameters including determining a plurality of 2d landmarks from captured images including faces, to evaluate more camera parameters to better estimate shape of objects within an image for 3d reconstruction.
Re claim 2, Rematas, Sun, and Oz teaches claim 1. Furthermore, Rematas teaches processing the image with a segmentation model to generate one or more segmentation outputs;
evaluating a second loss function that evaluates a difference between the one or more segmentation outputs and the reconstruction output; and adjusting one or more parameters of the generative neural radiance field model based at least in part on the second loss function (see p. 7, Variant 3: ShapeFromMask, wherein estimated from a given segmentation mask of the object during stage 1; optimize the shape network and the latent code to fit the mask by minimizing the projection of the estimated shape with the given mask).
Re claim 3, Rematas and Sun and Oz teach claim 1. Furthermore, Rematas teaches adjusting one or more parameters of the generative neural radiance field model based at least in part on a third loss, wherein the third loss comprises a term for incentivizing hard transitions (see p. 4, see 4.2. Shape Network, Training, voxel to image projection loss, with comparing with corresponding object silhouette).
Re claim 8, Rematas and Sun and Oz teach claim 1. Furthermore, Oz teaches wherein the plurality of 2d landmarks are associated with one or more facial features ([0164] In another method, many 2D images of real people can be obtained and then a 3D model can be created from these multiple 2D images by using photogrammetric software. In yet another method a depth camera that also includes an RGB camera, such as the Kinnect camera or the Intel RealSense camera can be used to obtain both 3D depth models and corresponding 2D images. After training the network using the methods described above, at run time it may be supplied with a 2D image as an input and the network outputs the 3D model. The 3D model can be output as a point cloud, a mesh or a set of parameters that describe a 3D model in a given parametric space), [0168] Specifically, a Generative Adversarial Network (GAN) may be trained based on many images of a certain person or on many images of multiple people to generate images of people from angles that may be different than the angle at which the camera may be currently seeing the person), ([0366] Approximated deformation parameters of the person in the video and an approximated camera model can be found by standard 3DMM fitting techniques, for example by using a face landmark detection method to detect known face parts parameters and optimize the camera and pre-annotated landmarks in a least-squared sense. The initialization does not need to be precise but only approximated and can be generated via commonly known technique), ([0368] At each frame, the deformed mesh will be referred to as the current 3D face mesh, and its deformation parameters on top of the template may be chosen based on a set of landmarks deduced from the 2D face parts segmentation and the pre-annotated segmentation. To that end, the suggested method may use a 2D face parts segmentation method, in conjunction with a classical 2D rigid registration technique utilizing an ICP (Iterative Closest Point) method to track and deform a model of a 3D face based on an input RGB monocular video).
For motivation, see claim 1.
Re claim 17, Rematas teaches a computer-implemented method for generating a novel view of an object, the method comprising:
obtaining input data, wherein the input data comprises a single-view image, wherein the single-view image is descriptive of a first object of a first object class (see p. 1-2, introduction, In summary, our contributions are: (1) a new model to represent object classes that enables reconstructing objects from a single image, (2) a new representation that combines an intermediate volumetric shape representation to condition a high fidelity radiance field, and (3) optimization and fine-tuning strategies during inference that allow estimating radiance fields from real image), (see p. 4-5, 4.2. Shape Network, Training: “training data uses 3D objects of a ShapeNet class (eg. chairs or cars) and each instance has its own shape latent code”) and (see Fig. 5, wherein a plurality of input images such as different chairs of a shared chair class).
processing the input data with a machine-learned model to generate a view rendering, wherein the view rendering comprises a novel view of the first object that differs from the single-view image see p. 7, Fig. 5, wherein input views generate novel views of different respective scenes of a particular class, such as cars and chairs).
wherein the machine-learned model was trained on a plurality of training images associated with a plurality of second objects associated with the first object class (see p. 4-5, Training, wherein 3d objects of ShapeNet class of chairs and cars are used as training data) and (see p. 7-8, Novel View Synthesis on ShapeNet-SRN, wherein data sets of cars and chairs are used for training from different viewpoints)
wherein the first object and the plurality of second objects differ (see Fig. 5-8, wherein input images differs from second objects of different views, generated by training)
providing the view rendering as an output (see Fig. 7-8, showing view rendering as an output).
Rematas does not explicitly teach wherein the training comprises aligning training images by, processing the plurality of training images with a landmark estimator model to determine a respective set of one or more camera parameters for each image of the plurality of training images, wherein determining the respective set of one or more camera parameters comprises determine a plurality of 2d landmarks in each image.
However, Sun teaches wherein the training comprises aligning training images by, processing the plurality of training images with a landmark estimator model to determine a respective set of one or more camera parameters for each image of the plurality of training images, wherein determining the respective set of one or more camera parameters comprises determine a plurality of 2d landmarks in each image see p. 2975, Fig. 2, from data source images/pictures, align the shape with 2d silhouettes by minimizing the 2d coordinates of the keypoints, wherein keypoints are 2d landmarks in each image), and (see p. 2974, wherein Pix3D collects a large number of image-shape pairs, and collect 2d keypoint annotations of objects in the images).
For motivation, see claim 1.
Rematas and Sun do not explicitly teach wherein the first object class comprises a face class, wherein training comprises aligning training images by aligning the plurality of training images based on fitting of 2d landmark outputs to class specific canonical 3d keypoints for the faces class.
However, Oz teaches wherein the first object class comprises a face class, wherein training comprises aligning training images by aligning the plurality of training images based on fitting of 2d landmark outputs to class specific canonical 3d keypoints for the faces class ([0164] In another method, many 2D images of real people can be obtained and then a 3D model can be created from these multiple 2D images by using photogrammetric software. In yet another method a depth camera that also includes an RGB camera, such as the Kinnect camera or the Intel RealSense camera can be used to obtain both 3D depth models and corresponding 2D images. After training the network using the methods described above, at run time it may be supplied with a 2D image as an input and the network outputs the 3D model. The 3D model can be output as a point cloud, a mesh or a set of parameters that describe a 3D model in a given parametric space), [0168] Specifically, a Generative Adversarial Network (GAN) may be trained based on many images of a certain person or on many images of multiple people to generate images of people from angles that may be different than the angle at which the camera may be currently seeing the person), ([0366] Approximated deformation parameters of the person in the video and an approximated camera model can be found by standard 3DMM fitting techniques, for example by using a face landmark detection method to detect known face parts parameters and optimize the camera and pre-annotated landmarks in a least-squared sense. The initialization does not need to be precise but only approximated and can be generated via commonly known technique), ([0368] At each frame, the deformed mesh will be referred to as the current 3D face mesh, and its deformation parameters on top of the template may be chosen based on a set of landmarks deduced from the 2D face parts segmentation and the pre-annotated segmentation. To that end, the suggested method may use a 2D face parts segmentation method, in conjunction with a classical 2D rigid registration technique utilizing an ICP (Iterative Closest Point) method to track and deform a model of a 3D face based on an input RGB monocular video).
Rematas, Sun, and Oz teaches claim 17. For motivation, see claim 1.
Re claim 18, Rematas, Sun, and Oz teaches claim 17. Furthermore, Rematas teaches wherein the input data comprises a position and a view direction, wherein the view rendering is generated based at least in part on the position and the view direction (see p. 3, Fig. 1 and 3. Background, wherein input image is of an object in position in volumetric space and viewpoint) and (see p. 7, Fig. 5, wherein novel views are generated at least in part on input views).
Re claim 20, Rematas, Sun, and Oz teaches claim 17. Furthermore, Rematas teaches the view rendering is generated based at least in part on learned latent table (see p. 1, Abstract, wherein formulation is based on a generative process that first maps a latent code to a voxelized shape, and then renders it to an image) and (see Figs. 2-3, shape code, in reference to Table 1, as a table of codes learned).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rematas et al. (“ShaRF: Shape-conditioned Radiance Fields from a Single View”) in view of Sun et al. (“Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”), Oz et al. (US 20210358195), and Boss et al. (“NeRD: Neural Reflectance Decomposition from Image Collections”).
Re claim 4, Rematas, Sun, and Oz teach claim 1. Furthermore, Rematas, Sun, and Oz do not explicitly teach evaluating a third loss function that evaluates an alpha value of the reconstruction output, wherein the alpha value is descriptive of one or more opacity values of the reconstruction output; and adjusting one or more parameters of the generative neural radiance field model based at least in part on the third loss function.
However, Boss teaches evaluating a third loss function that evaluates an alpha value of the reconstruction output, wherein the alpha value is descriptive of one or more opacity values of the reconstruction output; and adjusting one or more parameters of the generative neural radiance field model based at least in part on the third loss function (see Fig. 2, NeRD Architecture, wherein samples along the ray need to be alpha composed based on the density along the ray), (see p. 3, Method; follow general architecture of NeRF…sampled volume parameters are combined along the ray via alpha composition, using density at each point), and (see p. 9, Supplementary, Loss and learning rate schedule: For adjusting the losses and learning…color loss and alpha loss is faded).
Rematas, Sun, Oz, and Boss teaches claim 4. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Rematas, Sun, and Oz’s system of estimating scene representation from an image input and using loss calculations for reconstruction to explicitly include loss function that evaluates an alpha value of the reconstruction output, as taught by Boss, as the references are in the analogous art of NeRF-based framework for estimating scenes. An advantage of the modification is that it achieves the result of explicitly using opacity values and adjusting parameters based at least in part on the third losses related to opacity, providing an alternative mathematical loss to computer to improve the quality of the estimated scene representation.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rematas et al. (“ShaRF: Shape-conditioned Radiance Fields from a Single View”) in view of Sun et al. (“Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”), Oz et al. (US 20210358195), and Schwarz et al. (“GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis”).
Re claim 6, Rematas, Oz, and Sun teaches claim 1. Rematas, Oz, and Sun teaches faces class, but do not explicitly teach wherein a first object of the plurality of different objects comprises a first face associated with a first person, and wherein a second object of the plurality of different objects comprises a second face associated with a second person.
However, Schwarz teaches wherein a first object of the plurality of different objects comprises a first face associated with a first person, and wherein a second object of the plurality of different objects comprises a second face associated with a second person (see p. 8, Fig. 6-7, wherein objects include faces with first and second faces associated with people).
Rematas, Sun, Oz, and Schwarz teaches claim 6. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Rematas, Oz, and Sun’s generative neural radiance field model of processing images with different objects in a shared class of faces to explicitly teach wherein a first object of the plurality of objects comprises a first face associated with a first person and wherein a second object of the plurality of different objects comprises a second face associated with a second person, as taught by Schwarz, as the references are in the analogous art of generative radiance field systems including object classes. An advantage of the modification is that it achieves the result of explicitly include object classes such as faces, including different faces of a plurality of people to improve synthesis for generating 3d models of different object classes.
Claim(s) 9-10, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rematas et al. (“ShaRF: Shape-conditioned Radiance Fields from a Single View”) in view of Sun et al. (“Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling”) Oz et al. (US 20210358195), and Xie et al. (“Fig-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling”).
Re claim 9, Rematas, Oz, and Sun teaches claim 1. Rematas, Oz, and Sun do not explicitly teach wherein the generative neural radiance field model comprises a foreground model and a background model, wherein the background model comprises a lower-capacity model than the foreground model.
However, Xie teaches wherein the generative neural radiance field model comprises a foreground model and a background model (see abstract, Fig. 3 and section 3.3. “Object and Background separation: wherein NeRF comprises foreground and background models).
Wherein the background model comprises a lower-capacity model than the foreground model (see p. 4, Separation Regularization, wherein the background model does not vary its density, Ff is force to represent any varying geometry of the scenes, which includes the objects). Thus, the background model is interpreted to comprise a lower-capacity model because it does not vary its density and does not represent varying geometry of the scene.
Rematas, Sun, Oz, and Xie teach claim 9. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Rematas, Oz, and Sun’s system of NeRF system of detecting objects in an image to explicitly include processing foreground and background models, as taught by Xie, as the references are in the analogous art of NeRF based processing of input images. An advantage of the modification is that it achieves the result of explicitly using foreground and background models to separate objects within an input image.
Re claim 10, Rematas, Sun, Oz, and Xie teach claim 9. Furthermore, Xie teaches wherein the foreground model comprises a concatenation block (see Fig. 3, wherein concatenation is noted in the foreground). For motivation, see claim 9.
Re claim 19, Rematas, Sun, and Oz teaches claim 17. Furthermore, Oz teaches a landmark model ([0164] In another method, many 2D images of real people can be obtained and then a 3D model can be created from these multiple 2D images by using photogrammetric software. In yet another method a depth camera that also includes an RGB camera, such as the Kinnect camera or the Intel RealSense camera can be used to obtain both 3D depth models and corresponding 2D images. After training the network using the methods described above, at run time it may be supplied with a 2D image as an input and the network outputs the 3D model. The 3D model can be output as a point cloud, a mesh or a set of parameters that describe a 3D model in a given parametric space), [0168] Specifically, a Generative Adversarial Network (GAN) may be trained based on many images of a certain person or on many images of multiple people to generate images of people from angles that may be different than the angle at which the camera may be currently seeing the person), ([0366] Approximated deformation parameters of the person in the video and an approximated camera model can be found by standard 3DMM fitting techniques, for example by using a face landmark detection method to detect known face parts parameters and optimize the camera and pre-annotated landmarks in a least-squared sense. The initialization does not need to be precise but only approximated and can be generated via commonly known technique), ([0368] At each frame, the deformed mesh will be referred to as the current 3D face mesh, and its deformation parameters on top of the template may be chosen based on a set of landmarks deduced from the 2D face parts segmentation and the pre-annotated segmentation. To that end, the suggested method may use a 2D face parts segmentation method, in conjunction with a classical 2D rigid registration technique utilizing an ICP (Iterative Closest Point) method to track and deform a model of a 3D face based on an input RGB monocular video).
For motivation, see claim 1.
Rematas, Sun and Oz do not explicitly teach wherein the machine-learned model comprises a foreground neural radiance field model, and a background neural radiance field model.
However, Xie teaches wherein the machine-learned model comprises a a foreground neural radiance field model, and a background neural radiance field model (see abstract, Fig. 3 and section 3.3. “Object and Background separation: wherein NeRF comprises foreground and background models).
Rematas, Sun, Oz, and Xie teaches claim 19. For motivation, see claim 9.
Claim(s) 11, 13, 14, is/are rejected under 35 U.S.C. 103 as being unpatentable over by Rematas et al. (“ShaRF: Shape-conditioned Radiance Fields from a Single View”) in view of Bertel et al. (US 20220139036).
Re claim 11, Rematas teaches a computing system generating class-specific view rendering outputs, the system comprising: one or more processors; one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: (see p.1, Abstract and 1. Introduction, wherein 3d scene reconstruction is implicitly directed towards computer processing, execution, and storage media)
obtaining a training dataset, wherein the training dataset comprises a plurality of single-view images, wherein the plurality of single-view images are descriptive of a plurality of different respective scenes (see p. 1-2, introduction, In summary, our contributions are: (1) a new model to represent object classes that enables reconstructing objects from a single image, (2) a new representation that combines an intermediate volumetric shape representation to condition a high fidelity radiance field, and (3) optimization and fine-tuning strategies during inference that allow estimating radiance fields from real image), (see p. 4-5, 4.2. Shape Network, Training: “training data uses 3D objects of a ShapeNet class (eg. chairs or cars) and each instance has its own shape latent code”) and (see Fig. 5, wherein a plurality of input images such as different chairs of a shared chair class, as different respective scenes).
processing the training dataset with a machine-learned model to train the machine-learned model (see section 4.2. shape network, Training, wherein 3d objects are trained on dataset of images).
to learn a volumetric three-dimensional representation associated with a particular class, wherein the particular class is associated with the plurality of single-view images (see section 4.2. shape network, Training, wherein 3d objects are trained on dataset of images) and (see 4.4. Inference on a test image, incorporate prior knowledge of the object class using learning of shape and appearance).
and generating a view rendering based on the volumetric three-dimensional representation (see Fig. 7-8, wherein a 3d view is rendered).
Wherein the view rendering is associated with the particular class (see p. 1-2, introduction, In summary, our contributions are: (1) a new model to represent object classes that enables reconstructing objects from a single image, (2) a new representation that combines an intermediate volumetric shape representation to condition a high fidelity radiance field, and (3) optimization and fine-tuning strategies during inference that allow estimating radiance fields from real image), (see p. 4-5, 4.2. Shape Network, Training: “training data uses 3D objects of a ShapeNet class (eg. chairs or cars) and each instance has its own shape latent code”) and (see Fig. 5, wherein a plurality of input images such as different chairs of a shared chair class).
and wherein the view rendering is descriptive of a novel scene that differs from the plurality of different respective scenes (see p. 4, Generative Neural Rendering, “…The goal of our method is to estimate the radiance field of an object from a single image so that we can render novel views), and (See Fig. 5, input view and novel views that are novel scene different from a respective scene) and (see p. 7, 5.2. Novel View Synthesis on ShapeNet -
Rematas does not explicitly teach wherein the novel scene comprises an object of the particular class that differs from objects depicted in the training dataset.
However, Bertel teaches wherein the novel scene comprises an object of the particular class that differs from objects depicted in the training dataset (abstract: Novel images may be generated using an image generator implemented on a processor. The image generator may receive as input neural features selected from a neural texture atlas. The image generator may also receive as input one or more position guides identifying position information for a plurality of input image pixels. The novel images may be evaluated using an image discriminator to determine a plurality of optimization values by comparing each of the plurality of novel images with a respective one of a corresponding plurality of input images. Each of the novel images may be generated from a respective camera pose relative to an object identical to that of the respective one of the corresponding plurality of input images. The image generator and the neural features may be updated based on the optimization values and stored on a storage device) and ([0041] According to various embodiments, the pose information determined at 104, the 3D mesh determined at 106, and/or the UV map determined at 108 are provided to a generative adversarial network at 110. The generative adversarial network generates a new, artificial image of the object at 116 by sampling textures at 112 and applying a generator at 114. The artificial image is then fed back in to a discriminator at 120 along with the input images at 118 to optimize the generative adversarial network at 122.
Rematas and Bertel teaches claim 11. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Rematas system of generating class-specific view rendering outputs to explicitly include wherein the novel scene comprises an object of the particular class that differs from objects depicted in the training dataset, as taught by Bertel, as the references are in the analogous art of Nerf based system for rendering outputs and training. An advantage of the modification is that it achieves the result of explicitly generating objects that differ from objects in different views/training datasets, as the novel scenes include novel objects that are generated.
Re claim 13, Rematas and Bertel teaches claim 11. Furthermore, Rematas teaches wherein the view rendering is descriptive of a second view of a scene depicted in at least one of the plurality of single-view images (see Fig. 7, ShapeNet-SRN, view rendered is descriptive of a novel view of the scene depicted in the input single view image).
Re claim 14, Rematas and Bertel teaches claim 11. Furthermore, Rematas teaches generated a learned latent table based at least in part on the training dataset, wherein the view rendering is generated based on the learned table (see p. 1, Abstract, wherein formulation is based on a generative process that first maps a latent code to a voxelized shape, and then renders it to an image) and (see Figs. 2-3, shape code, in reference to Table 1, as a table of codes learned).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rematas et al. (“ShaRF: Shape-conditioned Radiance Fields from a Single View”) in view of in view of Bertel et al. (US 20220139036) and Boss et al. (“NeRD: Neural Reflectance Decomposition from Image Collections”).
Re claim 15, Rematas and Bertel teaches claim 11. Rematas and Bertel do not explicitly teach wherein the machine-learned model is trained based at least in part on a red-green-blue loss, a segmentation mask loss, and a hard surface loss.
However, Boss teaches wherein the machine-learned model is trained based at least in part on a red-green-blue loss, a segmentation mask loss, and a hard surface loss (see p. 3, 3. Method, loss from input color and color loss, loss applied to RGB prediction, and Foreground/Background mask loss) and (see p. 9, Supplementary, Loss and learning rate schedule, teachings of loss directed towards direct color loss and alpha loss).
Rematas, Bertel, and Boss teaches claim 15. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Rematas and Bertel’s system of estimating scene representation from an image input and using loss calculations for reconstruction to explicitly include loss function that evaluates an alpha value of the reconstruction output, as well as losses to color and segmentation, as taught by Boss, as the references are in the analogous art of NeRF-based framework for estimating scenes. An advantage of the modification is that it achieves the result of explicitly using opacity values and adjusting parameters based at least in part on the third losses related to opacity, providing an alternative mathematical loss to computer to improve the quality of the estimated scene representation.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rematas et al. (“ShaRF: Shape-conditioned Radiance Fields from a Single View”) in view of in view of Bertel et al. (US 20220139036) and Bojanowski et al. (“Optimizing the Latent Space of Generative Networks”).
Re claim 16, Rematas and Bertel teaches claim 11. Furthermore, Rematas teaches wherein the machine-learning model comprises GLO (see section 4.2., Shape Network, Training, wherein training is akin to Generative Latent Optimization technique), but Rematas and Bertel do not explicitly teach an autoencoder.
However, Bojanowski teaches machine-learning model comprises an auto-encoder model (see p. 2, Contributions, GLO, as an autoencoder).
Rematas, Bertel, and Bojanowski teaches claim 16. It would have been obvious skill in the art before the effective filing date of the claimed invention to modify Rematas and Bertel’s training system of incorporated 3d objects akin to Generative Latent Optimization, to explicitly include autoencoder models, as taught by Bojanowski, as the references are in the analogous art of training data systems for optimizing generative networks. An advantage of the modification is that it achieves the result of explicitly Generative Latent Optimizations that include auto-encoders for training using latent codes.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Peter Hoang whose telephone number is (571)270-1346. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hajnik F. Daniel can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER HOANG/ Primary Examiner, Art Unit 2616
Prosecution Insights