Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Shugo (US10460461B2), Son (US20220148127A1), and Ha (US20220284663A1).
Regarding claim 1, teaches a scene restoration method performed by at least one processor (Shugo; ¶0028, describes the information processing apparatus comprising a CPU (processor).) the scene restoration method comprising: obtaining an input image of an object (¶0040, describes input-image acquiring unit acquiring image data captured by cameras with objects present. This teaches obtaining an input image of an object.) and a plurality of viewpoints, surrounding the object (¶0027, describes a plurality of cameras arranged around the object and, ¶0029, describes viewpoint information for those cameras. This teaches a plurality of viewpoints surrounding the object in 3D space.)
However, Shugo does not explicitly disclose determining augmented viewpoints based on an input viewpoint, generating augmented images at the augmented viewpoints using a view change model, generating a scene restoration model, or restoring a scene image of a target view using the scene restoration model.
Son teaches based on an input viewpoint corresponding to the input image, determining a plurality of augmented viewpoints surrounding the object in a three-dimensional (3D) space comprising the object (Son; ¶0093-0094, describes unprojection of the input image to 3D space, 3D transformation, and projection to an arbitrary viewpoint to generate a new view, T(x, d), and, ¶0104, describes that even when a single input image is transformed it is possible to maintain a 3D geometric structure in the transformed image. This teaches determining an augmented viewpoint in 3D space from a single input image and its depth. Combined with Shugo, ¶0027 and 0029, this teaches a plurality of viewpoints surrounding the object in 3D space based on an input viewpoint.) generating a plurality of augmented images at the plurality of augmented viewpoints, wherein each augmented image from among the plurality of augmented images corresponds to a view of the object from a corresponding augmented viewpoint from among the plurality of augmented viewpoints, and wherein each augmented image is generated based on an image at a different viewpoint using a view change model (Son; ¶0098-0099, describes the input image to a new view, T(x,d), and applying that transformed view to an image generation neural network to generate a second synthesized image, G(T(x,d)), and, ¶0111, describes the trained image generation neural network generating a synthesized image by maintaining geometric information of the input image and transforming the remaining image information. This teaches generating an augmented image at an augmented viewpoint generated based on the input image transformed to the augmented viewpoint using an image generation model.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the image processing method of Shugo with the 3D viewpoint transformation and image generation of Son. The motivation for such a combination would have been to provide the benefit of consistent novel view synthesis from a single input image.
However, Shugo in view of Son does not explicitly disclose generating a scene restoration model based on the input image and the plurality of augment ted images, or restoring a scene image of a target view using the scene restoration model.
Ha teaches generating a scene restoration model based on the input image at the input viewpoint and the plurality of augmented images at the plurality of augmented viewpoints (Ha; ¶0023-0024, describes neural network-based extraction models by iteratively correcting model parameters such that a difference between reconstructed image data and training image data is reduced and, ¶0094-0096, describes generating reconstruction image data, comparing the reconstructed image data with second training image data, defining a photometric reconstruction loss, and updating the extraction models based on that loss. Together with Son (Son; ¶0098-0099); a model is trained using the input image and addition images generated from it. This teaches generating a scene restoration model based on the input image and the plurality of augmented images.) restoring a scene image of a target view of the object using the scene restoration model (Ha; ¶0026, describes using the trained extraction models to generate reconstructed image data from input image data including an object and, ¶0064, describes rendering the object into various viewpoints and poses. The trained scene restoration model is used to generate a reconstructed mage of the object at a target viewpoint. This teaches restoring a scene image of a target view of the object using the scene restoration model.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to further modify the image processing method of Shugo in view of Son with the neural network-based image reconstruction of Ha. The motivation for such a combination would have been to provide the benefit of increased accuracy of the image of the object at the target view.
Claim 11, has similar limitations as of claim 1, therefore it is rejected under the same rationale as claim 1, except claim 11 recites, “a memory” and “at least one processor”. Shugo; ¶0085 describes a computing system including one or more processors and storage medium including RAM and hard disk.
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Shugo (US10460461B2), Son (US20220148127A1), Ha (US20220284663A1), and Funt (US11544894B2).
Regarding claim 2, Shugo in view of Son, Ha teaches the scene restoration method of claim 1, but does not explicitly disclose wherein the determining of the plurality of augmented viewpoints comprises determining positions on a surface of a virtual solid figure surrounding the object in the 3D space as the plurality of augmented viewpoints.
However, Funt discloses wherein the determining of the plurality of augmented viewpoints comprises determining positions on a surface of a virtual solid figure surrounding the object in the 3D space as the plurality of augmented viewpoints. Funt (¶0022) describes a view hierarchy as a layout of multiple viewpoints surrounding an object where all viewpoints are placed a fixed distance away from the center of the object and, ¶0036, describing a geodesic structure where the location of each vertex of the geodesic structure is a location where a viewpoint may be located. Fig. 10 shows viewpoints distributed across the surface of the virtual structure surrounding the object. This teaches determining positions on a surface of a virtual solid figure surrounding the object in the 3D space as the plurality of augmented viewpoints.
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son and Ha with the geodesic viewpoint distribution of Funt. The motivation for such a combination would have been to provide the benefit of uniform coverage of viewpoints surrounding an object.
Claim 12, has similar limitations as of claim 2, therefore it is rejected under the same rationale as claim 2.
Claims 3-4 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Shugo (US10460461B2), Son (US20220148127A1), Ha (US20220284663A1), Funt (US11544894B2), and Kroon (US11856223B2).
Regarding claim 3, Shugo in view of Son, Ha teaches the scene restoration method of claim 1. However, Shugo in view of Son and Ha do not explicitly disclose that generating each augmented image comprises determining a plurality of reference viewpoints around each augmented viewpoint, generating a plurality of candidate images at each augmented viewpoint based on reference images at the plurality of reference viewpoints, and selecting an augmented image from among the plurality of candidate images.
Funt teaches determining a plurality of reference viewpoints around the each augmented viewpoint (Funt; ¶0026, describes identifying the view triangle enclosing the hit point for the desired viewpoint and selecting the three viewpoints as primary views. This teaches determining a plurality of reference viewpoints around the augmented viewpoint.) generating a plurality of candidate images at the each augmented viewpoint based on a plurality of reference images at the plurality of reference viewpoints using the view change model (Funt; ¶0027, describes rendering, for each of the primary views, an image of the object from the user’s viewpoint based on the meshlets and RGB data corresponding to the primary view. Candidate images are generated at the augmented viewpoint from reference images at the reference viewpoints. As previously discussed in claim 1, Son (Son; ¶0098-0099, ¶0111) teaches generating view transformed images using the view change model (see claim 1). In the combination, this teaches generating a plurality of candidate images at the augmented viewpoint based on reference images at the plurality of reference viewpoints using the view change model.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son and Ha with the reference viewpoint selection and candidate image generation of Funt. The motivation for such a combination would have been to provide the benefit of more complete and accurate image generation at each viewpoint.
However, Shugo in view of Son, Ha, and Funt does not explicitly disclose selecting one augmented image from among the plurality of candidate images.
Kroon teaches selecting an augmented image at the each augmented viewpoint from among the plurality of candidate images (Kroon; ¶0017, describes generating predicted images for a candidate set of images and selecting a set of selected images from the candidate images in response to prediction quality. This teaches selecting the augmented image from among the plurality of candidate images.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to further modify the system of Shugo in view of Son, Ha, and Funt with the image selection of Kroon. The motivation for such a combination would have been to provide the benefit of improved augmented image quality and more efficient generation.
Claim 13, has similar limitations as of claim 3, therefore it is rejected under the same rationale as claim 3.
Regarding claim 4, Shugo in view of Son, Ha, Funt, and Kroon teaches the scene restoration method of claim 3. However, this combination does not explicitly disclose selecting the augmented image comprises obtaining a retransformed image by transforming each candidate image to a corresponding reference viewpoint using the view change model, and selecting the augmented image based on a comparison between the retransformed image and a corresponding reference image.
Son teaches obtaining a retransformed image by transforming each candidate image from among the plurality of candidate images to a corresponding reference viewpoint using the view change model (Son; ¶0113, describes inversely transforming the second synthesized image to the original view based on the transformation relationship. A retransformed image is obtained by transforming a candidate image to a corresponding reference viewpoint using the view change model. This teaches obtaining a retransformed image by transforming each candidate image to a corresponding reference viewpoint using the view change model.)
Kroon teaches selecting the augmented image based on a comparison between the retransformed image and a corresponding reference image (Kroon; ¶0017, describes determining a prediction quality measure for each candidate image indicative of a difference between the candidate image and the predicted version and, ¶0022, determining a prediction quality based on a comparison between corresponding pixels of the candidate image and the predicted image. An image is selected based on a comparison between a retransformed image and a corresponding reference image. This teaches selecting the augmented image based on a comparison between the retransformed image and the corresponding reference image.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to further modify the system of Shugo in view of Son, Ha, Funt, and Kroon with the transformation of Son and the comparison-selection of Kroon. The motivation for such a combination would have been to provide the benefit of increased accuracy when selecting the image representing the object at the viewpoint.
Claim 14, has similar limitations as of claim 4, therefore it is rejected under the same rationale as claim 4.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Shugo (US10460461B2), Son (US20220148127A1), Ha (US20220284663A1), Funt (US11544894B2), Kroon (US11856223B2), and Bradley (US20220237751A1).
Regarding claim 5, Shugo in view of Son, Ha, Funt, and Kroon teaches the scene restoration method of claim 4. However, this combination does not explicitly disclose calculating an LPIPS loss between the retransformed image and the corresponding reference image having the smallest LPIPS loss as the augmented image.
Bradley teaches calculating a learned perceptual image patch similarity (LPIPS) loss between the retransformed image and the corresponding reference image (Bradley; ¶0047, describes calculating an LPIPS loss to compare a rendered image ( Ik ) and a projection image (Pk). This teaches calculating an LPIPS loss between the retransformed image and the corresponding reference image.) selecting a candidate image having a smallest LPIPS loss from among the plurality of candidate images as the augmented image (As previously discussed in claim 4, Kroon; ¶0017 and ¶0022, describe selecting an image from among candidate images based on a prediction quality of a difference between the candidate image and its predicted version. In the combination, the comparison metric is used to select from candidate images, including selecting the image having the smallest LPIPS loss. This teaches selecting the candidate image having the smallest LPIPS loss as the augmented image.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son, Ha, Funt, and Kroon with the with the LPIPS loss-image evaluation of Bradley. The motivation for such a combination would have been to provide the benefit of improved selection of the augmented image that is most similar to the corresponding reference image.
Claim 15, has similar limitations as of claim 5, therefore it is rejected under the same rationale as claim 5.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Shugo (US10460461B2), Son (US20220148127A1), Ha (US20220284663A1), and Watanabe (JP2972175B2).
Regarding claim 6, Shugo in view of Son and Ha teaches the scene restoration method of claim 1. However, Shugo in view of Son and Ha does not explicitly disclose generating an augmented image at each augmented viewpoint sequentially in an order of increasing distance from the input viewpoint.
Watanabe teaches generating an augmented image at each augmented viewpoint sequentially in an order of increasing distance from the input viewpoint (Watanabe; ¶0006-0007, describes processing objects in ascending order of distance from a viewpoint by integrating drawing data starting from the closest object and repeating outward until the farthest object in the visible region is reaches; sequentially processing viewpoints in order of increasing distance from a reference viewpoint. This teaches generating an augmented image at each augmented viewpoint sequentially in an order of increasing distance from the input viewpoint.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son and Ha with the sequential distance processing of Watanabe. The motivation for such a combination would have been to provide the benefit of more efficient and organized generation of augmented images.
Claims 7-9 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Shugo (US10460461B2), Son (US20220148127A1), Ha (US20220284663A1), and Gu (Gu, Jiatao, et al. "Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion." International Conference on Machine Learning. PMLR, 2023.).
Regarding claim 7, Shugo in view of Son and Ha teaches the scene restoration method of claim 1. However, this combination does not explicitly disclose the view change model comprises a diffusion model, or generating of the plurality of augmented images comprises providing rotation and translation parameters with a reference viewpoint to the diffusion model to generate a candidate image and providing a parameter for transformation of the augmented viewpoint back to the reference viewpoint together with the candidate image to generate a retransformed image.
Gu teaches wherein the view change model comprises a diffusion model (Gu; ABST, describes a 3D-aware conditional diffusion model (CDM) used to synthesize novel views of an object and, pg. 5 section 4.2, the CDM is learned to iteratively refine renderings of the image to match target views. This teaches the view change model comprises a diffusion model.) wherein the generating of the plurality of augmented images comprises: providing parameters based on a rotation parameter and a translation parameter for transformation of a reference viewpoint into an augmented viewpoint to the diffusion model together with a reference image at the reference viewpoint to generate a candidate image at the augmented viewpoint (Gu; section 4.4 NeRF Guided Distillation, describes generating virtual views at relative camera poses to the input, where the pose define a target viewpoint. Gu; 4.2 and Fig. 3, describe target view rendering is used to condition the diffusion process so the CDM denoises at that target pose. Ha; ¶0081, describes the target pose value as a 3D transform matrix including elements for translation and rotation. This teaches providing rotation and translation parameters together with a reference image to the diffusion model to generate a candidate image at the augmented viewpoint.)
providing a parameter for transformation of the augmented viewpoint into the reference viewpoint to the diffusion model together with the candidate image at the augmented viewpoint to generate a retransformed image (Son; ¶0113, describes inversely transforming the second synthesized image to the original view based on the transformation relationship, teaching providing an inverse transformation parameter together with a candidate image to generate a retransformed image at the reference viewpoint. Gu, as discussed above, teaches using a diffusion model as the view change model for conditioned viewpoint image generation. In the combination, this teaches providing a parameter for transformation of the augmented viewpoint into the reference viewpoint to the diffusion model together with the candidate image at the augmented viewpoint to generate a retransformed image.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son and Ha with the diffusion model view synthesis of Gu and the inverse transformation of Son. The motivation for such a combination would have been to provide the benefit of more accurate augmented image generation.
Claim 17, has similar limitations as of claim 7, therefore it is rejected under the same rationale as claim 7.
Regarding claim 8, Shugo in view of Son and Ha teaches the scene restoration method of claim 1. However, Shugo in view of Son and Ha does not explicitly disclose restoring the scene image comprises generating scene information including color information and volume density information based on the scene restoration model and restoring the scene image by repeatedly determining pixel values by performing colume rendering on the scene information.
Gu teaches wherein the restoring of the scene image comprises: generating scene information comprising color information and volume density information based on the scene restoration model (Gu; section 3.1, describes NeRF as defining an implicit function, fθ : (x,d) → (c,σ), where c and σ are the color and density, respectively. This teaches generating scene information comprising color information and volume density information based on the scene restoration model.)
restoring the scene image by repeatedly determining a pixel value for each pixel from among a plurality of pixels in a view to be restored by performing volume rendering on the scene information (Gu; section 3.1, describes rendering a posed image by marching a camera ray through each pixel and calculating its color via an approximation of the volume rendering integral using the color and density scene information; repeatedly determining a pixel value for each pixel in a view by performing volume rendering on the scene information. This teaches restoring the scene image by repeatedly determining a pixel value for each pixel by performing volume rendering on the scene information.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son and Ha with the NeRF volume rendering of Gu. The motivation for such a combination would have been to provide the benefit of improved scene image restoration.
Claim 18, has similar limitations as of claim 8, therefore it is rejected under the same rationale as claim 8.
Regarding claim 9, Shugo in view of Son and Ha teaches the scene restoration method of claim 1, wherein the scene restoration model comprises: a deformation estimation model configured to convert coordinates of a point in the 3D space into coordinates corresponding to a canonical frame with reference to deformation code (Ha; ¶0059, describes determining albedo data and depth data in a canonical space from input image data, ¶0075, the canonical space is a deformation-free normalized pose space that aligns a deformable object in a single pose space, and, ¶0076-0077, application of a target shape deformation value as a 3D offset for movement of the point cloudl relative to the canonical space. A deformation-related model that represents object coordinates with reference to a canonical frame and a deformation value used as deformation code. This teaches a deformation estimation model configured to convert coordinates of a point in the 3D space into coordinates corresponding to a canonical frame with reference to deformation code.)
However, this combination does not explicitly disclose an NSR estimation model configured to estimate color information and volume density information based on the converted canonical frame coordinates.
Gu teaches a neural scene representation (NSR) estimation model configured to estimate color information and volume density information based on the converted coordinates according to the canonical frame (Gu; section 3.1, describes NeRF as defining an implicit function, fθ : (x,d) → (c,σ), where c and σ are the color and density, respectively, and describes the function as taking a 3D location and viewing direction as input to estimate color and volume density. In the combination, this teaches an NSR estimation model configured to estimate color information and volume density information based on the converted coordinates according to the canonical frame.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son and Ha with the canonical space deformation of Ha and the neural radiance field representation of Gu. The motivation for such a combination would have been to provide the benefit of more accurate scene restoration.
Claim 19, has similar limitations as of claim 9, therefore it is rejected under the same rationale as claim 9.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shugo (US10460461B2), Son (US20220148127A1), Ha (US20220284663A1), Bradley (US20220237751A1), and Gu (Gu, Jiatao, et al. "Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion." International Conference on Machine Learning. PMLR, 2023.).
Regarding claim 10, Shugo in view of Son and Ha teaches the scene restoration method of claim 1. However, this combination does not explicitly describe generating the scene restoration model comprises generating a temporary image by providing to the scene restoration model, a deformation code and coordinates for each point from among a plurality of points in the 3D space corresponding to a ray for each pixel in a two-dimensional scene corresponding to a view to be restored, updating parameters of the scene restoration model and the deformation code based on a loss between the generated temporary image and a training image corresponding to the 2D scene, and based on the updating of the parameters of the scene restoration model and the deformation code converging, mapping the converged deformation code to a frame identifier indicating the training image.
Gu teaches wherein the generating of the scene restoration model comprises: generating a temporary image by providing, to the scene restoration model, a deformation code and coordinates for each point from among a plurality of points in the 3D space corresponding to a ray for each pixel in a two-dimensional (2D) scene corresponding to a view to be restored (Gu; section 3.1, describes to render a posed image, a camera ray is marched through each pixel and its color is calculated via an approximation of the volume rendering integral; providing coordinates for points along a ray corresponding to each pixel to a scene model for image generation. Ha; ¶0091-0093, describe extracting a shape deformation value from second training image data and generating reconstructed image data from deformed data based on that value. In the combination, this teaches generating a temporary image by providing a deformation code and coordinates for each point corresponding to a ray for each pixel to the scene restoration model.) updating parameters of the scene restoration model and the deformation code based on a loss between the generated temporary image and a training image corresponding to the 2D scene (Ha; ¶0093-0096, describes comparing reconstructed image data with training image data, defining a loss based on the difference, and updating model parameters to reduce that difference. This teaches updating parameters of the scene restoration model and the deformation code based on a loss between the generated temporary image and the training image.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son and Ha with the ray-scene rendering of Gu. The motivation for such a combination would have been to provide the benefit of improved model training.
However, this combination does not explicitly disclose mapping the converged deformation code to a frame identifier indication the training image.
Bradley teaches based on the updating of the parameters of the scene restoration model and the deformation code converging, mapping the converged deformation code to a frame identifier indicating the training image (Bradley; ¶0050-0051, describes parameter vectors (xk) associated with temporally ordered images or frames; associating a converged parameter code with a corresponding fram e identifier. This teaches mapping the converged deformation code to a frame identifier indicating the training image.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the system of Shugo in view of Son, Ha, and Gu with the frame-parameter optimization of Bradley. The motivation for such a combination would have been to provide the benefit of more stable and flexible model training.
Claim 20, has similar limitations as of claim 10, therefore it is rejected under the same rationale as claim 10.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAN F KALHORI whose telephone number is (571)272-5475. The examiner can normally be reached Mon-Fri 8:30-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DEVONA E FAULK can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAN F KALHORI/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights