DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 13 and 20 recites the limitation "the outputted depth-encoded images" in line 12. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-11, 13-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yang (US 2018/0234671 A1) in view of Roy (WO 2021/203076 A1) and in further view of Zakharov et al., "Single-Shot Scene Reconstruction," Proceedings of the 5th Conference on Robot Learning, PMLR 164:501-512, November 2021 (https://proceedings.mlr.press/v164/zakharov22a/zakharov22a.pdf).
Regarding claims 1, 13, and 20 Yang teaches
A computer-implemented method for single-view three-dimensional (3D) few-view novel-view synthesis of a novel scene (Paragraph [0001] and FIG. 5), comprising:
inputting at least one posed RGB image of the novel scene into an encoder;
encoding with the encoder the at least one inputted posed RGB image (Paragraphs [0002] & [0067]) and inputting the at least one encoded RGB image into a far multi-layer perceptron (MLP) for representing a color of background images (Paragraphs [0099],[0100],[0107]) and a near multi-layer perceptron (MLP) for representing a color of foreground images (Paragraphs [0006] & [0107]);
outputting the color of background images from the far MLP as depth- encoded features, to be volumetrically rendered as background images (Paragraphs [0099],[0100],[0107]) and outputting the color of foreground images from the near MLP to be volumetrically rendered as foreground images (Paragraphs [0006] & [0107]);
creating a convolutional two-dimensional (2D) feature map from the depth-encoded features and the foreground images (Paragraphs [0031], [0067]);
extracting local and global features from the global features representation at projected pixel locations (Paragraph [0068], [0069], [0073], [0077]); and
inputting the extracted local and global features into a decoder to render local and global feature representations of the novel scene from the modeled 3D surroundings (paragraph [0067], [0077], [0080]).
Yang fails to describe representing a density and outputting a density;
aggregating the outputted depth-encoded images and the foreground images;
producing a triplanar representation from the 2D feature map;
transforming the triplanar representation into a global features representation to model 3D surroundings of the novel scene;
However, Roy teaches aggregating the outputted depth-encoded images and the foreground images (Paragraphs [0012] & [0030]); producing a triplanar representation from the 2D feature map (Paragraph [0009]); transforming the triplanar representation into a global features representation to model 3D surroundings of the novel scene (Paragraph [0004]);
Yang and Roy are considered analogous to the claimed invention as they are in the same field of view synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Yang with Roy to specify the input and output forms of a view synthesis method.
Yang in view of Roy fails to teach representing a density and outputting a density.
However, Zakharov teaches representing a density and outputting a density (Section 2, Sub-Section “Neural Implicit Representations”)
Zakharov is considered analogous to the claimed invention as it is in the same field of view synthesis. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yang in view of Roy with the teachings of Zakharov to include density as a part of the view synthesis generation process and possibly improve training and rendering performance.
Regarding claim 2, Yang and Roy teach the method of claim 1 and Yang further teaches wherein a number of the posed RGB images input into the encoder is from 1 to 5 (paragraph [0025], [0067]). Yang describes the use of a single image to generate a viewpoint and that image is encoded as 2D data.
Regarding claims 3, and 14, Yang and Roy teach the method of claim 1 and Yang further teaches wherein the encoder uses a trained encoder network (Paragraph [0067]). Yang describes its Geometric Flow Network including “a plurality of convolution (or encoding) layers” which “may include image encoders”. This is a form of a trained encoder network.
Regarding claims 4, and 15, Yang and Roy teach the method of claim 1 and Yang further teaches wherein the near and far MLPs are neural networks (Paragraph [0045]). Yang suggests the use of a deep neural network or artificial neural network its system.
Regarding claims 5, and 16, Yang and Roy teach the method of claim 1 and Roy further teaches wherein the triplanar representation comprises three axis- aligned orthogonal planes (105 & 107 in FIG. 3). Roy uses pose coordinates for in its representation, these are expressed in terms of three axis-aligned orthogonal planes which are the x, y and z axis.
Regarding claims 8, and 17, Yang and Roy teach the method of claim 1 and Yang further teaches wherein the decoder comprises one or more rendering MLPs. (Paragraph [0067]). Yang describes the use of multiple convolutional layers for the decoding portion of its network, this composition of multiple convolutional layers for the decoding portion reconstructs the features and provides information on image completion. The multiple layers of convolution can be considered analogous to one MLP and the reconstruction and production of image completion information can be considered analogous to rendering.
Regarding claims 9, and 18 Yang and Roy teach the method of claim 1. However Yang and Roy fail to teach wherein the decoder predicts a color and density for an arbitrary 3D location and a viewing direction from the triplanar representation.
However, Zakharov teaches wherein the decoder predicts a color and density for an arbitrary 3D location and a viewing direction from the triplanar representation (Section 2, Sub-Section “Neural Implicit Representations” paragraph 2) Zakharov describes the regression (prediction) of color and density along rays (arbitrary 3D locations and viewing directions) as opposed to predefined 3D coordinates.
Regarding claim 10, Yang and Roy teach the method of claim 1. However Yang and Roy fail to teach wherein the decoder uses near and far rendering MLPs to decode color and density used to render the local and global feature representations of the novel scene.
However, Zakharov teaches wherein the decoder uses near and far rendering MLPs to decode color and density used to render the local and global feature representations of the novel scene (Section 2, Sub-Section “Neural Implicit Representations”, Section 3.1, Sub-Section “3D Reasoning Block”). Zakharov describes a decoder calculating information for foreground and background information. Zakharov also describes regressing density and color values. Furthermore, Zakharov uses a "GAN-like encoder-decoder" which could be considered analogous an MLP. This is used for determining various features such as the object’s geometry and pose (global) and 3D points of the partial object shape in the frame of reference (local).
Regarding claim 11, Yang, Roy and Zakharov teach the method of claim 9. Zakharov further teaches wherein the near and far rendering MLPs output density and color for a 3D point and a viewing direction (Section 2 Sub-Section “6DoF Object Detection via Correspondence Regression” and “Neural Implicit Representation”, Section 3.1, Sub-Sections “Object Reasoning Block” and “3D Reasoning Block Zakharov describes the regression (prediction) of color and density along rays (arbitrary 3D locations and viewing directions). Zakharov describes the use of multiple MLPs. Zakharov describes calculating information for foreground and background information. While Zakharov doesn’t explicitly say the MLPs output density and color for a 3d point and viewing direction, it does say they are used to “predict important object properties” which implies prediction of the previously mentioned density and color values. Therefore it would have been obvious to one of ordinary skill in the art to specify the ”important object properties” as object or image properties that are present elsewhere in the reference. Furthermore, it would have been obvious to incorporate these teachings with Yang in view of Roy to possibly improve training and rendering performance.
Claim(s) 12, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Roy and Zakharov and in further view of Alexander (US 2020/0184221 A1).
Regarding claim 12, Yang, Roy and Zakharov teach the method of claim 1. However, Yang, Roy and Zakharov fail to teach wherein the novel scene is rendered in a 360 degree view.
However, Alexander teaches wherein the novel scene is rendered in a 360 degree view (paragraph [0036]).
Alexander is considered analogous to the claimed invention, Yang, Roy and Zakharov as they are all in the same field of computer graphics. Therefore it would have been obvious to modify the teachings of Yang, Roy and Zakharov with the 360 degree view of Alexander to improve user experience and better illustrate features of the scene.
Regarding claim 19, Yang, Roy and Zakharov teach the system of claim 13. However, they fail to teach A vehicle comprising the system of claim 13. Alexander teaches a vehicle comprising a system (paragraph [0026]). Alexander describes a high performance computing module “positioned on (or embodied at) an aircraft”. This is analogous to a vehicle comprising a system. It would be obvious to one of ordinary skill in the art to substitute the system of Alexander, with the system of Yang in view of Roy to implement a view reconstruction system in a vehicle.
Claim(s) 6, 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Roy and Zakharov and in further view of Baumgartner (US 11,200,693 B2) and Hinek (US 2021/0407124 A1).
Regarding claim 6, Yang in view of Roy and Zakharov teach the method of claim 5. However, Yang in view of Roy and Zakharov fail to teach obtaining a respective feature map for each of the three axis-aligned orthogonal planes.
Baumgartner teaches obtaining a respective feature map for each of the axis-aligned orthogonal planes (Col. 4 lines 6-18). Baumgartner describes saliency maps being generated for the respective orthogonal planes. The saliency maps of Baumgartner are analogous to feature maps.
Baumgartner is considered analogous to the claimed invention as it is in the same field of image processing. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Baumgartner with Yang in view of Roy and Zakharov to improve object detection and generation.
Baumgartner fails to teach three axis-aligned orthogonal planes.
However, Hinek teaches three axis-aligned orthogonal planes (paragraph [0016]). Hinek describes gathering training images from various angles. Some of the suggested angles of Hinek disclose three axis-aligned orthogonal planes, these being top-down, front and side views.
Hinek is considered analogous to the claimed invention as it is in the same field of image processing. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to incorporate the teachings of Hinek to improve the performance of generating image data for occluded views.
Regarding claim 7, Yang in view of Roy and Zakharov teach the method of claim 1. Roy further teaches modeling the 3D surroundings from a corresponding perspective (Paragraphs [0004], [0006]).
Yang in view of Roy and Zakharov fail to teach wherein the triplanar representation comprises three perpendicular cross-planes, each cross-plane modeling the 3D surroundings from a corresponding perspective, and the method further comprises merging the cross-planes to produce the global features.
However, Baumgartner teaches wherein the triplanar representation comprises perpendicular cross-planes, each cross-plane modeling the 3D surroundings from a corresponding perspective, and the method further comprises merging the cross-planes to produce the global features (Section 3.2.2).
Baumgartner fails to teach wherein the triplanar representation comprises three perpendicular cross-planes.
However, Hinek teaches three perpendicular cross-planes (paragraph [0016]). Hinek describes gathering training images from various angles. A subset of the suggested angles of Hinek are three perpendicular cross-planes, these being top-down, front and side views.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Tsai (WO 2022/216333 A1), Xu (CN 113808187 A), Zhang (CA 2553473 A1).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aidan W McCoy whose telephone number is (571)272-5935. The examiner can normally be reached 8:00 AM-5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AIDAN W MCCOY/Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611