Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 21 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 12 of U.S. Patent No. 11,308,659. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the patent disclose all of the features of the present application.
Re claim 21 Claim 12 of the patent discloses
A computer-implemented method for view synthesis with adjustable visual characteristics, the method comprising: (see claim 12 “A computer-implemented method for view synthesis with user-specifiable characteristics, the method comprising”)
obtaining a position within three-dimensional space and an embedding, wherein the embedding encodes one or more visual characteristics of a synthetic image to be generated; (see claim 12 “obtaining, by a computing system comprising one or more computing devices, a desired position within three-dimensional space and a user-specified generative embedding, wherein the generative embedding encodes one or more visual characteristics of a synthetic image to be generated”)
generating, using a first portion of a view synthesis model, an opacity (see claim 12 “processing, by the computing system, data descriptive of the position with a base portion of a machine-learned view synthesis model to generate an opacity and a latent representation”);
generating using a second portion of a view synthesis model, and based on the embedding, color data (see claim 12 “processing, by the computing system, the latent representation and the generative embedding with a content portion of the machine-learned view synthesis model to generate a color”);
and
rendering, based on the opacity and the color data, a synthetic pixel color for a synthetic pixel of the synthetic image; wherein the synthetic image exhibits the one or more visual characteristics encoded by the embedding (see claim 12 “performing, by the computing system, volumetric rendering to generate a synthetic pixel color for a synthetic pixel of the synthesized image from the opacity and the color, wherein the synthesized image exhibits the one or more visual characteristics encoded by the generative embedding” note while the language is phrased differently it discloses all the elements of this limitation).
Claim 22 and 40 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 12 of U.S. Patent No. US 11308659 B2 in view of ZIEGLER US 20210082185 A1
Re claim 22 claim 12 discloses all the elements of claim 21. Claim 12 does not disclose wherein the one or more visual characteristics correspond to at least one characteristic selected from: a lighting characteristic, time of day, weather, or style. Ziegler discloses wherein the one or more visual characteristics correspond to at least one characteristic selected from: a lighting characteristic, time of day, weather, or style ( see paragraph 26 “By mapping the target view image on the geometry representation, the high quality texture of the target view image may be combined with the position information of the one or more objects comprised in the geometry representation. Using the geometry representation may be beneficial for considering lighting effects. For example, the geometry representation may be used to compute lighting effects, such as computer generated illumination, shading, reflections or specularities, for example by raytracing, such that the lighting effects may be considered very accurately, leading to a very realistic final image” Note that the geometry may be modified to account for lighting effects). The motivation to combine is “provides a way to render CG scenes with high resolution light field objects and realistic lighting” (see paragraph 26). One of ordinary skill in the art could have modified the teachings of claim 12 with the teachings of Ziegler to reach the aforementioned advantage. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Ziegler and claim 12.
Re claim 40 Claim 12 of the patent discloses
A computer-implemented method for view synthesis with adjustable visual characteristics, the method comprising: (see claim 12 “A computer-implemented method for view synthesis with user-specifiable characteristics, the method comprising”)
obtaining a position within three-dimensional space and an embedding, wherein the embedding encodes one or more visual characteristics of a synthetic image to be generated; (see claim 12 “obtaining, by a computing system comprising one or more computing devices, a desired position within three-dimensional space and a user-specified generative embedding, wherein the generative embedding encodes one or more visual characteristics of a synthetic image to be generated”)
generating, using a first portion of a view synthesis model, an opacity (see claim 12 “processing, by the computing system, data descriptive of the position with a base portion of a machine-learned view synthesis model to generate an opacity and a latent representation”);
generating using a second portion of a view synthesis model, and based on the embedding, color data (see claim 12 “processing, by the computing system, the latent representation and the generative embedding with a content portion of the machine-learned view synthesis model to generate a color”);
and
rendering, based on the opacity and the color data, a synthetic pixel color for a synthetic pixel of the synthetic image; wherein the synthetic image exhibits the one or more visual characteristics encoded by the embedding (see claim 12 “performing, by the computing system, volumetric rendering to generate a synthetic pixel color for a synthetic pixel of the synthesized image from the opacity and the color, wherein the synthesized image exhibits the one or more visual characteristics encoded by the generative embedding” note while the language is phrased differently it discloses all the elements of this limitation).
Claim 40 does not expressly disclose A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations. In a similar field of endeavor ZIEGLER discloses A computing system, comprising: one or more processors; ( see paragraph 28 “According to a further aspect, computer programs are provided, wherein each of the computer programs is configured to implement the above-described method when being executed on a computer or signal processor, so that the above-described method is implemented by one of the computer programs.” Note that the invention is implement using software and a processor) and one or more non-transitory computer-readable media that store instructions that, (see paragraph 244 “the invention can be implemented in hardware or in software or at least partially in hardware or at least partially in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable” note that the software is stored on a computer readable medium) when executed by the one or more processors, cause the computing system to perform operations ( see paragraph 28 “According to a further aspect, computer programs are provided, wherein each of the computer programs is configured to implement the above-described method when being executed on a computer or signal processor, so that the above-described method is implemented by one of the computer programs.”). The motivation to combine is to implement the method using a computer (See paragraph 28). One of ordinary skill in the art could have easily modified the teachings of claim 40 to be implemented with the device of Ziegler to reach the aforementioned advantage. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 40 and Ziegler.
Claim 23 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 12 of U.S. Patent No. US 11308659 B2 in view of Lee US 2020/0389605
Re claim 23 claim 12 discloses all the elements of claim 21. claim 12 does not expressly disclose wherein the embedding is obtained based on a user input that describes the one or more visual characteristics. Lee discloses wherein the embedding is obtained based on a user input that describes the one or more visual characteristics (see paragraph 29 “In operation, one of the predefined viewing positions may be selected (e.g., by user or automatically chosen) as a new viewing position and, accordingly, view synthesis may be performed to create an effect of viewing the scene from the new viewing position via one or more see-through cameras” note that view synthesis parameters are input by the user). The motivation to combine is to allow the user to select the view parameters (See paragraph 29). One of ordinary skill in the art could have easily modified claim 12l with the teachings of Lee to allow the user to select the parameters for the view synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine claim 12 and Lee to reach the aforementioned advantage.
Claim 24 and 27 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 12 of U.S. Patent No. US 11308659 B2 in view of Mildenhall et al NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis arXiv:2003.08934v1 [cs.CV] 19 Mar 2020.
Re claim 24 claim 12 discloses all the elements of claim 21, claim 21 does not expressly discloses wherein the embedding does not affect the opacity. ( see section 3 second paragraph “We encourage the representation to be multiview consistent by restricting the network to predict the volume density σ as a function of only the location x, while allowing the RGB color c to be predicted as a function of both location and viewing direction” note that the density [opacity] is only affected by the location and is not affected by the viewing direction ). The motivation to combine is “We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis”. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 12 and Mildenhall to reach the aforementioned advantage.
Re claim 27 claim 12 discloses all the elements of claim 21, claim 21 does not expressly discloses wherein at least one of the first portion of the view synthesis model or the second portion of the view synthesis model comprises a neural network. Mildenhall discloses wherein at least one of the first portion of the view synthesis model or the second portion of the view synthesis model comprises a neural network. ( see claim 3 second paragraph “To accomplish this, the MLP FΘ first processes the input 3D coordinate x with 8 fully-connected layers (using ReLU activations and 256 channels per layer), and outputs σ and a 256-dimensional feature vector. This feature vector is then concatenated with the camera ray’s viewing direction and passed to 4 additional fully-connected layers (using ReLU activations and 128 channels per layer) that output the view-dependent RGB color.” Note that an MLP is a neural network.). The motivation to combine is “We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis”. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Claim 12 and Mildenhall to reach the aforementioned advantage.
Claim 28 29 and 31-32 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 and 2 of U.S. Patent No. 11,308,659. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the patent disclose all of the features of the present application.
Re claim 28 claim 1 discloses
A computer-implemented method, comprising: (see claim 1 “and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising” note that the operations correspond to a method)
generating, using a view synthesis model, a static color based on a position within three- dimensional space that is associated with the scene ( see claim 1 “processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color” note that a static color is generated using the static portion of the model).
generating, using the view synthesis model, a transient color based on the position (see claim 1 “processing the latent representation with the transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color”); and
generating, using the view synthesis model, and based on the static color and the transient color, a synthetic pixel color for a synthetic pixel of a synthetized image (see claim 1 “performing volumetric rendering to generate a synthetic pixel color for a synthetic pixel of a synthesized image from the static opacity, the static color, the transient opacity, and the transient color”) that depicts the scene (see also claim 1 “computing system to generate synthetic imagery of a scene” ).
Re claim 29 claim 2 discloses modifying, based on a loss function that compares the synthetic pixel color with the reference pixel color (see claim 2 “evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color”), the view synthesis model.( see claim 2 “modifying one or more parameter values for one or more parameters of the machine-learned view synthesis model based at least in part on the loss function”)
Re claim 31 claim 1 discloses generating, by a static content portion of the view synthesis model that models static content within the scene, the static color (see claim 1 “processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color” ); and generating, by a transient content portion of the view synthesis model that models transient content within the scene, the transient color (see claim 1 ”processing the latent representation with the transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color”).
Re claim 32 Claim 1 dislcoses generating, by a static content portion of the view synthesis model that models static content within the scene, the static color and a static opacity; (see claim 1 “obtaining a position within three-dimensional space; processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color“) and generating, by a transient content portion of the view synthesis model that models transient content within the scene, the transient color and a transient opacity (see claim 1 ” processing the latent representation with the transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color”).
Claim 28, 31 and 32 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 3 of U.S. Patent No. US 11704844 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the patent disclose all of the features of the present application.
Re claim 28 claim 3 of the patent discloses A computer-implemented method, comprising (see claim 1 [from which claim 3 depends] “store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising”) :generating, using a view synthesis model, a static color based on a position within three- dimensional space that is associated with the scene (see claim 3 “processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color”); generating, using the view synthesis model, a transient color based on the position (see claim 3 “processing the latent representation with the transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color” note that the latent representation is derived from the position); and generating, using the view synthesis model, and based on the static color and the transient color, a synthetic pixel color for a synthetic pixel (see claim 3 “performing volumetric rendering to generate a synthetic pixel color for a synthetic pixel of the synthetic image from the static opacity, the static color, the transient opacity, and the transient color”) of a synthetized image that depicts the scene (see claim 1 “generate a synthetic image of a scene from the position”).
Re claim 31 claim 3 further discloses generating, by a static content portion of the view synthesis model that models static content within the scene, the static color (see claim 3 “processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color”); and generating, by a transient content portion of the view synthesis model that models transient content within the scene, the transient color(see claim 3 “processing the latent representation with the transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color” note that the latent representation is derived from the position).
Re claim 32 Claim 3 discloses generating, by a static content portion of the view synthesis model that models static content within the scene, the static color and a static opacity (see claim 3 “processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color”); and generating, by a transient content portion of the view synthesis model that models transient content within the scene, the transient color and a transient opacity(see claim 3 “processing the latent representation with the transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color).
Claim 36 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 8 of U.S. Patent No. US 11704844 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the patent disclose all of the features of the present application.
Re claim 36 claim 8 of the patent discloses A computer-implemented method, comprising: obtaining, by a computing system comprising one or more processors, (see claim 8 “One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising: obtaining, by the computing system” note that the computing system implements the method)
a position and one or more camera parameters associated with a reference image, wherein the one or more camera parameters comprise: an orientation of a camera, a focal length of the camera, a principal point of the camera, a skew of the camera, a radial distortion, a tangential distortion of the camera, or camera intrinsics of the camera; (see claim 8 “a position within three-dimensional space and one or more camera parameters associated with an existing training image, wherein the one or more camera parameters comprise: an orientation of a camera, a focal length of the camera, a principal point of the camera, a skew of the camera, a radial distortion or a tangential distortion of the camera, or camera intrinsics of the camera”)
processing, by the computing system, the position and the one or more camera parameters with a view synthesis model to generate an opacity and a color; ( see claim 8 ” processing, by the computing system, the position within three-dimensional space and the one or more camera parameters with a machine-learned view synthesis model to generate an opacity and a color”)
rendering, by the computing system, a synthetic pixel color for a synthetic pixel of a synthetic image based on the opacity and the color (see claim 8 “volumetric rendering to generate a synthetic pixel color for a synthetic pixel of a synthesized image from the opacity and the color”);
and modifying, based on a loss function that compares the synthetic pixel color with a reference pixel color for a reference pixel in the reference image, one or more values of the one or more camera parameters (see claim 8 “evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color for a training pixel including in the existing training image; and modifying one or more values of the one or more camera parameters based at least in part on the loss function”).
Claims 28, 29, 31 and 32 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 12100074 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the patent disclose all of the features of the present application.
Re claim 28 claim 1 discloses A computer-implemented method, comprising: (see claim 1 “A computing system to generate synthetic imagery of a scene, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store: a machine-learned view synthesis model; and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:” note that the operations performed by the computer are the method )
generating, using a view synthesis model, a static color based on a position within three- dimensional space that is associated with the scene (see claim 1 “generating a static color based on descriptive data and the machine-learned view synthesis model, wherein the descriptive data is descriptive of a position within three-dimensional space that is associated with the scene”);
generating, using the view synthesis model, a transient color based on the position (see claim 1 “generating a transient color based on the descriptive data and the machine-learned view synthesis model” note the descriptive data is the position); and
generating, using the view synthesis model, and based on the static color and the transient color, a synthetic pixel color for a synthetic pixel of a synthetized image that depicts the scene (see claim 1 “generating a synthetic pixel color for a synthetic pixel of a synthetized image that depicts the scene, wherein the synthetic pixel color is generated based on the static color, the transient color, and the machine-learned view synthesis model”)
Re claim 29 Claim 1 discloses modifying, based on a loss function that compares the synthetic pixel color with the reference pixel color, the view synthesis model. (see claim 1 “evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color; and training the machine-learned view synthesis model based on the loss function”)
Re claim 31 Claim 1 discloses generating, by a static content portion of the view synthesis model that models static content within the scene, the static color;(see claim 1 “generating a static color based on descriptive data and the machine-learned view synthesis model, wherein the descriptive data is descriptive of a position within three-dimensional space that is associated with the scene”) and generating, by a transient content portion of the view synthesis model that models transient content within the scene, the transient color (see claim 1 “generating a transient color based on the descriptive data and the machine-learned view synthesis model”).
Re claim 32 Claim 5 discloses generating, by a static content portion of the view synthesis model that models static content within the scene, the static color see claim 1 “generating a static color based on descriptive data and the machine-learned view synthesis model, wherein the descriptive data is descriptive of a position within three-dimensional space that is associated with the scene”) and a static opacity (see claim 5 “generating a static opacity based on the descriptive data and the machine-learned view synthesis model”); and generating, by a transient content portion of the view synthesis model that models transient content within the scene, the transient color (see claim 1 “generating a transient color based on the descriptive data and the machine-learned view synthesis model”) and a transient opacity (see claim 5” generating a transient opacity based on a latent representation and the machine-learned view synthesis model”).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 28, 29 and 31 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Re claim 28
The limitation generating, using a view synthesis model, a static color based on a position within three- dimensional space that is associated with the scene, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, generating in the context of this claim encompasses the user generating a static color mentally based on a mental model.
The limitation generating, using the view synthesis model, a transient color based on the position, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, generating in the context of this claim encompasses the user generating a transient color mentally based on a mental model.
The limitation generating, using the view synthesis model, and based on the static color and the transient color, a synthetic pixel color for a synthetic pixel of a synthetized image that depicts the scene., as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, generating in the context of this claim encompasses the user mentally determining a color for a pixel.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim only recites one additional element – using a computer to implement the method. The computer is recited at a high-level of generality (i.e., as a generic computer performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a computer to perform both the steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible.
Re claim 29 The limitation modifying, based on a loss function that compares the synthetic pixel color with the reference pixel color, the view synthesis model, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, modifying in the context of this claim encompasses the user mentally modifying a mental model.
The analysis with respect to integration into an abstract idea and significantly more is not substantially changed from the claim from which this claim depends.
Re claim 31 The limitation generating, by a static content portion of the view synthesis model that models static content within the scene, the static color and a static opacity; and generating, by a transient content portion of the view synthesis model that models transient content within the scene, the transient color and a transient opacity, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind. For example, generating in the context of this claim encompasses the user mentally generating static and transient colors and opacities based on mental models.
The analysis with respect to integration into an abstract idea and significantly more is not substantially changed from the claim from which this claim depends.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 21, 24, 27 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mildenhall et al NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis arXiv:2003.08934v1 [cs.CV] 19 Mar 2020.
Re claim 21 Mildenhall discloses A computer-implemented (see section 2 first paragraph note the method is clearly intended to be performed on a computer) method for view synthesis with adjustable visual characteristics (see title and abstract note that the view synthesis is adjusted by position and angle), the method comprising:
obtaining a position within three-dimensional space and an embedding (see section 3 rd paragraph “We represent a continuous scene as a 5D vector-valued function whose input is
a 3D location x = (x,y,z) and 2D viewing direction (θ,φ)” note that the location corresponds to a position within 3 dimensional space, the viewing direction corresponds to the embedding ), wherein the embedding encodes one or more visual characteristics of a synthetic image to be generated ( section 3 first paragrah “We approximate this continuous 5Dscene representation with an MLP network FΘ : (x,d) → (c,σ) and optimize its weights Θ to map from each input 5D coordinate to its corresponding volume density and directional emitted color” and figure 2 caption note that direction encodes the viewing direction which is used to generate the particular view which is render using volume rendering techniques );
generating, using a first portion of a view synthesis model (see claim 3 second paragraph “To accomplish this, the MLP FΘ first processes the input 3D coordinate x with 8 fully-connected layers (using ReLU activations and 256 channels per layer),” Note first process corresponds to the first portion), an opacity (section 3 first paragrah “We approximate this continuous 5Dscene representation with an MLP network FΘ : (x,d) → (c,σ) and optimize its weights Θ to map from each input 5D coordinate to its corresponding volume density and directional emitted color” note that the density corresponds to an opacity)
generating using a second portion (see section 3 second paragraph “and outputs σ and a 256-dimensional feature vector. This feature vector is then concatenated with the camera ray’s viewing direction and passed to 4 additional fully-connected layers (using ReLU activations and 128 channels per layer) that output the view-dependent RGB color.” Note that this corresponds to the second portion of the model) of a view synthesis model, and based on the embedding, color data (section 3 first paragraph “We approximate this continuous 5Dscene representation with an MLP network FΘ : (x,d) → (c,σ) and optimize its weights Θ to map from each input 5D coordinate to its corresponding volume density and directional emitted color” note that the density corresponds to an opacity” note that the directional emitted color corresponds to the color data.);
and rendering, based on the opacity and the color data, a synthetic pixel color for a synthetic pixel of the synthetic image (see abstract and see figure 2 caption “feeding those locations into an MLP to produce a color and volume density (b), and using volume rendering techniques to composite these values into an image (c).” note that the density and color data are composited into an image using volume rendering);
wherein the synthetic image exhibits the one or more visual characteristics encoded by the embedding (see abstract “ whose output is the volume density and view-dependent emitted radiance at hat spatial location. We synthesize views by querying 5D coordinates long camera rays and use classic volume rendering techniques to project the output colors and densities into an image” note that the image is dependent on the view i.e. the embedding).
Re claim 24 Mildenhall discloses wherein the embedding does not affect the opacity ( see section 3 second paragraph “We encourage the representation to be multiview consistent by restricting the network to predict the volume density σ as a function of only the location x, while allowing the RGB color c to be predicted as a function of both location and viewing direction” note that the density [opacity] is only affected by the location and is not affected by the viewing direction ).
Re claim 27 Mildenhall discloses wherein at least one of the first portion of the view synthesis model or the second portion of the view synthesis model comprises a neural network. ( see claim 3 second paragraph “To accomplish this, the MLP FΘ first processes the input 3D coordinate x with 8 fully-connected layers (using ReLU activations and 256 channels per layer), and outputs σ and a 256-dimensional feature vector. This feature vector is then concatenated with the camera ray’s viewing direction and passed to 4 additional fully-connected layers (using ReLU activations and 128 channels per layer) that output the view-dependent RGB color.” Note that an MLP is a neural network.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mildenhall et al NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis arXiv:2003.08934v1 [cs.CV] 19 Mar 2020 in view of Lee US 2020/0389605
Re claim 23 Mildenhall discloses all the elements of claim 21. Mildenhall does not expressly disclose wherein the embedding is obtained based on a user input that describes the one or more visual characteristics. In a similar field of endeavor Lee discloses wherein the embedding is obtained based on a user input that describes the one or more visual characteristics (see paragraph 29 “In operation, one of the predefined viewing positions may be selected (e.g., by user or automatically chosen) as a new viewing position and, accordingly, view synthesis may be performed to create an effect of viewing the scene from the new viewing position via one or more see-through cameras” note that view synthesis parameters are input by the user). The motivation to combine is to allow the user to select the view parameters (See paragraph 29). One of ordinary skill in the art could have easily modified Mildenhall with the teachings of Lee to allow the user to select the parameters for the view synthesis. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mildenhall and Lee to reach the aforementioned advantage.
Claim(s) 22 and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mildenhall et al NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis arXiv:2003.08934v1 [cs.CV] 19 Mar 2020 in view of ZIEGLER US 20210082185 A1
Re claim 22 Mildenhall discloses all the elements of claim 21. Mildenhall does not disclose wherein the one or more visual characteristics correspond to at least one characteristic selected from: a lighting characteristic, time of day, weather, or style. In a similar field of endeavor Ziegler discloses wherein the one or more visual characteristics correspond to at least one characteristic selected from: a lighting characteristic, time of day, weather, or style ( see paragraph 26 “By mapping the target view image on the geometry representation, the high quality texture of the target view image may be combined with the position information of the one or more objects comprised in the geometry representation. Using the geometry representation may be beneficial for considering lighting effects. For example, the geometry representation may be used to compute lighting effects, such as computer generated illumination, shading, reflections or specularities, for example by raytracing, such that the lighting effects may be considered very accurately, leading to a very realistic final image” Note that the geometry may be modified to account for lighting effects). The motivation to combine is “provides a way to render CG scenes with high resolution light field objects and realistic lighting” (see paragraph 26). One of ordinary skill in the art could have modified the teachings of Mildenhall with the teachings of Ziegler to reach the aforementioned advantage. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Ziegler and Mildenhall.
Re claim 40 Mildenhall discloses computer-implemented (see section 2 first paragraph note the method is clearly intended to be performed on a computer) method for view synthesis with adjustable visual characteristics (see title and abstract note that the view synthesis is adjusted by position and angle), the method comprising:
obtaining a position within three-dimensional space and an embedding (see section 3 rd paragraph “We represent a continuous scene as a 5D vector-valued function whose input is
a 3D location x = (x,y,z) and 2D viewing direction (θ,φ)” note that the location corresponds to a position within 3 dimensional space, the viewing direction corresponds to the embedding ), wherein the embedding encodes one or more visual characteristics of a synthetic image to be generated ( section 3 first paragrah “We approximate this continuous 5Dscene representation with an MLP network FΘ : (x,d) → (c,σ) and optimize its weights Θ to map from each input 5D coordinate to its corresponding volume density and directional emitted color” and figure 2 caption note that direction encodes the viewing direction which is used to generate the particular view which is render using volume rendering techniques );
generating, using a first portion of a view synthesis model (see claim 3 second paragraph “To accomplish this, the MLP FΘ first processes the input 3D coordinate x with 8 fully-connected layers (using ReLU activations and 256 channels per layer),” Note first process corresponds to the first portion), an opacity (section 3 first paragrah “We approximate this continuous 5Dscene representation with an MLP network FΘ : (x,d) → (c,σ) and optimize its weights Θ to map from each input 5D coordinate to its corresponding volume density and directional emitted color” note that the density corresponds to an opacity)
generating using a second portion (see section 3 second paragraph “and outputs σ and a 256-dimensional feature vector. This feature vector is then concatenated with the camera ray’s viewing direction and passed to 4 additional fully-connected layers (using ReLU activations and 128 channels per layer) that output the view-dependent RGB color.” Note that this corresponds to the second portion of the model) of a view synthesis model, and based on the embedding, color data (section 3 first paragraph “We approximate this continuous 5Dscene representation with an MLP network FΘ : (x,d) → (c,σ) and optimize its weights Θ to map from each input 5D coordinate to its corresponding volume density and directional emitted color” note that the density corresponds to an opacity” note that the directional emitted color corresponds to the color data.);
and rendering, based on the opacity and the color data, a synthetic pixel color for a synthetic pixel of the synthetic image (see abstract and see figure 2 caption “feeding those locations into an MLP to produce a color and volume density (b), and using volume rendering techniques to composite these values into an image (c).” note that the density and color data are composited into an image using volume rendering);
wherein the synthetic image exhibits the one or more visual characteristics encoded by the embedding (see abstract “whose output is the volume density and view-dependent emitted radiance at hat spatial location. We synthesize views by querying 5D coordinates long camera rays and use classic volume rendering techniques to project the output colors and densities into an image” note that the image is dependent on the view i.e. the embedding).
Mildenhall does not expressly disclose A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations. In a similar field of endeavor ZIEGLER discloses A computing system, comprising: one or more processors; ( see paragraph 28 “According to a further aspect, computer programs are provided, wherein each of the computer programs is configured to implement the above-described method when being executed on a computer or signal processor, so that the above-described method is implemented by one of the computer programs.” Note that the invention is implement using software and a processor) and one or more non-transitory computer-readable media that store instructions that, (see paragraph 244 “the invention can be implemented in hardware or in software or at least partially in hardware or at least partially in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable” note that the software is stored on a computer readable medium) when executed by the one or more processors, cause the computing system to perform operations ( see paragraph 28 “According to a further aspect, computer programs are provided, wherein each of the computer programs is configured to implement the above-described method when being executed on a computer or signal processor, so that the above-described method is implemented by one of the computer programs.”). The motivation to combine is to implement the method using a computer (See paragraph 28). One of ordinary skill in the art could have easily modified the teachings of Mildenhall to be implemented with the device of Ziegler to reach the aforementioned advantage. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mildenhall and Ziegler.
Claim(s) 28, 30-31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hasinoff et al “Boundary matting for view synthesis” Computer Vision and Image Understanding Date: July 2006 in view of Pham US 20180174331 A1.
Re claim 28 Hasinoff discloses A computer-implemented (see section 1 first paragraph note that method is a computer vision application) method, comprising:
generating, using a view synthesis model, a background color (see section 4.2 background color estimation) based on a position within three- dimensional space that is associated with the scene (see section 3.1 and 4.2 “For a given boundary pixel, we find potentially corresponding background colors by forward-warping that pixel to all other views” note that background color is determined for points along a 3d boundary “)
generating, using the view synthesis model, a foreground color based on the position (see figure 3 “Geometric view of the system. (A) Stereo depth information is used to detect an occlusion boundary in the reference view, which is backprojected to 3D as our initial curve estimate. The 3D curve is refined, along with estimates for F color” note that foreground or f color is determined);
and generating, using the view synthesis model, and based on the foreground color and the background color, a synthetic pixel color for a synthetic pixel of a synthetized image that depicts the scene (see section 3 note that the color of pixels for view synthesis is determined by compositing the foreground and background with an opacity a see also figure 3 and figure 9).
Hasinoff does not expressly disclose Background is Static and foreground is transient. In a similar field of endeavor Pham discloses a transient foreground and a static background (See paragraph 9). The motivation to combine is “A dynamic scene modelling method also updates these mode models using the visual properties of incoming images. This updating step ensures the scene model is up to date with the dynamic changes happening in the scene including but not limited to illumination changes, or permanent changes to the background content such as addition, removal or one-off movement of fixed objects” (see paragraph 9) One of ordinary skill in the art could have modified Hasinoff to account for transient foreground objects as disclosed in Pham. Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Hasinoff and Pham to reach the aforementioned advantage.
Re claim 30 Hasinoff discloses computing a volumetric rendering along a ray that passes through the position (see figure 6 note that rays are computed along the boundary to determine background colors along the boundary also see figure 9 for the rendering).
Re claim 31 Hasinoff discloses generating, by a background content portion of the view synthesis model that models static content within the scene, the static color (see section 4.2 note that this section describes estimation of background colords); and generating, by a foreground content portion of the view synthesis model that models transient content within the scene, the foreground color (see section 4.3 note that this section estimates foreground colors ). Pham discloses a transient foreground and a static background (See paragraph 9).
Claim(s) 28 and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Stereo Magnification: Learning view synthesis using multiplane images arXiv:1805.09817v1 [cs.CV] 24 May 2018.
Re claim 28 Zhou discloses A computer-implemented method, comprising:
generating, using a view synthesis model, a background color (see figure 3 note that nueral network estimates the background color) based on a position within three- dimensional space that is associated with the scene (see section 3.2 first paragraph “In addition to the input images I1 and I2, we take as input their corresponding camera parameters c1 = (p1,k1) and c2 = (p2,k2), where pi and ki denote camera extrinsic (position and orientation) and intrinsics, respectively” note that the position of the camera is used as an input to the network see also figure 2);
generating, using the view synthesis model, a foreground color based on the position (See section 5.4 note that “Color layer prediction. In Section3.2, we propose that our network
create the color values for each MPI plane as a weighted average of a network predicted “background” image and the reference source image. Here we compare several variants of the color prediction format (ordered by increasing level of representation flexibility)” and “Foreground + background + blending weights. In contrast to the previous variant, instead of using the reference source as the foreground image, the network predicts an extra foreground image for blending with the background”;
and generating, using the view synthesis model, and based on the background color and the foreground color, a synthetic pixel color for a synthetic pixel of a synthetized image that depicts the scene (see figure 3 and section 3.2 “. Hence, for each depth plane, we compute each
RGB image Cd as a per-pixel weighted average of the foreground image I1 and the predicted background image ˆIb: Cd =wd ⊙I1 +(1−wd)⊙ ˆIb , (1)where ⊙ denotes the Hadamard product, and the blending weights wd are also predicted by the network. ).
Zhou does not expressly disclose Background is Static and foreground is transient. In a similar field of endeavor Pham discloses a transient foreground and a static background (See paragraph 9). The motivation to combine is “A dynamic scene modelling method also updates these mode models using the visual properties of incoming images. This updating step ensures the scene model is up to date with the dynamic changes happening in the scene including but not limited to illumination changes, or permanent changes to the background content such as addition, removal or one-off movement of fixed objects” (see paragraph 9).One of ordinary skill in the art could have modified Zhou to account for transient foreground objects as disclosed in Pham. Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Zhou and Pham to reach the aforementioned advantage.
Re claim 29 Zhou discloses modifying, based on a loss function that compares the synthetic pixel color with the reference pixel color, the view synthesis model. (see section 3.4 first paragraph including section 4 note that a loss function which compares a synthesized image to a ground truth is used to train the model.)
Allowable Subject Matter
Claims 25, 26, 33-35, 37-39 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 36 would be allowable if the double patenting rejection was overcome.
Claims 31 and 32 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claim and the double patenting rejections were overcome.
Cited Art
The following is a listing of cited art which is considered relevant but not cited in a rejection above:
Sunkavalli US 20210012561 A1 disclose mbodiments are generally directed to generating novel images of an object having a novel viewpoint and a novel lighting direction based on sparse images of the object. A neural network is trained with training images rendered from a 3D model. Utilizing the 3D model, training images, ground truth predictive images from particular viewpoint(s), and ground truth predictive depth maps of the ground truth predictive images, can be easily generated and fed back through the neural network for training. Once trained, the neural network can receive a sparse plurality of images of an object, a novel viewpoint, and a novel lighting direction. The neural network can generate a plane sweep volume based on the sparse plurality of images, and calculate depth probabilities for each pixel in the plane sweep volume. A predictive output image of the object, having the novel viewpoint and novel lighting direction, can be generated and output. (see abstract).
Lin US 20210272253 A1 discloses The present disclosure relates to an image merging system that automatically and seamlessly detects and merges missing people for a set of digital images into a composite group photo. For instance, the image merging system utilizes a number of models and operations to automatically analyze multiple digital images to identify a missing person from a base image, segment the missing person from the second image, and generate a composite group photo by merging the segmented image of the missing person into the base image. In this manner, the image merging system automatically creates merged group photos that appear natural and realistic. (see abstract).
Ito US 8849017 B2 discloses An image processing apparatus includes: an image feature outputting unit that outputs each of image features in correspondence with a time of the frame; a foreground estimating unit that estimates a foreground image at a time s by executing a view transform as a geometric transform on a foreground view model and outputs an estimated foreground view; a background estimating unit that estimates a background image at the time s by executing a view transform as a geometric transform on a background view model and outputs an estimated background view; a synthesized view generating unit that generates a synthesized view by synthesizing the estimated foreground and background views; a foreground learning unit that learns the foreground view model based on an evaluation value; and a background learning unit that learns the background view model based on the evaluation value by updating the parameter of the foreground view model. (see abstract).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEAN T MOTSINGER whose telephone number is (571)270-1237. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEAN T MOTSINGER/Primary Examiner, Art Unit 2673
Prosecution Insights