DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the claims at issue are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the reference application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/forms/. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to http://www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim(s) 1, 11, 20 is/are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-17 of patent US 12307573. Although the claims at issue are not identical, they are not patentably distinct from each other because the pending claim(s) is/are an obvious variation of the patented claims or entirely covered by the patented claims.
For example, claim 1 of the instant application discloses A method, comprising: receiving an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps, generating a rasterized RGB (Red Green Blue) image based on the object mesh, the viewpoint, and the set of maps; generating, using the neural network model, an output image of the object with the reflective surface based at least in part on the rasterized RGB image and the viewpoint; and providing for display the output image of the object with the reflective surface on a display of a computer client device. These limitations are all disclosed by claim 4 of the patent US 12307573. Therefore, claim 1 of the instant application is covered by claim 4 of the patent US 12307573, and is/are not patently distinct from the mentioned patent claim.
The following table illustrates a comparative mapping between the limitations of claim 1 of the instant application and the mapping claim 4 of patent US 12307573.
Claim 1 of the Instant Application 19186056
Claim 4 of the Patent 12307573
A method, comprising: receiving an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps,
A method, comprising: receiving an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps,
wherein the viewpoint comprises a vector from a point of a surface of the object mesh, and the set of maps comprises a set of textures that were modified from training a neural network model,
wherein the viewpoint comprises a vector from a point of a surface of the object mesh, and the set of maps comprises a set of textures that were modified from training a neural network model,
the set of textures comprising a BRDF (Bidirectional Reflectance Distribution Function) texture, a roughness texture, an irradiance map texture, and a set of specular textures;
the set of textures comprising a set of specular textures, wherein the set of specular textures comprises a plurality of specular textures including at least a first specular texture and a second specular texture, the second specular texture being a different size from the first specular texture;
wherein the set of specular textures comprises a plurality of specular textures including at least a first specular texture and a second specular texture, the second specular texture being half a size of the first specular texture
generating a rasterized RGB (Red Green Blue) image based on the object mesh, the viewpoint, and the set of maps;
generating a rasterized RGB (Red Green Blue) image based on the object mesh, the viewpoint, and the set of maps;
generating, using the neural network model, an output image of the object with the reflective surface based at least in part on the rasterized RGB image and the viewpoint; and
generating, using the neural network model, an output image of the object with the reflective surface based at least in part on the rasterized RGB image and the viewpoint; and
providing for display the output image of the object with the reflective surface on a display of a computer client device.
providing for display the output image of the object with the reflective surface on a display of a computer client device.
The following is a complete listing of the correspondence between the claim of the instant application and the patents:
Claims of the Instant Application 19186056
1
11, 20
Claims of the Patent 12307573
4
13
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 11-13, 15, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over SEVASTOPOLSKIY et al. (US 20220157014) in view of Lakshminarayana et al. (US 20230169715).
Regarding claim 1, SEVASTOPOLSKIY discloses A method, comprising: receiving an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps (“[0026] Predicted albedo describes the spatially varying reflectance properties (albedo ρ(x)) of the head surface (skin, hair, or other parts). [0036] An image and a camera viewpoint corresponding to the image are randomly sampled at S100 from the captured sequence of images. A predicted image (e.g., predicted albedo, normals, environmental shadow maps, and segmentation mask) for the camera viewpoint is obtained at S105 by the deep neural network. Auxiliary face meshes with corresponding texture mapping are predicted at S110 by a 3D face mesh reconstruction network for each of the images (or frames) in the captured sequence, including the image randomly sampled at S100.”),
wherein the viewpoint comprises a vector from a point of a surface of the object mesh (“[0032] The camera viewpoints estimated at step S205.1 may be used for generating the 3D point cloud as well as for training the deep neural network, latent descriptors for the 3D point cloud, and auxiliary parameters during the training stage. Estimating camera viewpoints at S205.1 may be performed using, for example, SfM. Points of the originally generated 3D point cloud at least partly corresponding to an upper body of the person. Claim 8, the latent descriptor includes a multi-dimensional latent vector characterizing properties of a corresponding point in the 3D point cloud”. Noted the point cloud measures points on the external surface of objects), and
generating a rasterized RGB (Red Green Blue) image based on the object mesh, the viewpoint, and the set of maps (“[0024] Latent descriptors of the generated 3D point cloud may be rasterized at different resolutions according to a requested new camera viewpoint to obtain rasterized images 15. [0031] One or more inputs defining a camera viewpoint and/or lighting conditions are received at S210. Latent descriptors of the 3D point cloud, which are previously obtained during the training stage, are rasterized at S215 at different resolutions according to the camera viewpoint to obtain rasterized images. [0037] A median reference texture is obtained at S115 by calculating a pixel-wise median of all images from the set of flash images resampled to the texture space by the bilinear interpolation. The auxiliary face meshes predicted at S110 are rasterized at S120 using color-coding vertices of the face mesh with their geometric normals and Z-buffering. Color-coding vertices with their geometric normals is a process of assigning R, G, B color channels for each point to X, Y, Z coordinates of a normal vector in that point, respectively. Then, the assigned RGB colors are rasterized via Z-buffering”);
generating, using the neural network model, an output image of the object with the reflective surface based at least in part on the rasterized RGB image and the viewpoint; and providing for display the output image of the object with the reflective surface on a display of a computer client device (figs.1&4, “[0026] Neural rendering may be performed by processing the rasterized images 15 with, for example, a deep neural network trained to predict albedo, normals, environmental shadow maps, and segmentation mask for the received camera viewpoint. Predicted albedo describes the spatially varying reflectance properties (albedo ?(x)) of the head surface (skin, hair, or other parts). [0031] Neural rendering is performed by processing at S220 the rasterized images with a deep neural network to predict albedo, normals, environmental shadow maps, and segmentation mask for the received (e.g., new) camera viewpoint. The relighted 3D portrait is rendered by fusing at S225 the predicted albedo, normals, environmental shadow maps, and segmentation mask into the relighted 3D portrait according to the received lighting conditions. [0037] Then, the assigned RGB colors are rasterized via Z-buffering”).
On the other hand, SEVASTOPOLSKIY fails to explicitly disclose but Lakshminarayana discloses the set of maps comprises a set of textures that were modified from training a neural network model, the set of textures comprising a set of specular textures, wherein the set of specular textures comprises a plurality of specular textures including at least a first specular texture and a second specular texture, the second specular texture being a different size from the first specular texture (“[0050] To determine the light-reflectance values for the textured surface, the present techniques include using a modified version of mipmapping referred herein as neural-texture pyramid. The neural-texture pyramid uses a set of learned power-of-2 feature textures (e.g., 512×512) to represent the material at different scales. The neural-texture pyramid can be queried to identify the input vector (e.g., a reflectance feature vector) for a particular texel. [0053] the present techniques can accurately render textured surfaces based on a neural network trained with a small batch of training data (e.g., about 200-400 random BTF queries per texel)”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined SEVASTOPOLSKIY and Lakshminarayana, to include all limitations of claim 1. That is, applying the neural-texture pyramid of Lakshminarayana to the texture mapping of SEVASTOPOLSKIY. The motivation/ suggestion would have been the neural-texture pyramid only requires storage on the order of traditional mipmapped texture pyramids (with 7 channels per texel), while enabling efficient prefiltering and multiscale appearance (Lakshminarayana, [0053]).
Regarding claim 2, SEVASTOPOLSKIY in view of Lakshminarayana discloses The method of claim 1.
SEVASTOPOLSKIY further discloses wherein the object mesh was generated based on photogrammetry techniques applied on a set of source images of a physical object, the set of source images including images captured from a set of different viewpoints of the physical object (“Abstract, the 3D point cloud is generated based on a sequence of images captured by a camera with a blinking flash while moving the camera at least partly around an upper body. [0023] A 3D point cloud 10 may be generated based on the sequence of captured images. The 3D point cloud 10 may be generated using, for example, and without limitation, a Structure-from-Motion (SfM) technique or any other known techniques allowing to reconstruct 3D structure of a scene or object based on the sequence of 2D images of that scene or object. The 3D point cloud is generated and camera viewpoints are estimated based on the set of flash images or the set of no-flash images.”).
Regarding claim 3, SEVASTOPOLSKIY in view of Lakshminarayana discloses The method of claim 2.
SEVASTOPOLSKIY further discloses sending the rasterized RGB image to the neural network model for rendering the object (“[0031] Neural rendering is performed by processing at S220 the rasterized images with a deep neural network to predict albedo, normals, environmental shadow maps, and segmentation mask for the received (e.g., new) camera viewpoint. The relighted 3D portrait is rendered by fusing at S225 the predicted albedo, normals, environmental shadow maps, and segmentation mask into the relighted 3D portrait according to the received lighting conditions”).
Regarding claim 5, SEVASTOPOLSKIY in view of Lakshminarayana discloses The method of claim 1.
SEVASTOPOLSKIY further discloses wherein the second specular texture is half a size of the first specular texture (“[0050] The neural-texture pyramid uses a set of learned power-of-2 feature textures (e.g., 512×512) to represent the material at different scales”. Therefore, for example, feature textures 512*512 is half size when adding 1 to the power of 2).
Regarding claim(s) 11-13, 15, they are interpreted and rejected for the same reasons set forth in claim(s) 1-3, 5, respectively.
Regarding claim(s) 20, it is interpreted and rejected for the same reasons set forth in claim(s) 1.
Particularly, claims 11, 20 further discloses “A system comprising: a processor; and a memory including instructions that, when executed by the processor, cause the processor to perform operations”, and “A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations”.
SEVASTOPOLSKIY further discloses “[0010] According to an example embodiment, a computing device is provided, the computing device comprising: a processor and a memory storing processor-executable instructions and weights of a deep neural network, latent descriptors, and auxiliary parameters obtained during a training stage, wherein upon execution of the processor-executable instructions by the processor the processor causes the computing device to carry out a method for rendering a relighted 3D portrait of a person according to the above example method or any further implementation of the above example method”.
Claim(s) 4, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over SEVASTOPOLSKIY et al. (US 20220157014) in view of Lakshminarayana et al. (US 20230169715), and further in view of CAI et al. (US 20190251697).
Regarding claim 4, SEVASTOPOLSKIY in view of Lakshminarayana discloses The method of claim 1.
On the other hand, SEVASTOPOLSKIY in view of Lakshminarayana fails to explicitly disclose but CAI discloses wherein the set of textures comprises a roughness texture, the roughness texture includes a set of pixels, each pixel having a value from 0 to 1 (“[0074] FIG. 9B shows mapping roughness by using V-system with degree k=1 in one example embodiment, where the global DEM called LDEM16 at a resolution of 16 pixels/degree by 16 pixels/degree is used, mapping of global roughness is achieved using 8×8 orthogonal matrix. In order to provide a convenient comparison, all the values in roughness maps are normalized from 0 to 1.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined CAI into the combination of Lakshminarayana and SEVASTOPOLSKIY, to include all limitations of claim 4. That is, adding the normalized roughness of CAI to the textures of SEVASTOPOLSKIY and Lakshminarayana. The motivation/ suggestion would have been to provide a method that generates an improved map showing surface roughness of a planetary surface (CAI, [0005]).
Regarding claim(s) 14, it is interpreted and rejected for the same reasons set forth in claim(s) 4.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over SEVASTOPOLSKIY et al. (US 20220157014) in view of Lakshminarayana et al. (US 20230169715), and further in view of Kaji et al. (Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging).
Regarding claim 10, SEVASTOPOLSKIY in view of Lakshminarayana discloses The method of claim 1.
On the other hand, SEVASTOPOLSKIY in view of Lakshminarayana fails to explicitly disclose but Kaji discloses wherein the neural network model comprises an image to image model (Kaji, “5.1 Hybrid approach, As reconstruction can be thought of as image-to-image translation, the DNNs discussed in Sect. 1.3 (e.g., AUTOMAP [27]) can be directly applicable if we have enough data consisting of pairs of original and translated images”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Kaji into the combination of Lakshminarayana and SEVASTOPOLSKIY, to include all limitations of claim 10. That is, applying the image-to-image translation of Kaji to the neural network of SEVASTOPOLSKIY and Lakshminarayana. The motivation/ suggestion would have been to achieve a better quality of reconstruction with a lesser amount of training image pairs (Kaji, 5.1).
Allowable Subject Matter
Claim(s) 6-9, 16-19 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 6, it recites, wherein the set of textures comprising a BRDF (Bidirectional Reflectance Distribution Function) texture, a roughness texture, an irradiance map texture, the BRDF texture comprises a first three channel image, the roughness texture comprises a one channel image, the irradiance map texture comprises second three channel image, and each specular texture comprises a particular three channel image corresponding to a particular level of an environment map. None of the prior arts on the record or any of the prior arts searched, alone or in combination, renders obvious the combination of elements recited in the claim(s) as a whole.
Regarding claim 16, it is interpreted and allowed under similar rationale set forth in claim 6.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRACE Q LI whose telephone number is (571)270-0497. The examiner can normally be reached Monday - Friday, 8:00 am-5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DEVONA FAULK can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GRACE Q LI/Primary Examiner, Art Unit 2618 10/8/2026
Prosecution Insights