DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 6, 7, 9, 10, 14 – 17, 19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Huang; Xudong et al. (RefSR-NeRF: Towards High Fidelity and Super Resolution View Synthesis; hereinafter simply referred to as Huang).
Regarding independent claim 1, Huang teaches:
A method of generating an image, performed by an electronic device (See Page 8248 Right Column Paragraph 1, Page 8245 Right Column Paragraph 2, wherein a method of generating an image (synthesizing HR images) is implemented via a device, comprising a V100 NVIDIA GPU with 32GB memory)
obtaining an image sequence of a target scene (See Page 8248 Right Column Paragraph 3, Figure 1, real-world scene made up of 20-62 images) and information about a first viewing angle (See Page 8246 Left Column Final Paragraph, viewing angle represented by the viewing direction ‘d’), the image sequence comprising a plurality of images captured from the first viewing angle and having a first resolution (See Figure 1, Page 8246 Left Column Final Paragraph wherein the image sequence/patch representing the scene is learned in low resolution (first resolution) from a first viewing angle/direction)
generating a plurality of rays corresponding to a plurality of pixels of an image plane of the first viewing angle of the target scene determining a plurality of spatial points by sampling the plurality of rays (See Page 8247 Left Column Paragraph 1 – 3, Page 8246 Left Column Final Paragraph, Page 8246 Right Column Paragraphs 1 – 3, Figure 1 and 2 wherein a plurality of rays ‘r’ are sampled corresponding to each pixel of the first viewing angle/direction, wherein spatial points (location ‘x’) are determined using the plurality of rays)
generating a first rendered image of the image plane by rendering, using a first neural network, the plurality of spatial points, the first rendered image having the first resolution (See Page 8246 Left Column Final Paragraph, Page 8246 Right Column Paragraphs 1 – 3Figure 1, 2 wherein a low resolution (first resolution) rendering is generated using a first neural network (MLP) and spatial points (location ‘x’))
determining a reference image of the first rendered image from among the plurality of images of the image sequence (See Page 8248 Left Column Paragraph 3 Figure 1, wherein a reference image is selected from among a plurality of images of the image sequence (selected reference middle image/frame))
and generating a second rendered image having a second resolution by upsampling, using a second neural network, the first rendered image based on the reference image. (See Figure 1, 2 wherein a second rendered image (novel view HR image) is generated having a second resolution (high resolution) using a second neural network (RefSR-CNN), and the first rendered image (Low Resolution LR rendering) based on the reference image (HR reference image)).
Regarding dependent claim 2, Huang teaches:
Obtaining the image sequence by downsampling an original image of the target scene. (See Page 8245 Left Column Paragraph 1, wherein the image sequence (low resolution images) is obtained by downsampling an original image (HR Training images) of the target scene).
Regarding dependent claim 6, Huang teaches:
Generating a first feature by extracting a first plurality of features from the first rendered image; (See Page 8248 Left Column Paragraph 1, Figure 2 wherein a first plurality of features (Flr features) are extracted from the first rendered image (Novel View LR image))
generating a second feature by extracting a second plurality of features from the reference image; (See Page 8248 Left Column Paragraph 1, Figure 2 wherein a second plurality of features (Fref features) are generated from the reference image (HR Ref image))
generating a third feature by fusing the first feature and the second feature; (See Page 8248 Left Column Paragraph 1 and 2, Figure 2, equations 7 and 8, wherein the Fref and Flr are fused by being fed into a ResBlock creating a third feature)
generating a fourth feature by performing cascading residual processing on the third feature; (See Page 8248 Left Column Paragraph 1, Figure 2, equations 7 and 8 wherein a fourth feature is generated by performing cascading residual processing on the third feature, being the output of the ResBlock which had the fused Fred and Flr image features as input)
and generating the second rendered image by decoding the fourth feature. (See Page 8248 Left Column Paragraph 1 and 2, Figure 2, equations 7 and 8, wherein the decoder layer (decoding) decodes the features output from the ResBlocks to create the Novel View HR image (second rendered image)).
Regarding dependent claim 7, Huang teaches:
Updating the first neural network based on neural radiance fields (NeRF). (See page 8246 Left Column Paragraph 2 wherein NeRF is used in the updating/training of the first neural network (RefSR model)).
Regarding independent claim 9, claim 9 is a device claim corresponding to claim 1. Please see the discussion of claim 1 above. Furthermore, Huang teaches of an electronic device comprising at least on processor and a memory storing instructions to be carried out by the at least one processor. (See Page 8248 Right Column first paragraph wherein the implementation of the reference based super resolution model is implemented using a device comprising at least one processor (Nvidia GPU) and a memory (32GB memory) storing instructions (program instructions via PyTorch) to execute the super resolution model, please see the explanation provided in claim 1 for further details).
Regarding dependent claim 10, claim 10 is a device claim corresponding to claim 2. Please see the discussion of claim 2 above. Furthermore, Huang teaches of an electronic device comprising at least on processor and a memory storing instructions to be carried out by the at least one processor. (See Page 8248 Right Column first paragraph wherein the implementation of the reference based super resolution model is implemented using a device comprising at least one processor (Nvidia GPU) and a memory (32GB memory) storing instructions (program instructions via PyTorch) to execute the super resolution model please see the explanation provided in claim 2 for further details).
Regarding dependent claim 14, claim 14 is a device claim corresponding to claim 6. Please see the discussion of claim 6 above. Furthermore, Huang teaches of an electronic device comprising at least on processor and a memory storing instructions to be carried out by the at least one processor. (See Page 8248 Right Column first paragraph wherein the implementation of the reference based super resolution model is implemented using a device comprising at least one processor (Nvidia GPU) and a memory (32GB memory) storing instructions (program instructions via PyTorch) to execute the super resolution model please see the explanation provided in claim 6 for further details).
Regarding dependent claim 15, claim 15 is a device claim corresponding to claim 7. Please see the discussion of claim 7 above. Furthermore, Huang teaches of an electronic device comprising at least on processor and a memory storing instructions to be carried out by the at least one processor. (See Page 8248 Right Column first paragraph wherein the implementation of the reference based super resolution model is implemented using a device comprising at least one processor (Nvidia GPU) and a memory (32GB memory) storing instructions (program instructions via PyTorch) to execute the super resolution model please see the explanation provided in claim 7 for further details).
Regarding independent claim 16, Huang teaches:
A method of updating an image generation model, performed by an electronic device, (See Page 8246 Left Column Paragraph 2, Page 8248 Right Column Paragraph 1, wherein an image generation model is updated/trained using a patch rays sampling strategy implemented via a device comprising an NVIDIA GPU and a 32GB Memory)
obtaining an image sequence of a target scene (See Page 8248 Right Column Paragraph 3, Figure 1, real-world scene made up of 20-62 images), the image sequence comprising a plurality of images having a first resolution(See Figure 1, wherein the image sequence/patch representing the scene is learned in low resolution (first resolution));
generating, for a first image of the image sequence, a plurality of rays corresponding to a plurality of pixels of an image plane of a first viewing angle of the target scene and determining a plurality of spatial points by sampling the plurality of rays; (See Page 8247 Left Column Paragraph 1 – 3, Page 8246 Left Column Final Paragraph, Page 8246 Right Column Paragraphs 1 – 3, Page 8247 Right Column Last Paragraph, Figure 1 and 2 wherein a plurality of rays ‘r’ are sampled corresponding to each pixel of the first viewing angle/direction ‘d’, for each of the low resolution, LR images of a scene, (including a first image), wherein spatial points (location ‘x’) are determined using the plurality of rays)
generating a first rendered image of the image plane by rendering the plurality of spatial points using a first neural network of the image generation model, the first rendered image having the first resolution; (See Page 8278 Right Column Last Paragraph, Page 8246 Left Column Final Paragraph, Page 8246 Right Column Paragraphs 1 – 3, Figure 1, 2, wherein a low resolution (first resolution) rendering is generated using a first neural network (MLP in NeRF pipeline) and spatial points (location ‘x’))
determining a reference image of at least one of the first image or the first rendered image; (See Page 8248 Left Column Paragraph 3 Figure 1, wherein a reference image is selected from among a plurality of images of the image sequence (wherein the image selected may be the first image of the image sequence))
generating a second rendered image having a second resolution by upsampling the first rendered image based on the reference image using a second neural network of the image generation model; (See Figure 1, 2 wherein a second rendered image (novel view HR image) is generated having a second resolution (high resolution) using a second neural network (RefSR-CNN), and the first rendered image (Low Resolution LR rendering) based on the reference image (HR reference image)).
and updating the image generation model based on the second rendered image and the first image. (See Page 8248 Left Column Last Paragraph wherein the loss is calculated and used to improve/update the model based on the second rendered image and first image (super resolved HR image based on first image)).
Regarding dependent claim 17, Huang teaches:
Determining the plurality of spatial points corresponding to a predetermined shape by sampling the plurality of rays according to the predetermined shape. (See Page 8246 Left Column Paragraph 2, Page 8247 Left Column Paragraph 3, Page 8248 Right Column Paragraph 1, Page 8249 Left Column Last Paragraph, Figure 1 – 4, wherein the plurality of rays are sampled according to a predetermined shape (image patch having a predefined geometry therefore being a shape)).
Regarding dependent claim 19, Huang teaches:
Generating the second rendered image by performing at least one of feature extraction, feature fusion, cascading residual processing, or feature decoding on the first rendered image and the reference image. (See Page 8248 Left Column Paragraph 1 and 2, wherein a second rendered image (Novel View HR image) is generated using feature extraction on the first rendered image (Flr being features extracted from the LR image/first rendered image) and the reference image (Fref being the features extracted from the HR reference image)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Huang; Xudong et al. (RefSR-NeRF: Towards High Fidelity and Super Resolution View Synthesis; hereinafter simply referred to as Huang) in view of Mendi; Engin et al. (Sports video summarization based on motion analysis; hereinafter simply referred to as Mendi).
Regarding dependent claim 3, Huang does not explicitly disclose:
Generating optical flow information of the target scene; and determining, based on the optical flow information, the reference image from among the plurality of images of the image sequence.
However, Mendi teaches of generating optical flow information of the target scene; and determining, based on the optical flow information, the reference image from among the plurality of images of the image sequence. (See Page 792 first paragraph, 793 Last two paragraphs, and Figure 3, wherein key frames (reference frames) are selected based on motion functions determined by optical flow information of a target scene).
As taught by Mendi the determination of key/reference frames using optical flow allows for the use of smaller image sizes. (See Page 792 first paragraph wherein smaller image sizes can be used as input due to the key/reference frames being chosen based on motion function determined by optical flow information). As both the teachings of Huang and Mendi deal with the technical field of selecting of a key/reference image/frame from a plurality of images making up a scene for further processing/analyzing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings Huang with Mendi to teach of generating optical flow information of the target scene; and determining, based on the optical flow information, the reference image from among the plurality of images of the image sequence in order to allow for smaller images to be used as input.
Regarding dependent claim 11, claim 11 is a device claim corresponding to claim 3. Please see the discussion of claim 3 above. Furthermore, Huang teaches of an electronic device comprising at least on processor and a memory storing instructions to be carried out by the at least one processor. (See Page 8248 Right Column first paragraph wherein the implementation of the reference based super resolution model is implemented using a device comprising at least one processor (Nvidia GPU) and a memory (32GB memory) storing instructions (program instructions via PyTorch) to execute the super resolution model please see the explanation provided in claim 3 for further details).
Regarding dependent claim 16, claim 16 is a method for updating an image generation model claim corresponding to claim 3. Please see the discussion of claim 3 above. Furthermore, Huang teaches of an electronic device comprising at least on processor and a memory storing instructions to be carried out by the at least one processor. (See Page 8248 Right Column first paragraph wherein the implementation of the reference based super resolution model is implemented using a device comprising at least one processor (Nvidia GPU) and a memory (32GB memory) storing instructions (program instructions via PyTorch) to execute the super resolution model, please see the explanation provided in claim 3 for further details).
Claims 8 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Huang; Xudong et al. (RefSR-NeRF: Towards High Fidelity and Super Resolution View Synthesis; hereinafter simply referred to as Huang) in view of Bigos; Andrew et al. (US 20220309745 A1; hereinafter simply referred to as Bigos).
Regarding dependent claim 8, Huang does not explicitly disclose:
A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.
However, Bigos teaches of a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1. (See ¶ 274, 275, and claim 13 wherein a non-transitory machine readable medium such as a hard disk is used to implement an image rendering method similar to that of claim 1 wherein instructions are stored in the medium and executed by a processor ‘20’ in figure 1. Furthermore, Please see the discussion of claim 1 above).
As taught by Bigos the use of a non-transitory computer-readable storage medium allows for the storing of instructions that can be executable by a processor perform an image rendering method on multiple different devices such a PlayStation 5. (See ¶ 275 – 277 wherein the use of a non-transitory storage medium allows for the rendering method to be carried out on multiple different types of devices such as game consoles like the PlayStation 5). As both the teachings of Huang and Bigos deal with the technical field of image processing regarding image rendering, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Huang with Bigos to teach of a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 in order for the method to be carried out on multiple different type of devices such as a PlayStation 5.
Regarding dependent claim 20, Huang does not explicitly disclose:
A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 16.
However, Bigos teaches of a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 16. (See ¶ 274, 275, and claim 13 wherein a non-transitory machine readable medium such as a hard disk is used to implement an image rendering method similar to that of claim 16 wherein instructions are stored in the medium and executed by a processor ‘20’ in figure 1. Furthermore, Please see the discussion of claim 16 above).
As taught by Bigos the use of a non-transitory computer-readable storage medium allows for the storing of instructions that can be executable by a processor perform an image rendering method on multiple different devices such a PlayStation 5. (See ¶ 275 – 277 wherein the use of a non-transitory storage medium allows for the rendering method to be carried out on multiple different types of devices such as game consoles like the PlayStation 5). As both the teachings of Huang and Bigos deal with the technical field of image processing regarding image rendering, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Huang with Bigos to teach of a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 16 in order for the method to be carried out on multiple different type of devices such as a PlayStation 5.
Allowable Subject Matter
Claims 4, 5, 12, and 13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indications of allowable subject matter:
Regarding clams 4 and 12, the reason of allowable subject matter is that the prior art fails to teach or reasonably suggest the limitations of claims 3 and 11 respectively, further comprising determining a first previous image from among one or more previous images of the first rendered image, based on forward optical flow information of the optical flow information; determining a first following image from among one or more following images of the first rendered image, based on backward optical flow information of the optical flow information; and determining at least one of the first previous image or the first following image as the reference image.
Prior Art Made of Record
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure and is as follows:
U.S. Patent Application No. US 20190045168 A1 (Chaudhuri) discloses the processing of multi view images using up sampling and optical flow in a process of performing super resolution for a scene. (Chaudhuri [0026])
Furthermore please See attached PTO-892.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEJANDRO HERNANDEZ whose telephone number is (703)756-1876. The examiner can normally be reached M-F 8 am - 5 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEJANDRO HERNANDEZ/Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights