DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made of Applicant’s claim of priority from Foreign Application No. KR10-2022-0096814 , filed August 3, 2022.
Status of Claims
Claims 1-20 are pending.
Response to Arguments
Applicant's arguments filed April 29, 2026 with respect to the 35 USC 103 rejections have been fully considered but are moot because of the new grounds of rejection presented in the sections below.
Applicant argues that the Munkberg and Ma references do not teach “estimating a current filter kernel by warping the previous filter kernel using the change-data”. Examiner respectfully disagrees. In the rejections below, Munkberg teaches warping the external state from time t-1 to align the external state from t-1 to the input data (i.e., rendered image frame) at time t (see Munkberg, Para. [0028]). Thus, Munkberg teaches warping based on change-data, but does not explicitly teach that the warping occurs on a previous filter kernel to estimate a current filter kernel. However, in an analogous field of endeavor, Ma teaches a new kernel “warps” the original kernel using data (see Ma, Section 3). The Ma references is only relied upon to teach a new kernel estimated by warping the previous filter kernel, and the specific warping process of Ma is not relied upon. Ma is combined with the Munkberg reference to teach the warping using change-data of Munkberg using a previous kernel to estimate a current filter kernel.
Additionally, Applicant argues that the Munkberg and Ma references do not teach the newly amended limitation. However, as described below, the newly proposed Pan reference is relied upon to teach the limitation “wherein the warping is configured to align a portion of the image corresponding to the difference between the rendered images” (see Pan, Para. [0041]). Therefore, the 35 USC 103 rejection of the claims is upheld, and consequently, THIS ACTION IS FINAL.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4 and 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over Munkberg et al. (US 2020/0126191 A1) in view of Ma et al. (“Learning Invariant Representations with Kernel Warping”) further in view of Pan et al. (US 2022/0284222 A1, filed March 4, 2021).
Regarding claim 1, Munkberg teaches an image reconstruction method comprising:
determining an image warping result by warping a previous reconstruction result using change-data comprising a difference between rendered images, wherein the change-data comprises a current change-data comprising a difference between a current rendered image and a previous rendered image (Munkberg, Para. [0028], the temporal warp function warps the external state based on the per-datum differences to produce the warped external state for time t-1. The warping aligns the external state from time t-1 to the input data at time t. Para. [0035], in an embodiment, the input data comprises rendered image frames);
determining a previous filter kernel by executing a first neural network model with a previous rendered image and the image warping result (Munkberg, Para. [0040], The warped external state and the processed second input data frame are processed by the encoder/decoder neural network model (i.e., first neural network model) to generated spatially-varying filter kernels. When the sequence of input data is an image sequence, the combiner function applies a first filter kernel to pixels of the reconstructed first data frame (from time t-1) and applies a second filter kernel to pixels of the processed second input data frame (from time t));
determining a current reconstruction result by executing a second neural network model with a current rendered image, the current filter kernel, and the image warping result (Munkberg, Para. [0031], by applying the warped external recurrent neural network (i.e., second neural network model) over an image sequence, one image at a time, the neural network model outputs a sequence of temporally-stable reconstructed images, one image at a time. Para. [0037], The second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts),
Although Munkberg teaches a previous filter kernel from time t-1 and a second filter kernel for time t (Munkberg, Para. [0040]), Munkberg does not explicitly teach “estimating a current filter kernel by warping the previous filter kernel using the current change-data”. However, in an analogous field of endeavor, Ma teaches performing data-dependent kernel warping for modeling invariance, wherein the new kernel “warps” the original kernel by accounting for the invariances (Ma, Section 3).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Munkberg with the teachings of Ma by including performing warping on the previous filter kernel to create the current filter kernel. One having ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to combine these references because doing so would allow for a learning algorithm for invariances that retains computational and spatial efficiency, as recognized by Ma.
Although Munkberg in view of Ma teaches warping based on differences (Munkberg, Para. [0028]), they do not explicitly teach “wherein the warping is configured to align a portion of the image corresponding to the difference between the rendered images”. However, in an analogous field of endeavor, Pan teaches the classification module 240 receives two consecutive images, X.sub.t and X.sub.t+1, and their associated semantic keypoint data (e.g., semantic keypoints 231a, 232a, 231b, and 232b). The semantic keypoint data indicates locations of rear-side light signals in the images. At 241 the classification module 240 aligns the images (e.g., based on the semantic keypoint positions in each image) and adjusts at least one of the images to fit the other image and thereby compensate for alignment differences. In one or more embodiments, the classification module 240 adjusts the at least one image by warping the at least one image to fit the other image (Pan, Para. [0041]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Munkberg in view of Ma with the teachings of Pan by including that the warping aligns a portion of the image (i.e., semantic keypoint positions) corresponding to the alignment differences between the images. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for aligning images based on differences, as recognized by Pan. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 2, Munkberg in view of Ma further in view of Pan teaches the image reconstruction method of claim 1, and further teaches wherein the determining of the image warping result comprises:
determining a first image warping result by warping a second previous reconstruction result reconstructed from a second previous rendered image using first change-data comprising a difference between a first previous rendered image and the second previous rendered image (Munkberg, Para. [0037], the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state); and
determining a current image warping result by warping a first previous reconstruction result reconstructed from the first previous rendered image using the current change-data corresponding to a difference between the current rendered image and the first previous rendered image (Munkberg, Para. [0037], the external state is warped by the temporal warp function 115, using difference data corresponding to changes between the first input data frame and the second input data frame (e.g., optical flow, motion vectors, or the like), to produce warped external state. Warping the external state anchors individual characteristics or features to regions within the data frames. The warped external state enables improved tracking over time by integrating information associated with changing features over multiple frames in a sequence, producing more temporally stable and higher quality reconstructed data),
wherein the first previous rendered image corresponds to a previous frame of the current rendered image and the second previous rendered image corresponds to a previous frame of the first previous rendered image (Munkberg, Para. [0035], a sequence of input data including artifacts is received by the warped external recurrent neural network. The sequence includes a first input data frame and a second input data frame. Para. [0031], by applying the warped external recurrent neural network (i.e., second neural network model) over an image sequence, one image at a time, the neural network model outputs a sequence of temporally-stable reconstructed images, one image at a time. The external state carries information about one or more previous images).
Examiner interprets the current rendered image, first previous rendered image, and second previous rendered image to mean rendered image frames at time t, t-1, and t-2. Because the Munkberg reference teaches the warped external recurrent neural network over an image sequence one image at a time, Examiner submits that the warping is performed multiple times between the first and second previous rendered images and then between the first previous rendered image and current rendered image. If applicant would like the claim to be interpreted differently, further clarification through amendment is required.
Regarding claim 3, Munkberg in view of Ma further in view of Pan teaches the image reconstruction method of claim 2, and further teaches wherein the determining of the previous filter kernel comprises executing the first neural network model with the previous rendered image and the first image warping result (Munkberg, Para. [0040], the warped external state and the processed second input data frame are processed by the encoder/decoder neural network model 110 to generate spatially-varying filter kernels. When the sequence of input data is an image sequence, the combiner function 120 applies a first filter kernel to pixels of the reconstructed first data frame (from time t−1)).
Regarding claim 4, Munkberg in view of Ma further in view of Pan teaches the image reconstruction method of claim 2, and further teaches wherein the determining of the current reconstruction result comprises executing the second neural network model with the current rendered image, the current filter kernel, and the current image warping result (Munkberg, Para. [0031], by applying the warped external recurrent neural network (i.e., second neural network model) over an image sequence, one image at a time, the neural network model outputs a sequence of temporally-stable reconstructed images, one image at a time. Para. [0037], The second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts).
Regarding claim 12, Munkberg in view of Ma further in view of Pan teaches the image reconstruction method of claim 1, and further teaches wherein the first neural network model comprises an auto-encoder model comprising an encoding block and a decoding block (Munkberg, Paras. [0049]-[0050], a convolutional autoencoder may be used as a starting point to develop the encoder/decoder neural network model. The autoencoder includes a first encoder block and a succession of decoder stages).
Regarding claim 13, Munkberg in view of Ma further in view of Pan teaches the image reconstruction method of claim 12, wherein the decoding block comprises a convolutional recurrent layer configured to determine a current feature based on a previous feature (Munkberg, Para. [0055], Instead of receiving hidden state generated by the first convolutional layer during processing of a previous frame, the first convolutional layer receives the warped external state generated by the encoder/decoder neural network model during processing of a previous frame. The encoder/decoder neural network model functions as a single recurrent layer).
Regarding claim 14, Munkberg in view of Man further in view of Pan teaches the image reconstruction method of claim 13, wherein the convolutional recurrent layer warps the previous feature using the change-data and determines the current feature based on a warping result (Ma, Section 3, performing data-dependent kernel warping for modeling invariance, wherein the new kernel “warps” the original kernel by accounting for the invariances).
The proposed combination as well as the motivation for combining the Munkberg, Ma and Pan references presented in the rejection of Claim 1, apply to Claim 14 and are incorporated herein by reference. Thus, the method recited in Claim 14 is met by Munkberg in view of Ma further in view of Pan.
Claim 15 recites a computer-readable storage medium storing a program with instructions corresponding to the steps recited in Claim 1. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Munkberg, Ma and Pan references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Munkberg, Ma and Pan references discloses a computer readable storage medium (Munkberg, Para. [0153], the memory, the storage, and/or any other storage are possible examples of computer-readable media).
Claims 5-8, 16-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Munkberg et al. (US 2020/0126191 A1) in view of Ma et al. (“Learning Invariant Representations with Kernel Warping”) further in view of Pan et al. (US 2022/0284222 A1, filed March 4, 2021)., as applied to claims 1-4 and 12-15 above, and further in view of Chen et al. (US 2021/0133434 A1).
Regarding claim 5, Munkberg in view of Ma further in view of Pan teaches the image reconstruction method of claim 1, as described above.
Although Munkberg in view of Ma further in view of Pan teaches the method may be executed by a GPU, CPU, or any processor capable of implementing the neural network (Munkberg, Para. [0034]), they do not explicitly teach “the determining of the previous filter kernel by executing the first neural network model is performed by a first processing unit” and “the determining of the current reconstruction result by executing the second neural network model is performed by a second processing unit”. However, in an analogous field of endeavor, Chen teaches a first neural network run by the first neural network processor and the second neural network run by the second neural network processor (Chen, Para. [0108]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Munkberg in view of Ma further in view of Pan with the teachings of Chen by including executing the first neural network for determining the previous filter kernel on a first processing unit and executing the second neural network for de4termining the current reconstruction result using a second processing unit. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for performing execution of two neural network models using separate processing units, as recognized by Chen. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 6, Munkberg in view of Ma further in view of Pan and Chen teaches the image reconstruction method of claim 5, and further teaches wherein the determining of the image warping result and the estimating of the current filter kernel are further performed by the first processing unit (Munkberg, Para. [0034], the method may be executed by a GPU, CPU, or any processor capable of implementing the warped external recurrent neural network).
Regarding claim 7, Munkberg in view of Ma further in view of Pan and Chen teaches the image reconstruction method of claim 5, and further teaches wherein the first processing unit is configured to determine the previous filter kernel by executing the first neural network model independent of whether the current rendered image is generated (Munkberg, Para. [0040], the combiner function applies a first filter kernel (i.e., previous filter kernel) to pixels of the reconstructed first data frame (from time t-1) (i.e., independent of the current rendered image)).
Regarding claim 8, Munkberg in view of Ma further in view of Pan and Chen teaches the image reconstruction method of claim 5, and further teaches wherein
the previous rendered image comprises a first previous rendered image corresponding to a previous frame of the current rendered image or a second previous rendered image corresponding to a previous frame of the first previous rendered image (Munkberg, Para. [0077], the sequence includes a previous rendered image frame and the rendered image frame),
the previous reconstruction result comprises a first previous reconstruction result reconstructed from the first previous rendered image or a second previous reconstruction result reconstructed from the second previous rendered image (Munkberg, Para. [0078], a reconstructed first rendered image frame that approximates the first rendered image frame without artifacts), and
the first processing unit is configured to determine the previous filter kernel by executing the first neural network model based on the previous rendered image and the second previous reconstruction result, independent of whether the current rendered image and the first previous reconstruction result are generated (Munkberg, Para. [0040], the combiner function applies a first filter kernel (i.e., previous filter kernel) to pixels of the reconstructed first data frame (i.e., second previous reconstruction result) (i.e., independent of the current rendered image and the first previous reconstruction result)).
Regarding claim 16, Munkberg teaches an image processing apparatus comprising:
a first image processing unit configured to warp a previous reconstruction result using change-data comprising a difference between rendered images, and configured to determine an image warping result, wherein the change-data comprises a current change-data comprising a difference between a current rendered image and a previous rendered image (Munkberg, Para. [0028], the temporal warp function warps the eternal state based on the per-datum differences to produce the warped external state for time t-1. The warping aligns the external stated from time t-1 to the input data at time t. Para. [0035], in an embodiment, the input data comprises rendered image frames); and
a second processing unit configured to determine a previous filter kernel by executing a first neural network model with the previous rendered image and the image warping result (Munkberg, Para. [0040], The warped external state and the processed second input data frame are processed by the encoder/decoder neural network model (i.e., first neural network model) to generated spatially-varying filter kernels. When the sequence of input data is an image sequence, the combiner function applies a first filter kernel to pixels of the reconstructed first data frame (from time t-1) and applies a second filter kernel to pixels of the processed second input data frame (from time t)),
wherein the first processing unit is configured to determine a current reconstruction result by executing a second neural network model with a current rendered image, the current filter kernel, and the image warping result (Munkberg, Para. [0031], by applying the warped external recurrent neural network (i.e., second neural network model) over an image sequence, one image at a time, the neural network model outputs a sequence of temporally-stable reconstructed images, one image at a time. Para. [0037], The second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second input data frame without artifacts).
Although Munkberg teaches a previous filter kernel from time t-1 and a second filter kernel for time t (Munkberg, Para. [0040]), Munkberg does not explicitly teach the processing unit is “configured to estimate a current filter kernel by warping the previous filter kernel using the current change-data”. However, in an analogous field of endeavor, Ma teaches performing data-dependent kernel warping for modeling invariance, wherein the new kernel “warps” the original kernel by accounting for the invariances (Ma, Section 3).
The proposed combination as well as the motivation for combining the Munkberg and Ma references presented in the rejection of Claim 1, apply to Claim 16 and are incorporated herein by reference.
Although Munkberg in view of Ma teaches warping based on differences (Munkberg, Para. [0028]), they do not explicitly teach “wherein the warping is configured to align a portion of the image corresponding to the difference between the rendered images”. However, in an analogous field of endeavor, Pan teaches the classification module 240 receives two consecutive images, X.sub.t and X.sub.t+1, and their associated semantic keypoint data (e.g., semantic keypoints 231a, 232a, 231b, and 232b). The semantic keypoint data indicates locations of rear-side light signals in the images. At 241 the classification module 240 aligns the images (e.g., based on the semantic keypoint positions in each image) and adjusts at least one of the images to fit the other image and thereby compensate for alignment differences. In one or more embodiments, the classification module 240 adjusts the at least one image by warping the at least one image to fit the other image (Pan, Para. [0041]).
The proposed combination as well as the motivation for combining the Munkberg, Ma and Pan references presented in the rejection of Claim 1, apply to Claim 16 and are incorporated herein by reference.
Although Munkberg in view of Ma further in view of Pan teaches the method may be executed by a GPU, CPU, or any processor capable of implementing the neural network (Munkberg, Para. [0034]), they do not explicitly teach “the determining of the previous filter kernel by executing the first neural network model is performed by a first processing unit” and “the determining of the current reconstruction result by executing the second neural network model is performed by a second processing unit”. However, in an analogous field of endeavor, Chen teaches a first neural network run by the first neural network processor and the second neural network run by the second neural network processor (Chen, Para. [0108]).
The proposed combination as well as the motivation for combining the Munkberg, Ma, Pan and Chen references presented in the rejection of Claim 5, apply to Claim 16 and are incorporated herein by reference. Thus, the system recited in Claim 16 is met by Munkberg in view of Ma further in view of Pan and Chen.
Regarding claim 17, Munkberg in view of Ma further in view of Pan and Chen teaches the image processing apparatus of claim 16, and further teaches wherein
the previous rendered image comprises a first previous rendered image corresponding to a previous frame of the current rendered image or a second previous rendered image corresponding to a previous frame of the first previous rendered image (Munkberg, Para. [0077], the sequence includes a previous rendered image frame and the rendered image frame),
the previous reconstruction result comprises a first previous reconstruction result reconstructed from the first previous rendered image or a second previous reconstruction result reconstructed from the second previous rendered image (Munkberg, Para. [0078], a reconstructed first rendered image frame that approximates the first rendered image frame without artifacts), and
the first processing unit is configured to determine the previous filter kernel by executing the first neural network model based on the previous rendered image and the second previous reconstruction result, independent of whether the current rendered image and the first previous reconstruction result are generated Munkberg, Para. [0040], the combiner function applies a first filter kernel (i.e., previous filter kernel) to pixels of the reconstructed first data frame (i.e., second previous reconstruction result) (i.e., independent of the current rendered image and the first previous reconstruction result)).
Regarding claim 19, Munkberg in view of Ma further in view of Pan and Chen teaches the image processing apparatus of claim 16, and further teaches wherein
the first neural network model corresponds to an auto-encoder model comprising an encoding block and a decoding block (Munkberg, Paras. [0049]-[0050], a convolutional autoencoder may be used as a starting point to develop the encoder/decoder neural network model. The autoencoder includes a first encoder block and a succession of decoder stages), and
the decoding block comprises a convolutional recurrent layer configured to determine a current feature based on a previous feature (Munkberg, Para. [0055], Instead of receiving hidden state generated by the first convolutional layer during processing of a previous frame, the first convolutional layer receives the warped external state generated by the encoder/decoder neural network model during processing of a previous frame. The encoder/decoder neural network model functions as a single recurrent layer).
Regarding claim 20, Munkberg in view of Ma further in view of Pan and Chen teaches the image processing apparatus of claim 19, and further teaches wherein the convolutional recurrent layer is configured to warp the previous feature using the change-data, and is configured to determine the current feature based on a warping result (Ma, Section 3, performing data-dependent kernel warping for modeling invariance, wherein the new kernel “warps” the original kernel by accounting for the invariances).
The proposed combination as well as the motivation for combining the Munkberg, Ma, Pan and Chen references presented in the rejection of Claim 5, apply to Claim 20 and are incorporated herein by reference. Thus, the system recited in Claim 20 is met by Munkberg in view of Ma further in view of Pan and Chen.
Allowable Subject Matter
Claims 9-11 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
For Examiner’s statement of reasons for allowance, see the Non-Final Office Action mailed May 9, 2025.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Emma Rose Goebel whose telephone number is (703)756-5582. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Emma Rose Goebel/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights