Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/21/2024 and 03/20/2025 are being considered by the examiner.
Drawings
The drawings are objected to because Fig. 3A and 3B show two outputs of S303, S304 or S305. However, no “YES” or “NO” indicator in Fig. 3A shows which step corresponds to the result of S303. Applicant is advised to amend Fig. 3A and 3B to reflect the described steps in specification paragraph [0054]. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The disclosure is objected to because of the following informalities: paragraph [0003] recites “for a move.” Applicant is advised to amend the line to “for a movie.”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
In Claim 1, the claim recites “inputting: the first image data that is outputted from the first processing unit at the first time or was outputted therefrom at a second time that is earlier than the first time, and the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time.” This limitation includes a case where the inputs to the neural network are first image data that is outputted from the first processing unit at a second time that is earlier than the first time and second image data that was generated for the first image data that was outputted at the second time, this second time being earlier than a first time. These inputs correspond to two pieces of image data, first and second, corresponding to the same time, a second time which is earlier than the first time. However, the specification does not describe an instance of either of the two individual neural networks taking two inputs of the same time. See Fig. 4, Fig. 8, Fig. 9 and Fig. 12, where all neural networks 111, 112, and 1120 each take exactly one input from each time t = 0, 1, 2, etc. That is, only the limitation of the first optional limitation, “the first image data that is outputted from the first processing unit at the first time” is supported by the specification, and the alternative limitation “or was outputted therefrom at a second time that is earlier than the first time” is not supported by the specification.
Dependent Claim 11 contains the all subject matter of Claim 1, and also contains the limitation “the second image data that was generated by the second image processing unit at the second time to be inputted into the neural network at the first time.” Specification paragraph [0170] and Fig. 10 discloses “a quality-enhanced image 1000 of the second network unit 112 corresponding to the preceding time, which is used as an input, is merged along the channel axis by skip concatenation 1002 with a quality-enhanced image that is the result of image processing performed by the second network unit 112 at the current time.” The specification does not sufficiently clarify the meaning of the additional term “to be inputted into the neural network at the first time” in Claim 11 such that one of ordinary skill in the art would enable one of ordinary skill in the art to make and/or use the invention.
Independent Claim 14 also contains the limitation “inputting: the first image data that is outputted from the first image processing unit at the first time or was outputted therefrom at the second time, and the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time” and the specification does not provide any additional description of the training apparatus of Claim 14 such as to enable one skilled in the art to make and/or use the invention, thus Claim 14 is also rejected under 35 U.S.C. 112(a).
Claim 19 recites a non-transitory computer-readable storage medium with elements corresponding to the apparatus of Claim 1 and thus is also rejected under 35 U.S.C. 112(a).
Claim 20 recites a non-transitory computer-readable storage medium with elements corresponding to the apparatus of Claim 14 and thus is also rejected under 35 U.S.C. 112(a).
Dependent Claims 2-13 and 15-18 contain all subject matter of their respective independent claims and do not remedy the omitted elements of Claims 1 and 14 and thus are also rejected under 35 U.S.C. 112(a).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential elements, such omission amounting to a gap between the elements. See MPEP § 2172.01. The omitted elements are:
In Claim 1, the claim recites, relating to a first image processing unit, “inputting each of a series of image data that are successive in a time series into a neural network configured to perform predetermined image processing.” Claim 1 also recites, relating to a second image processing unit, “for the first image data outputted from the first image processing unit one after another in a time series, based on the neural network, generate second image data” and “as an input into the neural network at the first time, inputting: the first image data that is outputted from the first image processing unit at the first time.” As best understood in light of the specification, first and second image processing units each contain a neural network, that is two individual neural networks (Specification, [0036]-[0037]). However, recitations of “the neural network” in Claim 1 are sufficiently unclear as to amount to an omission of an essential element of a second (separate) neural network. That is, it is unclear if the first and second image processing units share a singular neural network or if each image processing unit includes a neural network, as recited in the specification.
In Claim 14, the claim recites, relating to a first image processing unit, “inputting each of the series of manipulated image data into a neural network configured to perform second image processing.” Claim 14 also recites, relating to a second image processing unit, “for the first image data outputted from the first image processing unit one after another in a time series, based on the neural network, generate second image data.” Claim 14 also recites, relating to a training unit, “perform a training of the neural network for each of the first image processing unit and the second image processing unit” and “wherein the second image processing unit is configured to generate the second image data corresponding to the first time by, as an input into the neural network at the first time, inputting…” As best understood in light of the specification, first and second image processing units each contain a neural network, that is two individual neural networks (Specification, [0036]-[0037]). Claim 14 recites training “the neural network for each” of first and second image processing units, however this and other recitations of “the neural network” are sufficiently unclear as to amount to an omission of an essential element of a second (separate) neural network. That is, it is unclear if the first and second image processing units share a singular neural network or if each image processing unit includes a neural network, as recited in the specification.
Claim 19 recites a non-transitory computer-readable storage medium with elements corresponding to the apparatus of Claim 1 and thus is also rejected under 35 U.S.C. 112(b).
Claim 20 recites a non-transitory computer-readable storage medium with elements corresponding to the apparatus of Claim 14 and thus is also rejected under 35 U.S.C. 112(b).
Dependent Claims 2-13 and 15-18 contain all subject matter of their respective independent claims and do not remedy the omitted elements of Claims 1 and 14 and thus are also rejected under 35 U.S.C. 112(b).
Appropriate correction of all rejected claims is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 4-10, 12, and 19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Duan et al. (US 2023/0177652 A1).
Regarding Claim 1, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “An information processing apparatus, comprising:
at least one memory storing instructions; and
at least one processor that, upon execution of the stored instructions” (Duan, [0171] discloses “an embodiment of the present disclosure provides an electronic device for image restoration, including: a memory 2 and a processor 3; where the memory 2 is configured to store a program; the processor 3 is configured to execute the program in the memory 2”), “causes the information processing apparatus to function as:
a first image processing unit” (Duan, [0137] discloses “a first processing unit 200”) “configured to, by inputting each of a series of image data that are successive in a time series into a neural network configured to perform predetermined image processing on the inputted image data one after another” (Duan, [0179] discloses “That is, compression noise of any frame of image to be processed in the video to be processed needs to be removed according to the combination of the current frame of image to be processed and the previous frame image, so that the compression noise of any frame of image in the video to be processed is removed, and the display quality is improved. In addition, because the relation between the previous frame image and later frame image is used in the whole process of removing compression noises, a motion compensation between frames can be realized, and thus the video quality is improved”; where a video is a series of image data that are successive in time series; where inputting frames of images of video to the disclosed image restoration method and apparatus to improve video quality is inputting each of a series of image data into a neural network configured to perform predetermined image processing on the inputted image data one after another), “generate first image data that is image data after the predetermined image processing” (Duan, [0137] discloses “a first processing unit 200, configured to remove, via single-frame network, compression noise of the image to be processed to output a first image”; where removing compression noise is performing predetermined image processing; where a first image is first image data); “and
a second image processing unit configured to, for the first image data outputted from the first image processing unit one after another in a time series, based on the neural network, generate second image data that is image data after the predetermined image processing” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image, compression noise of the image to be processed via the recursive network to output a second image, where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a second image is second image data),
“wherein the second image processing unit is configured to generate the second image data corresponding to a first time by, as an input into the neural network at the first time, inputting:
the first image data that is outputted from the first image processing unit at the first time or was outputted therefrom at a second time that is earlier than the first time” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image”; see Fig. 11, where output of first processing unit is input to second processing unit), “and
the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time” (Duan, [0138] discloses “where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a previous frame image is first image data that was outputted at the second time).
PNG
media_image1.png
474
816
media_image1.png
Greyscale
Fig. 11 of Duan
Regarding Claim 4, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 1,
wherein the first image processing unit is configured to, by inputting one or more pieces of image data that represent an image or a feature amount generated from the image into the neural network, generate, as the first image data, one or more pieces of image data that represent the image or the feature amount to which the predetermined image processing has been applied” (Duan, [0137] discloses “a first processing unit 200, configured to remove, via single-frame network, compression noise of the image to be processed to output a first image”; where a frame is one or more pieces of image data; where outputting a first image is generating one of more pieces of image data that represent the image; see also Duan, [0152]-[0155], which disclose “feature images of the image to be processed”; thus, Duan teaches both inputting one or more pieces of image data that represent an image and one or more pieces of image data that represent a feature amount).
Regarding Claim 5, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 4, wherein the second image processing unit is configured to generate the second image data corresponding to the first time by, as the input into the neural network at the first time, inputting: one or more pieces of image data that represent an image or a feature amount generated from the image and are outputted as the first image data from the first image processing unit at the first time, and the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image”; see Fig. 11, where output of first processing unit is input to second processing unit. Duan, [0138] discloses “where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a previous frame image is first image data that was outputted at the second time).
Regarding Claim 6, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 1, wherein the first image processing unit is configured to, by inputting a series of image data that represent a plurality of images that are successive in a time series or a plurality of feature amounts generated from the plurality of images respectively into the neural network, generate, as the first image data, a series of image data that represent the plurality of images or the plurality of feature amounts to which the predetermined image processing has been applied” (Duan, [0179] discloses “According to the method, any frame of image to be processed in the video to be processed is input into the target denoising network composed of the single-frame network and the recursive network, compression noise of the image to be processed is removed via the single-frame network, and the first image is output”; where denoised first image is image data that represents the plurality of images to which predetermined image processing (denoising) has been performed. Duan is directed to improving video quality particular between previous and later image frames, thus teaches inputting a series of image data that represent a plurality of images successive in a time series, as stated above).
Regarding Claim 7, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 6, wherein the second image processing unit is configured to generate the second image data corresponding to the first time by, as the input into the neural network at the first time, inputting:
a series of image data that represent a plurality of images that are successive in a time series or a plurality of feature amounts generated from the plurality of images and are outputted as the first image data from the first image processing unit at the first time” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image”; see Fig. 11, where output of first processing unit is input to second processing unit), “and the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time” (Duan, [0138] discloses “where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a previous frame image is first image data that was outputted at the second time).
Regarding Claim 8, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 1, wherein the second image processing unit is configured to use, as the second image data that is inputted into the neural network at the first time, the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time” (Duan, Fig. 5 and [0077] discloses “In the embodiment of the present disclosure, as shown in FIG. 5, which is a schematic structural diagram of the target denoising network 1, specifically, when there are two first down-sampling layers 1031, two up-sampling layers included in the first up-sampling layer 1032, one second sub-convolution layer 1012, one first sub-feature series layer 1021, two second down-sampling layers 2021, two second up-sampling layers 2022, one fourth sub-convolution layer 2014 and two second feature series layers 203. The number of filters of each convolution layer in the network is shown as the number above a horizontal line in FIG. 5, such as 64 and 128”; see multiple connections between single-frame network 20 and recursive network 10 (second image processing unit); see also recursive training architecture).
PNG
media_image2.png
400
529
media_image2.png
Greyscale
Fig. 5 of Duan
Regarding Claim 9, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 1, wherein the second image processing unit is configured to use, as the first image data that is inputted into the neural network at the first time, the first image data that is outputted from the first image processing unit at the first time” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image, compression noise of the image to be processed via the recursive network to output a second image, where the previous frame image is one previous frame of the image to be processed in the video to be processed”; see Fig. 11, where output of first processing unit is input to second processing unit).
Regarding Claim 10, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 1, wherein the second image processing unit is configured to use, as the first image data that is inputted into the neural network at the first time, the first image data that was outputted from the first image processing unit at the second time” (Duan, [0174] discloses “removing, according to a content of a previous frame image, compression noise of the image to be processed via the recursive network to output a second image, where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a previous frame image is a first image data outputted at the second time).
Regarding Claim 12, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 1, wherein the predetermined image processing is processing of removing noise manifested in an image to be processed” (Duan, [0137] discloses “a first processing unit 200, configured to remove, via single-frame network, compression noise of the image to be processed to output a first image”; where removing compression noise is performing predetermined image processing; where a first image is first image data).
Regarding Claim 19, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “A non-transitory computer-readable storage medium storing a program for causing a computer to execute a method” (Duan, [0171] discloses “an embodiment of the present disclosure provides an electronic device for image restoration, including: a memory 2 and a processor 3; where the memory 2 is configured to store a program; the processor 3 is configured to execute the program in the memory 2”) “comprising:
generating, by inputting each of a series of image data that are successive in a time series into a neural network configured to perform predetermined image processing on the inputted image data one after another” (Duan, [0179] discloses “That is, compression noise of any frame of image to be processed in the video to be processed needs to be removed according to the combination of the current frame of image to be processed and the previous frame image, so that the compression noise of any frame of image in the video to be processed is removed, and the display quality is improved. In addition, because the relation between the previous frame image and later frame image is used in the whole process of removing compression noises, a motion compensation between frames can be realized, and thus the video quality is improved”; where a video is a series of image data that are successive in time series; where inputting frames of images of video to the disclosed image restoration method and apparatus to improve video quality is inputting each of a series of image data into a neural network configured to perform predetermined image processing on the inputted image data one after another), “first image data that is image data after the predetermined image processing” (Duan, [0137] discloses “a first processing unit 200, configured to remove, via single-frame network, compression noise of the image to be processed to output a first image”; where removing compression noise is performing predetermined image processing; where a first image is first image data); “and
generating, for the first image data outputted one after another in a time series, based on the neural network, second image data that is image data after the predetermined image processing” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image, compression noise of the image to be processed via the recursive network to output a second image, where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a second image is second image data), “wherein the second image data corresponding to a first time is generated by, as an input into the neural network at the first time, inputting:
“the first image data that is outputted at the first time or was outputted therefrom at a second time that is earlier than the first time” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image”; see Fig. 11, where output of first processing unit is input to second processing unit), “and
the second image data that was generated for the first image data that was outputted at the second time” (Duan, [0138] discloses “where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a previous frame image is first image data that was outputted at the second time).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Duan et al. (US 2023/0177652 A1) in view of Son et al. (Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes, published 2021).
Regarding Claim 13, Duan does not explicitly teach the apparatus of Claim 13.
However, in an analogous field of endeavor, as best interpreted in light of the 112(a) and 112(b) rejections above, Son teaches “The information processing apparatus according to claim 1, wherein the at least one processor causes the information processing apparatus to further function as:
an estimation unit” (Son, Section 3, paragraph 1 discloses “a blur-invariant motion estimation network (BIMNet)”) “configured to estimate noise manifested in an image represented by each of the series of image data, wherein the first image processing unit generates the first image data by using, as the input into the neural network, a result of noise estimation by the estimation unit” (Son, Section 3.1, paragraph 2 discloses “A successfully trained BIMNet should be able to produce accurate optical flow from a pair of video frames regardless of the amounts of blur that the frames include”; where blur estimation is noise estimation; see Fig. 2, where output of Blur Invariant Motion Estimation is used to generate deblurred result of the current frame, Itest).
PNG
media_image3.png
349
502
media_image3.png
Greyscale
Fig. 2 of Son
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Duan to incorporate the teachings of Son by performing a blur invariant motion estimation and using the result to deblur and input image. One of ordinary skill in the art would be motivated to combine the Duan and Son references in order to improve deblurring: Son, Section 4.2, paragraph 4 discloses “Compared to these, BIMNetFN2 that is trained using our blur invariant loss shows improvements in all cases. A similar trend can also be found from LiteFlowNetSS, LiteFlowNet∗∗, and BIMNetLFN. This result verifies that our blur-invariant loss is the source of the improvement in motion estimation accuracy in the presence of blur.” That is, it would have been obvious to one of ordinary skill in the art that incorporating the known motion estimation technique of Son to the teachings of Duan would result in predictable results of improving motion estimation accuracy and deblurring performance. Accordingly, the combination of Duan and Son discloses the invention of Claim 13.
Allowable Subject Matter
Claims 2, 3, and 11 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Regarding Claim 2, as best interpreted in light of the 112(a) and 112(b) rejections above, Duan teaches “The information processing apparatus according to claim 1, wherein the neural network is trained on a basis of an error between:
(Duan, [0121] discloses “Then, according to the first prediction deviation between the simulation denoised image and the true value image of the corresponding frame in the true value video, the first loss function for the denoising network to be trained is determined”; where deviation between simulation denoised image and true value image is a second difference between pieces of second image data; where adding noise is a manipulation (a simulation image has noise) and denoising is undoing the manipulation in predetermined image processing).
However, Duan does not explicitly teach “a first difference between, among a series of true-value image data that are successive in a time series, pieces of the true-value image data corresponding respectively to a first time and a second time that is earlier than the first time, and a second difference between pieces of the second image data generated by performing the predetermined image processing on manipulated image data generated by applying a manipulation to the pieces of the true-value image data corresponding respectively to the first time and the second time.”
Song et al. (TempFormer: Temporally Consistent Transformer for Video Denoising, published 2022) discloses several loss functions, including a loss between ground truth images, or true-value image data, and denoised frames, and a loss term between previous and following timestep denoised frames, however Song does not explicitly teach a difference calculation between ground truth images of different time steps. Thus, Song does not explicitly teach the difference calculation unit of Claim 2.
PNG
media_image4.png
321
921
media_image4.png
Greyscale
PNG
media_image5.png
344
922
media_image5.png
Greyscale
Excerpt from Song
Munkberg et al. (US 2020/126192 A1) teaches a loss comprising two loss values between a denoised frame and a reference frame at different frames i and i-1 (see excerpt below). However, Munkberg does not explicitly teach a difference between ground truth, or reference frames, to determine an error.
PNG
media_image6.png
317
446
media_image6.png
Greyscale
Excerpt of Munkberg
Finally, Son et al. (Recurrent Video Deblurring with Blur-Invariant Motion Estimation
and Pixel Volumes, published 2021) teaches a blur invariant loss function that learns optical flow between consecutive sharp frames, or ground truth images (see excerpt below). However, although Son teaches a loss function, or error, using consecutive sharp frames, Son does not explicitly teach a difference between sharp frames, rather teaching a mean squared error between a warped sharp image frame and a sharp image frame in a next time step. Even under the broadest reasonable interpretation, the teaching of Son cannot be construed to teach “a first difference between, among a series of true-value image data that are successive in a time series, pieces of the true-value image data corresponding respectively to a first time and a second time that is earlier than the first time.”
PNG
media_image7.png
310
532
media_image7.png
Greyscale
Excerpt of Son
Thus, none of the previously cited prior art references provide a motivation to teach, alone or in combination, the ordered combination of “The information processing apparatus according to claim 1, wherein the neural network is trained on a basis of an error between: a first difference between, among a series of true-value image data that are successive in a time series, pieces of the true-value image data corresponding respectively to a first time and a second time that is earlier than the first time, and a second difference between pieces of the second image data generated by performing the predetermined image processing on manipulated image data generated by applying a manipulation to the pieces of the true-value image data corresponding respectively to the first time and the second time, the manipulation being an image-restorable manipulation that is to be undone by the predetermined image processing.”
Regarding Claim 3, Claims 3 depends from Claim 2 and thus contain all allowable subject matter of Claim 2.
Regarding Claim 11¸ as best interpreted in light of the 112(a) and 112(b) rejections above, none of the previously cited prior art explicitly teaches the apparatus of Claim 11. That is, although concatenation of image data as part of a neural network architecture is known in the art (K S et al., US 2022/0398700 A1) discloses “Examples of the layers may be, but are not limited to, a convolutional layer, an activation layer, an average pool layer, a max pool layer, a concatenated layer, a dropout layer, a fully connected (FC) layer, a SoftMax layer, and so on”), none of the previously cited prior art provides a motivation to teach the ordered combination of “The information processing apparatus according to claim 1, wherein the second image processing unit is configured to, by concatenating the second image data that was generated by the second image processing unit at the second time to be inputted into the neural network at the first time with an output of the neural network at the second time and by performing convolution thereon, generate the second image data corresponding to the first time.”
Claims 14-18 and 20 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 112(a) and 35 U.S.C. 112(b), set forth in this Office action.
Regarding Claim 14, Duan teaches “A training apparatus, comprising:
at least one memory storing instructions; and
at least one processor that, upon execution of the stored instructions” (Duan, [0171] discloses “an embodiment of the present disclosure provides an electronic device for image restoration, including: a memory 2 and a processor 3; where the memory 2 is configured to store a program; the processor 3 is configured to execute the program in the memory 2”), “causes the training apparatus to function as:
a manipulated image data generation unit configured to generate a series of manipulated image data by performing first image processing on each of a series of true-value image data that are successive in a time series” (Duan, [0156]-[0158] discloses “In the embodiment of the present disclosure, before the input unit 100 inputs the image to be processed into the target denoising network, the apparatus for image restoration further includes: a training unit, configured to: obtain a plurality of groups of image frame sequences, where each group of image frame sequences includes a plurality of images; encode the plurality of groups of image frame sequences into a true value video and a simulation video respectively, where each frame of simulation image in the simulation video contains compression noise”; where simulation video is manipulated image data; where compression noise is first image processing);
“a first image processing unit” (Duan, [0137] discloses “a first processing unit 200”) “configured to, by inputting each of the series of manipulated image data into a neural network configured to perform second image processing on the inputted data one after another” (Duan, [0179] discloses “That is, compression noise of any frame of image to be processed in the video to be processed needs to be removed according to the combination of the current frame of image to be processed and the previous frame image, so that the compression noise of any frame of image in the video to be processed is removed, and the display quality is improved. In addition, because the relation between the previous frame image and later frame image is used in the whole process of removing compression noises, a motion compensation between frames can be realized, and thus the video quality is improved”; where a video is a series of image data that are successive in time series; where inputting frames of images of video to the disclosed image restoration method and apparatus to improve video quality is inputting each of a series of image data into a neural network configured to perform second image processing on the inputted image data one after another), “generate first image data that is image data after the second image processing” (Duan, [0137] discloses “a first processing unit 200, configured to remove, via single-frame network, compression noise of the image to be processed to output a first image”; where removing compression noise is performing predetermined image processing; where a first image is first image data);
“a second image processing unit configured to, for the first image data outputted from the first image processing unit one after another in a time series, based on the neural network, generate second image data that is image data after the second image processing” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image, compression noise of the image to be processed via the recursive network to output a second image, where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a second image is second image data);
a training unit configured to, based on the error, perform a training of the neural network for each of the first image processing unit and the second image processing unit” (Duan, [0156]-[0158] discloses “In the embodiment of the present disclosure, before the input unit 100 inputs the image to be processed into the target denoising network, the apparatus for image restoration further includes: a training unit, configured to: obtain a plurality of groups of image frame sequences, where each group of image frame sequences includes a plurality of images; encode the plurality of groups of image frame sequences into a true value video and a simulation video respectively, where each frame of simulation image in the simulation video contains compression noise”), “wherein
the second image processing unit is configured to generate the second image data corresponding to the first time by, as an input into the neural network at the first time, inputting:
the first image data that is outputted from the first image processing unit at the first time or was outputted therefrom at the second time” (Duan, [0138] discloses “a second processing unit 300, configured to remove, according to a content of a previous frame image”; see Fig. 11, where output of first processing unit is input to second processing unit), “and
the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time” (Duan, [0138] discloses “where the previous frame image is one previous frame of the image to be processed in the video to be processed”; where a previous frame image is first image data that was outputted at the second time).
However, Duan does not explicitly teach “a difference calculation unit configured to calculate a first difference and a second difference, the first difference being a difference between pieces of the true-value image data corresponding respectively to a first time and a second time that is earlier than the first time, the second difference being a difference between pieces of the second image data generated by performing the second image processing on the manipulated image data generated from the pieces of the true-value image data corresponding respectively to the first time and the second time; an error calculation unit configured to calculate an error between the first difference and the second difference.”
Song et al. (TempFormer: Temporally Consistent Transformer for Video Denoising, published 2022) discloses several loss functions, including a loss between ground truth images, or true-value image data, and denoised frames, and a loss term between previous and following timestep denoised frames, however Song does not explicitly teach a difference calculation between ground truth images of different time steps. Thus, Song does not explicitly teach the difference calculation unit of Claim 14.
PNG
media_image4.png
321
921
media_image4.png
Greyscale
PNG
media_image5.png
344
922
media_image5.png
Greyscale
Excerpt from Song
Munkberg et al. (US 2020/126192 A1) teaches a loss comprising two loss values between a denoised frame and a reference frame at different frames i and i-1 (see excerpt below). However, Munkberg does not explicitly teach a difference between ground truth, or reference frames, to determine an error.
PNG
media_image6.png
317
446
media_image6.png
Greyscale
Excerpt of Munkberg
Finally, Son et al. (Recurrent Video Deblurring with Blur-Invariant Motion Estimation
and Pixel Volumes, published 2021) teaches a blur invariant loss function that learns optical flow between consecutive sharp frames, or ground truth images (see excerpt below). However, although Son teaches a loss function, or error, using consecutive sharp frames, Son does not explicitly teach a difference between sharp frames, rather teaching a mean squared error between a warped sharp image frame and a sharp image frame in a next time step. Even under the broadest reasonable interpretation, the teaching of Son cannot be construed to teach “the first difference being a difference between pieces of the true-value image data corresponding respectively to a first time and a second time that is earlier than the first time.”
PNG
media_image7.png
310
532
media_image7.png
Greyscale
Excerpt of Son
Thus, none of the previously cited prior art references provide a motivation to teach, alone or in combination, the ordered combination of “A training apparatus, comprising: at least one memory storing instructions; and at least one processor that, upon execution of the stored instructions, causes the training apparatus to function as: a manipulated image data generation unit configured to generate a series of manipulated image data by performing first image processing on each of a series of true-value image data that are successive in a time series; a first image processing unit configured to, by inputting each of the series of manipulated image data into a neural network configured to perform second image processing on the inputted data one after another, generate first image data that is image data after the second image processing; a second image processing unit configured to, for the first image data outputted from the first image processing unit one after another in a time series, based on the neural network, generate second image data that is image data after the second image processing; a difference calculation unit configured to calculate a first difference and a second difference, the first difference being a difference between pieces of the true-value image data corresponding respectively to a first time and a second time that is earlier than the first time, the second difference being a difference between pieces of the second image data generated by performing the second image processing on the manipulated image data generated from the pieces of the true-value image data corresponding respectively to the first time and the second time; an error calculation unit configured to calculate an error between the first difference and the second difference; and a training unit configured to, based on the error, perform a training of the neural network for each of the first image processing unit and the second image processing unit, wherein the second image processing unit is configured to generate the second image data corresponding to the first time by, as an input into the neural network at the first time, inputting: the first image data that is outputted from the first image processing unit at the first time or was outputted therefrom at the second time, and the second image data that was generated by the second image processing unit for the first image data that was outputted from the first image processing unit at the second time.”
Regarding Claims 15-18, Claims 15-18 depend from Claim 14 and thus contain all allowable subject matter of Claim 14.
Regarding Claim 20, Claim 20 recites a non-transitory computer-readable storage medium with elements corresponding to the apparatus of Claim 14 and thus contains all allowable subject matter of Claim 14.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Maggioni et al. (Efficient Multi-Stage Video Denoising with Recurrent Spatio-Temporal Fusion, published 2021) discloses a video denoising method using a recurrent stage fusing prior and current frames.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROLINE TABANCAY DUFFY whose telephone number is (703)756-1859. The examiner can normally be reached Monday - Friday 8:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at 5712723382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROLINE TABANCAY DUFFY/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights