Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 22 August 2025 has been entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 4, 6-11, 13, and 15-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“generate, from the plurality of images, input image data and target image data”
“generate predicted output image data”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
The limitations:
“calculate a difference between the predicted output image data and the target image data”
“transform the calculated difference into frequency domain data”
“calculate a loss value using an L1 family norm of the transformed frequency domain data”
“as part of training the neural network, perform backpropagation on the neural network to update weights of the neural network based on the calculated loss value”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“A computer system for training a neural network that processes images, the computer system comprising: non-transitory computer readable storage configured to store image data for a plurality of images”
“at least one hardware processor that is coupled to the non-transitory computer readable storage, the at least one hardware processor configured to…”
“train, over a plurality of epochs, the neural network based on a defined training schedule, wherein for each of the plurality of epochs a learning rate hyperparameter is set based on the defined training schedule”
“by using the input image data as input to a neural network, wherein the input image data represents images of a first resolution and the target image data represents images of a second resolution that is greater than the first resolution”
“as part of training the neural network, executing a learning rate scheduler for the defined training schedule to adjust the learning rate hyperparameter over the training of the neural network”
“(a) increase the learning rate hyperparameter for each of a first plurality of epochs over a first portion of training the neural network”
“(b) decrease the learning rate hyperparameter for each of a second plurality of epochs across a second portion of training the neural network that occurs after the first portion”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the transformation of the calculated difference into frequency domain data is performed by using a Fourier Transform”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: See corresponding analysis of claim 1.
Step 2B Analysis: See corresponding analysis of claim 1.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“apply, as part of transformation of the calculated difference, a windowing function to the calculated difference, wherein transformation of the calculated difference into frequency domain data is further based on application of the windowing function to the calculated difference”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: See corresponding analysis of claim 1.
Step 2B Analysis: See corresponding analysis of claim 1.
Regarding Claim 6,
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein a rate of change of the learning rate hyperparameter during the first portion is greater than a rate of change during the second portion”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the loss value is a scalar value that is calculated based on (a) a sum of the frequency domain data of differences in pixel values of different pixel locations within the target and output image data, and (b) a total number of differences in pixel values”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: See corresponding analysis of claim 1.
Step 2B Analysis: See corresponding analysis of claim 1.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the neural network is implemented using separable block transforms”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 9,
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the L1 family norm is the L1 norm”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 10,
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“generating predicted output image data”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
The limitations:
“calculating a difference between the predicted output image data and target image data”
“transforming the calculated difference into frequency domain data”
“calculating a loss value using an L1 family norm of the transformed frequency domain data”
“as part of training the neural network, performing backpropagation on the neural network to update weights of the neural network based on the calculated loss value”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“training, over a plurality of epochs, the neural network based on a defined training schedule, wherein for each of the plurality of epochs a learning rate hyperparameter is set based on the defined training schedule”
“by using the input image data as input to a neural network, wherein the input image data represents images of a first resolution and the target image data represents images of a second resolution that is greater than the first resolution”
“as part of training the neural network, executing a learning rate scheduler for the defined training schedule to adjust the learning rate hyperparameter over the training of the neural network”
“(a) increasing the learning rate hyperparameter for each of a first plurality of epochs over a first portion of training the neural network”
“(b) decreasing the learning rate hyperparameter for each of a second plurality of epochs across a second portion of training the neural network that occurs after the first portion”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“storing, to non-transitory computer readable storage, image data for a plurality of images”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply” or “insignificant extra-solution activity”. Additionally, the storing limitation recites the well-understood, routine, and conventional activity of storing and retrieving information in memory. MPEP 2106.05(d)(II); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015). Mere instructions to apply an exception and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 11,
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 11 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the transformation of the calculated difference into frequency domain data is performed by using a Fourier Transform”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: See corresponding analysis of claim 10.
Step 2B Analysis: See corresponding analysis of claim 10.
Regarding Claim 13,
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 13 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“applying, as part of transforming the calculated difference, a windowing function to the calculated difference, wherein transformation of the calculated difference into frequency domain data is further based on application of the windowing function to the calculated difference”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: See corresponding analysis of claim 10.
Step 2B Analysis: See corresponding analysis of claim 10.
Regarding Claim 15,
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 15 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein a rate of change of the learning rate during the first portion is greater than a rate of change during the second portion”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 16,
Claim 16 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 16 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the loss value is a scalar value that is calculated based on (a) a sum of the frequency domain data of differences in pixel values of different pixel locations within the target and output image data, and (b) a total number of differences in pixel values”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: See corresponding analysis of claim 10.
Step 2B Analysis: See corresponding analysis of claim 10.
Regarding Claim 17,
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 17 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the neural network is implemented using separable block transforms”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 18,
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 18 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the L1 family norm is the L1 norm”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 19,
Claim 19 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 19 is directed to a method of training a neural network to process image data, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“as part of calculating the difference between the predicted output image data and target image data, calculating a difference in RGB pixel values between at least one pixel in the target image data and a corresponding pixel in the predicted output image data”
“wherein transforming the calculated difference includes performing a Fourier Transform on the two-dimensional array as part of obtaining the frequency domain data”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“storing the calculated differences to a two-dimensional array”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Additionally, the storing limitation recites the well-understood, routine, and conventional activity of storing and retrieving information in memory. MPEP 2106.05(d)(II); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 20,
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 20 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“generate, from the plurality of images, input image data and target image data”
“generate predicted output image data”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
The limitations:
“transform the target image data and output image data into, respectively, frequency domain target data and frequency domain output data”
“calculate the absolute value of each coefficient of the frequency domain target data and the frequency domain output data”
“calculate a loss value by using a difference between each respective coefficient of the absolute value of the frequency domain target data and the absolute value of the frequency domain output data”
“as part of training the neural network, perform backpropagation on the neural network to update weights of the neural network based on the calculated loss value”
As drafted, under their broadest reasonable interpretations, cover mathematical concepts, i.e., mathematical relationships, mathematical formulas or equations, and mathematical calculations. The above limitations in the context of this claim correspond to mathematical calculations and/or relationships.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“A computer system for training a neural network that processes images”
“non-transitory computer readable storage configured to store image data for a plurality of images”
“at least one hardware processor that is coupled to the non-transitory computer readable storage, the at least one hardware processor configured to”
“train, over a plurality of epochs, the neural network based on a defined training schedule, wherein for each of the plurality of epochs a learning rate hyperparameter is set based on the defined training schedule”
“by using the input image data as input to a neural network, wherein the input image data represents images of a first resolution and the target image data represents images of a second resolution that is greater than the first resolution”
“as part of training the neural network, executing a learning rate scheduler for the defined training schedule to adjust the learning rate hyperparameter over the training of the neural network”
“(a) increase the learning rate hyperparameter for each of a first plurality of epochs over a first portion of training the neural network”
“(b) decrease the learning rate hyperparameter for each of a second plurality of epochs across a second portion of training the neural network that occurs after the first portion”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are additional details that do not apply the exception in a meaningful way and “mere instructions to apply”. Additional details that do not apply the exception in a meaningful way and mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 21,
Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 21 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the learning rate hyperparameter is increased exponentially over the first portion”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 22,
Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 22 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 21.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the learning rate hyperparameter is increased from 1e-5 to 1e-3 over the first portion”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 23,
Claim 23 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 23 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 21.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the learning rate hyperparameter is increased by at least a factor of 10 from a beginning of the first portion to an end of the first portion”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 24,
Claim 24 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 24 is directed to a computer system for training a neural network that processes images, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are additional details that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein different loss functions are used for different epochs in the training of the neural network”
As drafted, are additional elements that do not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 4, 6-7, 9-11, 13, 15-16, 18-24 are rejected under 35 U.S.C. 103 as being unpatentable over Mardani Korani et al. (U.S. Patent Publication No. 2022/0101494) (“Mardani Korani”) in view of Fuoli et al. (Fourier Space Losses for Efficient Perceptual Image Super-Resolution) (“Fuoli”) in further view of Loshchilov et al. (Stochastic Gradient Descent with Warm Restarts) (“Loshchilov”).
Regarding claim 1, Mardani Korani teaches a computer system for training a neural network that processes images (Mardani Korani [0056] “FIG. 1 is a block diagram illustrating an architecture for training an untrained neural network 106 to infer or synthesize, after training, a larger textured image 112 from a smaller textured image 108, according to at least one embodiment.” Mardani Korani provides a computer system for training a neural network that processes images.), the computer system comprising: non-transitory computer readable storage configured to store image data for a plurality of images (Mardani Korani [0056] "In at least one embodiment, texture synthesis is used to generate image data sets comprising larger image sizes from smaller baseline input images 108."; [0570] " In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electric or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, cache, and queues) within transceivers of transitory signals." Mardani Korani provides non-transitory computer readable storage configured to store image data for a plurality of images.); at least one hardware processor that is coupled to the non-transitory computer readable storage the at least one hardware processor configured to (Mardani Korani [0570] “In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media having stored thereon executable instructions (or other memory to store executable instructions) that, when executed (i.e., as a result of being executed) by one or more processors of a computer system, cause computer system to perform operations” Mardani Korani provides a processor coupled to a non-transitory computer readable storage used for storing image data.): …generate, from the plurality of images, input image data and target image data (Mardani Korani [0057] “In at least one embodiment, training data 102 is input into a training framework 104 to train an untrained neural network 106 to synthesize an output 112, such as an output textured image, from an input 108, such as an input textured image. In at least one embodiment, training data 102 is one or more images used to train an untrained neural network 106 using a training framework 104. In at least one embodiment, training data 102 includes supervision or other information used to facilitate training by a training framework 104. In at least one embodiment, supervision or other information to facilitate training includes data that identifies features of an image that improve training by a training framework 104”; [0058] “In at least one embodiment, training data 102 is a set of K target textured images having variable dimensions, from which K textured images are extracted, prior to training, from target images” Mardani Korani provides training dataset 102, which is generated from a plurality of images and comprises input and target image data.); generate predicted output image data by using the input image data as input to a neural network (Mardani Korani [0056] “FIG. 1 is a block diagram illustrating an architecture for training an untrained neural network 106 to infer or synthesize, after training, a larger textured image 112 from a smaller textured image 108, according to at least one embodiment.” Mardani Korani provides using neural networks to infer a larger textured image from a smaller textured images corresponding to generate predicted output image data by using the input image data as input to a neural network.), wherein the input image data represents images of a first resolution and the target image data represents images of a second resolution that is greater than the first resolution (Mardani Korani [0057] “In at least one embodiment, training data 102 is input into a training framework 104 to train an untrained neural network 106 to synthesize an output 112, such as an output textured image, from an input 108, such as an input textured image. In at least one embodiment, training data 102 is one or more images used to train an untrained neural network 106 using a training framework 104. In at least one embodiment, training data 102 includes supervision or other information used to facilitate training by a training framework 104. In at least one embodiment, supervision or other information to facilitate training includes data that identifies features of an image that improve training by a training framework 104”; [0058] “In at least one embodiment, training data 102 is a set of K target textured images having variable dimensions, from which K textured images are extracted, prior to training, from target images…A training framework 104, in an embodiment, trains an untrained neural network 106 to learn an up-sampler that maps each small texture example to the target textured image in training data 102.” Mardani Korani provides training dataset 102, which is generated from a plurality of images and comprises input and target image data including an upscale function to target images, corresponding to the input image data represents images of a first resolution, and the target image data represents images of a second resolution that is greater than the first.); calculate a difference between the predicted output image data and the target image data (Mardani Korani [0059] “In at least one embodiment, a training framework 104 determines loss values that are backpropagated in an untrained neural network 106 in order to train said untrained neural network 106, as described below in conjunction with FIG. 5.” Mardani Korani provides determining a loss value from training corresponding to calculate a difference between the predicted output image data and the target image data.); …as part of training the neural network, perform backpropagation on the neural network to update weights of the neural network based on the calculated loss value (Mardani Korani [0059] “In at least one embodiment, a training framework 104 determines loss values that are backpropagated in an untrained neural network 106 in order to train said untrained neural network 106, as described below in conjunction with FIG. 5.” Mardani Korani provides performing backpropagation on the neural network based on loss values corresponding to perform backpropagation on the neural network to update weights of the neural network based on the calculated loss value.).
Mardani Korani fails to teach train, over a plurality of epochs, the neural network based on a defined training schedule, wherein for each of the plurality of epochs a learning rate hyperparameter is set based on the defined training schedule …transform the calculated difference into frequency domain data; calculate a loss value using an L1 family norm of the transformed frequency domain data; …as part of training the neural network, executing a learning rate scheduler for the defined training schedule to adjust the learning rate hyperparameter over the training of the neural network including: (a) increase the learning rate hyperparameter for each of a first plurality of epochs over a first portion of training the neural network; and (b) decrease the learning rate hyperparameter for each of a second plurality of epochs across a second portion of training the neural network that occurs after the first portion.
However, Fuoli also teaches calculate a difference between the predicted output image data and the target image data (Fuoli Section 3, Proposed Method “The task of image SR, is to increase the resolution of an image x ∈ R H×W×C from the LR domain X to the corresponding image y ∈ R rH×rW×C in HR domain Y with factor r.”; Section 3.3 Supervision Losses “Following the setting in [31] we calculate a VGG-loss using the pre-trained 19-layer VGG network.” Fuoli also provides calculating losses between predicted and target image data corresponding to calculating a difference between the predicted output image data and the target image data.).
Fuoli further teaches transform the calculated difference into frequency domain data (Fuoli Section 3.3, Supervision Losses “In addition to these spatial domain losses, we propose a Fourier space loss LF for supervision from the ground truth frequency spectrum during training. First, ground truth y and generated image yˆ are pre-processed with a Hann window, as described in Section 3.2. Afterwards, both images are transformed into Fourier space by applying the fast Fourier transform (FFT), where we calculate amplitude and phase of all frequency components.” Fuoli provides transforming calculated differences in images into the frequency domain using a Fast Fourier transform.); calculate a loss value using an L1 family norm of the transformed frequency domain data (Fuoli Section 3.3 Supervision Losses “The L1-norms of amplitude difference LF,|·| and phase angle difference LF,∠ between output image and target are averaged to produce the total frequency loss LF .” Fuoli provides calculating a loss value using an L1 family norm of the transformed frequency domain data.).
Mardani Korani and Fuoli are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically Fourier transform based image manipulation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Further, Loshchilov teaches train, over a plurality of epochs, the neural network based on a defined training schedule, wherein for each of the plurality of epochs a learning rate hyperparameter is set based on the defined training schedule (Loshchilov Figure 1 “Learning rate schedule”; Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs. Our SGDR simulates warm restarts by scheduling the learning rate to achieve competitive results on CIFAR-10 and CIFAR-100 roughly two to four times faster” Loshchilov provides training a neural network over a plurality of epochs based on a defined learning rate schedule, as shown in Figure 1, which shows the change in learning rate over consecutive epochs of training); …as part of training the neural network, executing a learning rate scheduler for the defined training schedule to adjust the learning rate hyperparameter over the training of the neural network (Loshchilov Figure 1 “Learning rate schedule”; Section 1 Introduction “In this paper, we propose to periodically simulate warm restarts of SGD, where in each restart the learning rate is initialized to some value and is scheduled to decrease. Four different instantiations of this new learning rate schedule are visualized in Figure 1. Our empirical results suggest that SGD with warm restarts requires 2× to 4× fewer epochs than the currently-used learning rate schedule schemes to achieve comparable or even better results.” Loshchilov provides a learning rate scheduler, as shown in Figure 1, as part of training a neural network.) including: (a) increase the learning rate hyperparameter for each of a first plurality of epochs over a first portion of training the neural network (Loshchilov Figure 1 “Learning rate schedule”; Section 3 Stochastic Gradient Descent with Warm Restarts “In this work, we consider one of the simplest warm restart approaches. We simulate a new warm started run / restart of SGD once Ti epochs are performed, where i is the index of the run. Importantly, the restarts are not performed from scratch but emulated by increasing the learning rate ηt while the old value of xt is used as an initial solution.” Loshchilov provides increasing learning rate, as shown in Figure 1.); and (b) decrease the learning rate hyperparameter for each of a second plurality of epochs across a second portion of training the neural network that occurs after the first portion (Loshchilov Figure 1 “Learning rate schedule”; Section 3 Stochastic Gradient Descent with Warm Restarts “The decrease of the learning rate is shown in Figure 1 for fixed Ti = 50, Ti = 100 and Ti = 200; note that the logarithmic axis obfuscates the typical shape of the cosine function.” Loshchilov provides decreasing learning rate, as shown in Figure 1.).
Mardani Korani, Fuoli and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli with the above teachings of Loshchilov. Doing so would accelerate the training of deep neural networks (Loshchilov Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs.”).
Regarding claim 2, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, as discussed above in the rejection of claim 1, wherein the transformation of the calculated difference into frequency domain data is performed by using a Fourier Transform (Fuoli Section 3.3, Supervision Losses “In addition to these spatial domain losses, we propose a Fourier space loss LF for supervision from the ground truth frequency spectrum during training. First, ground truth y and generated image yˆ are pre-processed with a Hann window, as described in Section 3.2. Afterwards, both images are transformed into Fourier space by applying the fast Fourier transform (FFT), where we calculate amplitude and phase of all frequency components.” Fuoli provides transforming calculated differences in images in the frequency domain using a Fast Fourier transform.).
Mardani Korani, Fuoli, and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Regarding claim 4, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, as discussed above in the rejection of claim 1, wherein the at least one hardware processor is further configured to: apply, as part of transformation of the calculated difference, a windowing function to the calculated difference, wherein transformation of the calculated difference into frequency domain data is further based on application of the windowing function to the calculated difference (Fuoli Section 3.3 Supervision Losses “First, ground truth y and generated image yˆ are pre-processed with a Hann window, as described in Section 3.2. Afterwards, both images are transformed into Fourier space by applying the fast Fourier transform (FFT), where we calculate amplitude and phase of all frequency components” Fuoli provides applying a Hann window function as part of transformation of the calculated difference and transformation into the frequency domain, corresponding to apply, as part of transformation of the calculated difference, a windowing function to the calculated difference, wherein transformation of the calculated difference into frequency domain data is further based on application of the windowing function to the calculated difference.).
Mardani Korani, Fuoli, and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Regarding claim 6, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, as discussed above in the rejection of claim 1, wherein a rate of change of the learning rate hyperparameter during the first portion is greater than a rate of change during the second portion (Loshchilov Figure 1 “Learning rate schedule”; Section 1 Introduction “In this paper, we propose to periodically simulate warm restarts of SGD, where in each restart the learning rate is initialized to some value and is scheduled to decrease. Four different instantiations of this new learning rate schedule are visualized in Figure 1. Our empirical results suggest that SGD with warm restarts requires 2× to 4× fewer epochs than the currently-used learning rate schedule schemes to achieve comparable or even better results.” Loshchilov provides a learning rate scheduler, as shown in Figure 1, including varying rates of change of the learning rate, corresponding to a rate of change of the learning rate hyperparameter during the first portion is greater than a rate of change during the second portion.).
Mardani Korani, Fuoli and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Loshchilov. Doing so would accelerate the training of deep neural networks (Loshchilov Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs.”).
Regarding claim 7, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, as discussed above in the rejection of claim 1, wherein the loss value is a scalar value that is calculated based on (a) a sum of the frequency domain data of differences in pixel values of different pixel locations within the target and output image data (Fuoli Section 3.3 Supervision Losses “First, ground truth y and generated image yˆ are pre-processed with a Hann window, as described in Section 3.2. Afterwards, both images are transformed into Fourier space by applying the fast Fourier transform (FFT), where we calculate amplitude and phase of all frequency components. The L1-norms of amplitude difference LF,|·| and phase angle difference LF,∠ between output image and target are averaged to produce the total frequency loss LF .” Fuoli provides calculating an L1 norm for frequency domain data of differences in input and target images, which corresponds to a scalar value that is calculated based on (a) a sum of the frequency domain data of differences in pixel values of different pixel locations within the target and output image data.), and (b) a total number of differences in pixel values (Fuoli Section 3.2 Fourier Transform and SR “The Fourier transform is widely used to analyze the frequency content in signals. It can also be applied to multidimensional signals such as images, where the spatial variations of pixel-intensities have a unique representation in the frequency domain…. Since images are composed of multiple color channels, we calculate the Fourier transform for each channel separately and perform the transform per channel.” Fuoli provides pixel-intensity variations between input and target images, corresponding to calculating a total number of differences in pixel values.).
Mardani Korani, Fuoli, and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Regarding claim 9, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, as discussed above in the rejection of claim 1, wherein the L1 family norm is the L1 norm (Fuoli Section 3.3 Supervision Losses “The L1-norms of amplitude difference LF,|·| and phase angle difference LF,∠ between output image and target are averaged to produce the total frequency loss LF .” Fuoli provides calculating a loss value using an L1 norm of the frequency domain data, corresponding to the L1 family norm is the L1 norm.).
Mardani Korani, Fuoli, and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Regarding claim 10, it is the method embodiment of claim 1 with similar limitations to claim 1 and is rejected using the same reasoning found in the rejection of claim 1.
Regarding claim 11, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Mardani Korani in view of Fuoli in further view of Loshchilov for the same reasons set forth above in the rejection of claim 2.
Regarding claim 13, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Mardani Korani in view of Fuoli in further view of Loshchilov for the same reasons set forth above in the rejection of claim 4.
Regarding claim 15, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Mardani Korani in view of Fuoli in further view of Loshchilov for the same reasons set forth above in the rejection of claim 6.
Regarding claim 16, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Mardani Korani in view of Fuoli in further view of Loshchilov for the same reasons set forth above in the rejection of claim 7.
Regarding claim 18, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Mardani Korani in view of Fuoli in further view of Loshchilov for the same reasons set forth above in the rejection of claim 9.
Regarding claim 19, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the method of claim 10, as discussed above in the rejection of claim 10, further comprising: as part of calculating the difference between the predicted output image data and target image data, calculating a difference in RGB pixel values between at least one pixel in the target image data and a corresponding pixel in the predicted output image data (Fuoli Section 3.2 Fourier Transform and SR “The Fourier transform is widely used to analyze the frequency content in signals. It can also be applied to multidimensional signals such as images, where the spatial variations of pixel-intensities have a unique representation in the frequency domain…. Since images are composed of multiple color channels, we calculate the Fourier transform for each channel separately and perform the transform per channel.” Fuoli provides pixel-intensity variations between input and target images, corresponding to calculating a difference in RGB pixel values between at least one pixel in the target image data and a corresponding pixel in the predicted output image data.); and storing the calculated differences to a two-dimensional array, wherein transforming the calculated difference includes performing a Fourier Transform on the two-dimensional array as part of obtaining the frequency domain data (Fuoli Section 3.3 Supervision Losses “First, ground truth y and generated image yˆ are pre-processed with a Hann window, as described in Section 3.2. Afterwards, both images are transformed into Fourier space by applying the fast Fourier transform (FFT), where we calculate amplitude and phase of all frequency components. The L1-norms of amplitude difference LF,|·| and phase angle difference LF,∠ between output image and target are averaged to produce the total frequency loss LF” Fuoli provides an L1 applied to a difference in image date using a Fourier transform corresponding to storing the calculated differences to a two-dimensional array, wherein transforming the calculated difference includes performing a Fourier Transform on the two-dimensional array as part of obtaining the frequency domain data.).
Mardani Korani, Fuoli, and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Regarding claim 20, Mardani Korani teaches a computer system for training a neural network that processes images (Mardani Korani [0056] “FIG. 1 is a block diagram illustrating an architecture for training an untrained neural network 106 to infer or synthesize, after training, a larger textured image 112 from a smaller textured image 108, according to at least one embodiment.” Mardani Korani provides a computer system for training a neural network that processes images.), the computer system comprising: non-transitory computer readable storage configured to store image data for a plurality of images (Mardani Korani [0056] "In at least one embodiment, texture synthesis is used to generate image data sets comprising larger image sizes from smaller baseline input images 108."; [0570] " In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electric or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, cache, and queues) within transceivers of transitory signals." Mardani Korani provides non-transitory computer readable storage configured to store image data for a plurality of images.); the at least one hardware processor configured to: …generate, from the plurality of images, input image data and target image data (Mardani Korani [0057] “In at least one embodiment, training data 102 is input into a training framework 104 to train an untrained neural network 106 to synthesize an output 112, such as an output textured image, from an input 108, such as an input textured image. In at least one embodiment, training data 102 is one or more images used to train an untrained neural network 106 using a training framework 104. In at least one embodiment, training data 102 includes supervision or other information used to facilitate training by a training framework 104. In at least one embodiment, supervision or other information to facilitate training includes data that identifies features of an image that improve training by a training framework 104”; [0058] “In at least one embodiment, training data 102 is a set of K target textured images having variable dimensions, from which K textured images are extracted, prior to training, from target images” Mardani Korani provides training dataset 102, which is generated from a plurality of images and comprises input and target image data.); generate predicted output image data by using the input image data as input to a neural network (Mardani Korani [0056] “FIG. 1 is a block diagram illustrating an architecture for training an untrained neural network 106 to infer or synthesize, after training, a larger textured image 112 from a smaller textured image 108, according to at least one embodiment.” Mardani Korani provides using neural networks to infer a larger textured image from a smaller textured images corresponding to generate predicted output image data by using the input image data as input to a neural network.), wherein the input image data represents images of a first resolution and the target image data represents images of a second resolution that is greater than the first resolution (Mardani Korani [0057] “In at least one embodiment, training data 102 is input into a training framework 104 to train an untrained neural network 106 to synthesize an output 112, such as an output textured image, from an input 108, such as an input textured image. In at least one embodiment, training data 102 is one or more images used to train an untrained neural network 106 using a training framework 104. In at least one embodiment, training data 102 includes supervision or other information used to facilitate training by a training framework 104. In at least one embodiment, supervision or other information to facilitate training includes data that identifies features of an image that improve training by a training framework 104”; [0058] “In at least one embodiment, training data 102 is a set of K target textured images having variable dimensions, from which K textured images are extracted, prior to training, from target images…A training framework 104, in an embodiment, trains an untrained neural network 106 to learn an up-sampler that maps each small texture example to the target textured image in training data 102.” Mardani Korani provides training dataset 102, which is generated from a plurality of images and comprises input and target image data including an upscale function to target images, corresponding to the input image data represents images of a first resolution, and the target image data represents images of a second resolution that is greater than the first.) …as part of training the neural network, perform backpropagation on the neural network to update weights of the neural network based on the calculated loss value (Mardani Korani [0059] “In at least one embodiment, a training framework 104 determines loss values that are backpropagated in an untrained neural network 106 in order to train said untrained neural network 106, as described below in conjunction with FIG. 5.” Mardani Korani provides backpropagation on the neural network based on loss values corresponding to perform backpropagation on the neural network to update weights of the neural network based on the calculated loss value.)
Mardani Korani fails to teach …train, over a plurality of epochs, the neural network based on a defined training schedule, wherein for each of the plurality of epochs a learning rate hyperparameter is set based on the defined training schedule …transform the target image data and output image data into, respectively, frequency domain target data and frequency domain output data; calculate the absolute value of each coefficient of the frequency domain target data and the frequency domain output data; calculate a loss value by using a difference between each respective coefficient of the absolute value of the frequency domain target data and the absolute value of the frequency domain output data; …as part of training the neural network, executing a learning rate scheduler for the defined training schedule to adjust the learning rate hyperparameter over the training of the neural network including: (a) increase the learning rate hyperparameter for each of a first plurality of epochs over a first portion of training the neural network; and (b) decrease the learning rate hyperparameter for each of a second plurality of epochs across a second portion of training the neural network that occurs after the first portion.
However, Fuoli teaches …transform the target image data and output image data into, respectively, frequency domain target data and frequency domain output data (Fuoli Section 3.3, Supervision Losses “In addition to these spatial domain losses, we propose a Fourier space loss LF for supervision from the ground truth frequency spectrum during training. First, ground truth y and generated image yˆ are pre-processed with a Hann window, as described in Section 3.2. Afterwards, both images are transformed into Fourier space by applying the fast Fourier transform (FFT), where we calculate amplitude and phase of all frequency components.” Fuoli provides transforming ground truth and generated images, corresponding respectively to target image data and output image data into, which are both transformed into frequency domain data using a fast Fourier transform.); calculate the absolute value of each coefficient of the frequency domain target data and the frequency domain output data (Fuoli Section 1, Introduction “To the best of our knowledge we are the first to apply a GAN loss directly on Fourier coefficients in SR”; Section 3.2, Fourier Transform and SR “Since images are composed of multiple color channels, we calculate the Fourier transform for each channel separately and perform the transform per channel. The explicit notation of channels is omitted in our formulas. Each complex component Xu;v can be represented by amplitude jFfxgu;vj and phase \Ffxgu;v, which provides a more intuitive analysis of the frequency content.”; Equation (2); Fuoli provides calculating the absolute value of the amplitude (which is a Fourier coefficient) for each channel as shown in Equation 2, which corresponds to calculating the absolute value of each coefficient of the frequency domain target data and the frequency domain output data.); calculate a loss value by using a difference between each respective coefficient of the absolute value of the frequency domain target data and the absolute value of the frequency domain output data (Fuoli Section 1, Introduction “To the best of our knowledge we are the first to apply a GAN loss directly on Fourier coefficients in SR”; Section 3.3 Supervision Losses “First, ground truth y and generated image ^y are pre-processed with a Hann window, as described in Section 3.2. Afterwards, both images are transformed into Fourier space by applying the fast Fourier transform (FFT), where we calculate amplitude and phase of all frequency components. The L1-norms of amplitude difference LF,|·| and phase angle difference LF,∠ between output image and target are averaged to produce the total frequency loss LF .” Fuoli provides calculating loss for amplitude in differences between ground truth and generated images corresponding to calculating a loss value by using a difference between each respective coefficient of the absolute value of the frequency domain target data and the absolute value of the frequency domain output data.).
Mardani Korani and Fuoli are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically Fourier transform based image manipulation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Further, Loshchilov teaches train, over a plurality of epochs, the neural network based on a defined training schedule, wherein for each of the plurality of epochs a learning rate hyperparameter is set based on the defined training schedule (Loshchilov Figure 1 “Learning rate schedule”; Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs. Our SGDR simulates warm restarts by scheduling the learning rate to achieve competitive results on CIFAR-10 and CIFAR-100 roughly two to four times faster” Loshchilov provides training a neural network over a plurality of epochs based on a defined learning rate schedule, as shown in Figure 1, which shows the change in learning rate over consecutive epochs of training); …as part of training the neural network, executing a learning rate scheduler for the defined training schedule to adjust the learning rate hyperparameter over the training of the neural network (Loshchilov Figure 1 “Learning rate schedule”; Section 1 Introduction “In this paper, we propose to periodically simulate warm restarts of SGD, where in each restart the learning rate is initialized to some value and is scheduled to decrease. Four different instantiations of this new learning rate schedule are visualized in Figure 1. Our empirical results suggest that SGD with warm restarts requires 2× to 4× fewer epochs than the currently-used learning rate schedule schemes to achieve comparable or even better results.” Loshchilov provides a learning rate scheduler, as shown in Figure 1, as part of training a neural network.) including: (a) increase the learning rate hyperparameter for each of a first plurality of epochs over a first portion of training the neural network (Loshchilov Figure 1 “Learning rate schedule”; Section 3 Stochastic Gradient Descent with Warm Restarts “In this work, we consider one of the simplest warm restart approaches. We simulate a new warm started run / restart of SGD once Ti epochs are performed, where i is the index of the run. Importantly, the restarts are not performed from scratch but emulated by increasing the learning rate ηt while the old value of xt is used as an initial solution.” Loshchilov provides increasing learning rate, as shown in Figure 1.); and (b) decrease the learning rate hyperparameter for each of a second plurality of epochs across a second portion of training the neural network that occurs after the first portion (Loshchilov Figure 1 “Learning rate schedule”; Section 3 Stochastic Gradient Descent with Warm Restarts “The decrease of the learning rate is shown in Figure 1 for fixed Ti = 50, Ti = 100 and Ti = 200; note that the logarithmic axis obfuscates the typical shape of the cosine function.” Loshchilov provides decreasing learning rate, as shown in Figure 1.).
Mardani Korani, Fuoli and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli with the above teachings of Loshchilov. Doing so would accelerate the training of deep neural networks (Loshchilov Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs.”).
Regarding claim 21, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, wherein the learning rate hyperparameter is increased exponentially over the first portion (Loshchilov Section 1 Introduction; Figure 1 “Alternative schedule schemes of learning rate ηt over batch index t: default schemes with η0 = 0.1 (blue line) and η0 = 0.05 (red line) as used by Zagoruyko & Komodakis (2016); warm restarts simulated every T0 = 50 (green line), T0 = 100 (black line) and T0 = 200 (grey line) epochs with ηt decaying during i-th run from η i max = 0.05 to η i min = 0 according to eq. (5); warm restarts starting from epoch T0 = 1 (dark green line) and T0 = 10 (magenta line) with doubling (Tmult = 2) periods Ti at every new warm restart” Loshchilov provides Figure 1, which displays a graph with learning rate on the y-axis which provides intervals by factors of 10 and Epochs measuring units of time on the x-axis, wherein the intervals of factors of 10 on the y-axis correspond to the plurality of exponentially increasing learning rates over a first portion (epochs).).
Mardani Korani, Fuoli and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Loshchilov. Doing so would accelerate the training of deep neural networks (Loshchilov Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs.”).
Regarding claim 22, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 21, as discussed above in the rejection of claim 21, wherein the learning rate hyperparameter is increased from 1e-5 to 1e-3 over the first portion (Loshchilov Section 1 Introduction; Figure 1 “Alternative schedule schemes of learning rate ηt over batch index t: default schemes with η0 = 0.1 (blue line) and η0 = 0.05 (red line) as used by Zagoruyko & Komodakis (2016); warm restarts simulated every T0 = 50 (green line), T0 = 100 (black line) and T0 = 200 (grey line) epochs with ηt decaying during i-th run from η i max = 0.05 to η i min = 0 according to eq. (5); warm restarts starting from epoch T0 = 1 (dark green line) and T0 = 10 (magenta line) with doubling (Tmult = 2) periods Ti at every new warm restart” Loshchilov provides Figure 1, which portrays a value at 1e-5 on the bottom left corner of the graph depicted in Figure 1, and shows a warm restart of a plurality of learning rates which include a value increase to 1e-3 in a first portion/plurality of epochs.).
Mardani Korani, Fuoli and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Loshchilov. Doing so would accelerate the training of deep neural networks (Loshchilov Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs.”).
Regarding claim 23, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 21, as discussed above in the rejection of claim 21, wherein the learning rate hyperparameter is increased by at least a factor 10 from a beginning of the first portion to an end of the first portion (Loshchilov Section 1 Introduction; Figure 1 “Alternative schedule schemes of learning rate ηt over batch index t: default schemes with η0 = 0.1 (blue line) and η0 = 0.05 (red line) as used by Zagoruyko & Komodakis (2016); warm restarts simulated every T0 = 50 (green line), T0 = 100 (black line) and T0 = 200 (grey line) epochs with ηt decaying during i-th run from η i max = 0.05 to η i min = 0 according to eq. (5); warm restarts starting from epoch T0 = 1 (dark green line) and T0 = 10 (magenta line) with doubling (Tmult = 2) periods Ti at every new warm restart” Loshchilov provides Figure 1, which displays on the y-axis values for learning increasing by at least a factor of 10 for a plurality of epochs corresponding to a beginning and end of a first portion.).
Mardani Korani, Fuoli and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Loshchilov. Doing so would accelerate the training of deep neural networks (Loshchilov Section 6 Conclusion “In this paper, we investigated a simple warm restart mechanism for SGD to accelerate the training of DNNs.”).
Regarding claim 24, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, as discussed above in the rejection of claim 1, wherein different loss functions are used for different epochs in the training of the neural network (Fuoli Section 3 Proposed Method “In contrast to the representation of an image in spatial domain, these missing frequencies can be clearly separated in Fourier domain. We therefore propose two losses in the frequency domain, to directly emphasize the training on the relevant frequencies. Additionally, the frequency components provide global guidance during training due to the nature of the Fourier transform.”; Section 4.1 Ablation “We do not use a learning rate scheduler for stability reasons and fairness due to the heterogeneous combinations of different loss types. The training parameters are set to α = 0.005, β = 0.01 and γ = 1 as proposed in state-of-the-art method ESRGAN [31].” Fuoli provides heterogeneous combinations of different loss types corresponding to different loss functions are used for different epochs in the training of the neural network.).
Mardani Korani, Fuoli, and Loshchilov are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli and Loshchilov with the above teachings of Fuoli. Doing so would help to generate plausible missing content in an image and provide better perceptual quality (Fuoli Section 5, Conclusion “The clear separation of images into LF (retained) and HF (missing) content and therefore the direct emphasis on the missing high frequencies in Fourier space, imposed by our proposed losses, helps the SR network to generate plausible HF content. At the same time, we also apply the corresponding spatial losses to leverage the complementary local information, which results in even better perceptual quality.”).
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Mardani Korani et al. (U.S. Patent Publication No. 2022/0101494) (“Mardani Korani”) in view of Fuoli et al. (Fourier Space Losses for Efficient Perceptual Image Super-Resolution) (“Fuoli”) in further view of Loshchilov et al. (Stochastic Gradient Descent with Warm Restarts) (“Loshchilov”) and Siekmann et al. (U.S. Patent Publication No. 2021/0084301) (“Siekmann”).
Regarding claim 8, Mardani Korani in view of Fuoli in further view of Loshchilov teaches the computer system of claim 1, as discussed above in the rejection of claim 1, but fails to teach wherein the neural network is implemented using separable block transforms.
However, Siekmann teaches wherein the neural network is implemented using separable block transforms (Siekmann [0046] “FIG. 7 shows a schematic block diagram of block-based residual coding using a set of transforms comprising one or more neural networks according to an embodiment of the present invention;”; [0138] “A further transform class is given by the fact that transforms may factorize into a vertical and a horizontal transform and referred to as separable transforms.”).
Mardani Korani, Fuoli, Loshchilov and Siekmann are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to neural network based image scaling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mardani Korani in view of Fuoli, and Loshchilov with the above teachings of Siekmann. Doing so would allow for a flexible and efficient testing and signaling of transforms for residual blocks (Siekmann [0009] “The innovative concept of introducing a mode dependent candidate list may allow for a flexible and efficient testing and signaling of transforms for residual blocks.”).
Regarding claim 17, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Mardani Korani in view of Fuoli in further view of Loshchilov and Siekmann for the same reasons set forth in the rejection of claim 8.
Response to Arguments
Regarding the rejection applied under 35 U.S.C. 101, Applicant firstly asserts that increasing and decreasing a learning rate during the training of a neural network is not a mental process (“Remarks”, Page 8). Applicant further asserts that the limitations “generate, from the plurality of images, input image data and target image data” and “generate predicted output image data” are not mentally performable, and that the BRI of image data or predicted output image data are not mere mental pictures in the mind of a person (“Remarks”, Page 8).
However “generate, from the plurality of images, input image data and target image data” is mentally performable. For example, “input image data and target image data”, under the broadest reasonable interpretation, could include any data related to an image. Accordingly, one could mentally generate “image data”, such as pixel values, RGB values, or other image defining characteristics. For example, one could mentally generate pixel values for an image, or a label characterizing an image. The “generate predicted output image data” is also mentally performable, as one could mentally generate that same data in a predictive manner. The “increasing” and “decreasing” learning rate limitations, as discussed in the 35 U.S.C. 101 rejection of claim 1 above, are being analyzed under prong two.
Applicant further asserts that these elements are more appropriately handled under prong two, and integrate any judicial exception into a practical application (“Remarks”, Page 8). Applicant further asserts that the argued improvements are in training neural networks (“Remarks”, Page 8). Applicant further asserts that the limitations “calculate a difference between the predicted output image data and the target image data; transform the calculated difference into frequency domain data; calculate a loss value using an L1 family norm of the transformed frequency domain data” integrates any alleged otherwise judicially excepted subject matter into a practical application because it improves the training of neural networks (“Remarks”, Page 8-9).
However, as discussed above, the “generating” image data limitations are mentally performable. Further, the limitations “calculate a difference between the predicted output image data and the target image data; transform the calculated difference into frequency domain data; calculate a loss value using an L1 family norm of the transformed frequency domain data”, are mathematical calculations and/or relationships, and are therefore analyzed under prong one, as discussed above in the 35 U.S.C. 101 rejection of claim 1 above. Therefore, even if the claims did recite an improvement related to “training”, it would be an improvement in the abstract idea of “perform backpropagation on the neural network to update weights of the neural network based on the calculated loss value”. As discussed in MPEP 2106.05(a)(II), it is important to keep in mind that an improvement in the abstract idea itself is not an improvement in technology. Therefore, the claims remain rejected under 35 U.S.C. 101.
Regarding the rejection applied under 35 U.S.C. 103, Applicant firstly asserts that the applied reference, Fuoli states “we do not use a learning rate scheduler for stability reasons and fairness due to the heterogenous combination of different loss types.”, which Applicant further asserts teaches away from combining the teachings of Fuoli with any teaching of a learning rate scheduler (“Remarks”, Page 9-10).
Examiner agrees that Fuoli states “We do not use a learning rate scheduler for stability reasons and fairness due to the heterogeneous combinations of different loss types.” in section 4.1. However, this does not teach away from the use a learning rate scheduler, because the claimed invention, as recited in amended claim 1, does not use a “heterogeneous combination of different loss types.” Rather, amended claim 1 of the present application only recites a single type of loss function (The L1 loss). Therefore, the claim does not recite a “heterogeneous combination of different loss types”, and a person of ordinary skill in the would have sought to use a learning rate scheduler, despite the disclosure from Fuoli. Further, Loshchilov, the reference relied upon for the teaching of a learning rate scheduler, does not disclose a “heterogeneous combination of different loss types.”. Therefore, the claims remain rejected under 35 U.S.C. 103.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KURT NICHOLAS PRESSLY whose telephone number is (703)756-4639. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KURT NICHOLAS PRESSLY/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125