DETAILED ACTION
[1] Remarks
I. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
II. Claims 1-20 are pending and have been examined, where claims 1-2, 4-5, 9-11, 13-14 and 16-18 is/are rejected and claim 3, 6-8, 12, 15, and 19-20 is/are objected. Explanations will be provided below.
III. Inventor and/or assignee search were performed and determined no double patenting rejection(s) is/are necessary.
IV. Patent eligibility (updated in 2019) shown by the following: Claims 1-20 pass patent eligibility test because there is/are no limitation or a combination of limitations amounting to an abstract idea. Also, the following limitation or the combinations of the limitations:
“remove at least one gradient from among the plurality of gradients by applying a gradient mask comprising gradient pruning information to the plurality of gradients; train the neural network model by updating the weights of the neural network model based on one or more remaining gradients from among the plurality of gradients; and obtain a quality-processed output image based on the input image, using the trained neural network model” effects a transformation or a reduction of a particular article to a different state or thing / adds a specific limitation(s) other than what is well-understood, routine and conventional in the field, or adding unconventional steps that confine the claim to a particular useful application and providing improvements to the technical field of deep learning, which recite additional elements that integrate the judicial exception into a practical application and amounting significant more.
[2] Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Use of the word “means” (or “step for”) in a claim with functional language creates a rebuttable presumption that the claim element is to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is invoked is rebutted when the function is recited with sufficient structure, material, or acts within the claim itself to entirely perform the recited function. Absence of the word “means” (or “step for”) in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim elements in this application that use the word “means” (or “step for”) are presumed to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Similarly, claim elements that do not use the word “means” (or “step for”) are presumed not to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
Claim(s) 1-9 are not interpreted under 35 U.S.C. 112(f) or pre-AIA U.S.C. 112 6th paragraph because of the following reason(s): limitations are modified by sufficient structure or material for performing the claimed function.
Claim(s) 10-20 do not require 35 U.S.C. 112(f) or pre-AIA U.S.C. 112 6th paragraph interpretation because they are method claims and / or they are CRM claims.
Upon examination of the specification and claims, the examiner has determined, under the best understanding of the scope of the claim(s), rejection(s) under 35 U.S.C. 112(a)/(b) is not necessitated because of the following reasons: sufficient support are provided in the written description / drawings of the invention.
[3] Grounds of Rejection
Claim Rejections - 35 USC § 103
1. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claims 1-2, 4-5, and 9-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Molchanov (US 20180114114) in view of BHALGAT (US 20220261648).
NOTE: BHALGAT qualify as U.S.C. 102(a)(1) reference.
Regarding claim 1, Molchanov discloses an image processing apparatus comprising:
at least one processor; and a memory configured to store one or more instructions which, when executed by the at least one processor (see figure 6, 601 and 604), cause the image processing apparatus to:
obtain a neural network model for performing image quality processing on an input image (see paragraph 24);
PNG
media_image1.png
117
619
media_image1.png
Greyscale
calculate a plurality of gradients by partially differentiating weights of the neural network model with respect to a loss of the neural network model, by applying training data corresponding to the input image to the neural network model (see paragraph 18, first-order gradients of a cost function with respect to layer parameters are received for a trained neural network, where the cost function is read as the loss function and first order gradients are read as the partial derivatives);
obtain a quality-processed output image based on the input image, using the trained neural network model (see figure 2B, 225, the model stops training when EPOCHs are sufficient).
Molchanov is silent in disclosing remove at least one gradient from among the plurality of gradients by applying a gradient mask comprising gradient pruning information to the plurality of gradients; train the neural network model by updating the weights of the neural network model based on one or more remaining gradients from among the plurality of gradients.
BHALGAT discloses
remove at least one gradient from among the plurality of gradients by applying a gradient mask comprising gradient pruning information to the plurality of gradients (see paragraph 65, whether it should generate a new set of indices for gradients to be used in refining the model, or if it should re-use a previously-generated set of indices, see paragraph 81, the processing system may determine whether the pruning criteria specifies to prune or remove any gradients);
train the neural network model by updating the weights of the neural network model based on one or more remaining gradients from among the plurality of gradients (see figure 4, 435 using the gradients that are retrained, update weight tensor based on gradient tensor and indices).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to include remove at least one gradient from among the plurality of gradients by applying a gradient mask comprising gradient pruning information to the plurality of gradients in order to permanently or temporarily zeroing out unimportant gradients, the model training and learns to ignore redundant to improve a neural network's computational efficiency and generalization capabilities.
Regarding claim 2, BHALGAT discloses the image processing apparatus of claim 1, wherein the gradient mask comprises information for removing the at least one gradient from among the plurality of gradients and information for maintaining the one or more remaining gradients from among the plurality of gradients (see paragraph 81, the processing system may determine whether the pruning criteria specifies to prune or remove any gradients). See the motivation for claim 1.
Regarding claim 4, BHALGAT discloses the image processing apparatus of claim 1, wherein the gradient mask comprises information for removing gradients corresponding to n minimum values from among a plurality of values obtained by quantifying the plurality of gradients, where n is a natural number (see paragraph 24, Gradient pruning may be performed in various manners, such as directly pruning selected gradients in a gradient tensor by replacing them with a zero value such that the gradient tensor may then be converted to a compressed data representation, where the number of pruned values is finite and is read as n). See the motivation for claim 4. In addition, removing gradients corresponding to n minimum can skip operations associated with zeroed-out gradients, speeding up backpropagation.
Regarding claim 5, BHALGAT discloses the image processing apparatus of claim 4, wherein the gradient mask comprises information for removing gradients corresponding to values that are less than or equal to a threshold value from among the plurality of values obtained by quantifying the plurality of gradients (see paragraph 23, model's parameters by selectively pruning such gradients based on, for example, one or more criteria, such as a threshold value). See the motivation for claim 1, Also selectively pruning small gradients beneficially reduces the computational cost of training a model and thus improving training efficiency while having little impact on the performance of the model, or even improving the performance.
Regarding claim 9, Molchanov discloses the image processing apparatus of claim 1, wherein the one or more instructions further cause the image processing apparatus to: determine an image quality value of the input image; and obtain a low-resolution image by performing at least one of compression deterioration, blurring deterioration, resolution adjustment, or noise addition on the input image, based on the image quality value of the input image (see paragraph 71, the ROP unit 450 includes a ROP Manager 455, a Color ROP (CROP) unit 452, and a Z ROP (ZROP) unit 454, the CROP unit 452 performs raster operations related to pixel color, such as color compression, pixel blending, and the like).
Regarding claim 10, see the rationale and rejection for claim 1.
Regarding claim 11, see the rationale and rejection for claim 2.
Regarding claim 12, see the rationale and rejection for claim 3.
3. Claims 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over LEVINSHTEIN (US 20200320748) in view of Munkberg (US 20180357537).
Regarding claim 13, LEVINSHTEIN discloses a method of training a neural network model, the method comprising:
obtaining a plurality of
are degraded into a plurality of types of deteriorated image quality based on a training image (see figure 6, 602 is read as the low resolution image, see figure 6 illustration below, 602);
training the neural network model by applying the training image and the plurality of
calculating a plurality of gradients of the trained neural network model by applying the
training image and the plurality of low-resolution training images to the trained neural network
model (see paragraph 11, where each value in the masks, Mx, My and Mmag, are gradients); and
PNG
media_image2.png
187
521
media_image2.png
Greyscale
generating a gradient mask for removing at least one gradient from among the plurality
of gradients (see figure 6, the model with skip connection and mask image grad loss is rad as the gradient mask, also see paragraph 11 below):
PNG
media_image3.png
254
1027
media_image3.png
Greyscale
.
LEVINSHTEIN is silent in disclosing obtaining a plurality of low-resolution training images having image qualities which are degraded into a plurality of types of deteriorated image quality based on a training image.
Munkberg discloses
obtaining a plurality of low-resolution training images having image qualities which are degraded into a plurality of types of deteriorated image quality based on a training image (see paragraph 30, a low resolution, dense, input vector X is upscaled to generate a sparse input vector X. In one embodiment, the sparse input data and/or sparse target data for the training dataset is computed on-the-fly rather than storing the entire training dataset, also see paragraph 3, a noisy image can also be input);
training the neural network model by applying the training image and the plurality of low-resolution training images to the neural network model (see paragraph 8, FIG. 1C illustrates a conceptual diagram of neural network training using sparse input data and sparse ground truth training targets).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to include obtaining a plurality of low-resolution training images having image qualities which are degraded into a plurality of types of deteriorated image quality based on a training image because it enable algorithm to learn generalized features allowing it to accurately reconstruct high-resolution images from real-world or distorted inputs rather than just perfectly downsampled ones.
Regarding claim 14, Munkberg discloses the method of claim 13, wherein the training of the neural network model comprises:
obtaining a plurality of modified neural networks corresponding to the plurality of low-resolution training images (see figure 1C, the input is low resolution images, also are low-resolution training images);
calculating a plurality of test losses for the plurality of modified neural networks by
applying a test image to each modified neural network of the plurality of modified neural
networks (see paragraph 4, image processing networks, the L2 norm is often used as the loss function, where the sparse backpropagation is read as the modify neural network); and
updating a weight of the neural network model so that a sum of the plurality of test
losses for the plurality of modified neural networks is minimized (see paragraph 41, a differentiable function g describes the neural network model 125 with a set of trainable parameters, Θ, that map a dense input vector X={x1, x2, . . . , xn}, to an image, as close as possible to a dense target Y, the neural network 125 is trained by minimizing a loss function).
See the motivation for claim 1. In addition, enable algorithm to learn detailed features allowing it to accurately reconstruct high quality images.
Regarding claim 17, LEVINSHTEIN discloses the method of claim 13, wherein the gradient mask has a vector value of "0" associated with gradients corresponding to n minimum values from among values for the plurality of gradients, wherein the values are obtained by quantifying the plurality of gradients, and a vector value of "1" associated with the one or more remaining gradients, wherein n is a natural number (see figure 6, in the mask the dark portion is the “0” value and the bright areas are the “1” value, where each row in the image mask is read as a vector, see also paragraph 11, where each value in the masks, Mx, My and Mmag, are gradients); and
PNG
media_image2.png
187
521
media_image2.png
Greyscale
PNG
media_image3.png
254
1027
media_image3.png
Greyscale
Regarding claim 18, Munkberg discloses the method of claim 16, further comprising quantifying the plurality of gradients by summing pieces of data in a gradient data matrix for each gradient filter of a plurality of gradient filters corresponding to the plurality of gradients (see paragraph 43, epsilon is read as the gradient filters):
PNG
media_image4.png
253
630
media_image4.png
Greyscale
See the motivation for claim 13. Also summing pieces of data train each segment of the data for include any local features in local image.
4. Claims 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over LEVINSHTEIN (US 20200320748) in view of Munkberg (US 20180357537) and BHALGAT (US 20220261648).
Regarding claim 16, the combination of LEVINSHTEIN and Munkberg as a whole, discloses all the limitations of claim 13, but is silent in disclosing the method of claim 13, wherein the gradient mask comprises a first value for removing the at least one gradient of the plurality of gradients and a second value for maintaining one or more remaining gradients of the plurality of gradients.
BHALGAT discloses the method of claim 13, wherein the gradient mask comprises a first value for removing the at least one gradient of the plurality of gradients and a second value for maintaining one or more remaining gradients of the plurality of gradients (see paragraph 65, whether it should generate a new set of indices for gradients to be used in refining the model, or if it should re-use a previously-generated set of indices, see paragraph 81, the processing system may determine whether the pruning criteria specifies to prune or remove any gradients).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to include remove at least one gradient from among the plurality of gradients by applying a gradient mask comprising gradient pruning information to the plurality of gradients in order to permanently or temporarily zeroing out unimportant gradients, the model training and learns to ignore redundant to improve a neural network's computational efficiency and generalization capabilities.
[4] Claim Objections
Claim(s) 3, 6-8, 12, 15, 19-20 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
With regards to claim 3, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the image processing apparatus of claim 1, wherein the one or more instructions further cause the image processing apparatus to: calculate the plurality of gradients for updating weights of a plurality of filters, corresponding to each filter of the plurality of filters within at least one convolutional layer included in the neural network model, remove the at least one gradient corresponding to at least one filter from among the plurality of filters, based on the gradient mask; and update weights of one or more remaining filters from among the plurality of filters based on the one or more remaining gradients corresponding to the one or more remaining filters; in combination with the rest of the limitations of claim 1.
Molchanov discloses the image processing apparatus of claim 1, wherein the one or more instructions further cause the image processing apparatus to: calculate the plurality of gradients for updating weights of a plurality of filters, corresponding to each filter of the plurality of filters within at least one convolutional layer included in the neural network model (see paragraph 27, a criteria-based pruning technique is preferred, starting with a full set of the parameters W and pruning as a backward filter by iteratively identifying and removing at least one least important layer parameter to satisfy the l0 bound on W′), but is silent in disclosing remove the at least one gradient corresponding to at least one filter from among the plurality of filters, based on the gradient mask; and update weights of one or more remaining filters from among the plurality of filters based on the one or more remaining gradients corresponding to the one or more remaining filters.
Regarding claim 6, the method of claim 1, wherein, for the neural network model comprising at least one convolutional layer, a size of the gradient mask for a first convolutional layer is equal to a number of gradients for updating each filter of a plurality of filters within the first convolutional layer; in combination with the rest of the limitations of claim 1.
Molchanov discloses the method of claim 1, wherein, for the neural network model comprising at least one convolutional layer,
PNG
media_image1.png
117
619
media_image1.png
Greyscale
With regards to claim 7, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the image processing apparatus of claim 1, wherein the memory is further configured to store floating-point (FP) weights, and wherein the one or more instructions further cause the image processing apparatus to: obtain an adjusted plurality of gradients by adjusting a range of the plurality of gradients corresponding to FP data such that the range of the plurality of gradients corresponds to a range of the weights of the neural network model; update the adjusted plurality of gradients to the FP weights; convert the FP weights into integer (INT) weights by quantizing the updated FP weights; and perform a convolution operation on the neural network model, based on the INT weights; in combination with the rest of the limitations of claim 1.
BHALGAT discloses the image processing apparatus of claim 1, wherein the memory is further configured to store (see figure 4, 425 is recomputed after previous iteration); update the adjusted plurality of gradients to the FP weights (see figure 4, 440); but is silent in disclosing convert the FP weights into integer (INT) weights by quantizing the updated FP weights; and perform a convolution operation on the neural network model, based on the INT weights.
With regards to claim 8, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the image processing apparatus of claim 1, wherein the one or more instructions further cause the image processing apparatus to: remove at least one FP gradient of a plurality of FP gradients by applying the gradient mask to the plurality of FP gradients; update some FP weights of the FP weights, based on remaining FP gradients from among the plurality of FP gradients; and convert some of the updated FP weights to INT weights; in combination with the rest of the limitations of claim 1.
BLALGAT discloses the image processing apparatus of claim 1, wherein the one or more instructions further cause the image processing apparatus to: remove at least one tensor gradient of a plurality of tensor gradients by applying the gradient mask to the plurality of tensor gradients (see paragraph 25, a gradient tensor is pruned to remove gradients that fall below a predefined value, magnitude, percentile, or the like); update some tensor weights of the tensor weights, based on remaining tensor gradients from among the plurality of
LO (US 20190347553) discloses calculate an integer gradient value in the forward pass calculations and convert the integer gradient value to a floating point value before performing the updating of the weights, wherein the floating point value is at least a 16-bit value (see paragraph 76), but is silent in disclosing convert some of the updated FP weights to INT weights.
Regarding claim 12, see the rationale and rejection for claim 8.
Regarding claim 15, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the method of claim 13, wherein the obtaining of the plurality of low-resolution training images comprises obtaining a first low-resolution training image having an image quality degraded into a first type of deteriorated image quality and a second low-resolution training image having an image quality degraded into a second type of deteriorated image quality based on the training image, and wherein the training of the neural network model comprises: obtaining a first neural network by applying the training image and the first low-resolution training image to the neural network model and obtaining a second neural network by applying the training image and the second low-resolution training image to the neural network model; calculating a first test loss by applying the test image to the first neural network and calculating a second test loss by applying the test image to the second neural network; and updating the weight of the neural network model based on the first test loss and
the second test loss; in combination with the rest of the limitations of claim 13.
Munkberg discloses the method of claim 13, wherein the obtaining of the plurality of low-resolution training images comprises obtaining a first low-resolution training image having an image quality degraded into a first type of deteriorated image quality
obtaining a first neural network by applying the training image and the first low-resolution training image to the neural network model and obtaining a second neural network by applying the training image and the second low-resolution training image to the neural network model (see figure 1C, low resolution image are employed to train the neural network);
calculating a first test loss by applying the test image to the first neural network
updating the weight of the neural network model based on the first test loss
Regarding claim 19, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): the method of claim 13, wherein, for the neural network model comprising at least one convolutional layer, a size of the gradient mask for a first convolutional layer is equal to a number of gradients for updating each filter of a plurality of filters within the first convolutional layer; in combination with the rest of the limitations of claim 13.
Molchanov discloses the method of claim 13, wherein, for the neural network model comprising at least one convolutional layer,
PNG
media_image1.png
117
619
media_image1.png
Greyscale
Claim(s) 20 is/are objected as well because it is dependent on a claim with allowable subject matter.
CONTACT INFORMATION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEX LIEW (duty station is located in New York City) whose telephone number is (571)272-8623 (FAX 571-273-8623), cell (917)763-1192 or email alexa.liew@uspto.gov. Please note the examiner cannot reply through email unless an internet communication authorization is provided by the applicant. The examiner can be reached anytime.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MISTRY ONEAL R, can be reached on (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEX KOK S LIEW/Primary Examiner, Art Unit 2674 Telephone: 571-272-8623
Date: 6/13/26
Prosecution Insights