Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner's Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Information Disclosure Statement
The information disclosure statement (IDS) was submitted on 03/06th/2024, and 02/28th/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Allowable Subject Matter
Claims 2-18 are objected to as being dependent upon rejected base claim 1. Claims 2-18 would be allowable if claim 2 is rewritten in independent form.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over IOFFE (Batch Normalization Accelerating Deep Network Training by Reducing Covariate Shift - 2015), in view of ZHAO (US20210133535A1).
Regarding claim 1, IOFFE teaches A method performed by one or more data processing apparatus, the method comprising: obtaining a network input; processing the network input using a neural network to generate a network output ([Abstract] Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters).
wherein the neural network includes a normalization block ([Page 2, Section 1] In Sec. 4.2, we apply Batch Normalization to the best performing ImageNet classification network).
that is between a first neural network layer and a second neural network layer in the neural network ([Page 2, Section 2] As each layer observes the inputs produced by the layers below, it would be advantageous to achieve the same whitening of the inputs of each layer)
wherein the normalization block comprises one or more standardization neural network layers ([Page 2, Section 2] consider a layer with the input u that adds the learned bias b, and normalizes the result by subtracting the mean of the activation computed over the training data).
wherein processing the network input using the neural network comprises: receiving a first layer output from the first neural network layer; processing data derived from the first layer output using the standardization neural network layers of the normalization block to generate one or more adaptive standardization values; standardizing the first layer output using the adaptive standardization values to generate a standardized first layer output ([Page 2, Section 2] consider a layer with the input u that adds the learned bias b, and normalizes the result by subtracting the mean of the activation computed over the training data. The examiner notes that IOFFE teaches a neural network layer that received an input u and outputs a normalized output that is normalized using a learned bias).
However, IOFFE is not relied upon to explicitly teach generating a normalization block output from the standardized first layer output. IOFFE is also not relied upon to explicitly teach providing the normalization block output as an input to the second neural network layer.
On the other hand, ZHAO teaches generating a normalization block output from the standardized first layer output ([0035] The first decoder 205 further comprises a first normalization layer 230 that takes as input the input embedding 220, the position embedding 225, and the attention output of the masked multi-head attention block 215, normalizes the inputs across each of the features, and outputs a first normalization ( e.g., compute mean and variance from all of the summed inputs to the neurons in a layer on a single training or deployment case). The examiner notes that IOFFE and ZHAO are both directed to machine learning and are thus considered to be reasonably analogous. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified IOFFE’s learning process to incorporate generating a normalization block output from the standardized first layer output as taught by ZHAO [0035] in order to project the attention outputs potentially giving it a richer representation [0035]).
Furthermore, ZHAO teaches providing the normalization block output as an input to the second neural network layer ([0035] The first decoder 205 further comprises a feed forward network 235 (e.g., used to project the attention outputs potentially giving it a richer representation) that takes as input the first normalization and generates a feed forward output. The examiner notes that IOFFE and ZHAO are both directed to machine learning and are thus considered to be reasonably analogous. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified IOFFE’s learning process to incorporate providing the normalization block output as an input to the second neural network layer as taught by ZHAO [0035] in order to project the attention outputs potentially giving it a richer representation [0035]).
Claims 19 and 20 are rejected under 35 U.S.C 103 based upon the same rationale as claim 1 as they are the system and non-transitory computer storage media claims corresponding to the method claim.
Conclusion
The following reference have been determined to be related to the application, but were not applied in any specific rejection. They are nonetheless listed below for reference.
KOVACHKI (Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks)
“KOVACHKI teaches an efficient, gradient-free algorithm using ensemble Kalman inversion”
IOFFE (US20160217368A1)
“IOFFE teaches a method for processing inputs using a neural network system that includes a batch normalization layer”
HINTON (Reducing the Dimensionality of Data with Neural Networks)
“HINTON teaches an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data”
BIRNBAUM (Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulation)
“BIRNBAUM teaches Temporal Feature-Wise Linear Modulation (TFiLM) - a novel architectural component inspired by adaptive batch normalization and its extensions - that uses a recurrent neural network to alter the activations of a convolutional model”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAMCY ALGHAZZY whose telephone number is (571)272-8824. The examiner can normally be reached Monday-Friday between 9AM and 6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, OMAR FERNANDEZ RIVAS can be reached on (571)272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAMCY ALGHAZZY/Examiner, Art Unit 2128
/KYLE R STORK/Primary Examiner, Art Unit 2128
Prosecution Insights