Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s argument filed 07/22/2026 have been fully considered but they are not persuasive.
Applicant’s Argument: On page 10-11 of Applicant’s response to rejections under 35 U.S.C. 112(b), applicant states the amended claims overcome the rejection because it now recites “the complexity is based on statistics of the first intermediate activation data”.
Examiner’s Response: Applicant’s argument is not persuasive. The amendment does not provide a clear standard for ascertaining the requisite degree of what constitutes as complexity of intermediate activation data. One of ordinary skill in the art would not be reasonably apprised of the scope of the invention because the amended claims does not disclose details on how one can determine the complexity using statistics of the first intermediate activation data. The broad use of the terms “complexity” and “statistics” makes it difficult to interpret the scope of the claims. The amended claims does not clearly define how the use of statistics of intermediate data is used to determine whether the data is complex or not complex.
Applicant’s Argument: On page 11-18 of Applicant’s response to rejections under 35 U.S.C. 101, applicant provides additional remarks that was not already addressed in the Office Action mailed 03/23/2026 in regards to Ex parte Desjardins. Applicant argues the instant application is similar to Desjardins improvement to machine learning by reducing storage space and increasing model efficiency.
Examiner’s Response: Applicant’s argument is not persuasive. In Ex Parte Desjardins, Appeal No. 2024-000567 (PTAB September 26, 2025, Appeals Review Panel Decision), the claimed invention was a method of training a machine learning model on a series of tasks. The Appeals Review Panel (ARP) overall credited benefits including reduced storage, reduced system complexity and streamlining, and preservation of performance attributes associated with earlier tasks during subsequent computational tasks as technological improvements that were disclosed in the patent application specification. Specifically, the ARP upheld the Step 2A Prong One finding that the claims recited an abstract idea (i.e., mathematical concept). In Step 2A Prong Two, the ARP then determined that the specification identified improvements as to how the machine learning model itself operates, including training a machine learning model to learn new tasks while protecting knowledge about previous tasks to overcome the problem of “catastrophic forgetting” encountered in continual learning systems. Importantly, the ARP evaluated the claims as a whole in discerning at least the limitation “adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task” reflected the improvement disclosed in the specification. Accordingly, the claims as a whole integrated what would otherwise be a judicial exception instead into a practical application at Step 2A Prong Two, and therefore the claims were deemed to be outside any specific, enumerated judicial exception.
An important consideration in determining whether a claim improves technology is the extent to which the claim covers a particular solution to a problem or a particular way to achieve a desired outcome, as opposed to merely claiming the idea of a solution or outcome (see MPEP 2106.05(a)). The amended claims do not provide sufficient details to describe any technological improvement. If the specifications explicitly set forth an improvement but in a conclusory manner (see MPEP 2106.04(d)(1): a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art), the examiner should not determine the claim improves technology.
Applicant’s Argument: On page 18-20 of Applicant’s response to rejections under 35 U.S.C. 101, applicant provides additional remarks that was not already addressed in the Office Action mailed 03/23/2026 in regards to CosmoKey. Applicant argues that the amended claims recite a particular arrangement of steps that is directed to a technical improvement.
Examiner’s Response: Applicant’s argument is not persuasive. An important consideration in determining whether a claim improves technology is the extent to which the claim covers a particular solution to a problem or a particular way to achieve a desired outcome, as opposed to merely claiming the idea of a solution or outcome (see MPEP 2106.05(a)). The amended claims do not provide sufficient details to describe any technological improvement. The claim broadly recites the gate making a determination to exit or continue processing data based on complexity that consists of statistics of the intermediate data. There is not sufficient details in the particular steps to conclude that the claim as a whole is directed to a particular technological improvement.
Applicant’s Argument: On page 20-22 of Applicant’s response to rejections under 35 U.S.C. 102, applicant states that Zhang discloses that the determination to exit processing of the model is based on confidence and not complexity.
Examiner’s Response: Applicant’s argument is not persuasive. The Specification (par. 39) describes an embodiment where the gate consists of a threshold comparator and compare the output of the gate model to a decision threshold to determine whether to exit or continue. The output of the gate model can consist of confidence of an intermediate classifier. If the confidence exceeds a threshold, the intermediate classifier outputs a result.
Applicant’s Argument: On page 23-25 of Applicant’s response to rejections under 35 U.S.C. 103, applicant states that the cited references does not teach the amended claim element of “a similarity value”.
Examiner’s Response: Applicant’s argument is not persuasive. Applicant’s arguments with respect to claim 24 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-9, 11, 13-23, 25, 27-29, and 31-33 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “complexity” in claims 1, 15, and 29 is a relative term which renders the claim indefinite. The term “complexity” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. There is no degree to ascertain what constitutes as complex intermediate activation data. The claims need to be amended to provide additional details for what constitutes as complex intermediate data and what does not constitute as complex intermediate activation data. The claims have been amended to recite “wherein the complexity is based on statistics of the first intermediate activation data”. Under the broadest reasonable interpretation, statistics of the first intermediate activation data relates to the process of collecting, analyzing, interpreting, and presenting data. Statistics of the first intermediate data may include making predictions or drawing conclusions based on the intermediate activation data.
The Examiner interprets “complexity” to be defined as a numerical metric of how well the classification model is able to classify an input data such as model accuracy or confidence scores. When a model can accurately classify data, processing can terminate and the output is generated. When a model cannot classify data accurately, additional processing is required to obtain a more accurate output. The determination of complexity can compare the score to a threshold to further determine whether or not to exit processing.
Claims 1, 15, and 29 recites “making a determination by a first gate, based on a complexity of the first intermediate activation data, to exit processing by the classification model, wherein the complexity is based on statistics of the first intermediate activation data; in response to the determination, exiting the processing by the classification model before a final classifier in a plurality of classifiers of the classification model. The claims are unclear and indefinite because the claims fail to disclose what constitutes as exiting the processing step. Does the classification model exit processing when the intermediate activation data is complex or is the exit processing step executed when data is not complex? The Examiner interprets the exiting the processing step to be executed when the determination by the first gate is that the intermediate activation data is not complex. The Specification (par. 23) aligns with this interpretation because less complex input data is handled earlier and does not require additional processing.
Claims 2-9, 11, 13-14, and 31-32 are dependent claims of independent claim 1. Claims 16-23, 25, 27-28, and 33 are dependent claims of independent claim 15. The dependent claims are rejected on the same basis as the parent independent claims.
Claims 6 and 20 recites “processing data input to a second portion of the classification model to generate second intermediate activation data, wherein the second portion of the classification model is before the first portion of the classification model in the directional sequence of model portions”. It is not clear whether the data input from claims 6 and 20 is the same or is it different as the data input from independent claims 1 and 15, respectively. In the Specification (par. 31 and Figure 1), when a gate determines to continue processing, intermediate model data is provided to the next portion of the classification model. Data input is only provided to the beginning of the classification model. Claim 6 recites the section portion of the model before the first portion of the model. Thus, the first portion of the classification model does not process input data, but rather the first portion of the model processes intermediate data from a previous portion of the model. Examiner interprets data input to a second portion as original input data and data input to a first portion as intermediate model data from a previous portion of the model.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 4-9, 11, 13-15, 18-25, 27-29, and 31-34 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites “A processor-implemented method of machine learning, comprising” and is thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
“making a determination wherein the complexity is based on statistics of the first intermediate activation data” (a mental process that can be performed in the human mind, i.e. judgement)
“in response to the determination, exiting the processing by the classification model before processing with a final classifier in a plurality of classifiers of the classification model” (a mental process that can be performed in the human mind, i.e. judgement)
“after exiting the processing, generating a classification result from one of the plurality of classifiers of the classification model” (a mental process that can be performed in the human mind, i.e. evaluation)
Claim 1 therefore recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
“processing data input to a first portion of a classification model to generate first intermediate activation data, the classification model being a machine learning model” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“making a determination by a first gate,
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above. Therefore, Claim 1 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
“processing data input to a first portion of a classification model to generate first intermediate activation data, the classification model being a machine learning model” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“making a determination by a first gate,
The additional elements as disclosed above alone or in combination do not recite significantly more than the abstract idea itself as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above. Therefore, Claim 1 is subject-matter ineligible.
Regarding Claim 15:
The claim recites a system (“A processing system for machine learning”) that performs the method as described in claim 1. Therefore, claim 15 is rejected for the same reasons as disclosed for claim 1. The limitations for additional elements of claim 15 are analyzed below.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Please see Step 2A Prong 1 analysis of claim 1
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“at least one memory comprising computer-executable instructions” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“one or more processors configured to execute the computer-executable instructions and cause the processing system to” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 4 and 18:
Subject Matter Eligibility Analysis Step 2A Prong 1: None
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“processing the first intermediate activation data with a first classifier of the plurality of classifiers to generate the classification result” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 5 and 19:
Subject Matter Eligibility Analysis Step 2A Prong 1: None
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“each of the plurality of classifiers is associated with a model portion” (merely specifies a particular technological environment in which the abstract idea is to take place, ie. a field of use, and thus does not integrate the abstract idea into a practical application nor cannot provide significantly more than the abstract idea itself - see MPEP 2106.05(h))
“the classification model comprises a directional sequence of model portions” (merely specifies a particular technological environment in which the abstract idea is to take place, ie. a field of use, and thus does not integrate the abstract idea into a practical application nor cannot provide significantly more than the abstract idea itself - see MPEP 2106.05(h))
Regarding Claims 6 and 20:
Subject Matter Eligibility Analysis Step 2A Prong 1:
“making a determination based on the second intermediate activation data, to continue processing by the classification model” (a mental process that can be performed in the human mind, i.e. judgement)
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“processing data input to a second portion of the classification model to generate second intermediate activation data, wherein the second portion of the classification model is before the first portion of the classification model in the directional sequence of model portions” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“providing the second intermediate activation data to a second gate” (This step is directed to data gathering, which is understood to be insignificant extra solution activity (2106.05(g) in step 2A prong 2) and well understood, routine and conventional activity of transmitting and receiving data as identified by the court (2106.05(d) in step 2B))
“making a determination by the second gate, ” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 7 and 21:
Subject Matter Eligibility Analysis Step 2A Prong 1: None
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“the input data comprises image data” (merely specifies a particular technological environment in which the abstract idea is to take place, ie. a field of use, and thus does not integrate the abstract idea into a practical application nor cannot provide significantly more than the abstract idea itself - see MPEP 2106.05(h))
“the classification model comprises an image classification model” (merely specifies a particular technological environment in which the abstract idea is to take place, ie. a field of use, and thus does not integrate the abstract idea into a practical application nor cannot provide significantly more than the abstract idea itself - see MPEP 2106.05(h))
Regarding Claims 8 and 22:
Subject Matter Eligibility Analysis Step 2A Prong 1:
“” (a mathematical calculation, see Spec. par. 58)
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“wherein the first gate has been trained ” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 9 and 23:
Subject Matter Eligibility Analysis Step 2A Prong 1: None
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“wherein the first gate comprises a temporal comparison model configured to compare the first intermediate activation data from a current time step to previous intermediate activation data from a previous time step” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 24:
Subject Matter Eligibility Analysis Step 1:
Claim 24 recites “A processing system for machine learning comprising” and is thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
“make a determination value indicating comparison between the first intermediate activation data from a current time step and previous intermediate activation data from a previous time step, time step” (a mental process that can be performed in the human mind, i.e. judgement)
“exiting the processing of the classification model before processing with a final classifier in a plurality of classifiers of the classification model based on the determination by the gate” (a mental process that can be performed in the human mind, i.e. judgement)
“generating a classification result from one of the plurality of classifiers of the classification model” (a mental process that can be performed in the human mind, i.e. judgement)
Claim 24 therefore recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
“at least one memory comprising computer-executable instructions” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“one or more processors configured to execute the computer-executable instructions and cause the processing system to” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“process data input to a portion of a classification model to generate first intermediate activation data, the classification model being a machine learning model” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“make a determination by a gate ” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“output the classification result” (This step is directed to data gathering, which is understood to be insignificant extra solution activity - see MPEP 2106.05(g))
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above. Therefore, Claim 24 is directed to the abstract idea.
Subject Matter Eligibility Analysis Step 2B:
“at least one memory comprising computer-executable instructions” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“one or more processors configured to execute the computer-executable instructions and cause the processing system to” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“process data input to a portion of a classification model to generate first intermediate activation data, the classification model being a machine learning model” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“make a determination by a gate ” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“output the classification result” (This step is directed to transmitting or receiving information, which is understood to be insignificant extra solution activity and well understood, routine and conventional activity of gathering and analyzing information using conventional techniques and displaying the result as identified by the court - see MPEP 2106.05(d))
The additional elements as disclosed above alone or in combination do not recite significantly more than the abstract idea itself as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above. Therefore, Claim 24 is subject-matter ineligible.
Regarding Claims 11 and 25:
Subject Matter Eligibility Analysis Step 2A Prong 1:
“making, before making the determination to exit processing by the classification model, another determination other determination ” (a mental process that can be performed in the human mind, i.e. judgement)
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“making, , another determination by the first gate other determination by the first gate ” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 13 and 27:
Subject Matter Eligibility Analysis Step 2A Prong 1:
“further comprising convolving the first intermediate activation data using one or more convolution layers prior to ” (a mathematical calculation)
“” (a mental process that can be performed in the human mind, i.e. judgement)
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B: None
Regarding Claims 14 and 28:
Subject Matter Eligibility Analysis Step 2A Prong 1: None
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“the input data comprises video data” (merely specifies a particular technological environment in which the abstract idea is to take place, ie. a field of use, and thus does not integrate the abstract idea into a practical application nor cannot provide significantly more than the abstract idea itself - see MPEP 2106.05(h))
“the classification model comprises an video classification model” (merely specifies a particular technological environment in which the abstract idea is to take place, ie. a field of use, and thus does not integrate the abstract idea into a practical application nor cannot provide significantly more than the abstract idea itself - see MPEP 2106.05(h))
Regarding Claim 29:
The claim recites an article of manufacture that performs the method as described in claim 1. Therefore, claim 29 is rejected for the same reasons as disclosed for claim 1. The limitations for additional elements of claim 29 are analyzed below.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Please see Step 2A Prong 1 analysis of claim 1
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“A non-transitory computer-readable medium comprising computer- executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method of machine learning, the method comprising” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claim 31:
Subject Matter Eligibility Analysis Step 2A Prong 1:
“” (a mental process that can be performed in the human mind, i.e. judgement)
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“wherein the first gate comprises a multi-layer perceptron trained to ” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claims 32 and 33:
Subject Matter Eligibility Analysis Step 2A Prong 1:
“” (a mental process that can be performed in the human mind, i.e. judgement)
“making a determination ” (a mental process that can be performed in the human mind, i.e. judgement)
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“processing data input to a second portion of the classification model to generate second intermediate activation data, wherein the second portion of the classification model is before the first portion of the classification model in the directional sequence of model portions” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“providing the second intermediate activation data to a second gate, ” (This step is directed to data gathering, which is understood to be insignificant extra solution activity (2106.05(g) in step 2A prong 2) and well understood, routine and conventional activity of transmitting and receiving data as identified by the court (2106.05(d) in step 2B))
“” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
“making a determination by the second gate, ” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Regarding Claim 34:
Subject Matter Eligibility Analysis Step 2A Prong 1:
“wherein the similarity value is based on a difference between the first intermediate activation data from the current time step to the previous intermediate activation data from the previous time step and ” (a mental process that can be performed in the human mind, i.e. judgement)
Subject Matter Eligibility Analysis Step 2A Prong 2 & 2B:
“” (mere instructions to apply the exception using a generic computer component - see MPEP 2106.05(f))
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 4-7, 13, 15, 18-21, 27, 29, and 31 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Zhang (US20210056357A1).
Regarding claim 1, Zhang teaches:
“A processor-implemented method of machine learning, comprising” ([0015], The system contains a processor to perform the methods of implementing input-adaptive neural networks. Input data is processed using a neural network.)
“processing data input to a first portion of a classification model to generate first intermediate activation data, the classification model being a machine learning model” ([0063-0070, 0105, Figure 15], A CNN can receive images as input data in the input layers. The CNN may include feature-extraction layers that consist of convolution and activations layer to obtain useful features from the input data for processing. A convolution layer applies a convolution operation to the input data and outputs an activation map. The CNN model consists of classification layers that classify the input image into various classes. Figure 15 shows an adaptive and flexible early exit architecture that consists of early exit branches and decision modules. The decision module decides on whether to exit the inference process or continue the inference process.)
“making a determination by a first gate, based on a complexity of the first intermediate activation data, to exit processing by the classification model, wherein the complexity is based on statistics of the first intermediate activation data” ([0083, 0105, 0116, Figure 15], The smaller early exit branch network takes the intermediate features generated by the internal convolutional layers of the base model and transforms them into early predictions. The decision module (first gate) then takes the early prediction results generated by the early exit branch and makes a decision on whether to exit the inference process and output the early prediction results, or to continue the inference process and pass the generated feature maps to the next layer. The early exit branch allows for a model to be flexible and adaptive to the difficulty of the input data. In one embodiment, a video frame is exit if its entropy-based confidence score (complexity) is higher than a pre-determined threshold. Under the broadest reasonable interpretation, the early exit branch takes the intermediate features and analyzes the features to generate early predictions, which is a form of statistics of the first intermediate activation data.)
“in response to the determination, exiting the processing by the classification model before processing with a final classifier in a plurality of classifiers of the classification model” ([0105, 0142-144, Figure 15], The early exit branch is a small-size neural network (classifier) and contains convolutional, activation, pooling, and fully-connected layers. Each early exit consists of an early exit branch and decision modules. A neural network may have one or more early exits. When an easy input data is processed by the neural network, the first early exit point can determine an acceptable level of inference confidence and output the results without further processing until the information reaches the final exit point.)
“after exiting the processing, generating a classification result from one of the plurality of classifiers of the classification model” ([0105, 0117, Figure 15], Multiple early exit branches may be places in the DNN architecture and each early exit branch is a small-size neural network. When the confidence score of the early exit prediction results meets a threshold, the model may exit the inference process and output the early prediction results.)
Regarding claim 15:
Claim 15 recites a system (“A processing system for machine learning”) that performs the same process as described in Claim 1. Therefore claim 15 is rejected under the same reasons mention for claim 1. The additional elements of claim 15 is addressed below:
“at least one memory comprising computer-executable instructions” ([0015], The system consist of a memory having a set of instructions stored.)
“one or more processors configured to execute the computer-executable instructions and cause the processing system to” ([0015], The system includes a processor, at least one data input source, a memory having a set of instructions stored thereon, which when executed by the processor, cause the system to acquire a series of input data units from the data input source, process a first data unit of the series using an input-adaptive neural network.)
Regarding claims 5 and 19, Zhang teaches:
“each of the plurality of classifiers is associated with a model portion” ([0105,0145-0146, Figure 7B], Figure 7B shows early exit 108A receiving input data from base layer 104B and early exit 108B receiving input from a different portion of the model, base layer 104C.)
“the classification model comprises a directional sequence of model portions” ([0105,0143-0146, Figure 7B], A standard data processing flow path 103 represents how the neural network 100 would classify the input 102 without the early exits 108-B, passing data through every base layer 104-D regardless of input, culminating in a final exit output 110. Figure 7B shows a sequential progression of the data from one layer to the next.)
Regarding claims 4 and 18, Zhang teaches:
“further comprising processing the first intermediate activation data with a first classifier of the plurality of classifiers to generate the classification result” ([0105,0143-0144, Figure 7A], The first early exit branch may process the intermediate data from the base layers. The first early exit branch may process the data and output a classification result from the neural network.)
Regarding claims 6 and 20, Zhang teaches:
“processing data input to a second portion of the classification model to generate second intermediate activation data, wherein the second portion of the classification model is before the first portion of the classification model in the directional sequence of model portions” ([0105,0145-0146, Figure 7B], Figure 7B shows that the input data is processed by layer 104A and the data is provided to layer 104B to generate intermediate data. In one embodiment, layer 104B is a second portion of the classification model and layer 104C can be the first portion of the classification model.)
“providing the second intermediate activation data to a second gate” ([0105,0145-0146, Figure 7B], Figure 7B shows layer 104B processing the intermediate data from the previous layer and sending the output (second intermediate activation data) to the early exit point 108A.)
“making a determination by the second gate, based on the second intermediate activation data, to continue processing by the classification model” ([0105, 0145-0146, Figure 7B], When the early exit branch 108A (second gate) receives the second intermediate data from base layer 104B as input, the network will process the information and determine an inference confidence score. In one embodiment, the decision module decides to continue inference and provides the data to layer 104C.)
Regarding claims 7 and 21, Zhang teaches:
“the input data comprises image data” ([0063,0116, Figure 1A], Input data can be image data.)
“the classification model comprises an image classification model” ([0062-0063], The neural network can be convolutional neural network for image analysis.)
Regarding claims 13 and 27, Zhang teaches:
“further comprising convolving the first intermediate activation data using one or more convolution layers prior to making the determination based on the complexity of the first intermediate activation data” ([0069-0070, 0105], The base model consist of convolutional layers that processes the input data and aggregate the outputs prior to providing the intermediate data to the early exit architecture. The smaller early exit branch network takes the intermediate features generated by the internal convolutional layers of the base model and transforms them into early predictions.)
Regarding claim 29:
Claim 29 recites a program product that performs the same process as described in Claim 1. Therefore claim 29 is rejected under the same reasons mention for claim 1. The additional elements of claim 29 is addressed below:
“A non-transitory computer-readable medium comprising computer- executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method of machine learning, the method comprising” ([0015], The system includes a processor, at least one data input source, a memory having a set of instructions stored thereon, which when executed by the processor, cause the system to acquire a series of input data units from the data input source, process a first data unit of the series using an input-adaptive neural network.)
Regarding claim 31, Zhang teaches:
“wherein the first gate comprises a multi-layer perceptron trained to estimate the complexity of the first intermediate activation data” ([0089,0105], The fully connected layers may be feed-forward multi-layer perceptron layers. The early exit branch may consist of fuller connected layers. The decision module takes the output of the early exit branch to make the determination of exit or continue inference.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 8, 16-17, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US20210056357A1) in view of Herrmann, "An End-To-End Approach For Speeding Up Neural Network Inference".
Regarding claims 2 and 16, Zhang teaches:
“wherein the first gate comprises: a pooling layer configured to reduce a dimensionality of the first intermediate activation data” ([0086-0088,0105], The early exit branch may consist of pooling layers to reduce the spatial size of the representation.)
“one or more neural network layers configured to generate the determination of whether or not to exit processing by the classification model” ([0105], The early exit branch may consist of convolutional, activation, pooling, and fully-connected layers to generate early predictions. The decision module then takes the early prediction results generated by the early exit branch and makes a decision on whether to exit the inference process and output the early prediction results.)
“a ” ([0116], Zhang does not explicitly disclose a Gumbel distribution, but Zhang does disclose using a softmax classification probability distribution to determine the confidence score of an early exit.)
Zhang does not explicitly disclose an implementation of “a Gumbel sampling component”. However, Herrmann discloses in the same field of endeavor:
“a Gumbel sampling component” ([pg. 9, section 5, par. 1-2], The model uses Gumbel distribution to make a decision as to whether the instance can be successfully classified at that stage and whether an early exit is necessary at the intermediate classifier.)
It would be obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of a “Gumbel sampling component” from Herrmann into the teaching of Zhang. Doing so can improve the inference speed of a neural network by using Gumbel reparameterization (Herrmann, abstract).
Regarding claims 3 and 17, Zhang teaches:
“wherein the one or more neural network layers comprise a plurality of multi-layer perceptron layers” ([0089,0105], The fully connected layers may be feed-forward multi-layer perceptron layers. The early exit branch may consist of fuller connected layers.)
Regarding claims 8 and 22, Zhang teaches:
“wherein the first gate has been trained using a ” ([0111-0112, 0131, 0135], The early exit branch may consist of a loss function to minimize the model’s error. The system may modify the model’s accuracy to account for resource demands and availability.)
Zhang does not explicitly disclose an implementation of “a batch-shaping loss function”. However, Herrmann discloses in the same field of endeavor:
“wherein the first gate has been trained using a batch-shaping loss function to minimize classification error and to minimize processing resource usage” ([pg. 3, section 3, par. 1-2], The approach uses a per-batch activation loss function.)
It would be obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of a “a batch-shaping loss function” from Herrmann into the teaching of Zhang. Doing so can improve the inference speed of a neural network by using batch activation loss function (Herrmann, abstract).
Claims 9, 11, 14, 23, 25, 28, and 32-33 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US20210056357A1) in view of Wu, "LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition".
Regarding claims 9 and 23, Zhang teaches:
“wherein the first gate comprises a ” ([0025, 0105], The early exit branch may consist of a neural network to process the intermediate features from the internal convolutional layers of the base model.)
Zhang does not explicitly disclose an implementation of “wherein the first gate comprises a temporal comparison model configured to compare the first intermediate activation data from a current time step to previous intermediate activation data from a previous time step”. However, Wu discloses in the same field of endeavor:
“wherein the first gate comprises a temporal comparison model configured to compare the first intermediate activation data from a current time step to previous intermediate activation data from a previous time step” ([pg. 3, section 2.1, par. 1-3, Figure 1], The system consists of gating modules and temporal modeling that processes and computes the features of the incoming video frames (current time step) and features from a previous time step. The framework consists of a coarse and a fine LSTM, which is a recurrent neural network that compares current input data with previous hidden states to process sequence.)
It would be obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of a “wherein the first gate comprises a temporal comparison model configured to compare the first intermediate activation data from a current time step to previous intermediate activation data from a previous time step” from Wu into the teaching of Zhang. Doing so can improve the computation efficiency of the classification model by implementing the method of adjusting the computing power for an incoming video frame (Wu, abstract).
Regarding claims 11 and 25, Zhang teaches:
“making, before making the determination to exit processing by the classification model, another determination by the first gate based on other determination by the first gate comprises a determination to continue processing by the classification model” ([0105], The decision module makes a decision on whether to exit the inference process of the classification model or not based on processing the intermediate features of the internal convolutional layers of the base model. The decision module may continue the inference process and pass the generated feature maps to the next layer.)
Zhang does not explicitly disclose an implementation of “making … another determination by the first gate based on a dissimilarity of the first intermediate activation data from the current time step to previous intermediate activation data from the previous time step”. However, Wu discloses in the same field of endeavor:
“making, before making the determination to exit processing by the classification model, another determination by the first gate based on a dissimilarity of the first intermediate activation data from the current time step to previous intermediate activation data from the previous time step, wherein the other determination by the first gate comprises a determination to continue processing by the classification model” ([pg. 2, par. 1-2; pg. 3, section 2, par. 1-4, Figure 1 & 4], The gating module computes a probability and based on the computed probability, determines whether to process the fine features of the video frame or skip the processing step. The system decides to process the fine features of the frame in order to make better predictions of the label, such as distinguishing “making latte” from “making “cappuccino. The framework is to learn a policy that determines at each time step whether an input video frame contains discriminative features by computing current and historical features. The claims do not explicitly recite additional information to limit the scope of what constitutes as a dissimilarity between intermediate activation data. Under the broadest reasonable interpretation, the LSTM proposed by Zhang teaches dissimilarity because the LSTM analyze sequences of image features to determine spatial similarity. When distinct features are determined, the gating module outputs a high probability to compute fine features that requires additional processing.)
It would be obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of a “making … another determination by the first gate based on a dissimilarity of the first intermediate activation data from the current time step to previous intermediate activation data from the previous time step, wherein the other determination by the first gate comprises a determination to continue processing by the classification model” from Wu into the teaching of Zhang. Doing so can improve the computation efficiency of the classification model by implementing the method of adjusting the computing power for an incoming video frame (Wu, abstract).
Regarding claims 14 and 28, Zhang teaches:
“the input data comprises video data” ([0058], Video streams can be processed by the system.)
“the classification model comprises a video classification model” ([0062-0063, 0151], The CNN can be used for video analysis such as labeling of human activities in videos.)
Regarding claims 32 and 33, Zhang in view of Wu teaches:
“processing data input to a second portion of the classification model to generate second intermediate activation data, wherein the second portion of the classification model is before the first portion of the classification model in the directional sequence of model portions” ([Zhang, 0105,0145-0146, Figure 7B], Figure 7B shows that the input data is processed by layer 104A and the data is provided to layer 104B to generate intermediate data. In one embodiment, layer 104B is a second portion of the classification model and layer 104C can be the first portion of the classification model.)
“providing the second intermediate activation data to a second gate, wherein the second gate comprises a temporal comparison model configured to compare the second intermediate activation data from a current time step to previous intermediate activation data from a previous time step” ([Zhang, 0025, 0105], The early exit branch may consist of a neural network to process the intermediate features from the internal convolutional layers of the base model. [Wu, pg. 3, section 2.1, par. 1-3, Figure 1], The system consists of gating modules and temporal modeling that processes and computes the features of the incoming video frames (current time step) and features from a previous time step. The framework consists of a coarse and a fine LSTM, which is a recurrent neural network that compares current input data with previous hidden states to process sequence.)
“making a determination by the second gate, based on a dissimilarity of the second intermediate activation data from the current time step to the previous intermediate activation data from the previous time step, to continue processing by the classification model” ([Zhang, 0105, 0116], The decision module makes a decision on whether to exit the inference process of the classification model or not based on processing the intermediate features of the internal convolutional layers of the base model. The early exit branch is a small-size neural network. The decision module compares the confidence level of the early exit branch to a threshold to determine if an early exit is appropriate. [Wu, pg. 2, par. 1-2; pg. 3, section 2, par. 1-4, Figure 1 & 4], The gating module computes a probability and based on the computed probability, determines whether to process the fine features of the video frame or skip the processing step. Under the broadest reasonable interpretation, the LSTM proposed by Zhang teaches dissimilarity because the LSTM analyze sequences of image features to determine spatial similarity. When distinct features are determined, the gating module outputs a high probability to compute fine features that requires additional processing.)
Claims 24, and 34 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US20210056357A1) in view of Wu, "LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition" and Hou, “Time Series Similarity Measure via Siamese Convolutional Neural Network”.
Regarding claim 24, Zhang teaches:
“A processing system for machine learning, comprising: at least one memory comprising computer-executable instructions” ([0015], The system consist of a memory having a set of instructions stored to perform the methods of implementing input-adaptive neural networks. Input data is processed using a neural network.)
“one or more processors configured to execute the computer-executable instructions and cause the processing system to” ([0015], The system includes a processor, at least one data input source, a memory having a set of instructions stored thereon, which when executed by the processor, cause the system to acquire a series of input data units from the data input source, process a first data unit of the series using an input-adaptive neural network.)
“processing data input to a portion of a classification model to generate first intermediate activation data, the classification model being a machine learning model” ([0063-0070, 0105, Figure 15], A CNN can receive images as input data in the input layers. The CNN may include feature-extraction layers that consist of convolution and activations layer to obtain useful features from the input data for processing. A convolution layer applies a convolution operation to the input data and outputs an activation map. The CNN model consists of classification layers that classify the input image into various classes.)
“make a determination by a gate to exit processing of the classification model based on ” ([0105, 0116], The decision module makes a decision on whether to exit the inference process of the classification model or not based on processing the intermediate features of the internal convolutional layers of the base model. The early exit branch is a small-size neural network. The decision module compares the confidence level of the early exit branch to a threshold to determine if an early exit is appropriate.)
“exiting the processing of the classification model before processing with a final classifier in a plurality of classifiers of the classification model based on the determination by the gate” ([0105, 0142-144, Figure 15], Each early exit consists of an early exit branch and decision modules. A neural network may have one or more early exits. When an easy input data is processed by the neural network, the first early exit point can determine an acceptable level of inference confidence and output the results without further processing until the information reaches the final exit point.)
“generating a classification result from one of the plurality of classifiers of the classification model” ([0105, 0117, Figure 15], Multiple early exit branches may be places in the DNN architecture and each early exit branch is a small-size neural network. When the confidence score of the early exit prediction results meets a threshold, the model may exit the inference process and output the early prediction results.)
“output the classification result” ([0144, Figure 7A], The early exit branch may determine it has already reached an acceptable level of inference confidence and output the results of the classification model. Figure 7A shows the early exit output 112.)
Zhang does not explicitly disclose an implementation of “make a determination by a gate to exit processing of the classification model based on a similarity value indicating comparison between the first intermediate activation data from a current time step and previous intermediate activation data from a previous time step, wherein the gate comprises a temporal comparison model configured to compare the first intermediate activation data from the current time step to the previous intermediate activation data from the previous time step”. However, Wu discloses in the same field of endeavor:
“make a determination by a gate to the current time step to the previous intermediate activation data from the previous time step” ([pg. 2, par. 1-2; pg. 3, section 2, par. 1-4, Figure 1 & 4], The gating module computes a probability and based on the computed probability, determines whether to process the fine features of the video frame or skip the processing step. The framework is to learn a policy that determines at each time step whether an input video frame contains discriminative features by computing current and historical features. Under the broadest reasonable interpretation, the LSTM proposed by Zhang teaches similarity because the LSTM analyze sequences of image features to determine spatial similarity.)
It would be obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of a “make a determination by a gate to exit processing of the classification model based on a similarity value indicating comparison between the first intermediate activation data from a current time step and previous intermediate activation data from a previous time step, wherein the gate comprises a temporal comparison model configured to compare the first intermediate activation data from the current time step to the previous intermediate activation data from the previous time step” from Wu into the teaching of Zhang. Doing so can improve the computation efficiency of the classification model by implementing the method of adjusting the computing power for an incoming video frame (Wu, abstract).
Zhang in view of Wu does not explicitly disclose an implementation of “… a similarity value indicating comparison between the first intermediate activation data from a current time step and previous intermediate activation data from a previous time step …”. However, Hou discloses in the same field of endeavor:
“… processing of the classification model based on a similarity value indicating comparison between the first intermediate activation data from a current time step and previous intermediate activation data from a previous time step, …” ([pg. 1, Section 1, par. 3; pg. 3, section C, par. 1, pg. 3, Figure 1], A Siamese convolutional neural network consists of two CNN-based branches, which aims to extract the essential features of two time series inputs. The L1 norm is employed to calculate the distance of the output features and a similarity value of the features are generated based on Equation 3. The model uses the computed similarity value to generate a prediction.)
It would be obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of a “… a similarity value indicating comparison between the first intermediate activation data from a current time step and previous intermediate activation data from a previous time step …” from Hou into the teaching of Zhang in view of Wu. Doing so can improve the accuracy for time series classification task by implementing a framework that determines similarity in extracted features of time series data (Hou, abstract).
Regarding claim 34, Zhang in view of Wu and Hou teaches:
“wherein the similarity value is based on a difference between the first intermediate activation data from the current time step to the previous intermediate activation data from the previous time step and wherein the gate is configured to make the determination to exit when the difference is below a predefined threshold” ([Zhang, 0105, 0116], The decision module can decide to early exit the input frame if the confidence exceeds a predetermined threshold. A high confidence value indicates that the model estimates a strong probability that the classification output is correct. [Hou, pg. 2, Section A, par. 1, pg. 3, section C, par. 1, pg. 3, Figure 1], The similarity is computed between two time series input. Features are similar when the difference of the features are small and features that are dissimilar have a large distance. The similarity is computed using Equation 3. Time series input that have similar features would tend to produce the same output.)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GARY MAC whose telephone number is (703)756-1517. The examiner can normally be reached Monday - Friday 8:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached on (571) 270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GARY MAC/Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127
