Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/23/2026 has been entered.
DETAILED ACTION
Claims 1-20 are presented for examination in this application (17940159) filed
September 8, 2022. The Examiner cites particular sections in the references as applied to the claims below for the convenience of the applicant(s). Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant(s) fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner.
Response to Arguments
Applicant’s arguments and remarks filed 04/23/2026 have been fully considered. The arguments and remarks regarding the 35 U.S.C 101 and 103 rejections were not found to be persuasive. The arguments and remarks made regarding the 35 U.S.C 103 rejections were found to be persuasive and have been withdrawn, however a new ground of rejection, necessitated by amendment, has been made for the claims that were rejected under 35 U.S.C 103.
35 U.S.C 101
Applicant asserts:
Applicant asserts “Under Step 2A, Prong 2, the claims integrate any alleged abstract idea into a practical application by reciting a specific improvement to the operation of a neural network, rather than merely performing data analysis or selection.
In particular, the claims require that the activation function is configured to control feed- forward propagation at a fully connected dense layer such that outputs of nodes: preserve one or more statistical properties of the input data, and are constrained within an interval corresponding to the input data, thereby modifying how signals are transformed and propagated across interconnected neurons during execution of the neural network. This is not merely a "selection" step or mental process. Rather, the claims recite a specific mechanism that governs how data is transformed within the neural network during forward propagation, resulting in data-dependent constraints on activation behavior that improve the internal functioning of the model. In particular, the claims recite that, when the neural network is executed, the activation function actively transforms node outputs to preserve statistical properties and constrain outputs within a defined interval. This is a machine-implemented transformation applied across interconnected neurons during forward propagation, which cannot practically be performed in the human mind and is fundamentally different from the mental association described by the Examiner.”
Examiner’s response:
Examiner respectfully disagrees. The examiner notes that the process of selecting an activation function based on a determination of properties, is a mental process that can be practically performed within the human mind. A person having ordinary skill in the art could find that the properties of data such as skewness, kurtosis, and range-boundedness could be associated with specific activation functions and once an identification of the property has been made, a person having ordinary skill in the art can make the determination of which specific activation function to use for that input. The applicant’s response of 2A prong 2 and step 2B pertains to the elements of selecting activation functions based on properties of the input data which have been deemed abstract ideas. The examiner notes that the inventive concept cannot come from the abstract idea itself as noted in MPEP 2106.05 I. “An inventive concept "cannot be furnished by the unpatentable law of nature (or natural phenomenon or abstract idea) itself." Genetic Techs. Ltd. v. Merial LLC, 818 F.3d 1369, 1376, 118 USPQ2d 1541, 1546 (Fed. Cir. 2016).”. Adding limitations that characterize the type of neural network or type of general propagation does not, in this case, does not take away from being practically performed by the human mind, as the selection of activation functions based on observations of input data can be practically performed by the human mind.
35 U.S.C 103
Applicant’s response:
Applicant asserts “Applicant disagrees with the rejection for at least the reason that the references relied upon in the Office Action, even if properly combined, do not teach or suggest each and every feature of the claimed invention. Nonetheless, solely to expedite prosecution of this application, the claims have been amended to clarify aspects of the claimed invention.” “The Examiner's rationale effectively requires modifying Teder such that, based on historical input data, the system would generate or adapt a custom activation function in the manner described by Jie. This is materially different from the claimed invention. By incorporating Jie's approach into Teder, the resulting system would no longer be selecting an activation function in the manner required by the claims, but instead would be developing or adapting a new activation function. This replaces Teder's use of predefined activation functions with a fundamentally different mechanism for determining activation behavior. Such a modification would change the principle of operation of Teder, transforming it from a system that applies predefined activation functions within a fixed architecture into one that constructs or adapts activation functions based on data. A person of ordinary skill in the art would not make such a modification absent impermissible hindsight, as it requires rearchitecting Teder's neural network to introduce a different paradigm for activation function determination and application. Accordingly, the proposed combination is both improper and insufficient to render the claims obvious. For at least this reason, independent claims 1, 9 and 15, and their dependent claims are allowable over the references relied upon in the Office Action. Moreover, even if such a modification were proper, the combined references would still fail to teach or suggest the claimed invention. Neither Teder nor Jie discloses configuring an activation function at a fully connected dense layer such that, during feed-forward propagation, outputs preserve statistical properties of the input data and are constrained within a data-derived interval. These recitations relate to how the activation function operates within the neural network during execution, and are not taught or suggested by the references relied upon in the Office Action. None of the references relied upon in the Office Action cure at least these deficiencies. For at least this additional reason, independent claims 1, 9 and 15, and their dependent claims are allowable over the references relied upon in the Office Action.”.
Examiner’s response:
Arguments regarding the amended limitations are considered but are moot in view of the new grounds of rejections.
Information Disclosure Statement
Acknowledgement is made of the information disclosure statement filed on 01/28/2026.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C 101 because the claimed invention is directed
to an abstract idea without significantly more. The analysis of the claims will follow the
2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57
(January 7, 2019) (“2019 PEG”).
Regarding claim 1 (currently amended):
Step 1 – Is the claim directed to a process, machine, manufacture, or composition of matter?
Yes, the claim is directed to a system.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites abstract ideas:
identify one or more properties of historical data relating to the input data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
select an activation function for a neural network based on the one or more properties, the activation function controlling data that is fed forward in the neural network — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
wherein the activation function is selected to ensure that: outputs of nodes at a fully connected dense layer in the neural network are fed forward to preserve the one or more properties of the input data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
and constrain outputs of the nodes within an interval of the input data during feed-forward propagation through the fully connected dense layer— this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
generate a prediction for the input data based on the executed neural network with the activation function at the fully connected dense layer of the neural network — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
a system of identifying and using an activation function of a neural network based on input data, the system comprising a processor — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
execute the neural network with the activation function at a fully connected dense layer of the neural network — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
wherein, when the neural network is executed, the selected activation function is configured to: (i) feed forward outputs of the nodes that preserve the one or more properties if the input data, and (ii) constrain the outputs of the nodes within an interval if the input data — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
the neural network being trained on the historical data — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
transmit for display data indicating the prediction — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception. Any additional elements that were determined to be insignificant extra-solution activity in step 2A prong 2 are further evaluated in step 2B on whether they are well-understood, routine, and conventional activities. The “transmit for display data indicating the prediction” limitation was found to be an insignificant extra solution activity in claim 1. This limitation is recited at a high level of generality and amount to transmitting data over a network, which are well-understood, routine, and conventional activities (see MPEP 2106.05(d) II.). (MPEP 2106.05(f)) cannot integrate the abstract ideas into a practical application.
Regarding claim 2:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 1 which recited abstract ideas. The claim recites additional abstract ideas:
compare the one or more properties to a threshold value — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
select the activation function based on whether the one or more properties exceeds the threshold value — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein to select the activation the processor is programmed — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 3 (currently amended):
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 1 which recited abstract ideas. The claim recites additional abstract ideas:
to identify the one or more properties, the processor is further programmed to identify skewness, kurtosis and/or range boundedness of the input data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 4:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 1 which recited abstract ideas. The claim recites additional abstract ideas:
select a Rectified Linear Unit (ReLU) activation function when the one or more properties include a skew in the input data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 5:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 1 which recited abstract ideas. The claim recites additional abstract ideas:
wherein the processor is further programmed to: select a Sigmoid activation function when the one or more properties includes a range boundedness or quasi-range boundedness of the input data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 6:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 1 which recited abstract ideas. The claim recites additional abstract ideas:
wherein the processor is further programmed to: select a second activation function to be executed in a layer of the neural network adjacent to the selected activation at the fully connected layer — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 7:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 1 which recited abstract ideas. The claim recites additional abstract ideas:
wherein the selected activation function and the second activation function are different from one another — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 8 (currently amended):
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 1 which recited abstract ideas.
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein the input data comprises a time series of data values — this limitation is directed to the field of use (see MPEP 2106.05(h) VI.) as it merely relates the machine learning model’s training to power consumption.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of using a generic computer components to perform the abstract idea amounts to no more than field of use to apply the exception. Generally linking the use of a judicial exception to a particular technological environment or field of use cannot provide an inventive concept. Thus the claim is not patent eligible.
Regarding claim 9 (currently amended):
Step 1 – Is the claim directed to a process, machine, manufacture, or composition of matter?
Yes, the claim is directed to a method.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites abstract ideas:
identifying, by a processor, one or more properties of historical data relating to the input data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
selecting, by the processor, a first activation function for execution at a first layer that is a fully connected dense layer in a neural network to preserve the one or more properties, the first activation function controlling data that is fed forward in the neural network at the first layer of the neural network at which the first activation function executes — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
selecting, by the processor, a second activation function for the neural network based on the one or more properties, the second activation function controlling data that is fed forward in the neural network at a second layer of the neural network at which the second activation function executes — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
generating, by the processor, a prediction for the input data based on the executed neural network — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
executing, by the processor, the neural network with the first activation function at the first activation at the first layer and the second activation function at the second layer — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
wherein, when the neural network is executed, the selected activation function is configured to: (i) feed forward outputs of the nodes that preserve the one or more properties if the input data, and (ii) constrain the outputs of the nodes within an interval if the input data — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
transmitting, by the processor, for display data indicating the prediction — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception. Any additional elements that were determined to be insignificant extra-solution activity in step 2A prong 2 are further evaluated in step 2B on whether they are well-understood, routine, and conventional activities. The “transmitting, by the processor, for display data indicating the prediction” limitation was found to be an insignificant extra solution activity in claim 9. This limitation is recited at a high level of generality and amount to transmitting data over a network, which are well-understood, routine, and conventional activities (see MPEP 2106.05(d) II.). (MPEP 2106.05(f)) cannot integrate the abstract ideas into a practical application.
Regarding claim 10:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 9 which recited abstract ideas.
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein the first layer and the second layer are adjacent to one another — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 11:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 9 which recited abstract ideas.
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein executing the neural network comprises: executing the neural network with the second activation function at a fully connected dense layer — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 12:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 9 which recited abstract ideas.
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein the second activation function comprises a Rectified Linear Unit (ReLU) activation function — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 13:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 9 which recited abstract ideas.
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein executing the neural network comprises: executing the neural network with the second activation function at a layer that is adjacent to the fully connected dense layer — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 14:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 9 which recited abstract ideas.
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein the first activation function comprises a sigmoid activation function and the second activation function comprises a Rectified Linear Unit (ReLU) activation function — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 15 (currently amended):
Step 1 – Is the claim directed to a process, machine, manufacture, or composition of matter?
Yes, the claim is directed to a manufacture.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites abstract ideas:
identify one or more properties of historical data relating to input data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
select an activation function for a neural network based on the one or more properties, the activation function controlling data that is fed forward in the neural network — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
the learned data to be used in the neural network to make a prediction based on the stored data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
wherein, when the neural network is executed, the selected activation function is configured to: (i) feed forward outputs of nodes that preserve the one or more properties if the input data, and (ii) constrain the outputs of the nodes within an interval if the input data — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
A non-transitory storage medium storing instructions that, when executed by a processor, programs the processor to — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
train, based on the historical data, the neural network with the activation function at a fully connected dense layer of the neural network — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
store learned data, which was learned during training — this limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception. Any additional elements that were determined to be insignificant extra-solution activity in step 2A prong 2 are further evaluated in step 2B on whether they are well-understood, routine, and conventional activities. The “transmit for display data indicating the prediction” limitation was found to be an insignificant extra solution activity in claim 15. This limitation is recited at a high level of generality and amount to transmitting data over a network, which are well-understood, routine, and conventional activities (see MPEP 2106.05(d) II.). (MPEP 2106.05(f)) cannot integrate the abstract ideas into a practical application.
Regarding claim 16:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 15 which recited abstract ideas.
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim recites additional elements that do not integrate the judicial exception into a practical application:
wherein the learned data comprises weights learned at each node of the neural network — the process of classifying and organizing data amounts to mere instructions to apply an exception, as the use of a computer or other machinery in its machinery in its ordinary capacity amounts to invoking computer components merely as a tool to perform an existing process (see MPEP 2106.05(f)(2)).
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 17:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 15 which recited abstract ideas. The claim recites additional abstract ideas:
select a Rectified Linear Unit (ReLU) activation function when the one or more properties include a skew in the historical data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 18:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 15 which recited abstract ideas. The claim recites additional abstract ideas:
select a Sigmoid activation function when the one or more properties includes range boundedness or quasi-range boundedness of the historical data — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 19:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 15 which recited abstract ideas. The claim recites additional abstract ideas:
select a second activation function adjacent to be executed in a layer of the neural network adjacent to the selected activation function at the fully connected layer — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 20:
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim is dependent on claim 15 which recited abstract ideas. The claim recites additional abstract ideas:
wherein the selected activation function and the second activation function are different from one another — this limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, there are no additional elements that amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3, 5-6, 9-11, 13, 15-16, and 18-19 are rejected under 35 U.S.C 102 as being unpatentable under Mayer et al. (US20210287089A1 hereinafter, Mayer).
Regarding claim 1:
Mayer teaches a system identifying and using an activation function of a neural network based on input data, the system comprising (see para [0007]: “The systems and methods also utilize techniques for designing and constructing neural network models, for example, to select appropriate model architectures, loss functions, and activation functions (e.g., output activation functions). Additional techniques are presented for determining appropriate values for hyperparameters, which can be used to control the neural network training process.”):
a processor programmed to (see para [0014]: “In another aspect, the subject matter described in this specification can be embodied in a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors to perform operations”):
identify one or more properties of historical data relating to the input data (see para [0062]: “As used herein, “time-series data” may refer to data collected at different points in time. For example, in a time-series data set, each data sample may include the values of one or more variables sampled at a particular time. In some embodiments, the times corresponding to the data samples are stored within the data samples (e.g., as variable values) or stored as metadata associated with the data set.”);
select an activation function for a neural network based on the one or more properties (see para [0007]: “The systems and methods also utilize techniques for designing and constructing neural network models, for example, to select appropriate model architectures, loss functions, and activation functions (e.g., output activation functions). Additional techniques are presented for determining appropriate values for hyperparameters, which can be used to control the neural network training process. For example, values for hyperparameters can be determined automatically based on one or more training data characteristics and/or on a type of modeling problem to be solved (e.g., regression or classification)”)
the activation function controlling data that is fed forward in the neural network, wherein the activation function is selected such that: outputs of nodes at a fully connected dense layer in the neural network are fed forward to preserve the one or more properties of the input data, and constrain outputs of the nodes within an interval of the input data during feed-forward propagation through the fully connected dense layer (see para [0080]: “For example, the training processes can repeatedly take a small batch of data (e.g., a mini-batch of training data), calculate a difference between predictions and actuals, and adjust weights (e.g., parameters within a neural network that transform input data within each of the network's hidden layers) in the model by a small amount, layer by layer, to generate predictions closer to actual values. Neural network models are flexible and allow for inclusion or composition of arbitrary functions. A universal approximation theorem states that feed-forward networks with a finite number of neurons (also referred to as “width”) can approximate any continuous function and can do so with a single-layer. For example, networks using a rectified linear activation function (ReLU) can approximate any continuous function with n-dimensional input variables using a single hidden layer of width (e.g., number of neurons) n+4.”. Also see para [0079]: “For example, the two layers can be fully connected with each neuron in one layer connected to each neuron in the other layer, as depicted.”)
execute the neural network with the activation function at a fully connected dense layer of the neural network, wherein, when the neural network is executed, the selected activation function is configured to: (i) feed forward outputs of the nodes that preserve the one or more properties of the input data (see para [0080]: “For example, the training processes can repeatedly take a small batch of data (e.g., a mini-batch of training data), calculate a difference between predictions and actuals, and adjust weights (e.g., parameters within a neural network that transform input data within each of the network's hidden layers) in the model by a small amount, layer by layer, to generate predictions closer to actual values. Neural network models are flexible and allow for inclusion or composition of arbitrary functions. A universal approximation theorem states that feed-forward networks with a finite number of neurons (also referred to as “width”) can approximate any continuous function and can do so with a single-layer. For example, networks using a rectified linear activation function (ReLU) can approximate any continuous function with n-dimensional input variables using a single hidden layer of width (e.g., number of neurons) n+4.”)
(ii) constrain the outputs of the nodes within an interval of the input data (see para [0079]: “In general, the activation function of a node can define a range for the output of the node, for a given input or set of inputs.”)
the neural network being trained on the historical data (see para [0063]: “Time-series data may be useful for tracking and inferring changes in the data set over time. In some cases, a time-series data analytics model (or “time-series model”) may be trained and used to predict the values of a target Z at time t and optionally times t+1, . . . , t+i, given observations of Z at times before t and optionally observations of other predictor variables P at times before t.”)
generate a prediction for the input data based on the executed neural network with the activation function at the fully connected dense layer of the neural network (see para [0063]: “Time-series data may be useful for tracking and inferring changes in the data set over time. In some cases, a time-series data analytics model (or “time-series model”) may be trained and used to predict the values of a target Z at time t and optionally times t+1, . . . , t+i, given observations of Z at times before t and optionally observations of other predictor variables P at times before t.”) and
transmit, for display, data indicating the prediction (see para [0170]: “To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the compute”. Also see para [0063]: “Time-series data may be useful for tracking and inferring changes in the data set over time. In some cases, a time-series data analytics model (or “time-series model”) may be trained and used to predict the values of a target Z at time t and optionally times t+1, . . . , t+i, given observations of Z at times before t and optionally observations of other predictor variables P at times before t. ”).
Regarding claim 9:
Claim 9 recites analogous limitations to claim 1 and therefore is rejected on the same grounds as claim 1.
Regarding claim 15:
Claim 15 recites analogous limitations to claim 1 and therefore is rejected on the same grounds as claim 1. Claim 15, however, additionally adds elements of a non-transitory storage medium. Mayer further teaches the non-transitory storage medium (see para [0014]: “In another aspect, the subject matter described in this specification can be embodied in a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors to perform operations”).
Regarding claim 3:
Mayer teaches the system of claim 1.
Mayer further teaches wherein to identify the one or more properties, the processor is further programmed to identify skewness, kurtosis and/or range boundedness of the input data (see para [0017]: “Determining the one or more second hyperparameters can include determining the output activation function, and for regression problems the output activation function can be determined to be (i) an exponential function when the training data includes skewed targets and a loss function utilizes a Poisson distribution, a gamma distribution, or a Tweedie distribution or (ii) a linear function.”. See para [0017]: “Determining the one or more second hyperparameters can include determining the output activation function, and for classification problems the output activation function can be determined to be (i) a sigmoid function for binary classification problems or independent multiclass problems or (ii) a softmax function for mutually exclusive multiclass classification problems.”)
Regarding claim 5:
Mayer teaches the system of claim 3.
Mayer further teaches wherein the processor is further programmed to: select a Sigmoid activation function when the one or more properties includes range boundedness or quasi-range boundedness of the input data (see para [0017]: “Determining the one or more second hyperparameters can include determining the output activation function, and for classification problems the output activation function can be determined to be (i) a sigmoid function for binary classification problems or independent multiclass problems or (ii) a softmax function for mutually exclusive multiclass classification problems.”).
Regarding claim 18:
Claim 18 recites analogous limitations to claim 5 and therefore is rejected on the same grounds as claim 5.
Regarding claim 6:
Mayer teaches the system of claim 3.
Mayer further teaches wherein the processor is further programmed to select a second activation function to be executed in a layer of the neural network adjacent to the selected activation function at the fully connected layer (see para [0130]: “In various examples, “hidden activation” can be or include an activation function that follows one or more hidden layers in the neural network. Hidden activation can be used to introduce non-linearity, such that the network can learn non-linear patterns and/or utilize non-linear functions”. Also see fig. 5).
Regarding claim 11 and 19:
Claims 11 and 19 recite analogous limitations to claim 6 and therefore are rejected on the same grounds.
Regarding claim 10:
Mayer teaches the method of claim 9.
Mayer further teaches wherein the first layer and the second layer are adjacent to one another (see para [0076]: “FIG. 1 is a schematic diagram of an exemplary neural network 100, in accordance with certain examples. The neural network 100 can include an input layer 110, a first hidden layer 120, a second hidden layer 130, and an output layer 140. Each of these layers can further include neurons or nodes 150 connected to other nodes from adjacent layers via connections 160 (also referred to as “edges”).”. Also see fig. 1).
Regarding claim 13:
Mayer teaches the method of claim 9.
Mayer further teaches executing the neural network with the second activation function at a layer that is adjacent to the fully connected dense layer (see figs. 4 and 5 which show hidden layers that are fully connected , indicated by fig. 1, that have at least a first and second activation function adjacent to fully connected dense layers)
Regarding claim 16:
Mayer teaches the non-transitory storage medium of claim 15.
Mayer further teaches wherein the learned data comprises weights learned at each node of the neural network (see para [0078]: “In various examples, each edge or connection 160 in the neural network 100 can be associated with a weight and/or bias that can be tuned during a neural network training process, which can enable the model to “learn” to recognize patterns that may be present in the input data 170. In general, a weight for a connection 160 between two neurons can increase or decrease a “strength” (e.g., a contribution) for the connection 160. The weights can control how sensitive the network's predictions are to various features included in the input data 170.”).
Claims 2, 7, 8, 13, and 20 are rejected under 35 U.S.C 103 as being unpatentable under Mayer et al. (US20210287089A1 hereinafter, Mayer) in view of Marchisio et al. (“A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networks” hereinafter, Marchisio).
Regarding claim 2:
Mayer teaches the system of claim 1.
Mayer does not explicitly teach wherein to select the activation function, the processor is programmed to: compare the one or more properties to a threshold value; and select the activation function based on whether the one or more properties exceeds the threshold value.
Marchisio, however, analogously teaches wherein to select the activation function, the processor is programmed to: compare the one or more properties to a threshold value (see pg. 6 algorithm 1, specifically lines 15-16 which show when the accuracy is achieved using the activation f on a layer then that activation function from the library of activation functions is chosen) and
select the activation function based on whether the one or more properties exceeds the threshold value (see pg. 6 algorithm 1, specifically lines 15-16 which show when the accuracy is achieved using the activation f on a layer then that activation function from the library of activation functions is chosen).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art, having the teachings of Mayer and Marchisio before him or her, to modify the system of claim 2 to include attributes of wherein to select the activation function, the processor is programmed to: compare the one or more properties to a threshold value; and select the activation function based on whether the one or more properties exceeds the threshold value in order to improve accuracy of a deep neural network (see Marchisio at pg. 5 section 4.1: “Hence, our methodology, at the very first stage, focuses on an efficient way to extract useful information from the learning curve (i.e., the curve that describes the accuracy of the DNN as a function of the number of epochs) to obtain the Evaluation Point. Then, for each layer, we find the best combination (that produces the maximum test accuracy) of the activation function and the dropout rate. A layer-wise search is efficient for (1) improving the DNN accuracy and (2) not penalizing the computation efficiency, while using parallel processing and SIMD instructions in GPUs.”)).
Regarding claim 7:
Mayer teaches the system of claim 6.
Mayer does not explicitly teach wherein the selected activation function and the second activation function are different from one another.
Marchisio, however, analogously teaches wherein the selected activation function and the second activation function are different from one another (see fig. 1 which shows several activation functions that have been chosen from different activation functions in a library of activation functions).
Before the effective filing date of the claimed invention it would have been obvious to one of ordinary skill in the art, having the teachings of Mayer and Marchisio before him or her, to modify the system of claim 7 to include attributes of wherein the selected activation function and the second activation function are different from one another in order to avoid the overfitting problem using a dropout rate and automatically select activation functions per each layer of a deep neural network (see pg. 5 section 4: “We propose a simple yet effective methodology to automatically select the activation functions for each layer of a given DNN as well as its associated dropout rate, based on the accuracy obtained at the Evaluation Point.”).
Regarding claim 20:
Claim 20 recites analogous limitations to claim 7 and therefore is rejected on the same grounds as claim 5.
Regarding claim 8:
Mayer in view of Marchisio teaches the system of claim 7.
Mayer further teaches wherein the input data comprises a time series of data values (see para [0062]: “As used herein, “time-series data” may refer to data collected at different points in time. For example, in a time-series data set, each data sample may include the values of one or more variables sampled at a particular time. In some embodiments, the times corresponding to the data samples are stored within the data samples (e.g., as variable values) or stored as metadata associated with the data set.”).
Allowable Subject Matter
Claims 4, 12, 14, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims provided 101 rejections are overcome. Regarding claims 4, 12, 14, and 17, the closest prior art of record to the limitations of the aforementioned claims is to Mayer (US20210287089A1). Mayer teaches an exponential function, not reLU, in response to skewed input. Additionally, Mayer teaches reLU and sigmoid activation functions but not necessarily together in the same neural network.
The examiner has found that the distinct features of the applicant’s claimed invention over the prior art is the explicit claiming of the aforementioned limitations specified in claims 4, 12, 14, and 17. When viewed individually or in combination with other prior art of record, the limitations specified in claims 4, 12, 14, and 17 are distinct
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew A Bracero whose telephone number is (571)270-0592. The examiner can normally be reached Monday - Friday 9:00a.m. - 5:00 p.m. ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached Monday - Thursday 7:30a.m. - 5:00 p.m. ET at (571)270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW BRACERO/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights