Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed on May 2, 2024.
Claims 1-20 are pending.
Specification
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc.
The abstract of the disclosure is objected to because it uses phrases which can be implied and repeats information given in the title. For example, “Described herein are techniques for stacking machine learning models to better capture deterministic relations in a dataset” and “Techniques are described for…” A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-7 are directed to a method [process].
Claims 8-14 are directed to a system [machine].
Claims 15-20 are directed to a non-transitory computer-readable medium [machine].
Regarding Claim 1:
Step 2A, Prong 1: The following limitations are directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
determining that the training dataset has one or more additional deterministic relations not yet captured by the first machine learning model
determining whether the stacked machine learning model has captured more deterministic relations in the training dataset than the first trained machine learning model
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to integrate the judicial exception into a practical application.
training the second machine learning model to capture the one or more additional deterministic relations not yet captured
returning the stacked machine learning model when the stacked machine learning model has captured more deterministic relations in the training dataset than the first trained machine learning model
The following additional elements are directed to insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(g)].
receiving a first trained machine learning model, the first machine learning model having been previously trained to capture a plurality of deterministic relations within a training dataset containing a plurality of entries, each entry including a plurality of input variables and a plurality of output variables
requesting a second machine learning model in response to the determination
receiving the second machine learning model in response to the request
stacking the trained first machine learning model and the trained second machine learning model to form a stacked machine learning model
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to amount to significantly more than the judicial exception.
training the second machine learning model to capture the one or more additional deterministic relations not yet captured
returning the stacked machine learning model when the stacked machine learning model has captured more deterministic relations in the training dataset than the first trained machine learning model
The following additional elements are directed to receiving or transmitting data over a network. The courts have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
receiving a first trained machine learning model, the first machine learning model having been previously trained to capture a plurality of deterministic relations within a training dataset containing a plurality of entries, each entry including a plurality of input variables and a plurality of output variables
requesting a second machine learning model in response to the determination
receiving the second machine learning model in response to the request
The following additional elements are directed to stacking machine learning models at a high level of generality. Shafran et al. (US 20240338611), hereinafter Shafran, discloses in [0003] “Conventional approaches for combining the output predictions of each of the trained models to produce an optimal output prediction include weighted averaging and stacking…stacking refers to a technique which uses an additional machine learning model to learn how to best combine the output predictions from contributing ensemble members (e.g., the trained models) to generate the final output prediction” Shafran has recognized stacking machine learning models as a well-understood, routine, and conventional activity previously known in the industry [see MPEP 2106.05(d)].
stacking the trained first machine learning model and the trained second machine learning model to form a stacked machine learning model
Regarding Claim 2:
Step 2A, Prong 1: This claim recites the same abstract ideas as in the parent claim. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
for each entry in the training dataset:
(a) …generate a plurality of predicted outputs, each of the plurality of predicted outputs associated with one of the plurality of output variables from the training dataset
generating a plurality of first deviations, each first deviation generated by comparing one of the plurality of predicted outputs and its associated output variable
determining that there is stochastic dependence between the plurality of input variables in the training dataset and the plurality of first deviations
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to integrate the judicial exception into a practical application.
(a) …providing the plurality of input variables as input to the first machine learning model to…
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to amount to significantly more than the judicial exception.
(a) …providing the plurality of input variables as input to the first machine learning model to…
Regarding Claim 3:
Step 2A, Prong 1: This claim recites the same abstract ideas as in the parent claim. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein the plurality of first deviations models the difference between the actual distribution over classes and the plurality of predicted outputs given the plurality of input variables
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 4:
Step 2A, Prong 1: This claim recites the same abstract ideas as in the parent claim. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein generating the plurality of first deviations includes calculating the difference between an output variable and the predicted output associated with the output variable when the output variable is ordinal data or continuous data
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 2A, Prong 1: This claim recites the same abstract ideas as in the parent claim. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein generating the plurality of first deviations includes setting the deviation to a value zero when output variable is nominal data and the predicted output associated with the output variable accurately predicts the output variable
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 6:
Step 2A, Prong 1: This claim recites the same abstract ideas as in the parent claim. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein generating the plurality of first deviations includes setting the first deviation to a value representative of the output variable when the output variable is nominal data and the predicted output associated with the output variable inaccurately predicts the output variable
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 7:
Step 2A, Prong 1: This claim recites the same abstract ideas as in the parent claim.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to integrate the judicial exception into a practical application.
wherein training the second machine learning model comprises applying the plurality of input variables from the training dataset as input to the second machine learning model and applying the plurality of first deviations as output to the second machine learning model
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to amount to significantly more than the judicial exception.
wherein training the second machine learning model comprises applying the plurality of input variables from the training dataset as input to the second machine learning model and applying the plurality of first deviations as output to the second machine learning model
Regarding Claim 8:
Claim 8 corresponds to claim 1.
Step 2A, Prong 1: This claim recites the same abstract ideas as in the corresponding claim.
Step 2A, Prong 2: This claim recites the same additional elements as in the corresponding claim. There are no additional elements in this claim that integrate the judicial exception into a practical application. The analysis of this claim at this step mirror that of corresponding claim, with the addition/exception of the following limitations.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to integrate the judicial exception into a practical application.
A system comprising: one or more processors; a non-transitory computer-readable medium storing a program executable by the one or more processors, the program comprising sets of instructions for:
Step 2B: This claim recites the same additional elements as in the corresponding claim. There are no additional elements in this claim that amount to significantly more than the judicial exception. The analysis of this claim at this step mirror that of claim corresponding, with the addition/exception of the following limitations.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to amount to significantly more than the judicial exception.
A system comprising: one or more processors; a non-transitory computer-readable medium storing a program executable by the one or more processors, the program comprising sets of instructions for:
Regarding Claims 9-14:
Claims 9-14 correspond to claims 2-7. In particular, 9:2, 10:3, 11:4, 12:5, 13:6, 14:7.
Step 2A, Prong 1: Claims 9-14 recite the same abstract ideas as in claims 2-7.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application. The analysis of claims 9-14 at this step mirror that of claims 2-7.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception. The analysis of claims 9-14 at this step mirror that of claims 2-7.
Regarding Claim 15:
Claim 15 corresponds to claim 1.
Step 2A, Prong 1: This claim recites the same abstract ideas as in the corresponding claim.
Step 2A, Prong 2: This claim recites the same additional elements as in the corresponding claim. There are no additional elements in this claim that integrate the judicial exception into a practical application. The analysis of this claim at this step mirror that of corresponding claim, with the addition/exception of the following limitations.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to integrate the judicial exception into a practical application.
A non-transitory computer-readable medium storing a program executable by one or more processors, the program comprising sets of instructions for:
Step 2B: This claim recites the same additional elements as in the corresponding claim. There are no additional elements in this claim that amount to significantly more than the judicial exception. The analysis of this claim at this step mirror that of claim corresponding, with the addition/exception of the following limitations.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)], and therefore fails to amount to significantly more than the judicial exception.
A non-transitory computer-readable medium storing a program executable by one or more processors, the program comprising sets of instructions for:
Regarding Claims 16-20:
Claims 16-20 correspond to claims 2-6. In particular, 16:2, 17:3, 18:4, 19:5, 20:6.
Step 2A, Prong 1: Claims 16-20 recite the same abstract ideas as in claims 2-6.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application. The analysis of claims 16-20 at this step mirror that of claims 2-6.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception. The analysis of claims 16-20 at this step mirror that of claims 2-6.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 8, and 15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Kiraly et al. (US 20260228561), hereinafter Kiraly.
Regarding Claim 1:
Kiraly discloses:
A method, comprising: receiving a first trained machine learning model, the first machine learning model having been previously trained to capture a plurality of deterministic relations within a training dataset containing a plurality of entries, each entry including a plurality of input variables and a plurality of output variables
Kiraly, [0006], “Another example aspect of the present disclosure is directed to a computer-implemented method for deterministic inference, the method performed by a computing system comprising one or more computing devices. The method includes obtaining input data and one or more seed values associated with the input data. The method includes accessing a machine-learned model, wherein the machine-learned model is configured to variably apply one or more of a plurality of different processing operations when processing an input. The method includes processing the input data and the one or more seed values with the machine-learned model to generate one or more inferences from the input data, wherein the one or more seed values deterministically control which of the plurality of different processing operations are applied by the machine-learned model when processing the input data.”
[0103], “In some implementations, the one or more machine-learned models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112.”
In para. 6, Kiraly discloses a machine learning model. Para. 103 further specifies the machine learning models can be received from a server [receiving a first trained machine learning model]. Back in para. 6, Kiraly disclosed that the machine learning models are able to deterministically generate inferences from input data [the first machine learning model having been previously trained to capture a plurality of deterministic relations within a training dataset containing a plurality of entries, each entry including a plurality of input variables and a plurality of output variables].
determining that the training dataset has one or more additional deterministic relations not yet captured by the first machine learning model
Kiraly, [0076], “Stacking: In stacking, a second machine learning model is trained to make a final inference based on the intermediate inferences made by the first model. This can be useful if the intermediate inferences contain valuable information that is not captured by the final inference made by the first model.”
Kiraly discloses using stacking if intermediate inferences contain valuable information that is not captured by the first inference made by the first model [determining that the training dataset has one or more additional deterministic relations not yet captured by the first machine learning model]. As discussed above, Kiraly discloses working with deterministic machine learning models, and is interpreted as models capturing deterministic relations.
requesting a second machine learning model in response to the determination
As cited above within para. 76, Kiraly discloses training a second model if the first model’s inference has not captured valuable information. The addition of a second model is interpreted as requesting a second model [requesting a second machine learning model in response to the determination].
receiving the second machine learning model in response to the request
As cited above within para. 103, Kiraly discloses receiving their machine learning models from a server [receiving the second machine learning model in response to the request].
training the second machine learning model to capture the one or more additional deterministic relations not yet captured
As cited above within para. 76, Kiraly discloses training the second model using the first model’s inference to capture information that was not captured by the first model [training the second machine learning model to capture the one or more additional deterministic relations not yet captured].
stacking the trained first machine learning model and the trained second machine learning model to form a stacked machine learning model
As cited above within para. 76, Kiraly discloses stacking the second machine learning model with the first machine learning model [stacking the trained first machine learning model and the trained second machine learning model to form a stacked machine learning model].
determining whether the stacked machine learning model has captured more deterministic relations in the training dataset than the first trained machine learning model
Kiraly, [0076], “Stacking: In stacking, a second machine learning model is trained to make a final inference based on the intermediate inferences made by the first model. This can be useful if the intermediate inferences contain valuable information that is not captured by the final inference made by the first model.”
[0077], “By aggregating or combining the multiple intermediate inferences 320a-n in these or other ways, it is possible to generate the aggregated inference 322 that is typically more accurate or reliable than any individual intermediate inference.”
In para. 76, Kiraly discloses stacking the second machine learning model with the first machine learning model to capture valuable information that was not captured by the first machine learning model [determining whether the stacked machine learning model has captured more deterministic relations in the training dataset than the first trained machine learning model].
returning the stacked machine learning model when the stacked machine learning model has captured more deterministic relations in the training dataset than the first trained machine learning model
As cited above within para. 76, Kiraly discloses stacking the second machine learning model with the first machine learning model to capture valuable information that was not captured by the first machine learning model. Para. 77 further discloses aggregating or combining the stacked machine learning model inferences generates a more accurate or reliable aggregated inference as opposed to the individual intermediate inference [returning the stacked machine learning model when the stacked machine learning model has captured more deterministic relations in the training dataset than the first trained machine learning model].
Regarding Claim 8:
Claim 8 corresponds to claim 1 and is rejected for at least the same reasons as given in the rejection of claim 1, with the exception of the following limitations.
Kiraly discloses:
A system comprising: one or more processors; a non-transitory computer-readable medium storing a program executable by the one or more processors, the program comprising sets of instructions for:
Kiraly, [0005], “One example aspect of the present disclosure is directed to a computer system that performs deterministic inference for a machine-learned model with variable behavior. The computer system includes one or more processors and one or more non-transitory computer-readable media that collectively store: the machine-learned model, wherein the machine-learned model is configured to variably apply one or more of a plurality of different processing operations when processing an input; and instructions that, when executed by the one or more processors, cause the computer system to perform inference-time operations.”
Regarding Claim 15:
Claim 15 corresponds to claim 1 and is rejected for at least the same reasons as given in the rejection of claim 1, with the exception of the following limitations.
Kiraly discloses:
A non-transitory computer-readable medium storing a program executable by one or more processors, the program comprising sets of instructions for:
Kiraly, [0005], “One example aspect of the present disclosure is directed to a computer system that performs deterministic inference for a machine-learned model with variable behavior. The computer system includes one or more processors and one or more non-transitory computer-readable media that collectively store: the machine-learned model, wherein the machine-learned model is configured to variably apply one or more of a plurality of different processing operations when processing an input; and instructions that, when executed by the one or more processors, cause the computer system to perform inference-time operations.”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2-4, 9-11, and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Kiraly in view of Beregovaya et al. (US 20260057195), hereinafter Beregovaya, and further in view of Rajarathinam et al. (US 20200202179), hereinafter Rajarathinam.
Regarding Claim 2:
As discussed above, Kiraly teaches [the] method as in claim 1, but does not explicitly disclose:
wherein determining that the training dataset has one or more additional deterministic relations not yet captured by the first machine learning model comprises: for each entry in the training dataset: providing the plurality of input variables as input to the first machine learning model to generate a plurality of predicted outputs, each of the plurality of predicted outputs associated with one of the plurality of output variables from the training dataset
generating a plurality of first deviations, each first deviation generated by comparing one of the plurality of predicted outputs and its associated output variable
determining that there is stochastic dependence between the plurality of input variables in the training dataset and the plurality of first deviations
However, in the same field, analogous art Beregovaya teaches:
wherein determining that the training dataset has one or more additional deterministic relations not yet captured by the first machine learning model comprises: for each entry in the training dataset: providing the plurality of input variables as input to the first machine learning model to generate a plurality of predicted outputs, each of the plurality of predicted outputs associated with one of the plurality of output variables from the training dataset
Beregovaya, [0102], “the set of machine learning models may be trained by the set of supervised machine learning algorithms based on mutual information between the set of linguistic features identified in the set of unstructured texts and the set of user engagement analytic parameters measured for the set of unstructured texts to correlate how the set of linguistic features identified in the set of unstructured texts is predicted to impact the set of user engagement analytic parameters.”
Beregovaya teaches a set of linguistic features for the machine learning models [for each entry in the training dataset: providing the plurality of input variables as input to the first machine learning model] correlated with a set of user engagement analytic parameters measured for the set of unstructured tests [to generate a plurality of predicted outputs, each of the plurality of predicted outputs associated with one of the plurality of output variables from the training dataset].
determining that there is stochastic dependence between the plurality of input variables in the training dataset and the plurality of first deviations
Beregovaya, [0130], “the application program 202 (or another suitable logic running on the computing instance 104) performs feature selection by identifying importance of each feature in machine learning algorithms and removing (or ignoring) unnecessary features…For example, the engineer user profile may run a script (e.g., Python, Perl) with a library (e.g., a FeatureWiz library) on the application program 202 (or another suitable logic running on the computing instance 104) to find (a) all the pairs of highly correlated variables exceeding a correlation threshold such as 0.75 or (b) a mutual information score (MIS) of each feature to a target variable.”
Beregovaya teaches determining if a feature is highly correlated to a target variable [determining that there is stochastic dependence between the plurality of input variables in the training dataset and the plurality of first deviations].
Kiraly, Beregovaya, and the instant application are analogous art because they are all directed to deterministic machine learning models.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly with Beregovaya to determine correlation in order to increase the model’s robustness. “The grade correlates how the set of linguistic features identified in the unstructured text is predicted to impact the set of user engagement analytic parameters for the unstructured text” (Beregovaya, [0107]). Beregovaya teaches correlation directly grades how much of an impact certain features have a target variable.
Kiraly in view of Beregovaya do not explicitly disclose:
generating a plurality of first deviations, each first deviation generated by comparing one of the plurality of predicted outputs and its associated output variable
However, in the same field, analogous art Rajarathinam teaches:
generating a plurality of first deviations, each first deviation generated by comparing one of the plurality of predicted outputs and its associated output variable
Rajarathinam, [0005], “The logic circuitry may perform residual modeling on each model in the set during the monitoring period, to determine a list of input features that contribute to a residual for each model of the set to two or more models. The residual may comprise a difference between a result predicted by each model and an expected result.”
Rajarathinam teaches a difference between a result predicted by each model and an expected result [generating a plurality of first deviations, each first deviation generated by comparing one of the plurality of predicted outputs and its associated output variable].
Kiraly, Beregovaya, Rajarathinam, and the instant application are analogous art because they are all directed to deterministic machine learning models.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly and Beregovaya with Rajarathinam in order to increase the machine learning model’s accuracy. “Identification of features that contribute to the residuals can describe key features associated with the breakdown of models. The description of the key features associated with the breakdown of models may facilitate creation of a new model that accounts for these key features and/or identify features to target with corrective measures to improve or repair existing models” (Rajarathinam, [0019]).
Regarding Claim 3:
As discussed above, Kiraly in view of Beregovaya, further in view of Rajarathinam teach [the] method as in claim 2, and Rajarathinam further discloses:
wherein the plurality of first deviations models the difference between the actual distribution over classes and the plurality of predicted outputs given the plurality of input variables
Rajarathinam, [0005], “The logic circuitry may perform residual modeling on each model in the set during the monitoring period, to determine a list of input features that contribute to a residual for each model of the set to two or more models. The residual may comprise a difference between a result predicted by each model and an expected result.”
As discussed above in claim 2, Rajarathinam teaches determining a residual which is a difference between a result predicted by each model and an expected result [the plurality of first deviations]. This is performed for the entire list of input features, the results predicted by the model, and the expected results [models the difference between the actual distribution over classes and the plurality of predicted outputs given the plurality of input variables].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly and Beregovaya with Rajarathinam in order to increase the machine learning model’s accuracy. “Identification of features that contribute to the residuals can describe key features associated with the breakdown of models. The description of the key features associated with the breakdown of models may facilitate creation of a new model that accounts for these key features and/or identify features to target with corrective measures to improve or repair existing models” (Rajarathinam, [0019]).
Regarding Claim 4:
As discussed above, Kiraly in view of Beregovaya, further in view of Rajarathinam teach [the] method as in claim 2, and Kiraly further discloses:
…when the output variable is ordinal data or continuous data
Kiraly, [0036] and [0039], “There are many different types of input data 16 that the machine learned model 12 can process, including:…Time series data: Time series data can include sequential data points collected over time. This can include data such as inventory data, weather measurements, traffic patterns, or medical images of the same patient over the course of time.”
Rajarathinam further discloses:
wherein generating the plurality of first deviations includes calculating the difference between an output variable and the predicted output associated with the output variable…
Rajarathinam, [0005], “The logic circuitry may perform residual modeling on each model in the set during the monitoring period, to determine a list of input features that contribute to a residual for each model of the set to two or more models. The residual may comprise a difference between a result predicted by each model and an expected result.”
Rajarathinam teaches a difference between a result predicted by each model and an expected result [generating the plurality of first deviations includes calculating the difference between an output variable and the predicted output associated with the output variable].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly and Beregovaya with Rajarathinam in order to increase the machine learning model’s accuracy. “Identification of features that contribute to the residuals can describe key features associated with the breakdown of models. The description of the key features associated with the breakdown of models may facilitate creation of a new model that accounts for these key features and/or identify features to target with corrective measures to improve or repair existing models” (Rajarathinam, [0019]).
Regarding Claim 9:
Claim 9 corresponds to claim 2 and is rejected for at least the same reasons as given in the rejection of claim 2.
Regarding Claim 10:
Claim 10 corresponds to claim 3 and is rejected for at least the same reasons as given in the rejection of claim 3.
Regarding Claim 11:
Claim 11 corresponds to claim 4 and is rejected for at least the same reasons as given in the rejection of claim 4.
Regarding Claim 16:
Claim 16 corresponds to claim 2 and is rejected for at least the same reasons as given in the rejection of claim 2.
Regarding Claim 17:
Claim 17 corresponds to claim 3 and is rejected for at least the same reasons as given in the rejection of claim 3.
Regarding Claim 18:
Claim 18 corresponds to claim 4 and is rejected for at least the same reasons as given in the rejection of claim 4.
Claims 5-6, 12-13, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kiraly and Beregovaya in view of Rajarathinam as applied to claims 2, 8, and 16 above, respectively, and further in view of Ciampiconi et al. (“A Survey of Taxonomy of Loss Functions in Machine Learning”, 2023).
Regarding Claim 5:
As discussed above, Kiraly in view of Beregovaya, further in view of Rajarathinam teach [the] method as in claim 2, and Kiraly further discloses:
…when output variable is nominal data…
Kiraly, [0036] and [0038], “There are many different types of input data 16 that the machine learned model 12 can process, including:… Categorical data: Categorical data can include discrete values that can be used to classify data points into categories. This includes data such as names, labels, or categories like ‘red’ or ‘blue.’”
Kiraly in view of Beregovaya, further in view of Rajarathinam do not explicitly disclose:
wherein generating the plurality of first deviations includes setting the deviation to a value zero when…the predicted output associated with the output variable accurately predicts the output variable
However, in the same field, analogous art Ciampiconi teaches:
wherein generating the plurality of first deviations includes setting the deviation to a value zero when…the predicted output associated with the output variable accurately predicts the output variable
Ciampiconi, p. 12, Margin Based Loss Functions, “Zero-One loss. The basic and more intuitive margin-based classification loss is the Zero-One loss. It assigns 1 to a misclassified observation and a 0 to a correctly classified one.”
Kiraly, Beregovaya, Rajaranthinam, Ciampiconi, and the instant application are analogous art because they are all directed to machine learning.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly, Beregovaya, and Rajaranthinam with Ciampiconi to set the deviation value to zero in view of accurate predictions in order to create a simple and intuitive loss function. “Zero-One loss. The basic and more intuitive margin-based classification loss is the Zero-One loss. It assigns 1 to a misclassified observation and a 0 to a correctly classified one” (Ciampiconi, p. 12).
Regarding Claim 6:
As discussed above, Kiraly in view of Beregovaya, further in view of Rajarathinam teach [the] method as in claim 2, and Kiraly further discloses:
…when output variable is nominal data…
Kiraly, [0036] and [0038], “There are many different types of input data 16 that the machine learned model 12 can process, including:… Categorical data: Categorical data can include discrete values that can be used to classify data points into categories. This includes data such as names, labels, or categories like ‘red’ or ‘blue.’”
Kiraly in view of Beregovaya, further in view of Rajarathinam do not explicitly disclose:
wherein generating the plurality of first deviations includes setting the first deviation to a value representative of the output variable when…the predicted output associated with the output variable inaccurately predicts the output variable
However, in the same field, analogous art Ciampiconi teaches:
wherein generating the plurality of first deviations includes setting the first deviation to a value representative of the output variable when…the predicted output associated with the output variable inaccurately predicts the output variable
Ciampiconi, p. 12, Margin Based Loss Functions, “Zero-One loss. The basic and more intuitive margin-based classification loss is the Zero-One loss. It assigns 1 to a misclassified observation and a 0 to a correctly classified one.”
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly, Beregovaya, and Rajaranthinam with Ciampiconi to set the deviation value to one in view of inaccurate predictions in order to create a simple and intuitive loss function. “Zero-One loss. The basic and more intuitive margin-based classification loss is the Zero-One loss. It assigns 1 to a misclassified observation and a 0 to a correctly classified one” (Ciampiconi, p. 12).
Regarding Claim 12:
Claim 12 corresponds to claim 5 and is rejected for at least the same reasons as given in the rejection of claim 5.
Regarding Claim 13:
Claim 13 corresponds to claim 6 and is rejected for at least the same reasons as given in the rejection of claim 6.
Regarding Claim 19:
Claim 19 corresponds to claim 5 and is rejected for at least the same reasons as given in the rejection of claim 5.
Regarding Claim 20:
Claim 20 corresponds to claim 6 and is rejected for at least the same reasons as given in the rejection of claim 6.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Kiraly and Beregovaya in view of Rajaranthinam as applied to claims 2 and 9 above, respectively, and further in view of Waegeman et al. (“Multi-Target Prediction: A Unifying View on Problems and Methods”).
Regarding Claim 7:
As discussed above, Kiraly in view of Beregovaya, further in view of Rajarathinam teach [the] method as in claim 2, and Rajarathinam further discloses:
applying the plurality of first deviations as output to the second machine learning model
Rajarathinam, [0005], “The logic circuitry may perform residual modeling on each model in the set during the monitoring period, to determine a list of input features that contribute to a residual for each model of the set to two or more models. The residual may comprise a difference between a result predicted by each model and an expected result.”
Rajarathinam teaches a difference between a result predicted by each model and an expected result [applying the plurality of first deviations as output to the second machine learning model].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly and Beregovaya with Rajarathinam in order to increase the machine learning model’s accuracy. “Identification of features that contribute to the residuals can describe key features associated with the breakdown of models. The description of the key features associated with the breakdown of models may facilitate creation of a new model that accounts for these key features and/or identify features to target with corrective measures to improve or repair existing models” (Rajarathinam, [0019]).
Kiraly in view of Beregovaya, further in view of Rajarathinam do not explicitly disclose:
wherein training the second machine learning model comprises applying the plurality of input variables from the training dataset as input to the second machine learning model and
However, in the same field, analogous art Waegeman teaches:
wherein training the second machine learning model comprises applying the plurality of input variables from the training dataset as input to the second machine learning model and
Waegeman, p. 308, “In addition, the second-level model b can be trained on the first-level predictions h(x) alone or in concatenation with the original features x.”
Waegeman teaches training the second-level model [training the second machine learning model] using the first-level predictions and the original features x [applying the plurality of input variables from the training dataset as input to the second machine learning model].
Kiraly, Beregovaya, Rajarathinam, Waegeman, and the instant application are analogous art because they are all directed to machine learning.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kiraly, Beregovaya, and Rajarathinam with Waegeman to apply the input variables from the training dataset as input to the second machine learning model in order to capture dependencies. “Obviously, these second-order models are able to capture dependencies between different targets, as they seek to represent the prediction of one target as a function of the (predictions of) the others. Thus, the goal of the second step is to discover dependencies among targets, and to identify, for each target in turn, a set of other targets that help improve predictions” (Waegeman, p. 308).
Regarding Claim 14:
Claim 14 corresponds to claim 7 and is rejected for at least the same reasons as given in the rejection of claim 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN PHUNG whose telephone number is (703) 756-1499. The examiner can normally be reached Monday-Thursday: 9:00AM-4:00PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KAMRAN AFSHAR can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEVEN PHUNG/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights