DETAILED ACTION
1. This communication is in response to the amendments filed on March 11, 2026 for Application No. 18/335,450 in which Claims 1-6, 8-13, 15-18, and 20 are presented for examination.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
3. The amendments filed on March 11, 2026 have been considered. Claims 1-3, 8, 15-16, and 20 have been amended. Claims 7, 14, and 19 have been cancelled. Thus, claims 1-6, 8-13, 15-18, and 20 are pending and presented for examination.
4. Applicant's arguments filed March 11, 2026 with respect to the 35 U.S.C. 101 rejection have been fully considered but they are not persuasive.
Applicant’s Arguments on Pg. 9 of Arguments/Remarks state:
“The Office Action alleges the claims are directed toward an abstract mental process without a practical application, or significantly more than, the alleged abstract idea. As an initial matter, Applicant disagrees with the characterization of the claims as an abstract mental process. It is well settled that claims do not recite a mental process when they do not contain limitations that can practically be performed in the human mind, for instance when the human mind is not equipped to perform the claim limitations. See SRIInt'l, Inc. v. Cisco Systems, Inc., 930 F.3d 1295, 1304 (Fed. Cir. 2019) (declining to identify the claimed collection and analysis of network data as abstract because "the human mind is not equipped to detect suspicious activity by using network monitors and analyzing network packets as recited by the claims"); CyberSource, 654 F.3d at 1376, 99 USPQ2d at 1699 (distinguishing Research Corp. Techs. v. Microsoft Corp., 627 F.3d 859, 97 USPQ2d 1274 (Fed. Cir. 2010), and SiRF Tech., Inc. v. Int'l Trade Comm'n, 601 F.3d 1319, 94 USPQ2d 1607 (Fed. Cir. 2010), as directed to inventions that "could not, as a practical matter, be performed entirely in a human's mind"). In the case of the instant claims, the human mind is not equipped to perform the claimed limitations, and as such the claims are not abstract.
Additionally, the claims include features that form a practical application of, or significantly more than, any alleged abstract idea. Specifically, the independent claims now recite the additional elements of "updating, based on the output signal, one or more parameters of the ML model", which, when read in combination with the limitations of claim 1, illustrates an improvement to the technical field of machine learning.”
Examiner respectfully disagrees. At Step 2A Prong 1, the limitations “interpreting, by the requirements management layer, the at least one requirement” and “updating, based on the output signal, one or more parameters of the ML model” may still be feasibly performed by mental process. A user is capable of observing/analyzing already received requirements (pass criteria, test target, file path of test data, or a test condition) and accordingly using judgement/evaluation to interpret the at least one requirement in context of the system. Further, a user may update one or more parameters of the ML model by observing/analyzing the output signal (simply comprising evaluated metrics of the machine learning model) and accordingly using judgement/evaluation to update (with the aid of pen and paper) one or more parameters of the ML model based on said analysis. There are no technical details presented which preclude these limitations from being practically performed by mental process and no supposed technical improvement is reflected in the currently drafted claim language. Although Applicant cites SRI Int’l, Inc. v. Cisco Systems, the claims presented in that case are not comparable/analogous to the instant claims for the reasons stated above.
Applicant’s Arguments on Pg. 10 of Arguments/Remarks state:
“The Supreme Court and Federal Circuit have identified a number of considerations as relevant to the evaluation of whether the claimed additional elements demonstrate that a claim is directed to patent-eligible subject matter. MPEP § 2106.04(d)(I). Limitations that the courts have found indicative that an additional element may have integrated an exception into a practical application, or significantly more, include an improvement in the functioning of a computer, or an improvement to other technology or technical field. MPEP § 2106.04(d)(I). In the instant case, the additional elements evidence improvements to the functioning of a computer or an improvement to other technology or technical field, specifically, machine learning.
As outlined in Applicant's specification, evaluation of machine learning models is limited, as each component is separate. See Applicant's Published Specification [0004]. Additionally, in prior solutions, human users are needed to interpret results in order to obtain a more optimal model. Id. In the instant claims, the combination of steps, culminating with the "updating, based on the output signal, one or more parameters of the ML model", allows the parameters of a machine learning model to be updated in a specific, and practical, manner, thereby providing improved machine learning models more efficiently and effectively. The present invention, as claimed, improves this technological field by providing a pipelined and layered approach, that iteratively, and autonomously, explores requirements and parameters, which may heretofore have not been discovered or implemented, for ML evaluation. See Applicant's Published Specification [0065]. Additionally, the present invention, as claimed, streamlines the entire process of configuring and executing ML evaluation into a single pipeline. See Applicant's Published Specification [0065].”
Examiner respectfully disagrees for substantially the same reasons as stated above. Although Applicant’s specification recites a supposed technological improvement, this improvement is not reflected in the currently drafted claim language. The claim does not clearly indicate such a “pipelined and layered approach” and furthermore does not seem to “iteratively and autonomously explore requirements and parameters”. Instead, the claim seemingly lists multiple steps of receiving/transmitting data (insignificant extra-solution activity at Step 2A Prong 2 & well-understood, routine, conventional activity at Step 2B) merely using broadly labeled layers which are seemingly not embodied within any specified “pipeline” or “model” which performs said evaluation and highly generic “interpreting” and “updating” steps which may feasibly be performed by a human user – this cannot provide an inventive concept. Applicant is encouraged to further amend the claims to expand on the technical details of the pipeline/layered approach and should consider amending the claim language to portray the steps of “interpreting” and “updating” in a more technical manner that better describes the evaluation and/or training phases of the approach, such that it is not merely “applied” without significantly more.
Applicant’s Arguments on Pgs. 11-12 of Arguments/Remarks state:
“Additionally, the ordered combination of claimed limitations evidences that the claims are significantly more than any alleged abstract idea. The currently amended independent claims constitute an "ordered combination" that provide a specific, discrete implementation that provides significantly more than the abstract idea itself.
The Office Action fails to take into account the additional elements of "the requirements layer", "the storage layer", "the execution layer", "receiving... at least one requirement obtained from a storage layer", "transmitting.... instructions to perform an ML evaluation ...wherein the execution layer transmits an output signal...", and "updating, based on the output signal, one or more parameters of the ML model."
To illustrate, the currently amended independent claims recite an ordered combination of features that are comparable to BASCOM Global Internet Services, Inc. v. AT&T Mobility LLC, 2016 WL 3514158, (Fed. Cir. June 27, 2016). In BASCOM, the Federal Circuit held that "an inventive concept can be found in the ordered combination of claim limitations that transform the abstract idea into a particular, practical application of that abstract idea." Id. As stated in BASCOM, the inventive concept inquiry requires more than recognizing that each claim element, by itself, was known in the art. As is the case here, an inventive concept can be found in the non-conventional and non-generic arrangement of pieces.
In particular, the currently amended independent claims are directed towards a specific, discrete implementation of machine learning evaluation, that provides more efficient and effective evaluation of machine learning models, and additionally, updates the models based on the evaluation. The present invention, as claimed, provides a pipelined and layered approach, that iteratively, and autonomously, explores requirements and parameters, which may heretofore have not been discovered or implemented, for ML evaluation. See Applicant's Published Specification [0065]. Additionally, as the present invention, as claimed, streamlines the entire process of configuring and executing ML evaluation into a single pipeline. See Applicant's Published Specification [0065].
Claims 1, 7 and 10 are subject matter eligible, at least for the reasons argued above. Withdrawal of the rejection is respectfully requested.
The dependent claims are eligible at least by virtue of their dependencies from claim 1.”
Examiner respectfully disagrees for substantially the same reasons as stated above. The technological improvement and supposed “pipelined and layered approach” as described by Applicant is not reflected in the currently drafted claim language. The Office Action does not fail to take into account the additional elements of the “requirements management layer”, “storage layer”, and “execution layer” – these layers are recited at a high-level of generality and seemingly already configured to perform their own specific operations without significantly more. Adding a generic label to a “layer” of a pipeline/system (which again is also not described by the currently drafted claim language) and merely applying said “layer” without significantly more cannot provide an inventive concept. It must also be noted that Applicant heavily relies on this “pipelined and layered approach” but the Independent claims themselves do not embody the layers on any specific model and/or pipeline – there is no specific arrangement or sequence of these layers indicated by the claim language. Furthermore, the aforementioned limitations of “receiving/transmitting” data are well-understood, routine, conventional activities and cannot provide an inventive concept at Step 2B. Thus, for the reasons stated above, the claims still recite an abstract idea.
Thus, the 35 U.S.C. 101 rejection is maintained.
5. Applicant’s arguments filed March 11, 2026 with respect to the 35 U.S.C. 112(b) rejections have been fully considered and are persuasive. Thus, the 35 U.S.C. 112(b) rejections have been withdrawn.
6. Applicant’s arguments with respect to the 35 U.S.C. 102/103 rejections have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Note: Applicant states on Pg. 16 of Arguments/Remarks that “As to “receiving, by the requirements management layer, the output signal […]”, this limitation was found in original claim 7, indicated by the Office Action as allowable. As such the cited art of record fails to teach, or reasonably suggest, the limitation, as admitted in the Office Action” – however, as noted in the Non-Final Rejection, Claim 6 was indicated as allowable, with claim 7 being allowable by virtue of its dependency on preceding claim 6. Simply rolling up the limitations of claim 7 into the Independent claim, without rolling up the corresponding limitations of claim 6, do not immediately render the claims allowable. As such, the 35 U.S.C. 102 rejection is updated to a 35 U.S.C. 103 rejection below, to address the newly added limitations.
Claim Rejections - 35 USC § 101
7. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
8. Claims 1-6, 8-13, 15-18, and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Step 1: Claim 1 is a method type claim. Therefore, Claims 1-6 are directed to either a process, machine, manufacture, or composition of matter.
2A Prong 1: If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation by mathematical calculation but for the recitation of generic computer components, then it falls within the “Mathematical Concepts” grouping of abstract ideas.
interpreting, by the requirements management layer, the at least one requirement (mental process – other than reciting “by the requirements management layer”, interpreting the at least one requirement may be performed manually by a user observing/analyzing the obtained requirement and accordingly using judgement/evaluation to interpret the requirement)
updating, based on the output signal, one or more parameters of the ML model (mental process – updating one or more parameters of the ML model may be performed manually by a user observing/analyzing the output signal, comprising evaluated metrics, and accordingly using judgement/evaluation to update one or more parameters of the ML model (with the aid of pen and paper) based on said analysis)
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
requirements management layer/storage layer/execution layer (recited at a high-level of generality (i.e., as generic layers already configured to perform the specific operations of the claim without significantly more) such that it amounts to no more than mere instructions to apply the exception using generic computer components)
receiving, by a requirements management layer, at least one requirement obtained from a storage layer, the at least one requirement comprising at least one of: a pass criteria, a test target, a file path of test data, or a test condition (Adding insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g))
transmitting, by the requirements management layer, instructions to perform an ML evaluation process to an execution layer based on the interpreted requirements, wherein the execution layer transmits an output signal with the results of the ML evaluation process upon completing the ML evaluation process (Adding insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g))
receiving, by the requirements management layer, the output signal comprising an instruction to add or update the at least one requirement in the storage layer, wherein the output signal is transmitted from the execution layer based on evaluated metrics (Adding insignificant extra-solution activity to the judicial exception – see MPEP 2106.05(g))
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
requirements management layer/storage layer/execution layer (mere instructions to apply the exception using generic computer components cannot provide an inventive concept)
receiving, by a requirements management layer, at least one requirement obtained from a storage layer, the at least one requirement comprising at least one of: a pass criteria, a test target, a file path of test data, or a test condition (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
transmitting, by the requirements management layer, instructions to perform an ML evaluation process to an execution layer based on the interpreted requirements, wherein the execution layer transmits an output signal with the results of the ML evaluation process upon completing the ML evaluation process (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
receiving, by the requirements management layer, the output signal comprising an instruction to add or update the at least one requirement in the storage layer, wherein the output signal is transmitted from the execution layer based on evaluated metric (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 2-6. The additional limitations of the dependent claims are addressed below.
Regarding Claim 2:
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 2 depends on.
Step 2A Prong 2 & Step 2B:
wherein the at least one requirement is in the form of a requirements as code (RaC) file (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the at least one requirement is in the form of a requirements as code (RaC) file does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 3:
Step 2A Prong 1:
See the rejection of Claim 1 above, which Claim 3 depends on.
Step 2A Prong 2 & Step 2B:
wherein the storage layer is in communication with a first user interface configured to allow a user to edit the at least one requirement (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the storage layer is in communication with a first user interface does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 4:
Step 2A Prong 1:
See the rejection of Claim 3 above, which Claim 4 depends on.
Step 2A Prong 2 & Step 2B:
wherein the execution layer is configured to transmit the output signal to a second user interface […] (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
[…] second user interface configured to display the results of the ML evaluation process (MPEP 2106.05(d)(II) indicates that merely “Presenting offers and gathering statistics” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 5:
Step 2A Prong 1: See the rejection of Claim 4 above, which Claim 5 depends on.
perform an inference process based on the test data and instructions (mental process – other than reciting “inference component”, performing an inference process may be performed manually by a user observing/analyzing the received test data and instructions and accordingly using judgement/evaluation to perform an inference process/cast an inference based on said analysis)
perform an evaluation process based on the output from the ML model and the instructions to obtain metrics (mental process – other than reciting “unit test component”, performing an evaluation process may be performed manually by a user observing/analyzing the output and the instructions and accordingly using judgement/evaluation to perform an evaluation process to obtain metrics based on said analysis)
Step 2A Prong 2 & Step 2B:
wherein the execution layer comprises an inference component and a unit test component (mere instructions to apply the exception using generic computer components cannot provide an inventive concept)
wherein upon receiving the instructions to perform an ML evaluation process, the inference component is configured to receive test data and […] obtain an output from the ML model (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
[…] unit test component […] (mere instructions to apply the exception using generic computer components cannot provide an inventive concept)
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Regarding Claim 6:
Step 2A Prong 1: See the rejection of Claim 5 above, which Claim 6 depends on.
[…] wherein the evaluation process comprises evaluating metrics from the inference log […] (mental process – other than reciting “unit test component”, evaluating metrics from the inference log may be performed manually by a user observing/analyzing the inference log and accordingly using judgement/evaluation to evaluate metrics from said analysis of the inference log)
Step 2A Prong 2 & Step 2B:
wherein the output from the ML model comprises an inference log (Field of Use – limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception does not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application; in this case specifying that the output from the ML model comprises an inference log does not integrate the exception into a practical application nor amount to significantly more – See MPEP 2106.05(h))
[…] wherein upon completing the inference process, the inference component is configured to transfer the inference log to the unit test component […] (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
[…] wherein the evaluated metrics are displayed in the second user interface (MPEP 2106.05(d)(II) indicates that merely “Presenting offers and gathering statistics” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
Accordingly, under Step 2A Prong 2 and Step 2B, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea, as discussed above in the rejection of claim 1.
Independent Claim 8 recites substantially the same limitations as Claim 1, in the form of an apparatus, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
For the reasons above, Claim 8 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 9-13. The additional limitations of the dependent claims are addressed below.
Claim 9 recites substantially the same limitations as Claim 2, in the form of an apparatus, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 10 recites substantially the same limitations as Claim 3, in the form of an apparatus, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 11 recites substantially the same limitations as Claim 4, in the form of an apparatus, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 12 recites substantially the same limitations as Claim 5, in the form of an apparatus, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 13 recites substantially the same limitations as Claim 6, in the form of an apparatus, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Independent Claim 15 recites substantially the same limitations as Claim 1, in the form of a non-transitory computer-readable recording medium, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
For the reasons above, Claim 15 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 16-18 and 20. The additional limitations of the dependent claims are addressed below.
Claim 16 recites substantially the same limitations as Claim 2, in the form of a non-transitory computer-readable recording medium, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 17 recites substantially the same limitations as Claim 3, in the form of a non-transitory computer-readable recording medium, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 18 recites substantially the same limitations as Claim 4, in the form of a non-transitory computer-readable recording medium, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim 20 recites substantially the same limitations as Claim 6, in the form of a non-transitory computer-readable recording medium, including generic computer components. The claim is also directed to performing mental processes without significantly more, therefore it is rejected under the same rationale.
Claim Rejections - 35 USC § 103
9. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
10. Claims 1-3, 8-10, and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Saxon et al. (hereinafter Saxon) (US PG-PUB 20210334078), in view of Lekivetz et al. (hereinafter Lekivetz) (US PG-PUB 20220269401).
Regarding Claim 1, Saxon teaches a method for evaluating a machine learning (ML) model (Saxon, Par. [0071], “During the testing phase, the ML model is presented with [id, feature1 . . . n] and the ML model is expected to supply the target value for the given features. Internally, the system has the real target value, so when the ML model gives back the estimated target value, the two values can be compared to evaluate the performance of the model”, therefore, methods for testing/evaluating a machine learning model are disclosed), the method comprising:
receiving, by a requirements management layer (Saxon, Par. [0027], “The base layer 202 is layered on top of the generic container. The base layer 202 comprises a package manager 203, hardware specifications 204, ML toolkits 205, a repository manager 206 and a programming language 207.”, therefore, model requirements (artifacts, packages, hardware, repository, toolkits, etc.) may be managed by the base layer, also illustrated by Figure 2A, label 202 depicting the base layer and its components. Examiner Note: The term “requirements management” is merely a label in the context of the claim limitation – a requirements management layer is analogous to any layer of a system/model capable of managing requirements), at least one requirement obtained from a storage layer (Saxon, Claim 1, “receiving user specified metadata for execution tasks associated with a machine learning (ML) model; receiving artifacts specifying program code for implementing the ML model; […] receiving environment variables for operation of the ML model […]”, therefore, requirements (user specified metadata, artifacts, environment variables, etc.) are received. Further, these requirements may be obtained from a storage layer (such as the repository manager, See Par. [0033]. The repository manager is analogous to the storage layer, as Examiner notes again that “storage” is merely a label in context of the claim limitation and the “storage layer” is analogous to any layer capable of storing data/requirements)), the at least one requirement comprising at least one of: a pass criteria, a test target, a file path of test data, or a test condition (While Saxon Par. [0043] discloses that the environment variables may include a testing_path (absolute path name of the file containing data to be used for testing/evaluating the ML model), Saxon does not explicitly disclose wherein the at least one requirement comprises at least one of a pass criteria – See introduction of Lekivetz reference below for teaching of the at least one requirement comprising at least one of a pass criteria);
interpreting, by the requirements management layer, the at least one requirement (Saxon, Par. [0036], “The packages 211, 212 are specified by the user artifacts 215. For example, the first layer of the user layer 210 at the individual model level can be conda and pip packages. These packages are handled by inspecting the user artifacts 215 that are specified by the user. If a file named conda_requirements.txt or pip_requirements.txt is present, these packages will be installed by their corresponding package management system and compose this layer.”, thus, the at least one requirement (user artifacts/metadata/variables) may be interpreted/inspected by the base layer/requirement management layer);
transmitting, by the requirements management layer, instructions to perform an ML evaluation process to an execution layer based on the interpreted requirements (Saxon, Par. [0064], “For communication with the ML model image 224, the environment variables 222 are used to transmit state and file system volumes (such as the data volume 223 and the predictions volume 225) to both transmit and receive data between the ML model managing system 221 and the ML model image 224. The data is transferred to the host system that will be executing the ML model image 224 before running a phase within the container 200.”, thus, instructions to perform an ML evaluation process may be transmitted to an execution layer of a host system that will be executing the model based on the interpreted requirements (reflected by the ML model image). Examiner notes again that “execution” is merely a label in context of the claim limitation and the “execution layer” is analogous to any layer capable of executing the machine learning model – in the case of Saxon, the execution layer is contained on a separate host system. This is still consistent with Applicant’s “execution layer” as it is not specified that the “execution layer” is within the same system/model as the requirements management layer or storage layer of Applicant’s Independent claim), wherein the execution layer transmits an output signal with the results of the ML evaluation process upon completing the ML evaluation process (Saxon, Par. [0068], “The testing phase will present the model with data that does not contain target values. Typically, the system performs deserializing 242 the trained ML model representation from the intermediate volume, computing predictions 243 of the target values in memory, and outputting 244 those predictions to the predictions volume 225 that can be used to capture that output from the ML model.”, therefore, as also shown by Figure 2B, the execution layer (host system, executing model image label 224) may transmit an output signal with the results of the ML evaluation/testing process (predictions volume, label 225) upon completing the evaluation/testing process);
receiving, by the requirements management layer, the output signal comprising an instruction to add or update the at least one requirement in the storage layer (Saxon, Par. [0064], “For communication with the ML model image 224, the environment variables 222 are used to transmit state and file system volumes (such as the data volume 223 and the predictions volume 225) to both transmit and receive data between the ML model managing system 221 and the ML model image 224. The data is transferred to the host system that will be executing the ML model image 224 before running a phase within the container 200.” & Par. [0068], “The test phase 240 comprises the steps of transforming the data 241, deserializing 242 the trained model, computing predictions 243 and outputting 244 the predictions. The testing phase will present the model with data that does not contain target values. Typically, the system performs deserializing 242 the trained ML model representation from the intermediate volume, computing predictions 243 of the target values in memory, and outputting 244 those predictions to the predictions volume 225 that can be used to capture that output from the ML model.”, thus, the output signal (predictions volume) may be received by the requirements management layer (base layer), as also shown by Figure 2B which depicts communications in both directions from the requirements management layer and the execution layer (host system). The output signal (predictions volume) may comprise an instruction to add/update at least one requirement in the repository manager (storage layer) based on training of the model – as depicted by Figure 2C), wherein the output signal is transmitted from the execution layer based on evaluated metrics (Saxon, Par. [0068], “The testing phase will present the model with data that does not contain target values. Typically, the system performs deserializing 242 the trained ML model representation from the intermediate volume, computing predictions 243 of the target values in memory, and outputting 244 those predictions to the predictions volume 225 that can be used to capture that output from the ML model.”, therefore, as also shown by Figure 2B, the execution layer (host system, executing model image label 224) may transmit an output signal with the results of the ML evaluation/testing process (predictions volume, label 225) upon completing the evaluation/testing process); and
updating, based on the output signal, one or more parameters of the ML model (Saxon, Par. [0069-0070], “To allow the most flexibility, a ML model does not have to explicitly define a training phase. Instead, the model can optionally perform both steps within the testing phase directly. While not recommended, the testing phase can expose both training/testing data and variables to skip the training phase. Diagnostics are supported by capturing model specific output as well as runtime output. From the ML model's perspective, all informational and error output should be directed to the standard output and standard error pipes. Any data across these pipes will be captured and made available to assist in debugging model or runtime errors.”, therefore, the model may be updated during the training/testing phase to reduce error, based on the predictions volume (output signal)).
While Saxon Par. [0043] discloses that the environment variables may include a testing_path (absolute path name of the file containing data to be used for testing/evaluating the ML model), Saxon does not explicitly disclose the at least one requirement comprising at least one of: a pass criteria, a test target, a file path of test data, or a test condition
However, Lekivetz teaches the at least one requirement comprising at least one of: a pass criteria, a test target, a file path of test data, or a test condition (Lekivetz, Par. [0445], “The user may be able to specify the response type (e.g., a threshold or match target type). For instance, a validation specification may comprise a request to identify a set of inputs that will exceed a threshold for a response of the system of operation according to a test case of a test suite. In FIG. 50B, a test suite 5050 shows test cases for recording responses according to each of the criteria shown in graphical user interface 5000 (e.g., if the threshold is passed or a target is reached). The responses may be specified by the user (e.g., to record pass or fail for displaying to a user).”, therefore, the at least one requirement (which may be user-specified) may include a pass or fail criteria)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for evaluating a machine learning model, as disclosed by Saxon to include the at least one requirement comprising at least one of: a pass criteria, a test target, a file path of test data, or a test condition, as disclosed by Lekivetz. One of ordinary skill in the art would have been motivated to make this modification to enable the use of a pass criteria, which may specify and/or indicate different criteria for evaluating a machine learning model, hence improving system reliability, efficiency, and ensuring the model is fit for its intended purpose (Lekivetz, Par. [0444], “In one or more embodiments, the validation specification comprises multiple validations or multiple criteria. A computing system can then generate output indicating the deviation from the specified result by outputting the deviation corresponding to one or more criteria. In the example shown in FIG. 50A, the graphical user interface 5000 shows a validation specification comprising a first criteria 5010 indicating a crash of the system of operation.”).
Regarding Claim 2, Saxon in view of Lekivetz teaches the method according to claim 1, wherein the at least one requirement is in the form of a requirements as code (RaC) file (Saxon, Par. [0036-0040], “The packages 211, 212 are specified by the user artifacts 215. For example, the first layer of the user layer 210 at the individual model level can be conda and pip packages. These packages are handled by inspecting the user artifacts 215 that are specified by the user. If a file named conda_requirements.txt or pip_requirements.txt is present, these packages will be installed by their corresponding package management system and compose this layer. The user artifacts 215 are specified by the user and must include the model specific code as well as any other binary file required for the successful running of their code.”, therefore, the at least one requirement (user artifacts and metadata) may be in the form of packages/files that specify such requirements. For examples, see Par. [0038-0440] which describe different files that may be used to specify user requirements/artifacts. This is similarly supported by Saxon Claims 7 and 8 which also mention that the artifacts include binary library files).
Regarding Claim 3, Saxon in view of Lekivetz teaches the method according to claim 1, wherein the storage layer is in communication with a first user interface configured to allow a user to edit the at least one requirement (Saxon, Par. [0081], “For example, computing environment 400 can facilitate in whole or in part receiving user specified metadata for execution tasks associated with a machine learning (ML) model; receiving artifacts specifying program code for implementing the ML model; creating a file system structure for a container to hold the ML model; and receiving environment variables for operation of the ML model.”, therefore, computing environment 400, depicted by Figure 4, exemplifies how the storage layer (repository manager, contained in computer label 402) may be in communication with a user interface for allowing a user to enter and edit information into the computer (label 402, See Par. [0058] & [0094] for further details on user specified artifacts and metadata)).
Regarding Claim 8, Saxon in view of Lekivetz teaches an apparatus for evaluating a machine learning (ML) model (Saxon, Par. [0071], “During the testing phase, the ML model is presented with [id, feature1 . . . n] and the ML model is expected to supply the target value for the given features. Internally, the system has the real target value, so when the ML model gives back the estimated target value, the two values can be compared to evaluate the performance of the model”, therefore, methods for testing/evaluating a machine learning model are disclosed), the apparatus comprising: at least one memory storing computer-executable instructions; and at least one processor configured to execute the computer-executable instructions (Saxon, Abstract, “Aspects of the subject disclosure may include, for example, a device, including a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations including receiving user specified metadata for execution tasks […]”, thus, an apparatus/device comprising at least one memory storing executable instructions and at least one processor configured to execute the instructions is disclosed) to: […]
The rest of the claim language in Claim 8 recites substantially the same limitations as Claim 1, in the form of an apparatus, therefore it is rejected under the same rationale.
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Claim 9 recites substantially the same limitations as Claim 2 in the form of an apparatus, therefore it is rejected under the same rationale.
Claim 10 recites substantially the same limitations as Claim 3 in the form of an apparatus, therefore it is rejected under the same rationale.
Regarding Claim 15, Saxon in view of Lekivetz teaches a non-transitory computer-readable recording medium having recorded thereon instructions executable by at least one processor to cause the processor to perform a method (Saxon, Par. [0015], “One or more aspects of the subject disclosure include a machine-readable medium with executable instructions that, when executed by a processing system including a processor, facilitate performance of operations”, therefore, a non-transitory computer-readable medium having instructions to be executed by at least one processor is disclosed) comprising: […]
The rest of the claim language in Claim 15 recites substantially the same limitations as Claim 1, in the form of a non-transitory computer-readable recording medium, therefore it is rejected under the same rationale.
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Claim 16 recites substantially the same limitations as Claim 2 in the form of a non-transitory computer-readable recording medium, therefore it is rejected under the same rationale.
Claim 17 recites substantially the same limitations as Claim 3 in the form of a non-transitory computer-readable recording medium, therefore it is rejected under the same rationale.
11. Claims 4, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Saxon et al. (hereinafter Saxon) (US PG-PUB 20210334078), in view of Lekivetz et al. (hereinafter Lekivetz) (US PG-PUB 20220269401), in view of Gao et al. (hereinafter Gao) (US PG-PUB 20220198340).
Regarding Claim 4, Saxon in view of Lekivetz teaches the method according to claim 3.
Saxon in view of Lekivetz does not explicitly disclose wherein the execution layer is configured to transmit the output signal to a second user interface configured to display the results of the ML evaluation process.
However, Gao teaches wherein the execution layer is configured to transmit the output signal to a second user interface configured to display the results of the ML evaluation process (Gao, Par. [0034], “Output interface 204 provides an interface for outputting information from user device 200, for example, to a user of user device 200 or to another device. For example, output interface 204 may interface with various output technologies including, but not limited to, display 216, a speaker 218, a printer 220, etc. User device 200 may have one or more output interfaces that use the same or a different interface technology. The output interface technology further may be accessible by user device 200 through communication interface 206.”, thus, the execution layer (host system/user device) may be configured to transmit the output to a second user interface (output interface) configured to display results of the ML evaluation process (See Par. [0092]. Further, it should be noted that Saxon provides both an input interface and output interface (See Par. [0033]), such that the output interface is separate and available for displaying results/output).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for evaluating a machine learning model, as disclosed by Saxon in view of Lekivetz to include wherein the execution layer is configured to transmit the output signal to a second user interface configured to display the results of the ML evaluation process, as disclosed by Gao. One of ordinary skill in the art would have been motivated to make this modification to improve system efficiency by having two separate user interfaces, one for user input and the other for displaying output (Gao, Par. [0034], “Output interface 204 provides an interface for outputting information from user device 200, for example, to a user of user device 200 or to another device. For example, output interface 204 may interface with various output technologies including, but not limited to, display 216, a speaker 218, a printer 220, etc. User device 200 may have one or more output interfaces that use the same or a different interface technology. The output interface technology further may be accessible by user device 200 through communication interface 206.”).
Claim 11 recites substantially the same limitations as Claim 4 in the form of an apparatus, therefore it is rejected under the same rationale.
Claim 18 recites substantially the same limitations as Claim 4 in the form of a non-transitory computer-readable recording medium, therefore it is rejected under the same rationale.
12. Claims 5, 12, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Saxon et al. (hereinafter Saxon) (US PG-PUB 20210334078), in view of Lekivetz et al. (hereinafter Lekivetz) (US PG-PUB 20220269401), in view of Gao et al. (hereinafter Gao) (US PG-PUB 20220198340), further in view of Sarsfield et al. (hereinafter Sarsfield) (US PG-PUB 20090006897).
Regarding Claim 5, Saxon in view of Lekivetz in view of Gao teaches the method according to claim 4.
Saxon in view of Lekivetz in view of Gao do not explicitly disclose wherein the execution layer comprises an inference component and a unit test component, wherein upon receiving the instructions to perform an ML evaluation process, the inference component is configured to receive test data and perform an inference process based on the test data and instructions to obtain an output from the ML model, and the unit test component is configured to perform an evaluation process based on the output from the ML model and the instructions to obtain metrics.
However, Sarsfield teaches wherein the execution layer comprises an inference component and a unit test component (Sarsfield, Figure 4 which depicts the execution system comprising an inference component (label 414) and a test case execution component (label 408)), wherein upon receiving the instructions to perform an Sarsfield, Par. [0052], “ The information can also be sent to an inference component 414 for further analysis of the data to make determination and/or decisions regarding the testing of the service. As previously described, the inference component 414 can create one or more additional test cases based in part on information received from a previous test case--for example, to pinpoint a threshold value causing unexpected results.”, therefore, the inference component is configured to receive test data and perform an inference process based on the test data. Although Par. [0060] recites the use of machine learning, Sarsfield is not relied upon for the explicit teaching of an “ML” evaluation process – instead, this is taught by Saxon in the preceding rejection of the Independent claim), and the unit test component is configured to perform an evaluation process based on the output from the Sarsfield, Par. [0051], “The test case execution component 408 can execute the one or more test cases created by the test case creation component 406. The test cases can be ordered, for example, and required to execute in that order. This can be helpful, for example, in trying to pinpoint a threshold value causing an undesired result as described supra. Additionally, however, the order of testing can be left to the test case execution component 408. […] This information can be collated in a single source and/or used to make further determinations regarding the data and/or the service. For example, the information can be passed to the reporting component 410 where it can be processed and output, such as to a log file”, thus, the unit test component/test case execution component is configured to perform evaluation/testing based on the output and instructions, in order to obtain metrics/information).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method for evaluating a machine learning model, as disclosed by Saxon in view of Lekivetz in view of Gao to include wherein the execution layer comprises an inference component and a unit test component, wherein upon receiving the instructions to perform an evaluation process, the inference component is configured to receive test data and perform an inference process based on the test data and instructions to obtain an output from the model, and the unit test component is configured to perform an evaluation process based on the output from the model and the instructions to obtain metrics, as disclosed by Sarsfield. One of ordinary skill in the art would have been motivated to make this modification to enable the inclusion of an inference component and unit test component, which may jointly provide useful insights to improve testing and system accuracy (Sarsfield, Par. [0052], “It is to be appreciated that the inference component 414 is not limited to the examples described, rather the inference component 414 can make many inferences from the output data of the test cases to improve the testing, the service, and/or the service definition, for example.”).
Claim 12 recites substantially the same limitations as Claim 5 in the form of an apparatus, therefore it is rejected under the same rationale.
Claim 19 recites substantially the same limitations as Claim 5 in the form of a non-transitory computer-readable recording medium, therefore it is rejected under the same rationale.
Allowable Subject Matter
13. No prior art rejection is made for Claims 6, 13, and 20. However, these claims are rejected under 35 U.S.C. 101 – abstract idea, and are dependent upon a rejected base claim.
14. Examiner has disclosed Sarsfield et al. (US PG-PUB 20090006897) which is the closest prior art as compared to instant application claims 6, 13, and 20. Sarsfield discloses a service test case generation and execution system, which automatically generates and executes test cases according to a service definition. More particularly, Sarsfield discloses the use of an inference component for further analysis of data to make determinations regarding testing and a test case execution component for executing one or more test cases. However, Sarsfield does not explicitly disclose the specific limitations of Claims 6, 13, and 20, including the limitations “wherein the output from the ML model comprises an inference log, and wherein upon completing the inference process, the inference component is configured to transfer the inference log to the unit test component, wherein the evaluation process comprises evaluating metrics from the inference log, wherein the evaluated metrics are displayed in the second user interface.” in combination with the remaining limitations of the Independent claims.
Conclusion
15. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
16. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Devika S Maharaj whose telephone number is (571)272-0829. The examiner can normally be reached Monday - Thursday 8:30am - 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVIKA S MAHARAJ/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights