DETAILED ACTION
This action is responsive to the application filed on 05/15/2024. Claims 1-20 are pending and have been examined.
This action is Non-final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.
120, 121, 365(c), or 386(c) is acknowledged.
Claim Objections
Claims 2, 4, 6, 10, 11, 15, 16, and 18 are objected to under 37 CFR 1.75 as being informal. Appropriate correction is required. Specifically:
Claim 2 recites “generate a data record includes the identifier and the initiation date,” which is grammatically incomplete; it appears “includes” should read “that includes” (or “including”).
Claim 4 recites “a corresponding element of configuration data,” which is inconsistent with the term “the first configuration data” recited elsewhere in the claim; consistent terminology is suggested.
Claims 6 and 15 recite “the second element of configuration data,” which appears to omit “first” relative to the earlier-recited “a second element of the first configuration data”; consistent terminology is suggested.
Claims 10 and 18 appear to omit a conjunction (e.g., “; and”) between the “generate … a plurality of feature vectors …” limitation and the following “at least a second one of the executed application engines …” limitation; claim 18 further recites “first elements of first configuration data,” which appears to omit “the.”
Claim 11 recites “at least one of a portion of the first configuration data or the second configuration data”; the construction “at least one of … and …” is suggested for clarity.
Claim 16 recites “determining …, and generate a modified element …,” mixing verb forms; consistent grammatical form is suggested.
Claim Interpretation - 35 U.S.C. § 112(f)
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims recite “application engine(s)” (e.g., claims 1, 13, and 20, and dependent claims 3-8, 10, 15, 16, and 18). Although this limitation does not use the word “means,” the term “engine” is a generic placeholder (a nonce term) that is coupled with functional language, “the executed application engines causing the at least one processor to perform operations that train a machine-learning process …,” and does not, by itself, recite sufficient structure to perform the claimed function. Accordingly, the limitation is being interpreted under 35 U.S.C. 112(f) as invoking corresponding structure described in the specification.
The corresponding structure disclosed in the specification for performing the recited functions includes the executable code/scripts of the enumerated application engines (e.g., retrieval engine 156, preprocessing engine 158, indexing engine 160, featurizer engine 166, and reporting engine 172), executed by the one or more processors of computing system 130, together with the associated algorithms described in the specification (see, e.g., ¶¶ [0159]-[0161] and the corresponding figures). Because the specification discloses corresponding structure and associated algorithms for the claimed functions, the limitation is definite, and no rejection under 35 U.S.C. 112(b) arises from this interpretation.
If applicant does not intend to invoke 35 U.S.C. 112(f), applicant may amend the claims to recite sufficient structure to perform the claimed function, or to otherwise avoid the nonce term. See MPEP 2181-2183.
Claim Rejections - 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) Conclusion. The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 5, 9, 17, 19, and 20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claim 5, the claim depends from claim 1 and recites “wherein: obtain pipeline data associated with the training pipeline from the memory …; and execute the training pipeline script …“The functions “obtain” and “execute” are recited without any structural element performing them, and it is unclear whether these functions are performed by “the at least one processor” recited in claim 1 or by some other element; the claim also fails to recite the antecedent phrasing “the at least one processor is [further] configured to execute the instructions to,” as recited in the other dependent claims (e.g., claim 4). As a result, the metes and bounds of the claim cannot be determined. For purposes of examination, the recited functions are being interpreted as performed by the at least one processor of claim 1. Appropriate correction is required.
Regarding claim 9, the claim recites “data specifying the modified composition of the feature vectors” (final limitation). There is insufficient antecedent basis for “the modified composition” and for “the feature vectors” in the claim, as parent claim 1 does not recite feature vectors or a modified composition thereof. It is noted that “feature vectors” are first recited in claim 6 and a “modified composition” is first recited in claim 8; applicant may wish to amend claim 9 to depend from a claim that provides the necessary antecedent basis. Claim 9 further recites “execute the instruction to receive,” for which there is insufficient antecedent basis, as claim 1 recites “instructions” (plural). Appropriate correction is required.
Regarding claim 17, the claim depends from claim 13 and recites “data specifying the modified composition of the feature vectors.” There is insufficient antecedent basis for “the modified composition” and for “the feature vectors” in the claim, as parent claim 13 does not recite feature vectors or a modified composition thereof. It is noted that “feature vectors” are first recited in claim 15 and a “modified composition” is first recited in claim 16; applicant may wish to amend claim 17 to depend from a claim that provides the necessary antecedent basis. Appropriate correction is required.
Regarding claim 19, the claim depends from claim 13 and recites “transmitting at least a portion of the explainability data to the computing system via the communications interface.” There is insufficient antecedent basis for “the communications interface” in the claim, as neither claim 19 nor parent claim 13 recites a communications interface. Appropriate correction is required.
Regarding claim 20, the claim recites “transmitting the explainability data to the computing system via the communications interface” (final limitation). There is insufficient antecedent basis for “the communications interface” in the claim, as claim 20 recites only a computer-readable medium storing instructions and “at least one processor,” and does not recite a communications interface. Appropriate correction is required.
Claim Rejections - 35 U.S.C. § 101
The following is a quotation of 35 U.S.C. 101:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: This claim is directed to an apparatus, which is one of the four statutory categories (a machine). Therefore, the claim satisfies Step 1.
Step 2A Prong 1:
“generate elements of explainability data that characterize the training of the machine-learning process within the training pipeline” - This limitation is directed to a mental process, as characterizing the training of the model is an act of observation, evaluation, and judgment that can be performed in the human mind, and thus it is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“An apparatus, comprising: a memory storing instructions; a communications interface; and at least one processor coupled to the memory and the communications interface, the at least one processor being configured to execute the instructions to: execute sequentially a plurality of application engines within a training pipeline in accordance with first configuration data, the executed application engines causing the at least one processor to perform operations that train a machine-learning process based on corresponding ones of a plurality of partitioned datasets;” - This limitation recites generic computer components at a high level of generality and amounts to mere instructions to apply the exception using generic computer components, which cannot integrate the exception into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“based on artifact data associated with the sequential execution of the application engines” - This limitation is directed to obtaining artifact data on which the exception operates, which is mere data gathering and insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, the retrieval of data is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“the explainability data being generated in accordance with second configuration data, and at least a portion of the second configuration data being generated by a computing system” - This limitation nominally recites data used by the exception and a generic computing system that generates a portion of that data, and amounts to mere instructions to apply the exception using generic computer components, which cannot integrate the exception into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
“transmit the explainability data to the computing system via the communications interface” - This limitation is directed to outputting the result of the exception, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, transmitting data over a network is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 1 is non-patent eligible. Claims 13 and 20 are analogous to claim 1, aside from claim type and minute difference, and thus will face the same rejection.
Regarding claim 2,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
There are no additional elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The apparatus of claim 1, wherein the at least one processor is further configured to execute the instructions to:” - The limitation recites that at least one processor will have further instructions to apply onto computer, and thus it does not integrate to practical application (see MPEP 2106.05(f)).
“perform operations that initiate the sequential execution of the application engines on an initiation date; generate an identifier associated with the initiation of the sequential execution of the application engines on the initiation date” - This limitation is directed to generating identifying/metadata information for the process, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, generating and recording such data is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“generate a data record includes the identifier and the initiation date, and store the data record within a portion of the memory” - This limitation is directed to storing data, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, storing and retrieving information in memory is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 2 is non-patent eligible.
Regarding claim 3,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
There are no additional elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“the artifact data comprises a plurality of output artifacts generated by corresponding ones of the executed application engines” - This limitation further defines the artifact data that is gathered, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, the gathering of such data is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“the at least one processor is further configured to execute the instructions to store each of the output artifacts within the data record” - This limitation is directed to storing data, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, storing and retrieving information in memory is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 3 is non-patent eligible.
Regarding claim 4,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
There are no additional elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The apparatus of claim 1, wherein: the first configuration data comprises a plurality of elements associated with corresponding ones of the executed application engines; and the at least one processor is configured to execute the instructions to execute sequentially each of the application engines in accordance with a corresponding element of configuration data” - This limitation recites a generic processor executing software modules in accordance with configuration data and amounts to mere instructions to apply the exception using generic computer components, which cannot integrate the exception into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 4 is non-patent eligible.
Regarding claim 5,
Step 1: This claim is directed to an apparatus (a machine). Therefore, claim 5 satisfies Step 1.
There are no additional elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The apparatus of claim 1, wherein: obtain pipeline data associated with the training pipeline from the memory, the pipeline data comprising a training pipeline script that establishes an execution flow for the sequential execution of the application engines within the training pipeline” - This limitation is directed to retrieving pipeline data from memory, which is mere data gathering and insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, retrieving information from memory is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“execute the training pipeline script, the executed training pipeline script causing the at least one processor to execute sequentially each of the application engines at a corresponding position within the execution flow” - This limitation recites a generic processor executing instructions (a script) and amounts to mere instructions to apply the exception using a generic computer component, which cannot integrate the exception into a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 5 is non-patent eligible. Claim 14 is analogous to claim 5, aside from claim type and minute difference, and thus will face the same rejection.
Regarding claim 6,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
Step 2A Prong 1:
“The apparatus of claim 1, wherein: at least a first one of the executed application engines causes the at least one processor to generate feature vectors associated with each of the partitioned datasets … each of the feature vectors comprising values of a plurality of features, and the first element of the first configuration data specifying a composition of the feature vectors” - This limitation is directed to a mathematical concept, as generating feature vectors comprising feature values involves mathematical calculation and/or representation, and thus it is directed to math.
“at least a second one of the executed application engines causes the at least one processor to perform operations … that train the machine-learning process based on the feature vectors associated with each of the partitioned datasets, the second element of configuration data comprising values of process parameters of the machine-learning process” - This limitation is directed to a mathematical concept, as training the machine-learning process is performed through mathematical operations and calculations, and thus it is directed to math.
Step 2A Prong 2 and Step 2B:
“in accordance with a first element of the first configuration data,…in accordance with a second element of the first configuration data,” - The limitation recites that in accordance of the first/second element of the first configuration data. The limitation amounts to no more than mere further limiting to a field of use/environment, and thus it does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 6 is non-patent eligible. Claims 10,15, and 18 are majority-analogous to claim 6, aside from claim type and minute difference, and thus will face the same rejection. Claim 10 will have their own rejection but for just the part that is different.
Regarding claim 7,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
Step 2A Prong 1:
“The apparatus of claim 6, wherein the explainability data comprises at least one of (i) a first value characterizing an importance of one or more features on an output of the trained machine-learning process or (ii) a second value characterizing a performance of the trained machine-learning process” - This limitation is directed to a mathematical concept, as computing a value characterizing feature importance or model performance involves mathematical calculation/operation/concepts, and thus the limitation is directed to math.
There are no additional elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 7 is non-patent eligible. The corresponding limitation of claim 15 is analogous to claim 7, and thus will face the same rejection.
Regarding claim 8,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
Step 2A Prong 1:
“based on at least a portion of the explainability data, determine a modified composition of the feature vectors, and generate a modified element of the first configuration data that specifies the modified composition” - This limitation is directed to a mental process, as determining a modified composition of feature vectors is an evaluation and judgment that can be performed in the human mind (see MPEP 2106.04(a)(2)(III)).
“generate additional feature vectors associated with each of the partitioned datasets … and … train the machine-learning process based on the additional feature vectors associated with each of the partitioned datasets” - This limitation is directed to a mathematical concept, as generating feature vectors and training the machine-learning process are performed through mathematical operations and calculations (see MPEP 2106.04(a)(2)(I)).
There are no additional elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 8 is non-patent eligible. Claim 16 is analogous to claim 8, and thus will face the same rejection.
Regarding claim 9,
Step 1: This claim is directed to an apparatus (a machine). Therefore, claim 9 satisfies Step 1.
There are no additional elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“the computing system is configured to present a graphical representation of a portion of the explainability data within a digital interface” - This limitation is directed to displaying the result of the exception, which is considered mere instructions to apply onto a computer, and thus it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
“the at least one processor is further configured to execute the instruction to receive, via the communications interface, data specifying the modified composition of the feature vectors from the computing system” - This limitation is directed to receiving data, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, receiving data over a network is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 9 is non-patent eligible. Claim 17 (reciting the corresponding method steps) is analogous to claim 9, and thus will face the same rejection.
Regarding claim 10,
Step 1: This claim is directed to an apparatus (a machine). Therefore, claim 10 satisfies Step 1.
Step 2A Prong 1:
“at least a first one of the executed application engines causes the at least one processor to generate, for each of the set of indexed data elements of corresponding ones of the partitioned datasets, a plurality of feature vectors based on first elements of the first configuration data, each of the first elements of the first configuration data specifying a composition of a corresponding one of the feature vectors” - This limitation is directed to a mathematical concept, as generating feature vectors involves mathematical calculation and/or representation, and thus it is directed to math.
“at least a second one of the executed application engines causes the at least one processor to perform operations … that train the machine-learning process based on the feature vectors associated with the indexed data elements of each of the partitioned datasets, the second element of configuration data comprising values of process parameters of the machine-learning process” - This limitation is directed to a mathematical concept, as training the machine-learning process is performed through mathematical operations and calculations, and thus it is directed to math.
“the elements of explainability data comprising ranked values characterizing an importance of each feature within the feature vectors on an output of the trained machine-learning process” - This limitation is directed to a mathematical concept, as computing and ranking feature-importance values involves mathematical calculation, and thus it directed to math.
Step 2A Prong 2 and Step 2B:
“in accordance with a first element of the first configuration data,…in accordance with a second element of the first configuration data,” - The limitation recites that in accordance of the first/second element of the first configuration data. The limitation amounts to no more than mere further limiting to a field of use/environment, and thus it does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
“each of the partitioned datasets comprises a set of indexed data elements” - This limitation nominally recites the data on which the exception operates. The limitation amounts to no more than mere further limiting to a field of use/environment, and thus it does not integrate to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 10 is non-patent eligible. Claim 18 is analogous to claim 10, and thus will face the same rejection.
Regarding claim 11,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
There are no additional elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The apparatus of claim 1, wherein; the at least one processor is further configured to execute the instructions to receive, via the communications interface, at least one of a portion of the first configuration data or the second configuration data from the computing system, the computing system being configured to generate the portion of the first configuration data” - This limitation is directed to receiving configuration data, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, receiving data over a network is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 11 is non-patent eligible.
Regarding claim 12,
Step 1: This claim is directed to an apparatus (a machine). Therefore, the claim satisfies Step 1.
Step 2A Prong 1:
“generate elements of performance data characterizing an operation of each of the executed application engines within the training pipeline;” - This limitation is directed to a mental process, as characterizing the operation of the executed engines is an evaluation and judgment that can be performed in the human mind, and thus the limitation is directed to mental process.
Step 2A Prong 2 and Step 2B:
“transmit at least a portion of the explainability data to the computing system via the communications interface” - This limitation is directed to outputting data, which is insignificant extra-solution activity, and thus does not integrate the exception into a practical application (see MPEP 2106.05(g)). Under Step 2B, transmitting data over a network is a well-understood, routine, and conventional computer function, and thus does not provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
Thus, claim 12 is non-patent eligible. Claim 19 is analogous to claim 12, aside from claim type and minute differences, and thus will face the same rejection.
Claim Rejections - 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 10,452,992 B2, by Lee et. al. (referred herein as Lee) in view of NPL reference, “A Unified Approach to Interpreting Model Predictions.”, by Lundberg et. al. (referred herein as Lundberg) further in view NPL reference “TFX: A TensorFlow-Based Production-Scale Machine Learning Platform.”, by Baylor et. al. (referred herein as Baylor).
Regarding claim 1, Lee teaches:
An apparatus, comprising: a memory storing instructions; a communications interface; and at least one processor coupled to the memory and the communications interface, the at least one processor being configured to execute the instructions to: (Lee, page 80, col. 10, lines 55-58] “In some embodiments, a pool of compute servers and/or storage servers may be pre-configured for the MLS, and the resources for a given job may be selected from such a pool.” AND [Lee, page 81, col. 11 lines ] “In system 100, the MLS may implement a set of programmatic interfaces 161 (e.g., APIs, command-line tools, web pages, or standalone GUIs) that can be used by clients 164”, wherein the examiner interprets a pool of compute servers and storage servers pre-configured for the MLS to be the same as at least one processor coupled to a memory storing instructions, and programmatic interfaces that can be used by clients to be the same as a communications interface, because they are both, respectively, the computing hardware that stores and executes the recited instructions and the interface through which the apparatus exchanges data with another system).
execute sequentially a plurality of application engines within a training pipeline in accordance with first configuration data ([Lee, page 84, col. 7, lines 36-40] “the execution of one job Jp cannot be started until another job Jq is completed successfully (e.g., because the final output of Jq is required as input for Jp)” AND [Lee, page 89, col. 28, lines 44-46] “In some embodiments, a text version 1101 of a transformation recipe may be passed as a parameter in a ‘createRecipe’ MLS API call by a client”, wherein the examiner interprets jobs that must execute one after another because one job’s output is required as another job’s input to be the same as a plurality of application engines executed sequentially within a training pipeline, and a transformation recipe passed to the MLS by a client to be the same as first configuration data, because they are both, respectively, discrete processing operations run in a required order within a workflow and a client-supplied specification governing how those operations are performed).
the executed application engines causing the at least one processor to perform operations that train a machine-learning process based on corresponding ones of a plurality of partitioned datasets (Lee, page 83, col. 16, lines 64-67] “Job J 4 may result in the generation of a model training plan 428 (which may in turn involve several iterations of training, e.g., with different sets of parameters).” AND [Lee, page 89, col. 28, lines 23-26] “Recipes may be applied either to data records that have already split into training and test subsets, or to the entire data set prior to splitting into training and test subsets.”, wherein the examiner interprets a model training plan involving several iterations of training to be the same as operations that train a machine-learning process, and data records split into training and test subsets to be the same as a plurality of partitioned datasets, because they are both, respectively, the training of a model and the input data divided into separate portions on which that model is trained).
based on artifact data associated with the sequential execution of the application engines ([Lee, page 84, col. 18, lines 63-67] “As shown, in the depicted embodiment, MLS artifacts 601 may include, among others, data sources 602, statistics 603, feature processing recipes 606, model predictions 608, evaluations 610, modifiable or in-development models 630, and published models or aliases 640” AND [Lee, page 81, col. 11, lines 60-61] “Results of some jobs may be stored as MLS artifacts within repository 120”, wherein the examiner interprets results of jobs stored as MLS artifacts to be the same as artifact data associated with the sequential execution of the application engines, because they are both the outputs produced and retained as the pipeline jobs run).
the explainability data being generated in accordance with second configuration data (Lee, page 86, col. 22, lines 45-47] “In the depicted embodiment, a client 164 may submit a model execution request 812 to the MLS control plane 180 via a programmatic interface 861”, wherein the examiner interprets a model execution request processed by the MLS control plane to be the same as second configuration data, because they are both service-side parameters that govern the generation of the data characterizing the trained model).
Lee does not teach generate elements of explainability data that characterize the training of the machine-learning process within the training pipeline, or that at least a portion of the second configuration data being generated by a computing system.
Lundberg teaches generate elements of explainability data that characterize the training of the machine-learning process within the training pipeline ([Lundberg, Abstract] “SHAP assigns each feature an importance value for a particular prediction.” AND [Lundberg, page 4, sec. 4] “We propose SHAP values as a unified measure of feature importance.” AND [Ludenberg, Page 3, sec. 2.4] “Shapley sampling values are meant to explain any model by: (1) applying sampling approximations to Equation 4, and (2) approximating the effect of removing a variable from the model by integrating over samples from the training dataset.”, wherein the examiner interprets SHAP values that assign each feature an importance value and provide a unified measure of feature importance and an explanation of how the approximation is performed over samples in a training dataset to be the same as elements of explainability data that characterize the training of the machine-learning process, because they are both quantified values that explain each feature’s contribution to the trained model’s output).
Lee and Lundberg do not teach at least a portion of the second configuration data being generated by a computing system.
Baylor teaches at least a portion of the second configuration data being generated by a computing system ([Baylor, page 1390] “We also provide tooling to help generate the first version automatically by analyzing a sample of the data as well as suggest concrete fixes to the schema as data evolves.”, wherein the examiner interprets tooling that generates the first version of the schema automatically by analyzing the data to be the same as at least a portion of the second configuration data being generated by a computing system, because they are both configuration produced automatically by the system rather than supplied by a user).
Lee, Lundberg, Baylor, and the instant application are analogous art because they are both directed to analyzing and characterizing trained machine-learning models in order to inform users about model quality and the contribution of input features.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the apparatus for interface learning disclosed by Lee to include the “unified measure of feature importance” disclosed by Lundberg. One would be motivated to do so to effectively provide interpretable insight into how each input feature contributes to the trained machine-learning model’s predictions, as suggested by Lundberg (Lundberg, Page 1, sec. 1] “It engenders appropriate user trust, provides insight into how a model may be improved, and supports understanding of the process being modeled.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the automatically generated configuration disclosed by Baylor. One would be motivated to do so to efficiently reduce the manual effort of specifying configuration and to keep the configuration current as the data changes, as suggested by Baylor ([Baylor, page 1390] “We also provide tooling to help generate the first version automatically by analyzing a sample of the data as well as suggest concrete fixes to the schema as data evolves.”). Claims 13 and 20 are analogous art to claim 1, aside from claim type and minute differences, and thus the same rejection applies as above.
Regarding claim 4, Lee, Lundberg, and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Lee further teaches:
wherein: the first configuration data comprises a plurality of elements associated with corresponding ones of the executed application engines ([Lee, page 89, col. 28, lines 34-37] “In at least one embodiment, a pipeline of successive transformations to be performed starting with a given input data set may be indicated within a single recipe.”, wherein the examiner interprets a single recipe indicating a pipeline of successive transformations to be the same as first configuration data comprising a plurality of elements associated with corresponding executed application engines, because they are both one configuration made up of multiple per-operation specifications).
the at least one processor is configured to execute the instructions to execute sequentially each of the application engines in accordance with a corresponding element of configuration data (Lee, page 84, col. 7, lines 36-40] “the execution of one job Jp cannot be started until another job Jq is completed successfully” AND [Lee, page 86, col. 21, lines 8-13] “In some embodiments, the MLS may support recurring scheduling of related jobs. For example, a client may create an artifact such as a model, and may want that same model to be re-trained and/or re-executed for different input data sets (e.g., using the same configuration of resources for each of the training or prediction iterations) at specified points in time.”, wherein the examiner interprets each job executed in its required order under the recipe and the same model being re-trained/re-executed for input datasets at specific time points to be the same as executing sequentially each of the application engines in accordance with a corresponding element of configuration data, because they are both running each operation as specified by its own portion of the configuration).
Regarding claim 5, Lee, Lundberg, and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Lee further teaches:
wherein: obtain pipeline data associated with the training pipeline from the memory, the pipeline data comprising a training pipeline script that establishes an execution flow for the sequential execution of the application engines within the training pipeline ([Lee, page 89, col. 28, lines 34-37] “In at least one embodiment, a pipeline of successive transformations to be performed starting with a given input data set may be indicated within a single recipe” AND [Lee, page 95, col. 39, lines 5-11] “A chunk-level filtering plan 1850 may be generated for the chunked data set 1810 in some embodiments, e.g., based on contents of a filtering descriptor (which may also be referred to as a retrieval descriptor) included in the client's request. The chunk-level filtering plan may indicate, for example, the sequence in which a plurality of in-memory filtering operations 1870” AND [Lee, page 87, col. 24, lines 59-61] “the MLS may select a workload distribution strategy and processing plan may be identified for Jk”, wherein the examiner interprets a recipe indicating a pipeline of successive transformations together with an identified processing plan to be the same as a training pipeline script that establishes an execution flow, because they are both a stored specification that defines the order in which the operations run).
execute the training pipeline script, the executed training pipeline script causing the at least one processor to execute sequentially each of the application engines at a corresponding position within the execution flow ([Lee, page 84, col. 7, lines 36-40] “the execution of one job Jp cannot be started until another job Jq is completed successfully (e.g., because the final output of Jq is required as input for Jp)”, AND [Lee, page 95, col. 39, lines 5-11] “A chunk-level filtering plan 1850 may be generated for the chunked data set 1810 in some embodiments, e.g., based on contents of a filtering descriptor (which may also be referred to as a retrieval descriptor) included in the client's request. The chunk-level filtering plan may indicate, for example, the sequence in which a plurality of in-memory filtering operations 1870”, wherein the examiner interprets jobs executed one after another in their required order and “a filtering descriptor (which may also be referred to as a retrieval descriptor) included in the client's request. The chunk-level filtering plan may indicate, for example, the sequence in which a plurality of in-memory filtering operations 1870” to be the same as executing each of the application engines at a corresponding position within the execution flow, because they are both running each operation in the position defined by the plan). Claim 14 is analogous to claim 5, aside from claim type and minute difference, and thus will face the same rejection.
Regarding claim 6, Lee, Lundberg and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Lee further teaches:
wherein: at least a first one of the executed application engines causes the at least one processor to generate feature vectors associated with each of the partitioned datasets in accordance with a first element of the first configuration data, each of the feature vectors comprising values of a plurality of features, and the first element of the first configuration data specifying a composition of the feature vectors ([Lee, page 81, col. 12, lines 51-59] “Any of a variety of feature processing approaches may be used depending on the problem domain: e.g., the recipes typically used for computer vision problems may differ from those used for voice recognition problems, natural language processing, and so on. The output 116 of the feature processing transformations may in turn be used as input for a selected machine learning algorithm 166, which may be executed in accordance with algorithm parameters 154 using yet another set of resources from pool 185”, wherein the examiner interprets the output of the feature processing transformations, as specified by the recipe, and “Any of a variety of feature processing approaches may be used depending on the problem domain: e.g., the recipes typically used for computer vision problems may differ from those used for voice recognition problems, natural language processing, and so on. The output 116 of the feature processing transformations may in turn be used as input for a selected machine learning algorithm 166, which may be executed in accordance with algorithm parameters 154 using yet another set of resources from pool 185” to be the same as feature vectors comprising values of a plurality of features whose composition is specified by a first element of the first configuration data, because they are both the transformed feature values produced according to the configuration and provided to the model).
at least a second one of the executed application engines causes the at least one processor to perform operations, in accordance with a second element of the first configuration data, that train the machine-learning process based on the feature vectors associated with each of the partitioned datasets, the second element of configuration data comprising values of process parameters of the machine-learning process. (Lee, page 83, col. 16, lines 64-67] “Job J4 may result in the generation of a model training plan 428 (which may in turn involve several iterations of training, e.g., with different sets of parameters).” AND [Lee, page 95, col. 39, lines 5-11] “The chunk-level filtering plan may indicate, for example, the sequence in which a plurality of in-memory filtering operations 1870 (e.g., 1870A, 1870B and 1870N) such as shuffles, splits, samples, or partitioning for parallel computations”, wherein the examiner interprets iterations of training performed with different sets of parameters to be the same as training the machine-learning process in accordance with a second element comprising values of process parameters, because they are both training the model using configured parameter values). Claims 10,15, and 18 are majority-analogous to claim 6, aside from claim type and minute difference, and thus will face the same rejection. Claim 10 will have their own rejection but for just the part that is different.
Regarding claim 7, Lee, Lundberg and Baylor teaches The apparatus of claim 6 (see rejection of claim 6).
Lundberg further teaches wherein the explainability data comprises at least one of (i) a first value characterizing an importance of one or more features on an output of the trained machine-learning process or (ii) a second value characterizing a performance of the trained machine-learning process ([Lundberg, page 2, sec. 1] “We then show that game theory results guaranteeing a unique solution apply to the entire class of additive feature attribution methods (Section 3) and propose SHAP values as a unified measure of feature importance that various methods approximate (Section 4).” AND [Lundberg, page 5, Fig. 1] “SHAP (SHapley Additive exPlanation) values attribute to each feature the change in the expected model prediction when conditioning on that feature.” AND [Lundberg, page 8, sec. 5.2] “The second used a max allocation problem to which DeepLIFT can be applied. AND [Lundberg, page 9, sec. sec 5.3] “Figure 5: Explaining the output of a convolutional network trained on the MNIST digit dataset.”, wherein the examiner interprets a SHAP value that attributes to each feature the change in the expected model prediction to be the same as a first value characterizing an importance of one or more features on an output of the trained machine-learning process, because they are both a value quantifying a feature’s effect on the model’s output).
Lee, Lundberg, Baylor, and the instant application are analogous art because they are all directed to interpreting a trained machine-learning model by quantifying the importance or contribution of one or more input features to the model's output.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the apparatus of claim 6 disclosed by Lee, Lundberg, and Baylor to include the "unified measure of feature importance" disclosed by Lundberg. One would be motivated to do so to effectively quantify and expose how much each input feature contributes to the trained machine-learning model's output, as suggested by Lundberg ([Lundberg, page 2, sec. 1] “We then show that game theory results guaranteeing a unique solution apply to the entire class of additive feature attribution methods (Section 3) and propose SHAP values as a unified measure of feature importance that various methods approximate (Section 4).”)
Regarding claim 8, Lee, Lundberg, and Baylor teaches The apparatus of claim 6 (see rejection of claim 6).
Lee further teaches wherein: the at least one processor is further configured to execute the instructions to, based on at least a portion of the explainability data, determine a modified composition of the feature vectors, and generate a modified element of the first configuration data that specifies the modified composition ([Lee, page 77, col. 4, lines 29-33] “FIG. 40 illustrates an examples of a machine learning service configured to generate feature processing proposals for clients based on an analysis of costs and benefits of candidate feature processing transformations, according to at least some embodiments”, wherein the examiner interprets generating feature processing proposals based on an analysis of candidate transformations to be the same as determining a modified composition of the feature vectors and generating a modified element of the first configuration data, because they are both producing a revised feature specification based on evaluating the features).
at least the first one of the executed application engines causes the at least one processor to generate additional feature vectors associated with each of the partitioned datasets in accordance with the modified element of the first configuration data; and at least the second one of the executed application engines further causes the at least one processor to perform operations, in accordance with the second element of the first configuration data, that train the machine-learning process based on the additional feature vectors associated with each of the partitioned datasets (Lee, [0046] “a model is re-evaluated using modified evaluation data sets to determine the impact on prediction quality of using various processed variables” AND [Lee, page 95, col. 39, lines 5-11] “The chunk-level filtering plan may indicate, for example, the sequence in which a plurality of in-memory filtering operations 1870 (e.g., 1870A, 1870B and 1870N) such as shuffles, splits, samples, or partitioning for parallel computations”, wherein the examiner interprets re-evaluating a model using modified sets of processed variables to be the same as generating additional feature vectors and training the machine-learning process based on the additional feature vectors, because they are both re-running the model on a revised set of features). Claim 16 recites the corresponding method steps and is rejected under the same combination and rationale.
Regarding claim 9, Lee, Lundberg and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Lee further teaches wherein: the computing system is configured to present a graphical representation of a portion of the explainability data within a digital interface; and the at least one processor is further configured to execute the instruction to receive, via the communications interface, data specifying the modified composition of the feature vectors from the computing system ([Lee, page 78, col. 5 lines 55-60] “FIG. 62 illustrates an example system environment in which a machine learning service implements an interactive graphical interface enabling clients to explore tradeoffs between various prediction quality metric goals, and to modify settings that can be used for interpreting model execution results, according to at least some embodiments.”, wherein the examiner interprets an interactive graphical interface that presents model execution results to be the same as presenting a graphical representation of a portion of the explainability data within a digital interface, because they are both a visual display of the model-characterizing output. The examiner further interprets receiving, through the interface, a client’s modification of the settings to be the same as receiving data specifying the modified composition of the feature vectors from the computing system, because they are both the interface accepting a user-specified change that alters how the model output is produced).
Regarding claim 10, Lee, Lundberg, and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Majority of the claim is analogous to other parts of the earlier (claim 6). Claims 10 and 18 share this unique limitation:
Lee further teaches:
wherein: each of the partitioned datasets comprises a set of indexed data elements ([Lee, page 89, col. 28, lines 23-25] “Recipes may be applied either to data records that have already split into training and test subsets” AND [Lee, page 95, col. 39, lines 5-11] “The chunk-level filtering plan may indicate, for example, the sequence in which a plurality of in-memory filtering operations 1870 (e.g., 1870A, 1870B and 1870N) such as shuffles, splits, samples, or partitioning for parallel computations”, wherein the examiner interprets the data records within the training and test subsets AND “or partitioning for parallel computations” to be the same as a set of indexed data elements, because they are both the individual records that make up each dataset).
Regarding claim 11, Lee, Lundberg, and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Lee further teaches wherein the at least one processor is further configured to execute the instructions to receive, via the communications interface, at least one of a portion of the first configuration data or the second configuration data from the computing system, the computing system being configured to generate the portion of the first configuration data. ([Lee, page 89, col. 28, lines 44-47] “In some embodiments, a text version 1101 of a transformation recipe may be passed as a parameter in a ‘createRecipe’ MLS API call by a client” AND ([Lee, page 78, col. 5 lines 55-60] “FIG. 62 illustrates an example system environment in which a machine learning service implements an interactive graphical interface enabling clients to explore tradeoffs between various prediction quality metric goals, and to modify settings that can be used for interpreting model execution results, according to at least some embodiments.”, wherein the examiner interprets a transformation recipe created by a client and passed to the MLS via an API call to be the same as receiving a portion of the first configuration data from the computing system that is configured to generate it, because they are both the other system producing configuration data and supplying it to the apparatus).
Regarding claim 12, Lee, Lundberg, and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Lee further teaches:
wherein the at least one processor is further configured to execute the instructions to: generate elements of performance data characterizing an operation of each of the executed application engines within the training pipeline (Lee, [0054] “the MLS control plane may comprise a set of monitoring agents that collect performance and other metrics from the resources used for the various phases of machine learning operations”, AND [Lee, page 86, col. 21, lines 22-23, 29-35] “A respective job may be placed in the MLS job queue for each recurring training or execution iteration…Such programmatic interfaces may be referred to as ‘pipelining APIs’ in some embodiments. In addition to the artifact types shown in FIG. 6, pipeline artifacts may be stored in the MLS artifact repository in some embodiments, with each instance of a pipeline artifact representing a named set of recurring operations requested via such APIs.”, wherein the examiner interprets monitoring agents that collect performance metrics from the resources used for the various phases, and recurring training or execution iteration…Such programmatic interfaces may be referred to as pipelining APIs…pipeline artifacts may be stored in the MLS artifact repository in some embodiments, with each instance of a pipeline artifact representing a named set of recurring operations requested via such APIs to be the same as generating elements of performance data characterizing an operation of each of the executed application engines, because they are both measurements of how each phase of the pipeline performs).
transmit at least a portion of the explainability data to the computing system via the communications interface (Lee, [0049] “clients 164 may be able to view at least a subset of the artifacts stored in repository 120, e.g., by issuing read requests 118 via programmatic interfaces 161” AND [Lee, page 83, col. 16, lines 16-21] “For certain types of tasks, the costs of transmitting data sets and/or results over long distances may be so high, or the time required for the transmissions may so long, that the MLS may restrict the tasks to within a single geographical region of the provider network (or even within a single data center).”, wherein the examiner interprets clients viewing a subset of the stored artifacts through the programmatic interfaces and “transmissions may so long, that the MLS may restrict the tasks to within a single geographical region of the provider network (or even within a single data center)” to be the same as transmitting at least a portion of the explainability data to the computing system via the communications interface, because they are both delivering part of the generated output to another system through the interface). Claim 19 is analogous to claim 12, aside from claim type and minute differences, and thus will face the same rejection.
Claims 2 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Lundberg in view of Baylor, and further in view of NPL reference “Accelerating the Machine Learning Lifecycle with MLflow,”, by Zaharia et. al. (referred herein as Zaharia).
Regarding claim 2, Lee, Lundberg, and Baylor teaches The apparatus of claim 1 (see rejection of claim 1).
Lee further teaches wherein the at least one processor is further configured to execute the instructions to: generate an identifier associated with the initiation of the sequential execution of the application engines ([Lee, page 94-85, col. 18-19, lines 67, 1-3] “In some embodiments, the MLS may generate a respective unique identifier for each instance of at least some of the types of artifacts shown and provide the identifiers to the clients.”, wherein the examiner interprets a generated unique identifier for each instance to be the same as an identifier associated with the initiation of the sequential execution, because they are both a unique label the system creates for a given execution).
generate a data record includes the identifier and the initiation date, and store the data record within a portion of the memory ([Lee, page 81, col. 11, lines 30-32, 60-61] “As mentioned earlier, each job object may indicate one or more operations that are to be performed as a result of the invocation of a programmatic interface 161…results of some jobs may be stored as MLS artifacts within repository 120”, wherein the examiner interprets a job object that is stored to be the same as a data record that is stored within a portion of the memory, because they are both a stored record describing an execution).
Lee, Lundberg, and Baylor do not teach that the sequential execution is initiated on an initiation date, nor that the stored data record includes the initiation date.
Zaharia teaches on an initiation date and a stored record that includes the initiation date ([Zaharia, page 5, sec. 3.3] “Each MLflow Model is simply stored as a directory containing arbitrary files and an MLmodel YAML file that lists the flavors it can be used in and additional metadata about how it was created [the line of code]”, wherein the examiner interprets a recorded time_created stored together with a run’s unique identifier to be the same as an initiation date included in the stored data record, because they are both the date recorded for when the run is initiated).
Lee, Lundberg, Baylor, Zaharia, and the instant application are analogous art because they are both directed to recording and managing the execution of machine-learning workflows.
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the apparatus of claim 1 disclosed by Lee, Lundberg, and Baylor to include the recorded run creation time disclosed by Zaharia. One would be motivated to do so to effectively track and reproduce each training run over time, as suggested by Zaharia ([Zaharia, page 5, sec. 3] “letting users discover how they were built.”).
Regarding claim 3, Lee, Lundberg, Baylor, and Zaharia teaches The apparatus of claim 2 (see rejection of claim 2).
Lee further teaches:
wherein: the artifact data comprises a plurality of output artifacts generated by corresponding ones of the executed application engines ([Lee, page 84, col. 18, lines 62-67] “As shown, in the depicted embodiment, MLS artifacts 601 may include, among others, data sources 602, statistics 603, feature processing recipes 606, model predictions 608, evaluations 610, modifiable or in-development models 630, and published models or aliases 640”, wherein the examiner interprets MLS artifacts such as statistics, model predictions, and evaluations produced by the jobs to be the same as a plurality of output artifacts generated by corresponding ones of the executed application engines, because they are both the outputs each pipeline operation produces).
the at least one processor is further configured to execute the instructions to store each of the output artifacts within the data record (Lee, [Lee, page 81, col. 11, lines 60-61] “results of some jobs may be stored as MLS artifacts within repository 120”, wherein the examiner interprets results of jobs stored as MLS artifacts within the repository to be the same as storing each of the output artifacts within the data record, because they are both retaining each produced output in a stored record).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights