DETAILED ACTION
This Action is Responsive to Claims filed 02/06/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1, 12, and 22 have been amended. Claims 2-3, 7-11, 13-14, 18-21, 24, 26, and 28 were previously withdrawn. Claim 29 is new. Claims 1, 4-6, 12, 15-17, 22-23, 25, 27, and 29 are currently pending.
Response to Arguments
Applicant's arguments, see Pages 13-16, filed 02/06/2026, regarding the 35 U.S.C. 101 Rejection have been fully considered but they are not persuasive.
The Examiner appreciates the Applicant’s reiteration of recent guidance surrounding 101 Rejections, but respectfully disagrees with the Applicant as to their application to the instant Claims. The improvements cited by the Applicant, based on information in the Specification (Page 15), are recited highly generally, and without specification structure or implementation represented in the claims. Generic training, assessing, and deploying; generic recitation of “an intuitive representation;” and generic recitation of an interface are not analogous to the level of detail referred-to by recent guidance and decisions pertaining to 101 analysis. The implementation described in the Specification (Arguments Page 15, 3rd paragraph) are recited at a similar level of generality to the instant claims, and do not preclude a human mind with the aid of pen and paper from performing them. Based on the BRI of the claim limitations, and the cited portion of the Specification, the limitations of “assessing…”, “selecting…”, “determining…”, and “generating…” are not recited in a way precluding a human mind from performing the claimed steps. The claimed method is not specifically tied to computer implementation until the “deploying…” step, where a generic processor is recited, and the plurality of models are recited in a similarly generic fashion (a “model” may be a set of equations, for example, and does not specifically tie to a computing environment). The “receiving…” step is recited highly generally and amounts to mere data-gathering, and the newly amended “deploying…” step amounts to instructions to apply the abstract idea mental process determinations and decisions made in the preceding steps to a generic processor. The Examiner respectfully reiterates MPEP 2106.05(a)(II), which states the specific improvement cannot come from the abstract idea mental process step(s). As presently drafted, the Examiner contends the instant claims recite a set of generic model assessment and manipulation steps practically performed within the human mind, from which the alleged improvement is rooted (the improvement is not realized by the mere data-gathering, the improvement is not realized without performing the abstract idea mental process steps). See the updated 35 U.S.C. 101 Rejection below.
Applicant's arguments, see Pages 16-19, filed 02/06/2026, regarding the prior arts Rejections of the claims have been fully considered but they are not persuasive.
As discussed in the interview conducted 02/05/2026, the newly amended term “operational resourcing levels” which “specify constraints on outputs generated” is recited highly broadly within the instant claims. The BRI of the term covers much more than the Specification recites, specifically in paragraphs [0022] (cited in the arguments) and [0024] (cited during the interview). The Examiner maintains that a POSITA would reasonably understand the plain meaning of the term to recite or represent constraints similar to those recited in Thornton, all of which affect the output of a model (number of parameters, number of trainable weights, memory or power constraints, etc.) If the Applicant feels this limitation is where their novelty is best expressed, the Examiner again respectfully encourages the Applicant to include language indicating the term pertains to the implementation or details of paragraphs [0022] and [0024] as discussed in the interview.
The Examiner fails to see the distinction the Applicant attempts to draw on Page 18 regarding the amended “generating…” limitation. Based on the prior mapping, the formulation of a piecewise model is taught by Thornton, as expressed by the Applicant in their arguments, where the single model is generated with the majority-vote fusion. The mappings from other limitations would also reasonably read on each voted-on portion of the single model being tied to some optimization. If the Applicant feels this area is where their novelty is best expressed, the Examiner encourages more detail or better description in order to differentiate from the prior art of reference. See the updated 35 U.S.C. 103 Rejection(s) below.
Claim Rejections - 35 USC § 101
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 4-6, 12, 15-17, 22-23, 25, 27, and 29 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more than the abstract idea itself, and hence is not patent-eligible subject matter.
Step 1 (all claims):
Claims 1, 4-6, 23, and 29 recite a method (representing a process);
Claims 12, 15-17, and 25 recite a system (representing a machine); and
Claims 22 and 27 recite a method (representing a process).
Therefore, each of the claims falls into one of the four statutory categories (i.e., process, machine, article of manufacture, or composition of matter).
Regarding amended Claim 1,
Step 2A Prong 1: This claim recites the following abstract ideas:
... assessing, using the set of resourcing levels, performance of each model within the set of models by evaluating at least one of a precision, recall, accuracy, loss, impact, or a custom optimization function for each model (Under its broadest reasonable interpretation, the term "optimization function" broadly indicates a function (i.e., a mathematical expression/equation) for optimizing a model. As indicated earlier, according to Applicant's specification paragraph [0024], custom training and evaluation functions representing a set of optimization functions can be used to generate and assess/optimize the performance for the set of models. In other words, Applicant's claimed invention applies one or more training/evaluation functions representing a set of optimization functions, where this set of optimization functions is collectively applied to the set of models. Therefore, the phrase" ... for each model ... " appended at the end of the above limitation is interpreted as describing the assessing step/operation referenced in the limitation as a whole (and not the optimization function), and as such, merely reiterates that the performance assessment/evaluation process is applied to each model. Hence this limitation broadly recites a process that uses a set of constraints on outputs of the set of models representing a set of resourcing levels to further determine (e.g., assess) the performance for each respective model, and assesses/evaluates the performance for each respective model within the set of models based on applying at least one of a precision, recall, accuracy, loss, impact metric, or a mathematical expression/equation for optimizing the set of models representing a custom optimization function. A person having ordinary skill in the art can analyze a set of prediction output produced by a set of models, and apply known performance metrics representing a precision, recall, accuracy, loss, or impact metric, and a set of known conditions and associated thresholds representing constraints to the respective output associated with the set of models to evaluate and determine respective performance values associated with each model in a set of models. Additionally, a person having ordinary skill in the art can also analyze a set of prediction output produced by the set of models, create and apply one or more mathematical expressions/equations representing custom optimization functions to further optimize the prediction outputs produced by the models, and apply these one or more custom optimization functions and the same set of known conditions and thresholds representing constraints to the respective output associated with the set of models to evaluate and determine (e.g., assess) respective performance values associated with each model in a set of models. Hence this above-described process is directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).) ...
. . . selecting, for each resourcing level, a respective model from the set of models based on the assessed performance, wherein the selected model has a higher assessed performance than other models for the resourcing level (This limitation broadly recites a process that selects a model from the set of models based on the determined/assessed performance for each model and using the set of constraints/conditions representing the set of resourcing levels. A person having ordinary skill in the art can sort and rank the respective determined/assessed performance values associated with each model in the set of models (where these models were determined based on applying the set of constraints/conditions representing the set of resourcing levels), identify the performance value with the highest assessed/evaluated performance value associated with each constraint/condition, and select the respective model that exhibits the highest assessed/evaluated performance value as the selected model associated with that constraint/condition. Hence this above-described process is also directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).) ...
. . . determining, using the assessed performance, a feasible performance region that maps each resourcing level to the selected model for the respective resourcing level… (In light of Applicant's specification paragraphs [0028]-[0029], the phrase "a feasible performance region" with respect to a model performance broadly indicates an optimal performance associated with a model. Under its broadest reasonable interpretation, the phrase " ... the assessed performance for the selected model ... " in the context of the above limitation merely identifies the collection of optimal performances representing the feasible performance region for the set of models that are constrained on the set of constraints/conditions representing a set of resourcing levels. Hence this limitation broadly recites a process that uses the performance assessment/evaluation to determine the optimal performance for the set of models representing a feasible performance region. A person having ordinary skill in the art can further analyze the earlier assessed/evaluated performance data associated with the set of models that are constrained on a set of resourcing levels and further identify and determine an optimal performance (such as a range of accuracy values) for the set of models representing a feasible performance region. Hence this above-described process is also directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).) ...
. . . including interpolating one or more intermediate resourcing levels between discrete resourcing levels (In light of Applicant's specification paragraph [0028], the term "interpolating'' in the context of a feasible performance region broadly indicates a traversal along different discrete elements of interest that represent different values associated with performance/accuracy. Additionally, under its broadest reasonable interpretation, the phrase one or more intermediate resourcing levels between discrete resourcing levels ' broadly indicates a set of possible values representing a set of possible resourcing levels that defines a range of values representing resourcing levels. Hence this limitation broadly recites that the determination of the feasible performance region further involves performing a traversal process (representing an interpolation process) along a set of possible resourcing levels that defines a range of consecutive resourcing levels. A person having ordinary skill in the art can further analyze the optimal performance/accuracy data representing the feasible performance region for the set of models and identify a particular grouping/set of constraints/conditions (representing a set of possible resourcing levels) associated with the certain performance/accuracy data, and apply additional mathematical analysis to further identify a range of performance/accuracy values that would still satisfy the identified grouping/set of constraints/conditions, where this determination and identification of a range of performance/accuracy values that still satisfy the range of constraints/conditions defined by the grouping/set of constraints/conditions represents a traversal process (representing an interpolation process) along a range of possible performance/accuracy values between a set of possible resourcing levels that also includes traversing/interpolating between consecutive resourcing levels. Hence this above-described process is also directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).) ...
. . . generating, based on the feasible performance region, a single deployable global model comprising a piecewise model having a plurality of intervals corresponding to the resourcing levels, each interval referencing the selected model for that interval; (The phrase "a single deployable global model comprising a piecewise model having a plurality of intervals corresponding to the resourcing levels, each interval referencing the selected model for that interval " broadly indicates a model representing a global model that is generated from a set of models whose collective performance spans across a set of intervals associated with the feasible performance region. Examiner notes that Applicant's earlier recited limitations (" ... assessing... " and " ... determining... ") represent operations/steps that identify/determine an optimal performance representing a feasible performance region for a set of models across a set of resource/constraint levels, where this set of models includes individually selected models that also represents a global model that includes a piecewise model spanning a set of resource/constraint levels under the broadest reasonable interpretation. In other words, this optimal performance for the set of models also represents the optimal performance for a global model that represents a collective set of models, where this optimal performance for this collective set of models representing a global model is determined through determining/assessing the respective performances for the set of models for each resourcing level in the set of resourcing levels, such that this identification of respective optimal performances for each model of the set of models as a feasible performance region also represents a piecewise arrangement of a set of models forming the global model whose collective performance spans across a set of resourcing levels representing a set of intervals. Therefore, the earlier recited limitations that are used to generate a feasible performance region also generates a global model containing the set of models spanning across a set of intervals associated with the feasible performance region. Hence this limitation is also directed to performing Applicant's earlier recited limitations (" ... assessing... " and " ... determining... ") to generate the collective set of models representing the global model using the combined respective performance data from each model that spans across a set of intervals to represent the feasible performance region, where these earlier recited limitations are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).)
Step 2A Prong 2: This claim further recites:
... receiving data characterizing a set of models trained ... a set of operational resourcing levels (This claim element broadly recites receiving data representing a set of trained models and data representing a set of constraints/conditions on outputs of models representing resourcing levels. This claim element is directed to necessary data gathering and outputting for use in a claimed process, as well as an insignificant extra-solution activity for use in a claimed process. See MPEP 2106.05(g). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
... a set of models trained on a dataset (This claim element broadly recites that the set of trained models were trained on a dataset. Specifying that the set of received trained models were trained on a dataset only serves to further describe the training of the set of models applied in the claimed invention, and hence this claim element is directed to a general linking of a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
... trained ... using a set of operational resourcing levels, wherein the set of operational resourcing levels specify constraints on outputs generated by models in the set of models (This claim element broadly recites that the set of trained models were trained using a set of constraints/conditions on outputs of models representing resourcing levels. Specifying that the models were trained using information that includes a set of constraints/conditions on outputs of models representing a set of resourcing levels only serves to further describe the type of training being applied to the claimed invention as well as the type of additional data applied to train the set of models in the claimed invention, and hence this claim element is directed to a general linking of a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
… deploying the global model such that a processor automatically selects and executes the selected model corresponding the resourcing level corresponding to an operational constraint on model outputs without retraining models in the set of models. Amounts to mere instructions to apply the abstract idea mental process steps recited above (See MPEP 2106.05(f)).
Step 2B: This claim further recites:
... receiving data characterizing a set of models trained ... a set of operational resourcing levels (This claim element of receiving data representing a set of trained models and a set of constraints/conditions on outputs of models representing resourcing levels is directed to aspects of general necessary data gathering and outputting activities of presenting offers and gathering statistics (where the received data representing a set of trained models and a set of constraints/conditions on outputs of models representing resourcing levels provided as input represents the gathering of statistical information to produce subsequent respective prediction output and performance data output as a presented offer), as well as aspects of storing and retrieving information in memory (e.g., storing the received data for subsequent use in generating and evaluating prediction output), both of which are well-known, understood, routine, and conventional activities, and hence they do not add significantly more than the judicial exception, alone or in combination with other elements in the claim. See MPEP 2106.05(d)(II), list 3, example iv and MPEP 2106.05(d)(II), list 1, example iv.) ...
. . . a set of models trained on a dataset (As analyzed in Step 2A Prong 2, further describing the training of the set of models applied in the claimed invention only serves to generally link a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). Hence this claim element does not add significantly more than the judicial exception, alone or in combination with other elements in the claim.) ...
... trained ... using a set of operational resourcing levels, wherein the set of operational resourcing levels specify constraints on outputs generated by models in the set of models (As analyzed in Step 2A Prong 2, further describing the type of training being applied to the claimed invention as well as the type of additional data applied to train the set of models in the claimed invention only serves to generally link a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). Hence this claim element does not add significantly more than the judicial exception, alone or in combination with other elements in the claim.) ...
… deploying the global model such that a processor automatically selects and executes the selected model corresponding the resourcing level corresponding to an operational constraint on model outputs without retraining models in the set of models. Amounts to mere instructions to apply the abstract idea mental process steps recited above (See MPEP 2106.05(f)).
Regarding amended Claim 12,
This claim comprises of claim limitations that are substantively similar in scope to corresponding claim limitations in Claim 1, and hence, this claim under the 101 analysis is not patent-eligible subject matter, for the same reasons provided in Claim 1. In addition, under both Step 2A Prong 2 and Step 2B, the limitations (" ... at least one data processor ... " and " ... memory storing instructions which, when executed by the at least one processor, causes the at least one processor to perform operations comprising ... ") represent additional claim elements that are directed to forms of applying mere instructions on a generic computer to implement a judicial exception, and hence these additional claim elements do not further integrate the judicial exception into a practical application, nor do they add significantly more than the judicial exception, alone or in combination with other elements in the claim. See MPEP 2106.05(f).
Regarding amended Claim 22,
This claim comprises of claim limitations that are substantively similar in scope to corresponding abstract idea claim limitations in Claim 1, and hence, this claim under the Step 2A Prong 1 101 analysis is not patent-eligible subject matter, for the same reasons provided in Claim 1.
Step 2A Prong 2: This claim further recites:
. . . training, using the optimization function and the set of constraints, the set of models (This claim element broadly recites using the received mathematical expression/equation representing an optimization function for a set of models and the received set of constraints representing conditions on outputs of a set of models to train and generate the set of models. Applicant's high-level recitation of performing training using a set of received data representing a dataset for a set of models, the constraints/conditions, and a mathematical expression/equation for optimizing the set of models representing an optimization function is directed to a series of operations/steps representing a generic computer component executed on a computer system that subsequently generates and trains the set of models. This series of operations/steps performed by the generic computer component executed on a computer system representing the training process is directed to a form of applying mere instructions on a generic computer to implement a judicial exception. See MPEP 2106.05(f). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
. . . providing the set of models (This claim element broadly recites a process that provides the set of models. This claim element is directed to an insignificant extra-solution activity for use in a claimed process. See MPEP 2106.05(g). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
. . . wherein each constraint in the set of constraints is associated with at least one model in the set of models (This claim limitation broadly recites that each constraint in the set of constraints is associated with at least one model. Specifying that each constraint/condition in the set of constraints is associated with at least one model in the set of models only serves to describe the relationship/functionality of each constraint/condition applied in the claimed invention, and hence this claim element is directed to a general linking of a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
Step 2B: This claim further recites:
. . . training, using the optimization function and the set of constraints, the set of models (As analyzed in Step 2A Prong 2, applying mere instructions on a generic computer to implement a judicial exception does not further integrate the judicial exception into a practical application. See MPEP 2106.05(f). Hence this claim element does not add significantly more than the judicial exception, alone or in combination with other elements in the claim.) ...
. . . providing the set of models (This claim element of providing a set of models is directed to storing and retrieving information in memory (e.g., storing the set of trained models to be further used during the model selection process), which is a well-known, understood, routine, and conventional activity, and hence does not add significantly more than the judicial exception, alone or in combination with other elements in the claim. See MPEP 2106.05(d)(II), list 1, example iv.) ...
. . . wherein each constraint in the set of constraints is associated with at least one model in the set of models (As analyzed in Step 2A Prong 2, describing the relationship/functionality of each constraint/condition applied in the claimed invention only serves to generally link a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). Hence this claim element does not add significantly more than the judicial exception, alone or in combination with other elements in the claim.) ...
Dependent Claims:
Claim 4 (Claim 15) recites the following abstract ideas under Step 2A Prong 1:
... generating, using the feasible performance region, an ensemble model (This limitation broadly recites using the optimal performance representing a feasible performance region to generate an ensemble model. According to According to Applicant's specification paragraphs [0021], [0040], [0043], and [0045], an ensemble model can be generated by merely combining a set of models that is bounded by (i.e., associated with) the determined feasible performance region representing the performance of the set of models. In other words, the ensemble model can be created based on an analysis of the optimal performances associated with each identified optimal model. As established earlier, a person having ordinary skill in the art can assess/evaluate the performance associated with each model across a range of constraints/conditions applied to the output associated with the model to determine a range of performance associated with each model representing a feasible performance region, and based on identifying the optimal performances of each model, select and combine these optimal models to represent an ensemble model. Hence this above-described process is also directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).) ...
Step 2A Prong 2: This claim further recites:
... the ensemble model selecting the model associated with a first resourcing level in the set of resourcing levels in response to receiving the first resourcing level (This claim element broadly recites that the ensemble model includes a selected model associated with a received first resourcing level that is used to select the model. Specifying that the ensemble model includes a selected model associated with a received first resourcing level that is used to select the model only serves to describe the selection process for the one or more selected models within the ensemble model that is being generated and provided in the claimed invention, and hence this claim element is directed to a general linking of a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
. . . providing the ensemble model (This claim element broadly recites providing the generated ensemble model. This claim element is directed to an insignificant extra-solution activity for use in a claimed process. See MPEP 2106.05(g). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.).
Step 2B: This claim further recites:
... the ensemble model selecting the model associated with a first resourcing level in the set of resourcing levels in response to receiving the first resourcing level (As analyzed in Step 2A Prong 2, describing the selection process for the one or more selected models within the ensemble model that is being generated and provided in the claimed invention only serves to generally link a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). Hence this claim element does not add significantly more than the judicial exception, alone or in combination with other elements in the claim .. ) ...
. . . providing the ensemble model (This claim element of providing a combined set of models and their associated outputs representing an ensemble model is directed to storing and retrieving information in memory (e.g., storing the generated ensemble model to be used to select an individual model associated with the ensemble model), which is a well-known, understood, routine, and conventional activity, and hence does not add significantly more than the judicial exception, alone or in combination with other elements in the claim. See MPEP 2106.05(d)(II), list 1, example iv.).
Claim 5 (Claim 16) recites the following abstract ideas under Step 2A Prong 1:
... selecting, in response to receiving the first resourcing level and by the ensemble model the model associated with the first resourcing level in the set of resourcing levels (Under its broadest reasonable interpretation, the phrase" ... in response to receiving the first resourcing level and by the ensemble model ... " in the context of Applicant's earlier recited limitation in dependent Claim 4 (" ... the ensemble model selecting the model associated with a first resourcing level in the set of resourcing levels in response to receiving the first resourcing level ... ") broadly indicates selecting one of the set of models representing the ensemble model associated with the first resourcing level. Hence this limitation broadly recites identifying and selecting the model based on the constraint/condition representing a first resourcing level. A person having ordinary skill in the art can apply a constraint/condition representing a first resourcing level to identify and select the model associated with the ensemble model that satisfies the constraint/condition. Hence this above-described process is also directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).) ...
Step 2A Prong 2: This claim further recites:
... receiving data characterizing the first resourcing level (This claim element broadly recites receiving data associated with the constraint/condition representing a first resourcing level. This claim element is directed to an insignificant extra-solution activity for use in a claimed process. See MPEP 2106.05(g). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
. . . performing, using the model associated with the first resourcing level, a prediction (This claim element broadly recites applying the selected model in the set of models representing an ensemble model that was selected based on the constraint/condition representing a first resourcing level to perform a prediction. As established earlier, the limitation reciting the selection of the model is directed to a mental process. Hence this claim element that applies the selected model (where the limitation reciting the selection of the model is identified as being directed to a mental process) to further perform a prediction is directed to applying mere instructions on a generic computer to implement a judicial exception. See MPEP 2106.05(f). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
... providing an output of the prediction performed by the model associated with the first resourcing level (This claim element broadly recites providing a prediction output generated by the selected model associated with the first resourcing level. This claim element is directed to an insignificant extra-solution activity for use in a claimed process. See MPEP 2106.05(g). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.).
Step 2B: This claim further recites:
... receiving data characterizing the first resourcing level (This claim element of receiving data associated with constraint/condition information describing/representing a first resourcing level is directed to storing and retrieving information in memory (e.g., storing the constraint/condition information to further identify and select a model to perform a subsequent prediction), which is a well-known, understood, routine, and conventional activity, and hence does not add significantly more than the judicial exception, alone or in combination with other elements in the claim. See MPEP 2106.05(d)(II), list 1, example iv.) ...
. . . performing, using the model associated with the first resourcing level, a prediction (As analyzed in Step 2A Prong 2, applying mere instructions on a generic computer to implement a judicial exception does not further integrate the judicial exception into a practical application. See MPEP 2106.05(f). Hence this claim element does not add significantly more than the judicial exception, alone or in combination with other elements in the claim.) ...
. . . providing an output of the prediction performed by the model associated with the first resourcing level (This claim element of providing a prediction outputs associated with the selected model associated with the received constraint/condition representing a first resourcing level is directed to storing and retrieving information in memory (e.g., retrieving the selected model and applying it to perform a prediction), which is a well-known, understood, routine, and conventional activity, and hence does not add significantly more than the judicial exception, alone or in combination with other elements in the claim. See MPEP 2106.05(d)(II), list 1, example iv.).
Claim 6 (Claim 17) recites the following abstract ideas under Step 2A Prong 1:
... wherein the assessing further comprises: evaluating, using the set of optimization functions, respective outputs provided by respective models in the set of models (As indicated earlier, according to Applicant's specification paragraph [0024], custom training and evaluation functions representing a set of optimization functions can be used to generate and assess/optimize the performance for the set of models. In other words, Applicant's claimed invention applies one or more training/evaluation functions representing a set of optimization functions, where this set of optimization functions is collectively applied to the set of models. Hence this limitation broadly recites further using a set of mathematical expressions/equations representing a set of optimization functions to evaluate respective outputs provided by the respective models. A person having ordinary skill in the art can apply a set of mathematical expressions/equations representing a set of optimization functions to evaluate and optimize the losses derived from the respective outputs associated with the respective models and the ground-truth/expected outputs as part of assessing/evaluating the performance of each model of the set of models. Hence this above-described process is also directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).).
Step 2A Prong 2: This claim further recites:
... wherein each optimization function in a set of optimization functions characterizes respective metrics for evaluating the performance of the set of models (As indicated earlier, according to Applicant's specification paragraph [0024], custom training and evaluation functions representing a set of optimization functions can be used to generate and assess/optimize the performance for the set of models. In other words, Applicant's claimed invention applies one or more training/evaluation functions representing a set of optimization functions, where this set of optimization functions is collectively applied to the set of models. Additionally, in light of Applicant's specification paragraph [0028], the term "respective metric' broadly indicates a set of discrete elements associated with the constraint space. Hence this limitation broadly recites further using a set of mathematical expressions/equations representing a set of optimization functions that include respective discrete elements representing respective metrics to evaluate performance of the set of models. Specifying that each optimization function includes respective metric data that is used to evaluate the performance of a set of models only serves to describe the content and functionality of the set of optimization functions in the claimed invention, and hence this claim element is directed to a general linking of a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). This additional claim element does not add a meaningful limitation to the claim, and hence does not integrate the judicial exception into a practical application.) ...
Step 2B: This claim further recites:
... wherein each optimization function in a set of optimization functions characterizes respective metrics for evaluating the performance of the set of models (As analyzed in Step 2A Prong 2, describing the content and functionality of the set of optimization functions in the claimed invention only serves to generally link a technological environment in which to apply a judicial exception. See MPEP 2106.05(h). Hence this claim element does not add significantly more than the judicial exception, alone or in combination with other elements in the claim.) ...
Claim 23 (Claims 25 and 27) recites the following abstract ideas under Step 2A Prong 1:
... wherein assessing the performance of each model within the set of models comprises evaluating an impact (Under its broadest reasonable interpretation, the term "impact' broadly indicates an influence or effect on an element/item of interest. Hence this limitation broadly recites that the process of assessing/evaluating the performance of each model includes evaluating an influence or effect associated with the performance of each model. A person having ordinary skill in the art can analyze the prediction output data associated with the performance of each model and identify and determine features that exhibit an influence or effect to the prediction output data, where this identification and determination of an influence/effect on the prediction output data represents an assessment/evaluation associated with the prediction performance of each model. Hence this above-described process is also directed to a series of analyses and decision-making processes, all of which are mental processes (involving observations, judgments, evaluations, and opinions) that are implementable in a human mind, with aid of pen and paper. See MPEP 2106.04(a)(2)(111).).
Step 2A Prong 2: This claim does not recite any additional elements to be further analyzed at this step.
Step 2B: This claim does not recite any additional elements to be further analyzed at this step.
Claim 29 recites refinements the instructions to apply step of Claim 1.
Step 2A Prong 2: This claim does not recite any additional elements to be further analyzed at this step.
Step 2B: This claim does not recite any additional elements to be further analyzed at this step.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4-6, 12, 15-17, 22-23, 25, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Thornton et al. (US 2019/0156178 A1), hereafter Thornton in view of Lee et al. (US 8,306,940 B2), hereafter Lee.
In regards to claim 1: Thornton reads on “... receiving data characterizing a set of models trained ... a set of operational resourcing levels, wherein the set of operational resourcing levels specify constraints on outputs generated by models in the set of models” in [0019], [0022]-[0025], [0029]: " ... an estimation module 160 that takes into account one or more resource constraints 120, a random model architecture generation module 130, an adaptive refinement module 140 ... The adaptive refinement module 140 adaptively refines one or more of the model architectures 110a, 110b to improve performance of the model architecture ... ", [0033]-[0034], [0057]: "The random model architectures 110a-110n are passed from the random model architecture generation model 130 as input for the adaptive refinement module to adaptively refine the model architectures 11 Oa-11 On to improve performance relative to a metric ... ", and [0077]-[0078], [0080]; and [0055]: "Each random model architecture 110a-110n is generated subject to resource constraints 120. In other words, a model architecture 110a-110n that is produced according to the methods and systems described herein, when realized with a set of trained weights, must be capable of operating within the given resource constraints 120 . ... resource constraints can include upper limits on parameter count, number of trained weights, available memory (during training or at runtime) or number of FLOP/s . ... ", [0069]-[0070], and [0075]: " ... one of ordinary skill in the art would appreciate that systems and methods described herein contemplate use of a range of resource constraints 120 ... other possible resource constructions can include power consumption of a model architecture ... or the speed of execution of a forward pass of the model architecture 110 measured in number of inferences per second (IPS) ... ").) ...
Thornton reads on “. . . a set of models trained on a dataset” in [0025]: "Once particular meta parameter 112 and layer parameter 114 values are chosen to define a unique model architecture 110, the model architecture can be trained to analyze and interpret data using backpropagation techniques and a set of annotated data ... ", [0051]: " ... After initial training, one or more of the final layers of the model architecture ... are re-trained using annotated data of the target data type ... ", [0055]-[0056]: " ... a model architecture 110a-110n that is produced according to the methods and systems described herein, when realized with a set of trained weights, must be capable of operating within the given resource constraints ... The iteration can proceed until a pre-determined number of model architectures 120 is identified ... ", [0062]: "To determine the performance of a model architecture 110 with respect to a metric using the estimation module 160, the weights of the model architecture 110 have to be trained on an annotated set of data using backpropagation ... ", [0077]-[0078]: "In all experiments, a portion of the available annotated data for training was divided out to form a set of heldout test data while the remainder was divided into sets of training and validation data used during individual model architecture learning runs ... The dataset contains 50000 training images and 10000 test images. Five-thousand images from the training set were selected for validation . ... ").) ...
Thornton reads on “. . . trained ... using a set of operational resourcing levels” in [0031][ 0033], including Equations (1) and (2): " ... the overall problem of model architecture selection can be formulated as an optimization task defined as an objective function that is sought to be optimized ... the optimization task is to select a model architecture A that, when A is realized with a set of trained weights, minimizes the objective function in the following form: ... <See Equation (1)> ... The objective function is essentially a weighted sum of a loss term L given the labeled data {x;,y;} and a regularization term R ... <See Equation (2)> ... In Equation (1), cj represents the cost of the jth resource of the particular model architecture 110, which together with the thresholds rj represent the hard constraint 120. The loss term measure the cross-entropy error of the model with respect to a labeled data set ... To direct measure classification accuracy on validation data, L can be formulated as such, which can be used as stop criteria for backpropagation training and model selection ... "; and [0055], [0060]: "The estimation module 160 tests the resulting modified model architecture 110 against the objective function subject to satisfaction of resource constraints as described above ... ", [0069]-[0070], and [0075]).) ...
Thornton reads on “assessing, using the set of resourcing levels, performance of each model within the set of models by evaluating at least one of a precision, recall, accuracy, loss, impact, or a custom optimization function for each model;” in Figure 2 and [0029], [0031]-[0033], including Equations (1) and (2); [0056]-[0060]: " ... the random model architecture generation module 130 can continue to generate and assess random model architectures 11 Oa-11 On in an iterative fashion until one or more model architectures 110a-110n are identified that satisfy the resource constraint 120. The iteration can proceed until a pre-determined number of model architectures 120 is identified ... The random model architectures 110a-110n are passed from the random model architecture generation module 130 as input to the adaptive refinement module 140 to adaptively refine the model architectures 11 Oa-11 On to improve performance relative to a metric ... information about random model architectures 110a-110n and their objective function values ... is used to determine how to modify the values of meta parameters 112 and layer parameters 114 in order to improve performance. In other words, adaptive refinement is an iterative process where at least one meta parameter or layer parameter ... is modified and a new objective function value is determined ... the adaptive refinement module 140 can perform adaptive refinement of the random model architectures 110a-110n using the sequence illustrated below as Algorithm II ... the starting model architecture 111 can be the model architecture with the best performance relative to a metric such as accuracy or efficiency ... the "best" performance can be defined as producing either the highest value or the lowest value of the objective function. In subsequent stages, the starting model architecture 111 may be selected from among all model architectures 110 ... in a multi-threaded process, other workers (e.g., processors) may simultaneously be improving model architectures and may have identified a new best-performing model architecture ... The estimation module 160 tests the resulting modified model architecture 110 against the objective function subject to satisfaction of resource constraints as described above ... ", [0061]-[0063], and Algorithm II: Adaptive Architecture Learning; [0064]-[0066]: " ... systems and methods described herein can utilize a N-best methodology. The N-best methodology uses the top N architectures for generating subsequent architectures ... The adaptive refinement module 140 can operate to refine more than one random model architecture 110 at the same time ... A further implementation variant ... includes fusing the N top model architectures 110 during or after the adaptive refinement process to produce an ensemble model ... "; [0077]-[0078], and Table 1, [0080]).) ...
Thornton reads on “selecting, for each resourcing level, a respective model from the set of models based on the assessed performance, wherein the selected model has a higher assessed performance than other models for the resourcing level;” As indicated earlier, Thornton teaches the adaptive refinement module performs an iterative optimization process that involves refining one or more model parameters (e.g., meta parameters and layer parameters) associated with the received/generated model architectures based on the objective function shown in Thornton Equations (1) and (2) that takes into account the constraints cj and the loss L of the generated model to further identify and determine the best performing model architecture relative to an accuracy metric (which analyzes the loss between the generated model and the labeled data in a labeled dataset), where each identified best performing model architecture at each iteration is then further applied to the estimation module to test whether the identified best performing model architecture still satisfies the existing resource constraints. Thornton additionally teaches that identifying the best performing model includes identifying the model that produces the highest value or lowest value associated with the objective function (representing a high score or a low score). See Thornton paragraphs [0031]-[0033]; [0056]-[0060], [0061]-[0063], and Algorithm II: Adaptive Architecture Learning; [0065]; and [0077]-[0078]).) ...
Thornton reads on “determining, using the assessed performance, a feasible performance region that maps each resourcing level to the selected model for the respective resourcing level…” in [0031]-[0033]; [0056]-[0060], [0061]-[0063], and Algorithm II: Adaptive Architecture Learning; [0064]-[0066], and Table 1, [0080]). Thornton further teaches performing additional experiments to analyze and evaluate the accuracy of a selected model architecture that satisfies the existing set of resource constraints according to different constraint levels (expressed as percentages of constraints applied, as shown in Thornton Table 2), where the selected model architecture includes the asynchronous ensemble model architecture identified in Thornton Table 1 (Thornton [0086]-[0087]: " ... Additional experiments were performed to demonstrate the impact of enforcing resource constraints on development of the selected model architecture in accordance with the present methods ... The experiments used in the asynchronous version of the method ... continued to generate new random model architectures until constraints were met ... The results of the additional experiments showed that the automated method produced model architectures subject to resource constraints that exhibited little loss of accuracy while respective the resource constraints. Table 2 shows the relationship between network size reduction and accuracy and provides details of selected model architectures given different target resource constraints ... specified as a percentage of the unconstrained baseline ... "). A person having ordinary skill in the art would recognize from these teachings that the additional experiments producing accuracy data based on respective levels of constraints being satisfied, starting from 10% of resource constraints being satisfied to 100% of resource constraints being satisfied also represents a process that identifies a feasible performance region across a set of resourcing levels under the broadest reasonable interpretation. This same teaching from Thornton also indicates that a person having ordinary skill in the art can also apply this same method and technique to further evaluate a selected model architecture that includes a set of models (such as an ensemble model). Hence this iterative optimization process taught in Thornton that can be applied to more than one model architecture to evaluate the accuracy of the selected models produced by the iterative optimization process across a range of different levels of constraints/conditions also corresponds to a process that uses the performance assessment/evaluation to determine an optimal performance for the set of selected models representing a feasible performance region (Thornton [0064]-[0066], [0077]-[0078], [0080]; and Table 2, [0086]-[0087]).) ...
Thornton reads on “one or more intermediate resourcing levels between discrete resourcing levels;” As indicated earlier, Thornton teaches performing additional experiments to analyze and evaluate the accuracy of selected model architectures that satisfy the existing set of resource constraints according to different constraint levels (expressed as percentages of constraints applied), thus also corresponding to a set of possible values representing a set of possible resourcing levels that defines a range of values representing resourcing levels (Thornton [0077]-[0078], [0080]; and Table 2, [0086]-[0087]).) ...
Thornton reads on “generating, based on the feasible performance region, a single deployable global model comprising a piecewise model having a plurality of intervals corresponding to the resourcing levels, each interval referencing the selected model for that interval;” (Examiner's note: The limitation broadly indicates a model representing a global model that is generated from a set of models whose collective performance spans across a set of intervals associated with the feasible performance region. Examiner notes that Applicant's earlier recited limitations (" ... assessing... " and " ...determining ... ") represent operations/steps that identify/determine an optimal performance representing a feasible performance region for a set of models across a set of resource/constraint levels, where this set of models includes individually selected models that also represents a global model that includes a piecewise model spanning a set of resource/constraint levels under the broadest reasonable interpretation. In other words, this optimal performance for the set of models also represents the optimal performance for a global model that represents a collective set of models, where this optimal performance for this collective set of models representing a global model is determined through determining/assessing the respective performances for the set of models for each resourcing level in the set of resourcing levels, such that this identification of respective optimal performances for each model of the set of models combined as a feasible performance region also represents a piecewise arrangement of a set of models forming the global model whose collective performance spans across a set of resourcing levels representing a set of intervals. Therefore, the earlier recited limitations that are used to generate a feasible performance region also generates a global model containing the set of models spanning across a set of intervals associated with the feasible performance region. As indicated earlier, Thornton teaches that the adaptive refinement process can be used to generate and assess/evaluate ensemble model architectures, where the process selects and fuses the best model architectures to produce respective asynchronous ensemble and N-best model architectures representing a set of models, thus also representing a process that assesses/evaluates performance for a set of models, where the resulting performance numbers for the different types of selected models (including ensemble models created by identifying asynchronous ensemble and N-best model architecture methods) represent feasible performance regions for each type of assessed ensemble model (Thornton [0031]-[0033]; [0056]-[0060], [0061]-[0063], and Algorithm II: Adaptive Architecture Learning; [0064]-[0066], and Table 1, [0080]-[0081]). As indicated earlier, Thornton teaches performing additional experiments to analyze and evaluate the accuracy of a selected model architecture that satisfies the existing set of resource constraints according to different constraint levels (expressed as percentages of constraints applied, as shown in Thornton Table 2), where the selected model architecture includes the asynchronous ensemble model architecture identified in Thornton Table 1 (Thornton [0086]-[0087]). Therefore, this above process that generates the combined respective performance data representing a feasible performance region for a set of models (as shown in Thornton Table 2) also identifies and determines (e.g., generates) the collective set of models representing the global model whose collective respective performance spans across a set of intervals associated with the feasible performance region.).
Thornton reads on “deploying the global model such that a processor automatically selects and executes the selected model corresponding to the resourcing level corresponding to an operational constraint on model outputs without retraining models in the set of models.” In “Optimization of the structure of the model architecture 110 (i.e., selection of meta parameters 112 and layer parameters 114) is of paramount concern for target deployment platforms that will be used to execute the model and that have limited resources to train and run the model architecture 110.” ([0026])
While Thornton teaches identifying/determining a feasible performance region between different constraint levels and displaying a feasible performance region for the set of models and [0070]: " ... the estimation module 160 can implement a particular resource constraint 120 as a hard constraint, as a soft constraint, or as a combination of a hard and a soft constraint. ... a soft constraint can be implemented as a penalty term that negatively impacts a model's architecture's performance during evaluation of the objective function by a factor proportional to the difference between the model's architecture resource requirements and the one or more resource constraints. A combination of hard and soft resource constraints can be used in which the resource constraint becomes progressively "harder" depending upon the stage of iteration of, for example, the adaptive refinement module 140 . ... the hard resource constraint can be imposed with an initial margin that is progressively tightened throughout the adaptive refinement process ... "). In other words, these teachings from Thornton indicate that these hard and soft constraints can be thought of as respective weighting factors (i.e., weights) that affect the overall model selection and the resulting optimal performance associated with the selected model.
Thornton does not explicitly teach “…including interpolating one or more intermediate resourcing levels…” however, Lee reads on this limitation in col.5 lines 33-54: " ... The real-time visual feedback ensemble classifier generator 400 includes a weighted linear combination module 440. This module combines the plurality of previously-trained component classifiers 415 in a linear manner to generate the ensemble classifier ... each of the component classifiers 415 has a weight. The amount of this weight dictates how much influence a particular component classifier has on the resultant ensemble classifier when the weighted component classifiers are linearly combined by the module 440 ... The input device 445 allows the user to change weights of the component classifiers using a variety of interactive tools ... Based on adjustments to the weights by the user, an ensemble classifier accuracy module 450 provides real-time accuracy feedback... ", col.6 lines 29-39, col.6 line 45-col.7 line 5, col.8 lines 28-35, col.9 lines 6-15: "In some embodiments... small bars 650 are located below each matrix 610 in the component classifier view 600. These small bars 650 depict accuracies for each corresponding classifier ... an overall accuracy bar 670 depicts an overall accuracy of the current ensemble classifier ... ", col.9 lines 20-42: " ... the real-time visual feedback ensemble classifier generator 400 assumes that the component classifiers can produce a vector of real numbers ... and have one number per possible class .... ", col.10 lines 41-46: " ... the two subset classifiers are restricted to only predicting classes which fall on the appropriate side of the vertical partition line. Embodiments of the real-time visual feedback ensemble classifier generator 400 accomplish this by maximizing over an appropriate subset of the numeric vector . ... ", and col.11 lines 13-24).) ...
Both Thornton and Lee are analogous art since they both teach methods and techniques for generating and analyzing a plurality of classifier models.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to take the methods and techniques taught in Thornton and enhance them to incorporate the methods and techniques taught in Lee as a way to identify, explore, and improve the performance of the various machine learning algorithms. The motivation to combine is taught in Lee, because these visualization methods that plot the model prediction boundaries allows for the generalization of machine learning algorithms at the expense of not showing the internal operation of the algorithms while also highlighting the relative merits (such as performance or accuracy) of various algorithms, thus allowing an exploration of a variety of machine learning algorithms based on their performance, thus improving the generalization of the system as well as identifying the data that can be used to further improve the respective algorithms (Lee col.2 lines 31-35, col.3 lines 3-17, col.6 line 45-col.7 line 5).
In regards to claim 4: Thornton in view of Lee reads on “... generating, using the feasible performance region, an ensemble model” (Examiner's note: This limitation broadly recites using the optimal performance representing a feasible performance region to generate an ensemble model. Thornton teaches applying an asynchronous variant of the adaptive refinement process to further select and fuse the N-best model architectures to produce an ensemble model (Thornton [0064][0066]: " ... systems and methods described herein can utilize a N-best methodology. The N-best methodology uses the top N architectures for generating subsequent architectures ... The adaptive refinement module 140 can operate to refine more than one random model architecture 110 at the same time ... A further implementation variant ... includes fusing the N top model architectures 110 during or after the adaptive refinement process to produce an ensemble model ... the best two, three, four, five, or more model architectures 110 can be chosen in order of descending performance relative to the
metric ... ", and Figure 5, [0068]). According to Applicant's specification paragraphs [0021], [0040], [0043], and [0045], an ensemble model can be generated by merely combining a set of models that is bounded by (i.e., associated with) the determined feasible performance region representing the performance of the set of models. In other words, the ensemble model can be created based on an analysis of the optimal performances associated with each identified optimal model. Thornton further teaches performing additional experiments to analyze and evaluate the accuracy of selected model architectures that satisfy the existing set of resource constraints according to different constraint levels (expressed as percentages of constraints applied, as shown in Thornton Table 2), where this evaluation of the accuracy of the selected models produced by the iterative optimization process across a range of different levels of constraints/conditions represents a process that uses the performance assessment/evaluation to determine an optimal performance for the set of selected models representing a feasible performance region (Thornton [0064]-[0066], [0077]-[0078], [0080]; and Table 2, [0086]-[00871).). A person having ordinary skill in the art can apply these methods and techniques taught in Thornton to further analyze and evaluate an ensemble model according to different constraint levels, thus also representing a process that uses the performance assessment/evaluation to determine a feasible performance region to produce the ensemble model. Hence the combination of these teachings from Thornton correspond to a process that generates an ensemble model based on applying the optimal performance representing a feasible performance region associated with an ensemble model.) ...
“the ensemble model selecting the model associated with a first resourcing level in the set of resourcing levels in response to receiving the first resourcing level” (Examiner's note: This limitation broadly recites that the ensemble model includes a selected model associated with a received first resourcing level that is used to select the model. As indicated earlier, Thornton teaches the adaptive refinement module performs an iterative optimization process that involves refining one or more model parameters (e.g., meta parameters and
layer parameters) associated with the received/generated model architectures based on the objective function shown in Thornton Equations (1) and (2) that takes into account the constraints cj and the loss L of the generated model to further identify and determine the best performing model architecture relative to an accuracy/loss metric, where each identified best performing model architecture at each iteration is then further applied to the estimation module to test whether the identified best performing model architecture still satisfies the existing resource constraints (Thornton [0031]-[0033]; [0056]-[0060], [0061]-[0063], and Algorithm II: Adaptive Architecture Learning; [0065]; and [0077]-[0078]). As indicated earlier, Examiner notes that an ensemble model is a form of model that includes a set of models. As indicated earlier, Thornton teaches that this adaptive refinement process can be used to generate and assess/evaluate ensemble model architectures, where the process further selects and fuses the N-best model architectures after the adaptive refinement process based on the ranked performance of the model architectures to produce an ensemble model (Thornton [0064]-[0066]). A person having ordinary skill in the art would recognize from these teachings that the produced ensemble model includes a set of N-best models, where each model was selected based on applying one or more of a plurality of constraints cj, with one of these constraints representing a first resourcing level. Hence this adaptive refinement process that is used to produce an ensemble model that is a set of Nbest models also corresponds to a process in which an ensemble model includes a selected model associated with a received first resourcing level that is used to select the model.) ...
“providing the ensemble model” (Examiner's note: This limitation broadly recites providing the ensemble model. As indicated earlier, Thornton teaches that this adaptive refinement process can be used to generate and assess/evaluate ensemble model architectures, where the process further selects and fuses the N-best model architectures after the adaptive refinement process based on the ranked performance of the model architectures to produce an ensemble model, with this ensemble model being provided for validation of its accuracy (as shown in Thornton Table 1 ), thus also corresponding to a process that provides an ensemble model (Thornton [0064]-[0066]: " ... systems and methods described herein can utilize a N-best methodology. The N-best methodology uses the top N architectures for generating subsequent architectures ... The adaptive refinement module 140 can operate to refine more than one random model architecture 110 at the same time ... A further implementation variant ... includes fusing the N top model architectures 110 during or after the adaptive refinement process to produce an ensemble model ... the best two, three, four, five, or more model architectures 110 can be chosen in order of descending performance relative to the metric ... "; and Table 1, [0080]: " ... the system discussed herein was able to match or exceed that performance in the model architecture developed using the asynchronous ensemble variant of the process based on the best 4 performers ... from among the 120 model architectures ... ").).
In regards to claim 5: Thornton in view of Lee reads on “receiving data characterizing the first resourcing level” (Examiner's note: This limitation broadly recites receiving data associated with the constraint/condition representing a first resourcing level. As indicated earlier, Thornton teaches an estimation module applying a series of resource constraints to a generated set of model architectures, where these resource constraints include upper limits on parameters count, number of trained weights, available memory during training or at runtime, number of floating point operations per second (FLOPs), power consumption of a model architecture, or speed of execution of a forward pass of the model architecture measured in number of inferences per second (IPS), thus representing a set of resourcing levels that are applied to the set of trained models on the respective outputs produced by the set of models (Thornton [0055], [0069]-[0070], and [0075]). As indicated earlier, Thornton teaches the adaptive refinement module performs an iterative optimization process that involves refining one or more model parameters (e.g., meta parameters and layer parameters) associated with the received/generated model architectures based on the objective function shown in Thornton Equations (1) and (2) that takes into account the constraints cj and the loss L of the generated model to further identify and determine the best performing model architecture relative to an accuracy/loss metric, where each identified best performing model architecture at each iteration is then further applied to the estimation module to test whether the identified best performing model architecture still satisfies the existing resource constraints (Thornton [0031 ]-[0033]; [0056]-[0060], [0061 ]-[0063], and Algorithm II: Adaptive Architecture Learning; [0065]; and [0077]-[0078]). A person having ordinary skill in the art would recognize from these teachings that at each iteration, one or more of a plurality of constraints cj is applied to select the model that satisfies the applied constraints, with one of these constraints representing a first resourcing level. Hence this adaptive refinement process also includes a step/operation that corresponds to a process that receives data associated with constraint/condition information representing a first resourcing level.)
“selecting, in response to receiving the first resourcing level and by the ensemble model the model associated with the first resourcing level in the set of resourcing levels” (Examiner's note: Under its broadest reasonable interpretation, the phrase " ... in response to receiving the first resourcing level and by the ensemble model ... " in the context of Applicant's earlier recited limitation in dependent Claim 4 (" ... the ensemble model selecting the model associated with a first resourcing level in the set of resourcing levels in response to receiving the first resourcing level ... ") broadly indicates selecting one of the set of models representing the ensemble model associated with the first resourcing level. Hence this limitation broadly recites identifying and selecting a model within a set of models representing an ensemble model based on the constraint/condition representing a first resourcing level. As indicated earlier, Thornton teaches applying an asynchronous variant of the adaptive refinement process to further select and fuse the N-best model architectures to produce an ensemble model (Thornton [0064]-[0066]: " ... systems and methods described herein can utilize a N-best methodology. The N-best methodology uses the top N architectures for generating subsequent architectures ... The adaptive refinement module 140 can operate to refine more than one random model architecture 110 at the same time ... A further implementation variant ... includes fusing the N top model architectures 110 during or after the adaptive refinement process to produce an ensemble model ... the best two, three, four, five, or more model architectures 110 can be chosen in order of descending performance relative to the metric ... ", and Figure 5, [0068]). As indicated earlier, Thornton teaches the adaptive refinement module performs an iterative optimization process that involves refining one or more model parameters (e.g., meta parameters and layer parameters) associated with the received/generated model architectures based on the objective function shown in Thornton Equations (1) and (2) that takes into account the constraints cj and the loss L of the generated model to further identify and determine the best performing model architecture relative to an accuracy/loss metric, where each identified best performing model architecture at each iteration is then further applied to the estimation module to test whether the identified best performing model architecture still satisfies the existing resource constraints (Thornton [0031 ]-[0033]; [0056]-[0060], [0061]-[0063], and Algorithm II: Adaptive Architecture Learning; [0065]; and [0077]-[0078]). Hence the combination of the above teachings taught in Thornton in which a model within the set of models representing an ensemble model is identified and selected based on satisfying one of the constraints cj during the adaptive refinement process represents an operation/step that corresponds to a process that identifies and selects a model within a set of models representing an ensemble model based on the constraint/condition representing a first resourcing level.)
“performing, using the model associated with the first resourcing level, a prediction” (Examiner's note: This limitation broadly recites applying the selected model in the set of models representing an ensemble model that was selected based on the constraint/condition representing a first resourcing level to perform a prediction. As indicated earlier, Thornton teaches the adaptive refinement module performs the iterative optimization process that uses the estimation module to apply the earlier described set of objective functions associated with respective resource constraints cj and respective thresholds rj to perform assessment/evaluation of a generated model's performance. As indicated earlier, Thornton teaches the objective function is realized as a sum of a loss term L with respect to a labeled dataset and a regularization term R, subject to a set of resource constraints cj that are associated with respective thresholds rj (Thornton Equations (1) and (2)). As shown in Thornton Equation (2), the loss term L is based on labeled data { xi, y;}, thus indicating that this loss is determined based on a difference between the output predictions produced by the generated model and the labeled data in a labeled dataset. A person having ordinary skill in the art would recognize from these teachings that this determination of the loss term involves applying the model that was selected/chosen based on one or more resource constraints to perform a resulting prediction that is compared with the labeled data. Hence this adaptive refinement process taught in Thornton also includes an operation/step that corresponds to a process in which the selected model in the set of models representing an ensemble model that was selected based on the constraint/condition representing a first resourcing level to perform a prediction (Thornton Figure 2 and [0029], [0031]-[0033], including Equations (1) and (2), [0056]-[0060], [0062], [0069]-[0070], and [0075]).)
“providing an output of the prediction performed by the model associated with the first resourcing level” (Examiner's note: This limitation broadly recites providing a prediction output generated by the selected model associated with the first resourcing level. As indicated earlier, Thornton teaches the adaptive refinement module performs the iterative optimization process that uses the estimation module to apply the earlier described set of objective functions associated with respective resource constraints cj and respective thresholds rj to perform assessment/evaluation of a generated model's performance. As indicated earlier, Thornton teaches the objective function is realized as a sum of a loss term L with respect to a labeled dataset and a regularization term R, subject to a set of resource constraints cj that are associated with respective thresholds rj (Thornton Equations (1) and (2)). As shown in Thornton Equation (2), the loss term Lis based on labeled data {xi, y;}, thus indicating that this loss is determined based on a difference between the output predictions produced by the generated model and the labeled data in a labeled dataset. A person having ordinary skill in the art would recognize from these teachings that this determination of the loss term involves applying the model that was selected/chosen based on one or more resource constraints to perform a resulting prediction that is compared with the labeled data, where this prediction is provided as an output for the estimation module to perform this loss/difference calculation. Hence this adaptive refinement process taught in Thornton also includes an operation/step that corresponds to a process that provides a prediction output generated by the selected model associated with the first resourcing level (Thornton Figure 2 and [0029], [0031]-[0033], including Equations (1) and (2), [0056]-[0060], [0062], [0069]-[0070], and [0075]).).
In regards to claim 6: Thornton in view of Lee reads on “wherein each optimization function in a set of optimization functions characterizes respective metrics for evaluating the performance of the set of models” (Examiner's note: As established earlier, this limitation broadly recites further using a set of mathematical expressions/equations representing a set of optimization functions that include respective discrete elements representing respective metrics to evaluate performance of the set of models. As indicated earlier, Thornton teaches the estimation module applies an objective function that is realized as a sum of a loss term L with respect to a labeled dataset and a regularization term R, subject to a set of resource constraints cj that are associated with respective thresholds rj (Thornton Equations (1) and (2)), where this set of mathematical expressions/equations for optimizing the set of models subject to one or more resource constraints representing a set of resource constraints cj also represents a set of optimization functions under the broadest reasonable interpretation, with these respective thresholds rj further representing respective metrics associated with the set of optimization functions under the broadest reasonable interpretation. As indicated earlier, Thornton additionally teaches that the adaptive refinement module performs an iterative optimization process that involves refining one or more model parameters (e.g., meta parameters and layer parameters) associated with the received/generated model architectures based on the objective function to further identify and determine the best performing model architecture relative to an accuracy metric (which is based on the identified loss between the generated model and the labeled data in a labeled dataset), where each identified best performing model architecture at each iteration is then further applied to the estimation module to further test whether the identified best performing model architecture still satisfies the existing resource constraints. Hence this process of applying a set of objective functions associated with respective resource constraints cj and respective thresholds rj to perform assessment/evaluation of a generated model's performance also corresponds to a process that further uses a set of mathematical expressions/equations representing a set of optimization functions that include respective discrete elements representing respective metrics to evaluate performance of the set of models (Thornton Figure 2 and [0029], [0031]-[0033], including Equations (1) and (2); [0056]-[0060]).)
“wherein the assessing further comprises: evaluating, using the set of optimization functions, respective outputs provided by respective models in the set of models” (Examiner's note: As established earlier, this limitation broadly recites further using a set of mathematical expressions/equations representing a set of optimization functions to evaluate respective outputs provided by the respective models. As indicated earlier, Thornton teaches the adaptive refinement module performs the iterative optimization process that uses the estimation module to apply the earlier described set of objective functions associated with respective resource constraints cj and respective thresholds rj to perform assessment/evaluation of a generated model's performance. As indicated earlier, Thornton teaches this set of objective functions are realized as a sum of a loss term L with respect to a labeled dataset and a regularization term R, subject to a set of resource constraints cj that are associated with respective thresholds rj (Thornton Equations (1) and (2)). As shown in Thornton Equation (2), the loss term L is based on labeled data { xi, y;}, thus indicating that this loss is determined based on a difference between the output predictions produced by the generated model and the labeled data in a labeled dataset, where this difference being evaluated in the context of the set of objective functions represents an evaluation of the respective outputs provided by respective models under the broadest reasonable interpretation. As indicated earlier, Thornton further teaches that the adaptive refinement module performs an iterative optimization process based on the set of objective functions to further identify and determine the best performing model architecture relative to an accuracy metric (which is based on the identified loss between the generated model and the labeled data in a labeled dataset), where each identified best performing model architecture at each iteration is then further applied to the estimation module to further test whether the identified best performing model architecture still satisfies the existing resource constraints. Hence this above described process taught in Thornton in which the adaptive refinement module applies the set of objective functions that includes a loss term L based on labeled data {xi, y;} also corresponds to a process that uses a set of mathematical expressions/equations representing a set of optimization functions to evaluate respective outputs provided by the respective models (Thornton Figure 2 and [0029], [0031]-[0033], including Equations (1) and (2), [0056]-[0060], [0062], [0069]-[0070], and [0075]).).
In regards to claim 12 and 15-17: Claims 12 and 15-17 recites similar limitations to claims 1 and 4-6, save for the recitation of a system, therefore, both sets of claims are similarly rejected.
In regards to claim 22: Claim 22 recites similar limitations to claims 1 and 12, which are herein similarly rejected as well as:
“receiving data characterizing a dataset” (Examiner's note: This limitation broadly recites a process that receives data representing a dataset. As indicated earlier, Thornton teaches training the initially identified unique model architectures and the generated models using a set of annotated training data, thus also corresponding to a process that receives data representing a dataset (Thornton [0025], [0051], [0055]-[0056], [0062], [0077]-[0078]).) ...
“an optimization function” (Examiner's note: This limitation broadly recites a process that receives data that includes a mathematical expression/equation for optimizing the set of models representing an optimization function. As indicated earlier, Thornton teaches the estimation module applies an objective function that is realized as a sum of a loss term L with respect to a labeled dataset and a regularization term R, subject to a set of resource constraints cj that are associated with respective thresholds rj (Thornton Equations (1) and (2)), where this set of mathematical expressions/equations for optimizing the set of models represents an optimization function. As indicated earlier, Thornton additionally teaches that the adaptive refinement module performs an iterative optimization process that involves refining one or more model parameters (e.g., meta parameters and layer parameters) associated with the received/generated model architectures based on the objective function to further identify and determine the best performing model architecture relative to an accuracy metric (which is based on the identified loss between the generated model and the labeled data in a labeled dataset), where each identified best performing model architecture at each iteration is then further applied to the estimation module to further test whether the identified best performing model architecture still satisfies the existing resource constraints. Hence this process of applying an objective function to perform assessment/evaluation of a generated model's performance also corresponds to a process that receives data that includes a mathematical expression/equation for optimizing the set of models representing an optimization function (Thornton Figure 2 and [0029], [0031]-[0033], including Equations (1) and (2); [0056]-[0060]).)
“a set of constraints specifying a condition on outputs of models in a set of models” (Examiner's note: This limitation broadly recites a process that receives data that includes a set of constraints/conditions on outputs of models representing resourcing levels. As indicated earlier, Thornton teaches an estimation module applying a series of resource constraints to a generated set of model architectures, where these resource constraints include upper limits on parameters count, number of trained weights, available memory during training or at runtime, number of floating point operations per second (FLOPs), power consumption of a model architecture, or speed of execution of a forward pass of the model architecture measured in number of inferences per second (IPS). As indicated earlier, a person having ordinary skill in the art would also recognize from these teachings that these resource constraints (e.g., memory requirements imposed during training or at runtime, the number of trained weights or number of parameters applied to the generated model, the number of FLOPs, or the number of inferences per second (IPS)) also represent conditions that affect the training of a model as well as a model's generated prediction output, thus also representing a set of resourcing levels that are applied to the outputs of the set of trained models under the broadest reasonable interpretation. Hence this above described process taught in Thornton in which an estimation module applies a series of resource constraints that affect the outputs of models also corresponds to a process that receives data that includes a set of constraints/conditions on outputs of models representing resourcing levels (Thornton [0055], [0069]-[0070], and [0075]).)
“training, using the optimization function and the set of constraints, the set of models” (Examiner's note: This limitation broadly recites using the mathematical expression/equation representing an optimization function for a set of models and the set of constraints representing conditions on outputs of a set of models to train and generate the set of models. As indicated earlier, Thornton teaches an estimation module applying a series of resource constraints to a generated set of model architectures, where these resource constraints include upper limits on parameters count, number of trained weights, available memory during training or at runtime, number of floating point operations per second (FLOPs), power consumption of a model architecture, or speed of execution of a forward pass of the model architecture measured in number of inferences per second (IPS), thus representing a set of resourcing levels. As indicated earlier, Thornton additionally teaches that these resource constraints are applied during the training of these model architectures as well as during model selection within the iterative model assessment/evaluation process performed by the adaptive refinement module, where the estimation module applies an objective function that is realized as a sum of a loss term L with respect to a labeled dataset and a regularization term R, subject to a set of resource constraints cj that are associated with respective thresholds rj (Thornton Equations (1) and (2)). As indicated earlier, Thornton further teaches that the adaptive refinement module performs an iterative optimization process that involves refining one or more model parameters (e.g., meta parameters and layer parameters) associated with the received/generated model architectures based on the objective function to further identify and determine the best performing model architecture relative to an accuracy metric (which is based on the identified loss between the generated model and the labeled data in a labeled dataset), where each identified best performing model architecture at each iteration is then further applied to the estimation module to further test whether the identified best performing model architecture still satisfies the existing resource constraints. Hence this above described process taught in Thornton in which the adaptive refinement module uses the estimation module to train/generate a set of models based on the identified set of resource constraints and an optimization function also corresponds to a process that uses the mathematical expression/equation representing an optimization function for a set of models and the set of constraints representing conditions on outputs of a set of models to train and generate the set of models (Thornton Figure 2 and [0029], [0031]-[0033], including Equations (1) and (2), [0056]-[0060], [0062], [0069]-[0070], and [0075]).)
“providing the set of models” (Examiner's note: This limitation broadly recites a process that provides the set of models. As indicated earlier, Thornton teaches that this adaptive refinement process can be used to assess/evaluate more than one model architecture, where the process further selects and identifies the N-best model architectures after the adaptive refinement process based on the ranked performance of the model architectures, thus also corresponding to a process that provides a set of models (Thornton [0064]-[0066]: " ... systems and methods described herein can utilize a N-best methodology. The N-best methodology uses the top N architectures for generating subsequent architectures ... The adaptive refinement module 140 can operate to refine more than one random model architecture 110 at the same time ... A further implementation variant ... includes fusing the N top model architectures 110 during or after the adaptive refinement process to produce an ensemble model ... the best two, three, four, five, or more model architectures 110 can be chosen in order of descending performance relative to the metric ... ").)
“wherein each constraint in the set of constraints is associated with at least one model in the set of models” (Examiner's note: This limitation broadly recites that each constraint in the set of constraints is associated with at least one model. As indicated earlier, Thornton teaches that the resource constraints are applied during the training of these model architectures as well as during model selection within the iterative model assessment/evaluation process performed by the adaptive refinement module, where the estimation module applies an objective function that is realized as a sum of a loss term L with respect to a labeled dataset and a regularization term R, subject to a set of resource constraints cj that are associated with respective thresholds rj (Thornton Equations (1) and (2)), thus corresponding to a process that associates each constraint in the set of constraints with at least one model in the set of models (Thornton Figure 2 and [0029], [0031 ]-[0033], including Equations (1) and (2); [0056]-[0060]).)
In regards to claim 23: Thornton in view of Lee reads on
“assessing the performance of each model within the set of models comprises evaluating an impact” (Examiner's note: Under its broadest reasonable interpretation, the term "impact' broadly indicates an influence or effect on an element/item of interest. Hence this limitation broadly recites that the process of assessing/evaluating the performance of each model includes evaluating an influence or effect representing an impact associated with the performance of each model. As indicated earlier, Thornton teaches that the resource constraints used during assessing/evaluation of the performance of each model (as part of the objective function) includes hard and soft constraints, where soft constraints can be implemented during the evaluation as a penalty term that negatively impacts a model's architecture performance. Thornton further teaches that these soft constraints in combination with the hard constraints can be implemented to impose a set of conditions in which the condition becomes progressively harder to overcome at each model's performance evaluation iteration, where the presence of these soft constraints represents a constraint/condition that influences or affects (i.e., impacts) the model performance evaluation. Hence the assessment/evaluation process of a model's architecture performance in which the set of resource constraints being applied includes soft constraints that are used as penalty terms representing negative impacts to a model's performance also corresponds to a process in which assessing/evaluating the performance of each model includes evaluating an influence or effect representing an impact associated with the performance of each model (Thornton [0070]: " ... the estimation module 160 can implement a particular resource constraint 120 as a hard constraint, as a soft constraint, or as a combination of a hard and a soft constraint. ... a soft constraint can be implemented as a penalty term that negatively impacts a model's architecture's performance during evaluation of the objective function by a factor proportional to the difference between the model's architecture resource requirements and the one or more resource constraints. A combination of hard and soft resource constraints can be used in which the resource constraint becomes progressively "harder" depending upon the stage of iteration of, for example, the adaptive refinement module 140 . ... the hard resource constraint can be imposed with an initial margin that is progressively tightened throughout the adaptive refinement process ... ").).
In regards to claims 25 and 27: Claims 25 and 27 recite similar limitations to claim 23, therefore, both claims are similarly rejected.
In regards to claim 29: The present invention claims: “wherein the global model replaces execution of the set of models during deployment, reducing the set of models during deployment relative to executing the set of models individually and reducing computational resources relative to executing the set of models individually.” Thornton teaches “Optimization of the structure of the model architecture 110 (i.e., selection of meta parameters 112 and layer parameters 114) is of paramount concern for target deployment platforms that will be used to execute the model and that have limited resources to train and run the model architecture 110.” ([0026]). Step 810 of Figure 8 also selects/fuses a final global model.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRIFFIN T BEAN whose telephone number is (703)756-1473. The examiner can normally be reached M - F 7:30 - 4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GRIFFIN TANNER BEAN/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights