Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/30/2026 has been entered.
Claims 1, 10, and 18 are independent claims.
Claims 1, 10, and 18 are currently amended.
Claims 17 and 20 have been canceled.
Claims 1-16, 18-19, and 21-22 are currently pending.
Response to Arguments
Applicant's arguments filed 04/30/2026 have been fully considered but they are not persuasive.
Regarding the 35 USC § 103 Rejections:
Applicant's arguments regarding the 35 U.S.C. § 103 rejections of the previous office action have been fully considered, but are unpersuasive.
Applicant notes the 103 rejections and asserts (Pages 9-10), that the amended independent claim is supported by the specification. Applicant further notes that Nakandala describes a system for optimized deep learning model selection but does not appear to describe the optimized deep learning model selection and scheduling/training in terms of receiving candidate models from robotic modeling equipment, determining, based on the receiving of the candidate models, that the candidate models are insufficient to solve a use case of a plurality of use cases, and obtaining, based on the determining that the candidate models are insufficient to solve the use case of the plurality of use cases, a plurality of models from databases sourced from model contributors comprised of data scientists, engineers, and technical staff, as now recited in claim 1. Wang and Aarts fail to remedy the deficiencies of Nakandala.
Applicant’s arguments with respect to the independent claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant asserts (Page 10-11), that the similar independent claims and their dependent claims are also allowable as they are not taught within Nakandala/Wang/Aarts. Thus, all 103 rejections should be withdrawn due to the amendments.
Applicant merely notes that Nakandala/Wang/Aarts do not teach explicitly the independent claims by reiterating the limitations. However, these newly amended limitations are explicitly taught by the new prior art reference (Hou); thus, the arguments pertaining to the application of the specific data and assessing the data for affirmations/correction/refinement are moot. More specific details are discussed below within the responses and 35 USC § 103 Rejections. The amended independent claim rejections have been updated with a new reference to explicitly teach elements of the newly added limitations. Applicant’s arguments regarding the other independent and dependent claims rely upon the same assertions as with respect to Claim 1, and are thus likewise unpersuasive. Therefore, for the reasons given above and in the updated rejections below, the rejection to all Claims (including Claim 1, analogous independent Claims, and all dependent Claims) are maintained and updated as necessitated by Claim amendments. More specific details are discussed below within the 35 USC § 103 Rejections.
Regarding the 35 USC § 101 Rejections:
Applicant's arguments regarding the 35 U.S.C. 101 rejections of the previous office action have been fully considered, but are unpersuasive.
Applicant disagrees with the 101 rejections (Pages 11-12), for claims 1-16, 18-19, and 21-22, as the Claims are not directed to an abstract idea without significantly more. However, the independent claims have been amended and further direct the claimed subject matter to one or more practical applications and technological environments and improvements. Thus, the rejections are rendered moot as the rejection is inapplicable to the currently amended claims.
Examiner respectfully disagrees. The 35 U.S.C. § 101 rejection is not rendered moot as the amended claims are directed to an abstract idea (Step 2A Prong 1) and do not integrate the abstract idea into a practical application (Step 2A Prong 2). The rejection follows the steps of the analysis as laid out in the MPEP which was followed for the previous and current examination (see MPEP 2106). Therefore, for the reasons given above and in the updated rejections below, the rejection to all Claims (including Claim 1, similar independent claims, and all dependent Claims) are maintained and updated as necessitated by Claim amendments. More specific details are discussed below within the responses and 35 USC § 101 Rejections.
Applicant notes (Pages 11-12) that the Examiner fails to substantively address the remarks at pages 11-14 of the paper submitted on October 14, 2025, in any meaningful way. Accordingly, those remarks are incorporated herein by way of reference. A further elaboration upon those remarks, in view of the statements included in the current Office Action of February 4, 2026, is provided in the discussion that follows. At various points the remarks below are directed to claim 1, with the understanding that the rationale/reasoning of the remarks is applicable to claims 10 and 18 by way of analogy. The dependent claims are directed to patent eligible subject matter for at least the same reasons as their respective base independent claims.
Examiner respectfully disagrees. The remarks/arguments laid forth by the applicant have been substantively and were fully responded to previously. The rejection also follows the steps of the analysis as laid out in the MPEP which was followed for the previous and current examination (see MPEP 2106). Applicant’s arguments regarding the other independent and dependent claims rely upon the same assertions as with respect to Claim 1, and are thus likewise unpersuasive. More specific details are discussed below within the 35 USC § 101 Rejections.
Applicant asserts (Pages 12-13), within Section A, that nowhere in MPEP 2106 is the specific claim language of claim 1 indicated as being directed to "an evaluation or judgment that can be performed in the human mind, or by a human using pen or paper, or a mathematical relationship"; that is merely an unsubstantiated conclusion on the part of the Examiner.
Examiner respectfully disagrees. As noted in the previous office action, independent claim 1 recites abstract ideas. The currently evaluated abstract ideas are noted within the office action as a-k within Step 2A Prong 1; where each limitation is broken down to which group of abstract ideas they fall under. The rejection follows the steps of the analysis as laid out in the MPEP which was followed for the previous and current examination (see MPEP 2106).
Applicant asserts (Page 13), within Section B, that the Examiner has failed to furnish actual proof/documentation to support the rejection. It is not the Applicants' burden to demonstrate compliance with the statute in the first instance; rather, under the statute, the Examiner must furnish actual proof/documentation to substantiate the rejection in the first instance - e.g., to effectuate a burden-switching as it pertains to the invocation of the exception. In this case, the Examiner has failed to furnish the requisite proof to date. The Examiner's rejection set forth in the Office Action continues to lack actual proof as required by the statute; in the absence of more, the rejection must be withdrawn. Stated differently, the net effect of the rejection is that it appears that the Examiner incorrectly believes that Congress allocated the initial burden of production to the Applicants to demonstrate that the claims comply with the statute.
Examiner respectfully disagrees. The office action establishes a proper and well-supported prima facie case as the claims are explained to be not patentable via the Patent Subject Matter Eligibility steps within MPEP 2106. The abstract idea limitations were not merely noted as abstract ideas. As noted above, within the previous Office Action, and below, each limitation was evaluated with the reasoning of why they were interpreted as mental process/mathematical concepts (noted within the parentheses).
Applicant asserts (Pages 13-14), within Section C, the Examiner was requested to furnish actual evidence/proof as to what is impractical about the features of, e.g., claim 1 and/or why the features of, e.g., claim 1 do not amount to an (integration in an) application. The current Office Action of February 4, 2026, fails to furnish any such proof This absence of actual proof implies that the Examiner cannot demonstrate that the exception in its totality applies. The skilled artisan will appreciate based on a review of the disclosure that the features of the independent claims are integrated as part of a practical application within the meaning of 35 U.S.C. 101. The remarks at pages 7-8 of the Office Action of February 4, 2026 (in purportedly addressing "C") fail to furnish actual proof that demonstrates that the features of claim 1 are not integrated into a practical application.
Examiner respectfully disagrees. The previous independent claim is no more detailed than applying the selection model logic for specific/restricted use cases to evaluate a specific result via evaluation metric/score (which is obtained based on specific restrictions), applying a mathematical relationship for the specific use cases to generate specific scores to determine a score which is associated with improvement, taking a snapshot, and exiting on specific criteria with no detail on the application of the determination/business snapshot. The newly added limitations are unable to provide improvement as they are currently being evaluated as either abstract idea(s) or additional elements that fall within MPEP 2106.05. The claims are directed towards the improvement of an abstract idea. Improvements to an abstract idea are still considered to an abstract idea. Additionally, the Claims do not reflect any improvement in the functioning of a computer or hardware processor rather the additional elements merely use a generic computer component to perform the abstract idea, is restricting the abstract idea to a particular technological environment, and/or data gathering. More discussed below with integrated as a practical application within (D).
Applicant asserts (Page 14), within Section D, was pointed out that the rejection failed to demonstrate that, e.g., claim 1 was directed to "well-understood, routine, or conventional activities". As discussed above in respect of the rejections under 35 U.S.C. 103, the claimed subject matter is believed to be distinguishable from the applied art. In this respect, it is appreciated and understood that the claimed subject matter is directed to "significantly more" than the alleged abstract idea identified by the Examiner. Should the Examiner persist in a rejection under 35 U.S.C. 101, it is requested that the Examiner specifically point out what condition stated in 35 U.S.C. 101 the Applicants have not complied with, particularly in view of the broad mandate 'any' set forth in 35 U.S.C. 101.
Examiner respectfully disagrees. Applicant’s arguments with respect to Claim 1’s limitations being "well-understood, routine, or conventional activities" are moot as the previous office action did not mention any additional limitations that were considered to fall within well-understood, routine, and conventional activity. The previous office action is not alleging that any of the elements are well-understood, routine, and conventional activities. Therefore, the claims do not integrate the judicial exception into a practical application nor amount to significantly more. The claim is not patent eligible. Although the Claims are interpreted in light of the specification, limitations from the specification are not read into the Claims.
MPEP 2106.05(a) recites:
After the examiner has consulted the specification and determined that the disclosed invention improves technology, the claim must be evaluated to ensure the claim itself reflects the disclosed improvement in technology … the claim must include the components or steps of the invention that provide the improvement described in the specification
…
It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. See the discussion of Diamond v. Diehr, 450 U.S. 175, 187 and 191-92, 209 USPQ 1, 10 (1981)) in subsection II, below.
Applicant fails to show how any alleged technical improvement would be provided by anything more than the judicial exception on its own. Additionally, applicant fails to show how the claim includes components or steps that would provide the alleged improvement described in the specification. By MPEP 2106.05(f)(1), "the claim recites only the idea of a solution or outcome, i.e. the claim fails to recite details of how a solution to a problem is accomplished". Moreover, the examiner maintains that the Claim does not impose any meaningful limits on the judicial exceptions. As noted in the rejection, the Claim does not include additional elements that are sufficient to amount to an integration of the identified abstract idea into a practical application, thus the claim is directed to an abstract idea.
Applicant asserts (Pages 15-17), within Section E, the Examiner was requested to identify where in the statute the "abstract idea" exception is contained to ensure/confirm that the statute is not being rewritten outside the legislative process reserved for the people's representatives. However, instead of providing the requested identification pertaining to where in the statute the "abstract idea" exception is contained, the Examiner at page 10 of the Office Action alleges that the rejection follows the steps and analysis as laid out in the MPEP (specifically, MPEP 2106). This is not a substantive reply by any reasonable measure4 , as it fails to identify where in the statute the exception is contained. As discussed above, it is in fact the Examiner's burden to demonstrate the applicability of a rejection with actual proof/evidence. The technology at issue in the cases referred to by MPEP 2106, including Alice, are inapplicable to the technology of the instant application. Thus, a citation to Alice ( or any of the other cases cited in MPEP 2106) proves nothing on a factual level/basis with respect to the claimed subject matter in the instant application. At the end of the day, the Examiner wants the reader of the Office Action to believe that the exception applies merely because the Examiner said as much. The blithe indifference demonstrated by the Examiner towards the Constitution (and the liberty interests protected in connection therewith) in respect of this application is astonishing.
Examiner respectfully disagrees. As stated previously, the rejection follows the steps of the analysis as laid out in the MPEP which was followed for the previous and current examination (see MPEP 2106). Thus, the office action does not fail to establish a proper and well-supported prima facie case as the claims are explained to be not patentable via the Patent Subject Matter Eligibility steps within MPEP 2106. The remarks/arguments laid forth by the applicant has been substantively and fully responded to.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-16, 18-19, and 21-22 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 1 further recites the device comprising of:
determining … that the candidate models are insufficient to solve a use case of a plurality of use cases (a human being can mentally apply evaluation to determine that specific models are insufficient to solve a specific use case)
selecting modeling logic for an artificial intelligence (Al) model that solves the use case of the plurality of use cases based on the plurality of models … (a human being can mentally apply evaluation and make a judgement to select modeling logic for an AI model that solves a specific use case within multiple use cases based on multiple models)
… forecasting groups of equipment and personnel to deploy … (a human being can mentally apply evaluation to forecast groups of equipment and personnel to deploy)
… determining destinations that have slots available to accommodate the groups of equipment and personnel … (a human being can mentally apply evaluation to determine destinations that have slots available to accommodate specific groups)
… determining transportation routes for bringing the groups of equipment and personnel from an origin to the destinations having the slots available … (a human being can mentally apply evaluation to determine specific transportation routes)
evaluating the sub-result based on an evaluation metric (a human being can mentally apply evaluation to evaluate the sub-result based on a metric)
combining the sub-result with other sub-results of the plurality of use cases to generate intermediate data … (a human being can mentally apply evaluation to combine results to generate intermediate data)
invoking a cost function for a business problem corresponding to the plurality of use cases on the intermediate data to obtain a score, wherein the cost function includes a length of time needed to achieve a deployment created by the plurality of use cases (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
determining … that the score is representative of an improvement (a human being can mentally apply evaluation to determine the score is representative of an improvement)
taking … a snapshot of a business solution corresponding to the … (a human being can mentally apply evaluation to take a snapshot of a business solution)
determining … whether an exit criteria has been met (a human being can mentally apply evaluation to determine whether an exit criteria has been met)
Claim 1 thus recites an abstract idea (that falls into the “mathematical concepts” or “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the additional elements consist of:
a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
receiving candidate models from robotic modeling equipment (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
… based on the … (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
obtaining … a plurality of models from databases sourced from model contributors comprised of data scientists, engineers, and technical staff (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
wherein the plurality of use cases includes a first use case for … , a second use case for … , and a third use case for … and wherein as part of the second use case an algorithm supplies: a list of new equipment, inventory constraints and availability (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
executing the AI model using holdout data to obtain a sub-result (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
wherein each of the other sub-results is obtained based on an invocation of a specific application program interface (API) of a plurality of APIs that communicates with the AI model (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself. Additional elements a and f are merely applying the abstract idea on a computer (MPEP 2106.05(f)) which cannot provide significantly more. Additional elements b and d fall within MPEP 2106.05(d) as well-understood, routine and conventional activities of receiving or transmitting data over a network (MPEP 2106.05(d)(II): buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)). Additional elements c, e, and g are only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claim 2:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 2 recites the device of Claim 1. Claim 1 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 2 further recites … determined based on a common pattern in the business problem (a human being can mentally apply evaluation to determination based on a specific pattern in a specific problem/scenario). Claim 2 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited. The judicial exception alone does not provide significantly more than the abstract idea itself. Thus, the claim is subject-matter ineligible.
Regarding Claim 3:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 3 recites the device of Claim 2. Claim 2 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 3 does not recite any additional abstract ideas and only inherits the abstract ideas from Claim 2. Claim 3 thus recites an abstract idea (that falls into the “mathematical concepts” or “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the new sole additional element recited consists of … comprises regression, classification, optimization, or a combination thereof (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h)).
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the new sole additional element recited, alone or in combination, does not provide significantly more than the abstract idea itself. The additional element is only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claim 4:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 4 recites the device of Claim 2. Claim 2 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 4 further recites … ranking the other sub-results based on the evaluation metric (a human being can mentally apply evaluation to rank the other sub-results based on a metric). Claim 4 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited. The judicial exception alone does not provide significantly more than the abstract idea itself. Thus, the claim is subject-matter ineligible.
Regarding Claim 5:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 5 recites the device of Claim 4. Claim 4 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 5 further recites … determining the exit criteria for the plurality of use cases … (a human being can mentally apply evaluation to determine the exit criteria for the plurality of use cases). Claim 5 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the new sole additional element recited consists of … options including: exit when the cost function is satisfied within a threshold, continue searching for better solutions until an execution time limit has expired, or execute for a predefined number of iterations (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h)).
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the new sole additional element recited, alone or in combination, does not provide significantly more than the abstract idea itself. The additional element is only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claim 6:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 6 recites the device of Claim 5. Claim 5 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 6 further recites … formulates the modeling logic for the AI model (a human being can mentally apply evaluation to formulate the modeling logic for the AI model). Claim 6 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited. The judicial exception alone does not provide significantly more than the abstract idea itself. Thus, the claim is subject-matter ineligible.
Regarding Claim 7:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 7 recites the device of Claim 6. Claim 6 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 7 does not recite any additional abstract ideas and only inherits the abstract ideas from Claim 6. Claim 7 thus recites an abstract idea (that falls into the “mathematical concepts” or “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the new sole additional element recited consists of … training the AI model using training data (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f)).
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the new sole additional element recited, alone or in combination, does not provide significantly more than the abstract idea itself. The additional element is merely applying the abstract idea on a computer (MPEP 2106.05(f)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claim 8:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 8 recites the device of Claim 7. Claim 7 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 8 further recites … performing data wrangling on the training data and the holdout data (a human being can mentally apply evaluation to perform data wrangling on data). Claim 8 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited. The judicial exception alone does not provide significantly more than the abstract idea itself. Thus, the claim is subject-matter ineligible.
Regarding Claim 9:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 9 recites the device of Claim 8. Claim 8 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 9 does not recite any additional abstract ideas and only inherits the abstract ideas from Claim 8. Claim 9 thus recites an abstract idea (that falls into the “mathematical concepts” or “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the new sole additional element recited consists of … comprises a plurality of processors operating in a distributed computing environment (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h)).
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the new sole additional element recited, alone or in combination, does not provide significantly more than the abstract idea itself. The additional element is only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claims 10-16:
Claims 10-16 incorporate substantively all the limitations of Claims 1-8 in a non-transitory, machine-readable medium (thus, a manufacture) and further recites comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising (these claim limitations appear to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f)) and does not appear to integrate the abstract idea into a particular application; thus, the claim is subject-matter ineligible as it does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself); thus, Claims 10, 11, 12, 13-16 are rejected for reasons set forth in the rejections of Claims 1, 4, 2-3, 5-8, respectively.
Regarding Claim 18:
Subject Matter Eligibility Analysis Step 1:
Claim 18 recites a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 18 further recites the method comprising of:
determining … that the candidate models are insufficient to solve a use case of a plurality of use cases (a human being can mentally apply evaluation to determine that specific models are insufficient to solve a specific use case)
formulating … modeling logic for an artificial intelligence (Al) model that solves the use case of a plurality of use cases based on the plurality of models … (a human being can mentally apply evaluation and make a judgement to formulate modeling logic for an AI model that solves a specific use case within multiple use cases based on multiple models)
… forecasting groups of equipment and personnel to deploy … (a human being can mentally apply evaluation to forecast groups of equipment and personnel to deploy)
… determining destinations that have slots available to accommodate the groups of equipment and personnel … (a human being can mentally apply evaluation to determine destinations that have slots available to accommodate specific groups)
… determining transportation routes for bringing the groups of equipment and personnel from an origin to the destinations having the slots available … (a human being can mentally apply e evaluation to determine transportation routes)
evaluating … the sub-result based on an evaluation metric (a human being can mentally apply evaluation to evaluate the sub-result based on a metric)
combining … plural sub-results of the plurality of use cases to generate intermediate data (a human being can mentally apply evaluation to combine results to generate intermediate data)
invoking … a cost function for a business problem corresponding to the plurality of use cases on the intermediate data to obtain a score, wherein the cost function includes a length of time needed to achieve a deployment created by the plurality of use cases (a mathematical relationship between variables and/or numbers using a mathematical formula/equations)
determining, … and based on the invoking, that the score is representative of an improvement (a human being can mentally apply evaluation to determine the score is representative of an improvement)
taking … a snapshot of a business solution corresponding to the … (a human being can mentally apply evaluation to take a snapshot of a business solution)
determining, … whether an exit criteria has been met (a human being can mentally apply evaluation to determine whether an exit criteria has been met)
Claim 18 thus recites an abstract idea (that falls into the “mathematical concepts” or “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because the additional elements consist of:
receiving … candidate models from robotic modeling equipment (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
… by a processing system including a processor … (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
… based on the … (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
obtaining … a plurality of models from databases sourced from model contributors comprised of data scientists, engineers, and technical staff (which is insignificant extra-solution activity of data gathering, by MPEP 2106.05(g))
wherein the plurality of use cases includes a first use case for … , a second use case for … , and a third use case for … and wherein as part of the second use case an algorithm supplies: a list of new equipment, inventory constraints and availability (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
executing … the AI model using holdout data … to obtain a sub-result (to perform a mental process and the performance of an abstract idea on a computer is no more than instructions to “apply it” on a computer, by MPEP 2106.05(f))
wherein each of the plural sub-results is obtained based on an invocation of a specific application program interface (API) of a plurality of APIs that communicates with the AI model (which is restricting the abstract idea to a Particular Technological Environment, by MPEP 2106.05(h))
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself. Additional elements a and d fall within MPEP 2106.05(d) as well-understood, routine and conventional activities of receiving or transmitting data over a network (MPEP 2106.05(d)(II): buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)). Additional elements b and f are merely applying the abstract idea on a computer (MPEP 2106.05(f)) which cannot provide significantly more. Additional elements c, e, and g are only restricting the abstract idea to a Particular Technological Environment (MPEP 2106.05(h)) which cannot provide significantly more. Thus, the claim is subject-matter ineligible.
Regarding Claim 19:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 19 recites the method of Claim 18. Claim 18 is a method, thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 19 further recites the method comprising of:
dividing … the business problem into the plurality of use cases (a human being can mentally apply evaluation to determine whether an exit criteria has been met)
ranking … the plural sub-results … (a human being can mentally apply evaluation to rank the sub-results based on the evaluation metric)
Claim 19 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited. The judicial exception alone does not provide significantly more than the abstract idea itself. Thus, the claim is subject-matter ineligible.
Regarding Claim 21:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 21 recites the device of Claim 1. Claim 1 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 21 further recites the device comprising of wherein the operations further comprise: cataloging the use case, resulting in a catalogued use case (a human being can mentally apply evaluation to organize and catalog the use case resulting in a cataloged use case). Claim 21 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited. The judicial exception alone does not provide significantly more than the abstract idea itself. Thus, the claim is subject-matter ineligible.
Regarding Claim 22:
Subject Matter Eligibility Analysis Step 1:
Dependent Claim 22 recites the device of Claim 21. Claim 21 is a device, thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
However, Claim 22 further recites selecting second modeling logic for the AI model to solve a second business problem using the catalogued use case, wherein the second modeling logic is different from the modeling logic, and wherein the determining that the score is representative of the improvement is based on a comparison of the score with another score (a human being can mentally apply select modeling logic for a second business problem using the catalogues use case). Claim 22 thus recites an abstract idea (that falls into the “mental processes” group of abstract ideas).
Subject Matter Eligibility Analysis Step 2A Prong 2:
This judicial exception is not integrated into a practical application because there are no new additional elements recited.
Subject Matter Eligibility Analysis Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because there are no new additional elements recited. The judicial exception alone does not provide significantly more than the abstract idea itself. Thus, the claim is subject-matter ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-16, 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Nakandala et al., “Cerebro: A Data System for Optimized Deep Learning Model Selection”, in view of Wang et al., “DeepSTCL: A Deep Spatio-temporal ConvLSTM for Travel Demand Prediction”, in view of Aarts et al., US-2021/0192314-A1, in view of Hou et al., US-20210325861-A1.
Regarding Claim 1:
Nakandala teaches:
A device, comprising: a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
(Nakandala, [4. SYSTEM OVERVIEW], Page 2163, Figure 4; [6. EXPERIMENTAL EVALUATION], Page 2166, Paragraph 4 “Experimental Setup. We use two clusters: CPU-only for Criteo and GPU-enabled for ImageNet, both on Cloud- Lab [19]. Each cluster has 8 worker nodes and 1 master node. Each node in both clusters has two Intel Xeon 10- core 2.20 GHz CPUs, 192GB memory, 1TB HDD and 10 Gbps network” Figure 4 shows the system architecture of Cerebro containing the Cluster, Task Executor, and Scheduler. The clusters contain the processor/memory and interacts with the Task Executor (unit training/validation on cluster and model hopping) which interacts with the Scheduler (responsible for workload).
…
selecting modeling logic for an artificial intelligence (Al) model that solves the use case of the plurality of use cases based on the plurality of models, …
(Nakandala, [4. SYSTEM OVERVIEW], Page 2163, Paragraph 4, “We present an overview of Cerebro, an ML system that uses MOP to execute deep net model selection workloads”; [1. INTRODUCTION], Page 2159, “Case Study. We present a real-world model selection scenario. Our public health collaborators at UC San Diego wanted to try deep nets for identifying different activities (e.g., sitting, standing, stepping, etc.) of subjects from body-worn accelerometer data… During model selection, we tried different deep net architectures such as…” Cerebro is a machine learning system that executes model selection for deep neural networks that are selected to identify a plurality of use cases (different types of activities in this scenario) through parsing input data).
executing the AI model using holdout data to obtain a sub-result;
(Nakandala, [4.1 User-facing API], Page 2163, Paragraph 6, “Cerebro takes the reference to the dataset, set of initial training configs, the AutoML procedure, and 3 user defined functions: input_fn, model_fn, and train_fn. It first invokes input_fn to read and pre-process the data. It then invokes model_fn to instantiate the neural architecture… The train_fn is invoked to perform one sub-epoch of training. We assume validation data is also partitioned and use the same infrastructure for evaluation”; Page 2163, Figure 4. Cerebro invokes the neural architecture (shown in Figure 4), executes the AI model (neural network) with holdout data (the Examiner interprets holdout data as data that is separate from training data and used for validation; thus synonymous with validation/test data) to obtain validation results (as Nakandala notes that the validation data is partitioned the same way for evaluation). A validation result (task result) is a model performance result of the executed model’s task; thus, a sub-result as it is one result out of a set of results used to compare model performances).
evaluating the sub-result based on an evaluation metric;
(Nakandala, Page 2165, Algorithm 1 & 2; Page 2163, [6.2 Drilldown Experiments], Page 2168, Paragraph 3, “We evaluate 5 batch sizes and report makespans and the validation error of the best model for each batch size after 10 epochs”; Page 2168, Figure 9. The validation error and makespans (shown in Figure 9) are evaluating the sub-results based on makespans/scheduling and validation errors/loss (which are interpreted by the examiner as the evaluation metrics)).
combining the sub-result with other sub-results of the plurality of use cases to generate intermediate data …
(Nakandala, Page 2163, Figure 4; Page 2164, Figure 5. Figure 5 shows the combining of the sub-results by the schedulers which is scheduling the task (validation) result; thus, the Cerebro scheduler is combining the sub-results with other sub-results (scheduling tasks together) to generate intermediate data (which the examiner interprets as the scheduling data (as the data has not been processed and is in a intermediatory form and merely scheduled to be executed within the Task Executor (Figure 4))).
invoking a cost function for a business problem corresponding to the plurality of use cases on the intermediate data to obtain a score,
(Nakandala, [5.1 Formal Problem Statement as MILP], Page 2164, Paragraph 6, “The objective and constraints of the MOP-based scheduling problem is as follows …
PNG
media_image1.png
172
329
media_image1.png
Greyscale
…”. Equation 1 is the objective function to minimize makespan workload (C) with respects to the constraints where the business problem is interpreted as scheduling for optimizing resource utilization and computational costs/runtimes with specific constraints; thus, the objective function is interpreted as a cost function for a business problem by the examiner where C is a makespan score).
wherein the cost function includes a length of time needed to achieve a deployment created by the plurality of use cases;
(Nakandala, [5.4 Comparing Different Scheduling Methods], Page 2165, Paragraph 8, “We set a maximum optimization time of 5min for tractability sake. We compare the scheduling methods on 3 dimensions … Sub-epoch training time (unit time) of a training config is directly proportional to the compute cost of the config and inversely proportional to compute capacity of the worker … heterogeneous setting, training config compute costs are randomly sampled (with replacement) from a set of popular deep CNNs (n=35) obtained from [3]. The costs vary from 360 MFLOPS to 21000 MFLOPS with a mean of 5939 MFLOPS and standard deviation of 5671 MFLOPS. Due to space constraints we provide these computational costs in the Appendix …”; Page 2166, Figure 6; Page 2167, Figure 7. C is the makespan score which includes a length of time as a makespan is a length of time (total time to complete a set of tasks (time needed to achieve a deployment of multi-model parallel task scheduling))).
determining, based on the invoking, that the score is representative of an improvement;
(Nakandala, Page 2166, Figure 6. Figure 6 shows a depiction of determining that the score (makespan) is representative of an improvement when scheduling with the randomized scheduler within Cerebro. Figure 9 shows runtime as well and validation error % to depict the improvement).
taking, based on the determining that the score is representative of the improvement, a snapshot of a business solution corresponding to the combining of the sub-result with the other sub-results; and
(Nakandala, Page 2166, Figure 6. Figure 7 shows the results corresponding to the makespan schedules for the different systems; thus, a snapshot of a business solution corresponding to the combining of the sub-result with the other sub-results. Figure 9 shows runtime as well and validation error % for the snapshots of Cerebro compared to Horovod).
determining, based on the combining, whether an exit criteria has been met.
(Nakandala, Page 2165, Algorithm 1 & 2; Algorithm 1 and 2 are used in the Randomized Algorithm-based Scheduler and will continue to execute until the exit criteria is met (which is when Q (set of all validation units) is empty and leaves the workers and models idle)).
While Nakandala teaches selecting modeling logic for an Al model that solves a use case of a plurality of use cases… Nakandala does not explicitly disclose the specific use cases.
However, Wang explicitly discloses:
wherein the plurality of use cases includes a first use case for forecasting groups of equipment and personnel to deploy,
(Wang, Page 1, Column 2, Paragraph 1, “Thus, the order data is used to predict travel demand, achieve appropriate urban resource scheduling and provide better services for passengers in this mode. In this paper, a Deep Spatio-Temporal Convolutional LSTM (DeepSTCL) is proposed to forecast travel demand which considers the time and space factors comprehensively and gets a great prediction performance”; Abstract, “Therefore, it is significant to predict travel demand for urban resource dispatching”. DeepSTCL is used for forecasting travel demand, where travel demand is the need or desire to travel based on geography, travel patterns, destinations, etc. The forecasting of travel demand is needed for urban resource dispatching (how a city manages and distributes (interpreted by the examiner as deploys) its resources, including personnel, equipment, vehicles and other assets to ensure effective service delivery). Thus, the use case of forecasting groups of equipment and personnel to deploy is taught by Wang).
a second use case for determining destinations that have slots available to accommodate the groups of equipment and personnel,
(Wang, Fig. 1, 2 & 9; Page 1, Column 2, Paragraph 2, “Travel demand data is typical spatio-temporal data”; Page 7, Paragraph 3, “Travel demand modeling is an inherent part of smarter transportation. Analyzing and forecasting travel demand can help us manage the hot spot of passenger demand in the next period, balance supply and demand and schedule vehicle resources for passengers”; Abstract, “Urban resource scheduling is an important part of the development of a smart city, and transportation resources are the main components of urban resources. Currently, a series of problems with transportation resources such as unbalanced distribution and road congestion disrupt the scheduling discipline”. The deep learning traffic demand forecasting framework is based on spatio-temporal data which allows for analyzing congestion and unbalanced deployment of equipment/personnel. Fig. 1 shows a pictorial example of a geographical rectangle and Fig. 2 shows an example of a snapshot of an order count (order demand/requests) where both are used to create the heatmaps shown in Fig. 9 (which highlights the forecasted scenarios/situations for travel demand). Thus, the method of DeepSTCL determines destinations (location for urban resource scheduling) that have slots available to accommodate (capacity based resource scheduling to avoid congestion/unbalanced distribution) the groups of equipment and personnel (deployable urban resources such as transportation resources)).
and a third use case for determining transportation routes for bringing the groups of equipment and personnel from an origin to the destinations having the slots available,
(Wang, Fig. 9; Page 1, Column 2, Paragraph 2, “Travel demand data is typical spatio-temporal data”; Abstract, “Urban resource scheduling is an important part of the development of a smart city, and transportation resources are the main components of urban resources. Currently, a series of problems with transportation resources such as unbalanced distribution and road congestion disrupt the scheduling discipline”. Urban resource scheduling’s main component is transportation resources as they cause issues such as unbalanced distribution and road congestion (both of which are capacity based and interpreted by the examiner as available slots). By forecasting travel demand accurately, transportation routes are able to be optimized for scheduling deployments (where scheduling is based off routing from origin to endpoint using travel demand prediction). Travel demand heatmaps (such as Fig. 9) are utilized for equipment/personnel deployment (which is the scheduling of urban resources to avoid unbalanced deployments and road congestion)).
and wherein as part of the second use case an algorithm supplies: a list of new equipment, inventory constraints and availability;
(Wang, Fig. 1, 2 & 9; Page 5, Column 2, Paragraph 4, “
PNG
media_image2.png
126
272
media_image2.png
Greyscale
”. As noted previously, Fig. 1 and 2 show example snapshots of an order count to create heatmaps shown in Fig. 9; where, the second use case is determining the destinations by the corresponding urban resource scheduling of the equipment (resources). As the resource scheduling is utilizing the heatmap for travel demand (interpreted as inventory constraint and availability) which is shown in the matrix (list) for order count (new requested trips; which is interpreted as new requested resources i.e. new equipment/trips. Thus, the algorithm is supplying the data for resource scheduling).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the Nakandala’s process of selecting of the modeling logic to solve a use case, with the plurality of specific use cases taught by Wang to illustrate the importance of being able to analyze and forecast travel demand based on spatio-temporal data to manage efficiency and optimize distribution scheduling (see Wang, Page 7, Column 1, Paragraph 3, “Analyzing and forecasting travel demand can help us manage the hot spot of passenger demand in the next period, balance supply and demand and schedule vehicle resources for passengers … ConvLSTM-based deep learning model for travel demand (ST Data) prediction is proposed that takes advantage of both temporal and spatial properties … Our models’ performances are significantly beyond two baseline models, confirming that it is better and more flexible for the travel demand prediction”).
Nakandala/Wang do not explicitly teach:
… wherein each of the other sub-results is obtained based on an invocation of a specific application program interface (API) of a plurality of APIs that communicates with the AI model;
However, Aarts teaches:
… wherein each of the other sub-results is obtained based on an invocation of a specific application program interface (API) of a plurality of APIs that communicates with the AI model;
(Aarts, Page 58, [0556], “… a software layer may be implemented as a … API through which … may be invoked (e.g., called) … for performing compute, AI, or … to perform processing tasks in an effective and efficient manner”; Page 5, [0098], “… an application programming interface comprises a concatenation function for combining forward and reverse outputs of a ragged bidirectional recurrent neural network.”; FIG. 8. FIG. 8 depicts an example process where an API call (invocation) can occur for a specific API that communicates with the example RNN (recurrent neural network -> AI Model); where the graph definition and the recurrence attribute are considered the sub-results within this example).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the process of Nakandala/Wang process of selecting modeling logic to solve specific use cases, with the API calls of Aarts to generate optimized processes/results, automation, reduce complexity, technical advantages, etc. (see Aarts, FIG 8, Page 2, [0062], “In at least one embodiment, a graph is made to represent a recurrent neural network by associating a recurrence attribute 110 with the graph. In at least one embodiment, said graph is a nested graph. In at least one embodiment, association of a recurrence property with a graph effectively makes recurrence or looping an attribute of said graph. In at least one embodiment, use of a recurrence property in an application programming interface provides a technical advantage over use of a while loop, or similar programming construct, which may require construction of separate graphs to represent header, body, and exit portions of a while loop”).
Nakandala/Wang/Aarts do not explicitly teach:
receiving candidate models from robotic modeling equipment;
determining, based on the receiving of the candidate models, that the candidate models are insufficient to solve a use case of a plurality of use cases;
obtaining, based on the determining that the candidate models are insufficient to solve the use case of the plurality of use cases, a plurality of models from databases sourced from model contributors comprised of data scientists, engineers, and technical staff;
However, Hou teaches:
receiving candidate models from robotic modeling equipment;
(Hou, Fig 8; Fig. 9; Page 12, Column 2, [0102], “FIG. 9 … there are 3 robots performing the same screw driving task in a factory ( e.g., Robot A, Robot B, and Robot C) … each robot gets 3 model candidates to predict if a screw driving process is faulty or not faulty …”. Figure 9: Rows 916, 918, and 920 are for 3 different model candidates (also shown in Fig. 8) for screw driving assembly via Robot A (which is for a screw driving task). Thus, Hou’s system teaches receiving candidate models from robotic modeling equipment).
determining, based on the receiving of the candidate models, that the candidate models are insufficient to solve a use case of a plurality of use cases;
(Hou, Fig. 8; Page 11, Column 1, [0087], “… the example intelligent deployment circuitry 510 outputs a prediction of model combinations to improve prediction performance over time. For example, the intelligent deployment circuitry 510 may output a prediction of model combinations to improve prediction performance over time by monitoring the plurality of models produced, removing outliers, and/or storing models that are performing near a threshold of quality …”. The Intelligent Deployment Circuity determines. based on the receiving of the candidate models (shown in Fig. 8), ways to improve a prediction of model combinations such as removing outliers, storing models and more. Thus, the examiner interprets the module 510 from Fig. 8 to be determining if a candidate model is insufficient (due to the module needing to remove a candidate model for poor performance)).
obtaining, based on the determining that the candidate models are insufficient to solve the use case of the plurality of use cases, a plurality of models from databases sourced from model contributors comprised of data scientists, engineers, and technical staff;
(Hou, Fig. 9; Page 13, Column 1, [0108], “… FIG. 9 is an example management console … wherein a biological entity such as a human being may track the evolving ensemble of models. For example, the human being (engineer, worker, foreman, data scientist, etc.) may track the performance of the models and intervene ( e.g., adjust the model) in response to a determination made by the human being …”; Page 20, Column 2, [0168], “… in response to the intelligent trigger circuitry triggering the automated model update process, generates a plurality of candidate artificial intelligence models or selects a plurality of candidate artificial intelligence models from a repository of trained artificial intelligence models”. The repository of trained artificial intelligence models and ability to manage and track the performance of models … allows a user to generate new candidate models or select candidate models (due to weak or poor or insufficient performance for a use case) from a repository (which have been contributed to by model contributes such as data scientists, engineers, and technical staff)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the process of Nakandala/Wang/Aarts process of selecting modeling logic to solve specific use cases, with the receiving of candidate models to determine insufficient performance for robotic equipment data and gathering new model data from specific model contributors to optimize models in the repository and improve prediction performance over time(see Hou, Fig. 6; Page 12, Column 1, [0099], “At block 722, the example intelligent deployment circuitry 510 stores optimized models in the model repository 404. For example, the example intelligent deployment circuitry 510 may store a candidate model that performs above a threshold in the example model repository 404 with the metadata corresponding to the factory production line that is monitored by the optimized candidate model. The optimized candidate model is stored and may be selected by the example automated model search circuitry 508 at a later time …”; Page 11, Column 1, [0087], “At block 606, the example intelligent deployment circuitry 510 outputs a prediction of model combinations to improve prediction performance over time. For example, the intelligent deployment circuitry 510 may output a prediction of model combinations to improve prediction performance over time by monitoring the plurality of models produced, removing outliers, and/or storing models that are performing near a threshold of quality …”).
Regarding Claim 2:
Nakandala/Wang/Aarts/Hou teach the device of Claim 1 and Nakandala further teaches:
wherein each use case in the plurality of use cases is determined based on a common pattern in the business problem.
(Nakandala, [1. INTRODUCTION], Page 2159, Paragraph 2, “Case Study. We present a real-world model selection scenario. Our public health collaborators at UC San Diego wanted to try deep nets for identifying different activities (e.g., sitting, standing, stepping, etc.) of subjects from body-worn accelerometer data…” The examiner interprets a business problem as a challenge an organization is facing. The case study taught by Nakandala notes a business problem (to identify different activities of subjects wearing accelerometers for the public health collaborators at UC San Diego) where the use cases are different activities based on a common pattern (e.g., sitting, standing, stepping, etc.).
Regarding Claim 3:
Nakandala/Wang/Aarts/Hou teach the device of Claim 2 and Nakandala further teaches:
wherein the common pattern comprises regression, classification, optimization, or a combination thereof.
(Nakandala, [7. DISCUSSION AND LIMITATIONS], Page 2169, Paragraph 4, “Applications. Cerebro is in active use for time series analytics for our public health collaborators. In the case study from Section 1, Cerebro helped us pick 16 deep net configs to compare. To predict sitting vs. not-sitting, these configs had accuracies between 62% and 93%, underscoring the importance of rigorous model selection… However, note that MOP and Cerebro's ideas are directly usable for model selection of any ML models trainable with SGD. Examples include linear/logistic regression, some support vector machines, low-rank matrix factorization, and conditional random fields.” The case study for the public health collaborators is for classification with the use of stochastic gradient descent (optimization algorithm). Nakandala teaches optimization (the Examiner interprets optimization in terms of accuracy) and other machine learning models that can be used instead of stochastic gradient descent such as different regression algorithms).
Regarding Claim 4:
Nakandala/Wang/Aarts/Hou teach the device of Claim 2 and Nakandala further teaches:
wherein the operations further comprise ranking the other sub-results based on the evaluation metric.
(Nakandala, Page 2164, Figure 5 and [5. CEREBRO SCHEDULER] Paragraph 4, “Consider the model selection workload shown in Figure 5(A). Assume workers are homogeneous and there is no data replication. For one epoch of training, Figure 5(B) shows an optimal task-parallel schedule for this workload with a 9-unit makespan. Figure 5(C) shows a non-optimal MOP schedulewith also 9 units makespan. But as Figure 5(D) shows, an optimal MOP schedule has a makespan of only 7 units. Overall, we see that MOP's training unit-based scheduling offers more flexibility to raise resource utilization”. Figure 5 denotes the different scheduling that occurs for model selection workloads. The ranking is done by optimization (resource utilization) and runtime (denoted in makespan units)).
Regarding Claim 5:
Nakandala/Wang/Aarts/Hou teach the device of Claim 4 and Nakandala further teaches:
wherein the operations further comprise determining the exit criteria for the plurality of use cases, wherein the exit criteria comprise options including: exit when the cost function is satisfied within a threshold, continue searching for better solutions until an execution time limit has expired, or execute for a predefined number of iterations.
(Nakandala, Page 2165, Algorithm 1 & 2; [5.4 Comparing Different Scheduling Methods], Page 2165, Paragraph 8,“We use simulations to compare the efficiency and makespans yielded by the three alternative schedulers… We set a maximum optimization time of 5min for tractability sake” Algorithm 1 & 2 are used within the randomized scheduler to schedule tasks and once Q is empty (all dataset units) the scheduler has met the exit criteria as all units were removed (leaving the workers and models idle). Section 5.4 discusses comparing different scheduling methods and Nakandala teaches time limit constraints (time limit expiry) when scheduling tasks for a model. Also, the amount of time it takes for Q to become empty can be interpreted as a cost function (the examiner interprets a cost function as mapping values with an event; in this scenario cost would be the total runtime of the scheduler)).
Regarding Claim 6:
Nakandala/Wang/Aarts/Hou teach the device of Claim 5 and Nakandala further teaches:
wherein the device formulates the modeling logic for the AI model.
(Nakandala, [4.2 System Architecture], Page 2163, Paragraph 8, “Supporting Multiple Deep Learning Tools. The functions input_fn, model_fn, and train_fn are written by users in the deep learning tool's APIs. We currently support TensorFlow and PyTorch (it is simple to add support for more). To support multiple such tools, we adopt a handler-based architecture…” Cerebro applies deep learning tools to formulate/configure the model logic that will be used by the scheduler).
Regarding Claim 7:
Nakandala/Wang/Aarts/Hou teach the device of Claim 6 and Nakandala further teaches:
wherein the operations further comprise training the AI model using training data.
(Nakandala, [4.1 User-facing API], Page 2163, Paragraph 6, “Cerebro takes the reference to the dataset, set of initial training configs, the AutoML procedure, and 3 user defined functions: input_fn, model_fn, and train_fn. It first invokes input_fn to read and pre-process the data. It then invokes model_fn to instantiate the neural architecture… The train_fn is invoked to perform one sub-epoch of training”).
Regarding Claim 8:
Nakandala/Wang/Aarts/Hou teach the device of Claim 7 and Nakandala further teaches:
wherein the operations further comprise performing data wrangling on the training data and the holdout data.
(Nakandala, Page 2166, Table 4. Table 4 provides the dataset details (all data used within the benchmark datasets for experimenting which contains training and holdout data). The values provided are after preprocessing which includes data wrangling (examiner interprets data wrangling as transforming data) as Nakandala notes the data being encoded and densified in [6. EXPERIMENTAL EVALUATION]: Page 2166, Paragraph 2).
Regarding Claim 9:
Nakandala/Wang/Aarts/Hou teach the device of Claim 8 and Nakandala further teaches:
wherein the processing system comprises a plurality of processors operating in a distributed computing environment.
(Nakandala, [4.3 System Implementation Details], Page 2164, Paragraph 3, “We prototype Cerebro in Python using XML-RPC client-server package. Scheduler runs on the client. Each worker runs a single service. Scheduling follows a push-based model-Scheduler assigns tasks and periodically checks the responses from the workers. We use a shared network file system (NFS) as the central repository for models. Model hopping is realized implicitly by workers writing models to and reading models from this shared file system”; [6. EXPERIMENTAL EVALUATION], Page 2166, “Experimental Setup. We use two clusters: CPU-only for Criteo and GPU-enabled for ImageNet, both on Cloud- Lab [19]. Each cluster has 8 worker nodes and 1 master node. Each node in both clusters has two Intel Xeon 10- core 2.20 GHz CPUs, 192GB memory, 1TB HDD and 10 Gbps network. Each GPU cluster worker node has an extra Nvidia P100 GPU. All nodes run Ubuntu 16.04.” The examiner interprets a distributed computing environment as computer network setup where databases are located within multiple nodes allowing access locally/remotely (Nakandala teaches both remote/local reading from the partitions within [6.2 Drill-down experiments], Page 2168, “In this setting, the dataset is partitioned, replicated, and stored on 8 workers. We then load all local data partitions into each worker's memory. Celery performs remote reads for nonlocal partitions”)).
Regarding Claims 10-16:
Claims 10-16 incorporate substantively all the limitations of Claims 1-8 in a non-transitory, machine-readable medium and further recites comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising (Nakandala, Page 2163, Figure 4; [6. EXPERIMENTAL EVALUATION], Page 2166, Paragraph 4 “Experimental Setup. We use two clusters: CPU-only for Criteo and GPU-enabled for ImageNet, both on Cloud- Lab [19]. Each cluster has 8 worker nodes and 1 master node. Each node in both clusters has two Intel Xeon 10- core 2.20 GHz CPUs, 192GB memory, 1TB HDD and 10 Gbps network”. Figure 4 shows the system architecture of Cerebro containing the Cluster, Task Executor, and Scheduler. The clusters contain the processor/memory and interacts with the Task Executor (unit training/validation on cluster and model hopping) which interacts with the Scheduler (responsible for workload). Thus, the experiments done on the clusters are being done on a processor and a CRM is inherent); thus, Claims 10, 11, 12, 13-16 are rejected for reasons set forth in the rejections of Claims 1, 4, 2-3, 5-8, respectively.
Regarding Claim 18:
Nakandala teaches:
A method, comprising: …
formulating, by the processing system, modeling logic for an artificial intelligence (AI) model that solves the use case of a plurality of use cases based on the plurality of models, …
(Nakandala, [4. SYSTEM OVERVIEW], Page 2163, Paragraph 4, “We present an overview of Cerebro, an ML system that uses MOP to execute deep net model selection workloads”; [1. INTRODUCTION], Page 2159, “Case Study. We present a real-world model selection scenario. Our public health collaborators at UC San Diego wanted to try deep nets for identifying different activities (e.g., sitting, standing, stepping, etc.) of subjects from body-worn accelerometer data… During model selection, we tried different deep net architectures such as…” Cerebro is a machine learning system that executes model selection method for deep neural networks that are selected to identify a plurality of use cases (different types of activities in this scenario) through parsing input data).
executing, by the processing system, the AI model using holdout data to obtain a sub-result;
(Nakandala, [4.1 User-facing API], Page 2163, Paragraph 6, “Cerebro takes the reference to the dataset, set of initial training configs, the AutoML procedure, and 3 user defined functions: input_fn, model_fn, and train_fn. It first invokes input_fn to read and pre-process the data. It then invokes model_fn to instantiate the neural architecture… The train_fn is invoked to perform one sub-epoch of training. We assume validation data is also partitioned and use the same infrastructure for evaluation”; Page 2163, Figure 4. Cerebro invokes the neural architecture (shown in Figure 4), executes the AI model (neural network) with holdout data (the Examiner interprets holdout data as data that is separate from training data and used for validation; thus synonymous with validation/test data) to obtain validation results (as Nakandala notes that the validation data is partitioned the same way for evaluation). A validation result (task result) is a model performance result of the executed model’s task; thus, a sub-result as it is one result out of a set of results used to compare model performances).
evaluating, by the processing system, the sub-result based on an evaluation metric; and
(Nakandala, Page 2165, Algorithm 1 & 2; Page 2163, [6.2 Drilldown Experiments], Page 2168, Paragraph 3, “We evaluate 5 batch sizes and report makespans and the validation error of the best model for each batch size after 10 epochs”; Page 2168, Figure 9. The validation error and makespans (shown in Figure 9) are evaluating the sub-results based on makespans/scheduling and validation errors/loss (which are interpreted by the examiner as the evaluation metrics)).
combining, by the processing system, plural sub-results of the plurality of use cases to generate intermediate data …
(Nakandala, Page 2163, Figure 4; Page 2164, Figure 5. Figure 5 shows the combining of the sub-results by the schedulers which is scheduling the task (validation) result; thus, the Cerebro scheduler is combining the sub-results with other sub-results (scheduling tasks together) to generate intermediate data (which the examiner interprets as the scheduling data (as the data has not been processed and is in a intermediatory form and merely scheduled to be executed within the Task Executor (Figure 4))).
invoking, by the processing system, a cost function for a business problem corresponding to the plurality of use cases on the intermediate data to obtain a score,
(Nakandala, [5.1 Formal Problem Statement as MILP], Page 2164, Paragraph 6, “The objective and constraints of the MOP-based scheduling problem is as follows …
PNG
media_image1.png
172
329
media_image1.png
Greyscale
…”. Equation 1 is the objective function to minimize makespan workload (C) with respects to the constraints where the business problem is interpreted as scheduling for optimizing resource utilization and computational costs/runtimes with specific constraints; thus, the objective function is interpreted as a cost function for a business problem by the examiner where C is a makespan score).
wherein the cost function includes a length of time needed to achieve a deployment created by the plurality of use cases;
(Nakandala, [5.4 Comparing Different Scheduling Methods], Page 2165, Paragraph 8, “We set a maximum optimization time of 5min for tractability sake. We compare the scheduling methods on 3 dimensions … Sub-epoch training time (unit time) of a training config is directly proportional to the compute cost of the config and inversely proportional to compute capacity of the worker … heterogeneous setting, training config compute costs are randomly sampled (with replacement) from a set of popular deep CNNs (n=35) obtained from [3]. The costs vary from 360 MFLOPS to 21000 MFLOPS with a mean of 5939 MFLOPS and standard deviation of 5671 MFLOPS. Due to space constraints we provide these computational costs in the Appendix …”; Page 2166, Figure 6; Page 2167, Figure 7. C is the makespan score which includes a length of time as a makespan is a length of time (total time to complete a set of tasks (time needed to achieve a deployment of multi-model parallel task scheduling))).
determining, by the processing system and based on the invoking, that the score is representative of an improvement;
(Nakandala, Page 2166, Figure 6. Figure 6 shows a depiction of determining that the score (makespan) is representative of an improvement when scheduling with the randomized scheduler within Cerebro. Figure 9 shows runtime as well and validation error % to depict the improvement).
taking, by the processing system and based on the determining, that the score is representative of the improvement, a snapshot of a business solution corresponding to the combining of the sub-result with the other sub-results; and
(Nakandala, Page 2166, Figure 6. Figure 7 shows the results corresponding to the makespan schedules for the different systems; thus, a snapshot of a business solution corresponding to the combining of the sub-result with the other sub-results. Figure 9 shows runtime as well and validation error % for the snapshots of Cerebro compared to Horovod).
determining, by the processing system and based on the combining, whether an exit criteria has been met.
(Nakandala, Page 2165, Algorithm 1 & 2; Algorithm 1 and 2 are used in the Randomized Algorithm-based Scheduler and will continue to execute until the exit criteria is met (which is when Q (set of all validation units) is empty and leaves the workers and models idle)).
While Nakandala teaches selecting modeling logic for an Al model that solves a use case of a plurality of use cases… Nakandala does not explicitly disclose the specific use cases.
However, Wang explicitly discloses:
wherein the plurality of use cases includes a first use case for forecasting groups of equipment and personnel to deploy,
(Wang, Page 1, Column 2, Paragraph 1, “Thus, the order data is used to predict travel demand, achieve appropriate urban resource scheduling and provide better services for passengers in this mode. In this paper, a Deep Spatio-Temporal Convolutional LSTM (DeepSTCL) is proposed to forecast travel demand which considers the time and space factors comprehensively and gets a great prediction performance”; Abstract, “Therefore, it is significant to predict travel demand for urban resource dispatching”. DeepSTCL is used for forecasting travel demand, where travel demand is the need or desire to travel based on geography, travel patterns, destinations, etc. The forecasting of travel demand is needed for urban resource dispatching (how a city manages and distributes (interpreted by the examiner as deploys) its resources, including personnel, equipment, vehicles and other assets to ensure effective service delivery). Thus, the use case of forecasting groups of equipment and personnel to deploy is taught by Wang).
a second use case for determining destinations that have slots available to accommodate the groups of equipment and personnel,
(Wang, Fig. 1, 2 & 9; Page 1, Column 2, Paragraph 2, “Travel demand data is typical spatio-temporal data”; Page 7, Paragraph 3, “Travel demand modeling is an inherent part of smarter transportation. Analyzing and forecasting travel demand can help us manage the hot spot of passenger demand in the next period, balance supply and demand and schedule vehicle resources for passengers”; Abstract, “Urban resource scheduling is an important part of the development of a smart city, and transportation resources are the main components of urban resources. Currently, a series of problems with transportation resources such as unbalanced distribution and road congestion disrupt the scheduling discipline”. The deep learning traffic demand forecasting framework is based on spatio-temporal data which allows for analyzing congestion and unbalanced deployment of equipment/personnel. Fig. 1 shows a pictorial example of a geographical rectangle and Fig. 2 shows an example of a snapshot of an order count (order demand/requests) where both are used to create the heatmaps shown in Fig. 9 (which highlights the forecasted scenarios/situations for travel demand). Thus, the method of DeepSTCL determines destinations (location for urban resource scheduling) that have slots available to accommodate (capacity based resource scheduling to avoid congestion/unbalanced distribution) the groups of equipment and personnel (deployable urban resources such as transportation resources)).
and a third use case for determining transportation routes for bringing the groups of equipment and personnel from an origin to the destinations having the slots available,
(Wang, Fig. 9; Page 1, Column 2, Paragraph 2, “Travel demand data is typical spatio-temporal data”; Abstract, “Urban resource scheduling is an important part of the development of a smart city, and transportation resources are the main components of urban resources. Currently, a series of problems with transportation resources such as unbalanced distribution and road congestion disrupt the scheduling discipline”. Urban resource scheduling’s main component is transportation resources as they cause issues such as unbalanced distribution and road congestion (both of which are capacity based and interpreted by the examiner as available slots). By forecasting travel demand accurately, transportation routes are able to be optimized for scheduling deployments (where scheduling is based off routing from origin to endpoint using travel demand prediction). Travel demand heatmaps (such as Fig. 9) are utilized for equipment/personnel deployment (which is the scheduling of urban resources to avoid unbalanced deployments and road congestion)).
and wherein as part of the second use case an algorithm supplies: a list of new equipment, inventory constraints and availability;
(Wang, Fig. 1, 2 & 9; Page 5, Column 2, Paragraph 4, “
PNG
media_image2.png
126
272
media_image2.png
Greyscale
”. As noted previously, Fig. 1 and 2 show example snapshots of an order count to create heatmaps shown in Fig. 9; where, the second use case is determining the destinations by the corresponding urban resource scheduling of the equipment (resources). As the resource scheduling is utilizing the heatmap for travel demand (interpreted as inventory constraint and availability) which is shown in the matrix (list) for order count (new requested trips; which is interpreted as new requested resources i.e. new equipment/trips. Thus, the algorithm is supplying the data for resource scheduling).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the Nakandala’s process of selecting of the modeling logic to solve a use case, with the plurality of specific use cases taught by Wang to illustrate the importance of being able to analyze and forecast travel demand based on spatio-temporal data to manage efficiency and optimize distribution scheduling (see Wang, Page 7, Column 1, Paragraph 3, “Analyzing and forecasting travel demand can help us manage the hot spot of passenger demand in the next period, balance supply and demand and schedule vehicle resources for passengers … ConvLSTM-based deep learning model for travel demand (ST Data) prediction is proposed that takes advantage of both temporal and spatial properties … Our models’ performances are significantly beyond two baseline models, confirming that it is better and more flexible for the travel demand prediction”).
Nakandala/Wang do not explicitly teach:
… wherein each of the other sub-results is obtained based on an invocation of a specific application program interface (API) of a plurality of APIs that communicates with the AI model;
However, Aarts teaches:
… wherein each of the other sub-results is obtained based on an invocation of a specific application program interface (API) of a plurality of APIs that communicates with the AI model;
(Aarts, Page 58, [0556], “… a software layer may be implemented as a … API through which … may be invoked (e.g., called) … for performing compute, AI, or … to perform processing tasks in an effective and efficient manner”; Page 5, [0098], “… an application programming interface comprises a concatenation function for combining forward and reverse outputs of a ragged bidirectional recurrent neural network.”; FIG. 8. FIG. 8 depicts an example process where an API call (invocation) can occur for a specific API that communicates with the example RNN (recurrent neural network -> AI Model); where the graph definition and the recurrence attribute are considered the sub-results within this example).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the process of Nakandala/Wang process of selecting modeling logic to solve specific use cases, with the API calls of Aarts to generate optimized processes/results, automation, reduce complexity, technical advantages, etc. (see Aarts, FIG 8, Page 2, [0062], “In at least one embodiment, a graph is made to represent a recurrent neural network by associating a recurrence attribute 110 with the graph. In at least one embodiment, said graph is a nested graph. In at least one embodiment, association of a recurrence property with a graph effectively makes recurrence or looping an attribute of said graph. In at least one embodiment, use of a recurrence property in an application programming interface provides a technical advantage over use of a while loop, or similar programming construct, which may require construction of separate graphs to represent header, body, and exit portions of a while loop”).
However, Hou teaches:
Receiving, by a processing system including a processor, candidate models from robotic modeling equipment;
(Hou, Fig 8; Fig. 9; Page 12, Column 2, [0102], “FIG. 9 … there are 3 robots performing the same screw driving task in a factory ( e.g., Robot A, Robot B, and Robot C) … each robot gets 3 model candidates to predict if a screw driving process is faulty or not faulty …”. Figure 9: Rows 916, 918, and 920 are for 3 different model candidates (also shown in Fig. 8) for screw driving assembly via Robot A (which is for a screw driving task). Thus, Hou’s system teaches receiving candidate models from robotic modeling equipment).
determining, by the processing system an based on the receiving of the candidate models, that the candidate models are insufficient to solve a use case of a plurality of use cases;
(Hou, Fig. 8; Page 11, Column 1, [0087], “… the example intelligent deployment circuitry 510 outputs a prediction of model combinations to improve prediction performance over time. For example, the intelligent deployment circuitry 510 may output a prediction of model combinations to improve prediction performance over time by monitoring the plurality of models produced, removing outliers, and/or storing models that are performing near a threshold of quality …”. The Intelligent Deployment Circuity determines. based on the receiving of the candidate models (shown in Fig. 8), ways to improve a prediction of model combinations such as removing outliers, storing models and more. Thus, the examiner interprets the module 510 from Fig. 8 to be determining if a candidate model is insufficient (due to the module needing to remove a candidate model for poor performance)).
obtaining, by the processing system and based on the determining that the candidate models are insufficient to solve the use case of the plurality of use cases, a plurality of models from databases sourced from model contributors comprised of data scientists, engineers, and technical staff;
(Hou, Fig. 9; Page 13, Column 1, [0108], “… FIG. 9 is an example management console … wherein a biological entity such as a human being may track the evolving ensemble of models. For example, the human being (engineer, worker, foreman, data scientist, etc.) may track the performance of the models and intervene ( e.g., adjust the model) in response to a determination made by the human being …”; Page 20, Column 2, [0168], “… in response to the intelligent trigger circuitry triggering the automated model update process, generates a plurality of candidate artificial intelligence models or selects a plurality of candidate artificial intelligence models from a repository of trained artificial intelligence models”. The repository of trained artificial intelligence models and ability to manage and track the performance of models … allows a user to generate new candidate models or select candidate models (due to weak or poor or insufficient performance for a use case) from a repository (which have been contributed to by model contributes such as data scientists, engineers, and technical staff)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the process of Nakandala/Wang/Aarts process of selecting modeling logic to solve specific use cases, with the receiving of candidate models to determine insufficient performance for robotic equipment data and gathering new model data from specific model contributors to optimize models in the repository and improve prediction performance over time(see Hou, Fig. 6; Page 12, Column 1, [0099], “At block 722, the example intelligent deployment circuitry 510 stores optimized models in the model repository 404. For example, the example intelligent deployment circuitry 510 may store a candidate model that performs above a threshold in the example model repository 404 with the metadata corresponding to the factory production line that is monitored by the optimized candidate model. The optimized candidate model is stored and may be selected by the example automated model search circuitry 508 at a later time …”; Page 11, Column 1, [0087], “At block 606, the example intelligent deployment circuitry 510 outputs a prediction of model combinations to improve prediction performance over time. For example, the intelligent deployment circuitry 510 may output a prediction of model combinations to improve prediction performance over time by monitoring the plurality of models produced, removing outliers, and/or storing models that are performing near a threshold of quality …”).
Regarding Claim 19:
Nakandala/Wang/Aarts/Hou teach the device of Claim 18 and Nakandala further teaches:
dividing, by the processing system a business problem into the plurality of use cases; and
(Nakandala, [1. INTRODUCTION], Page 2159, Paragraph 2, “Case Study. We present a real-world model selection scenario. Our public health collaborators at UC San Diego wanted to try deep nets for identifying different activities (e.g., sitting, standing, stepping, etc.) of subjects from body-worn accelerometer data…” The examiner interprets a business problem as a challenge an organization is facing. The case study taught by Nakandala notes a business problem (to identify different activities of subjects wearing accelerometers) from the public health collaborators at UC San Diego which is divided into different types of activities (e.g., sitting, standing, stepping, etc.).
ranking, by the processing system, the plural sub-results based on the evaluation metric.
(Nakandala, Page 2164, Figure 5 and [5. CEREBRO SCHEDULER] Paragraph 4, “Consider the model selection workload shown in Figure 5(A). Assume workers are homogeneous and there is no data replication. For one epoch of training, Figure 5(B) shows an optimal task-parallel schedule for this workload with a 9-unit makespan. Figure 5(C) shows a non-optimal MOP schedulewith also 9 units makespan. But as Figure 5(D) shows, an optimal MOP schedule has a makespan of only 7 units. Overall, we see that MOP's training unit-based scheduling offers more flexibility to raise resource utilization”. Figure 5 denotes the different scheduling that occurs for model selection workloads. The ranking is done by optimization (resource utilization) and runtime (denoted in makespan units)).
Claims 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Nakandala et al., “Cerebro: A Data System for Optimized Deep Learning Model Selection”, in view of Wang et al., “DeepSTCL: A Deep Spatio-temporal ConvLSTM for Travel Demand Prediction”, in view of Aarts et al., US-2021/0192314-A1, in view of Hou et al., US-20210325861-A1, in view of Deshpande et al. “A linearized framework and a new benchmark for model selection for fine-tuning”.
Regarding Claim 21:
Nakandala/Wang/Aarts/Hou teach the device of Claim 1 and Nakandala further teaches:
wherein the operations further comprise: cataloging the use case, resulting in a catalogued use case.
However, Deshpande teaches:
wherein the operations further comprise: cataloging the use case, resulting in a catalogued use case.
(Deshpande, Page 1, Column 1, Paragraph 1, “A “model zoo” is a collection of pre-trained models, obtained by training different architectures on many datasets covering a variety of tasks and domains. … typical use of a model zoo is to provide a good initialization which can then be fine-tuned for a new target task, for which we have few training data”; Page 5, Column 1, Paragraph 2, “We evaluate model selection and fine-tuning with both, a model zoo of single-domain experts … and a model zoo of multi-domain experts … We include publicly available large source … from different domains, e.g. … consists of aerial imagery, … consist of food, plant images, … contain scene images. This allows us to maximize the coverage of our model zoo to different domains and enables more effective transfer when fine-tuning on different target tasks. Model zoo of single-domain experts. We build a model zoo of a total of 30 models … to evaluate our model selection”. The model zoo is a collection/repository of pretrained models (different configurations/initializations that are saved); thus, the model zoo is cataloging the use cases, resulting in a catalogued use case).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the Nakandala’s process of selecting of the modeling logic to solve a use case, with the plurality of specific use cases taught by Wang to illustrate the importance of being able to analyze and forecast travel demand based on spatio-temporal data, with the model selection utilizing catalogued use cases of Deshpande to manage efficiency, boost efficiency, save cost, optimize distribution scheduling, using prior historical data to initialize model selections for other use cases, and comparing scores (see Deshpande, Page 8, Column 2, Paragraph 5, “Fine-tuning using model zoo is a simple method to boost accuracy. We show that while a model zoo may have modest gains in the high-data regime, it outperforms Imagenet experts networks in the low-data regime. We show that simple baseline methods derived from a linear approximation of fine-tuning – Label-Gradient Correlation (LGC) and Label-Feature Correlation (LFC) – can select good models (single-domain) or parameters (multi-domain) to fine-tune, and match or outperform relevant model selection methods in the literature. Our model selection saves the cost of bruteforce fine-tuning and makes model zoos viable”).
Regarding Claim 22:
Nakandala/Wang/Aarts/Hou/Deshpande teach the device of Claim 21. Nakandala in view of Wang fails to explicitly teach:
wherein the operations further comprise: selecting second modeling logic for the AI model to solve a second business problem using the catalogued use case, wherein the second modeling logic is different from the modeling logic, and wherein the determining that the score is representative of the improvement is based on a comparison of the score with another score.
However, Deshpande teaches:
wherein the operations further comprise: selecting second modeling logic for the AI model to solve a second business problem using the catalogued use case, wherein the second modeling logic is different from the modeling logic, and wherein the determining that the score is representative of the improvement is based on a comparison of the score with another score.
(Deshpande, Page 2, Figure 2; Page 8, Figure 6; Page 5, Column 1, Paragraph 2, “This allows us to maximize the coverage of our model zoo to different domains and enables more effective transfer when fine-tuning on different target tasks. … We build a model zoo of a total of 30 models … to evaluate our model selection”. Figure 2 shows the selecting of the second modeling logic using a pretrained model configurations (catalogued use case) for initialization versus different architectures; thus, selecting second modeling logic for the AI model to solve a second business problem using the catalogued use case, wherein the second modeling logic is different from the modeling logic. Figure 6 shows the LFC scores having the highest Spearman’s ranking correlation used to compare the predicted model selection method score versus the actual performance score after fine-tuning; thus, determining that the score is representative of the improvement is based on a comparison of the score with another score).
The motivation of Claim 21’s combination of Nakandala/Wang/Aarts/Hou/Deshpande is still maintained.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IBRAHIM RAHMAN whose telephone number is (703)756-1646. The examiner can normally be reached M-F 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.R./ Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights