DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Regarding Japanese Patent App. No. JP2023-101373 (filed June 21, 2023), receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement submitted on 6/19/2024 has been considered.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations and associated disclosure are:
Claim Limitation
Applicable Claims
Associated Disclosure
data storage unit
1-8
Fig. 1, 111, para. 0058
data attribute extraction unit
1-8
Fig. 1, 120, para. 0046
experiment record information storage unit
1-8
Fig. 1, 115, para. 0040
processing specification unit
1-8
Fig. 1, 140, para. 0046
range estimation unit
4-6
Fig. 1, 130, para. 0044
association processing unit
5-6
Fig. 1, 150, para. 0048
search unit
6
Fig. 1, 160, para. 0049
data storage step
9-15
Fig. 1, 111, para. 0058
data attribute extraction step
9-15
Fig. 1, 120, para. 0046
experiment record information storage step
9-15
Fig. 1, 115, para. 0040
processing specification step
9-15
Fig. 1, 140, para. 0046
range estimation unit
12-14
Fig. 1, 130, para. 0044
association processing unit
13-14
Fig. 1, 150, para. 0048
search unit
14
Fig. 1, 160, para. 0049
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) Applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Step 1 of the Alice/Mayo framework, Claims 1-8 are directed to a system (an apparatus) and Claims 9-15 are directed to a method (a process), which each fall within one of the four statutory categories of inventions.
Regarding Claim 1
Step 2A, prong 1 (Is the claim directed to a law of nature, a natural phenomenon or an abstract idea).
Claim 1 recites the following mental processes, that in each case under the broadest reasonable interpretation, covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper but for the recitation of generic computer components (e.g., hardware/software “units”).
… refer to, based on the training data received from the data storage unit, definition information in which a conversion condition for converting the training data into an attribute thereof is defined, and extract a plurality of data acquisition conditions indicating the attribute from the training data; (under the broadest reasonable interpretation, for example, a human can mentally refer to definition information with respect to a conversion condition (e.g., convert time from 0600-1800 as day and 1801-0559 as night), and use such conversion condition to mentally convert training data consisting of timesteps to day/night categories)
… refer to the experiment record information for each of the data acquisition conditions extracted by the data attribute extraction unit, and specify, for each of the data acquisition conditions, a machine learning product in which the accuracy of the inference is improved between old and new versions among the versions of the machine learning product recorded in the experiment record information (under the broadest reasonable interpretation, for example, a human can mentally review experiment record information (such as data records for different machine learning models printed on paper), and mentally review the differences in inference accuracy between different model version numbers)
Step 2A, prong 2 (Does the claim recite additional elements that integrate the judicial exception into a practical application?).
The judicial exception is not integrated into a practical application.
Regarding the “a data storage unit configured to store training data used in training processing of a machine learning program” limitation, such additional element of a data storage step is recited at a high level of generality and amounts to extra-solution activity of storing data, i.e. post-solution activity of data storage for use in the claimed process (see MPEP 2106.05(g)).
Regarding the “a data attribute extraction unit configured to” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “experiment record information storage unit configured to store experiment record information, in which a machine learning product including the machine learning program and a trained model trained by the training processing of the machine learning program is divided into a plurality of versions and recorded, and an accuracy of an inference obtained by each of the versions of the machine learning product is recorded in association with the version of the machine learning product, the trained model, and the training data” limitation, such additional element of a data storage step is recited at a high level of generality and amounts to extra-solution activity of storing data, i.e. post-solution activity of data storage for use in the claimed process (see MPEP 2106.05(g)).
Regarding the “a processing specification unit configured to” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Step 2B (Does the claim recite additional elements that amount to significantly more than the judicial exception?)
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception.
Regarding the “a data storage unit configured to store training data used in training processing of a machine learning program” limitation, as discussed above, the additional element of a data storage step is recited at a high level of generality and amounts to extra-solution activity of storing data, i.e. post-solution activity of storing data after or during use in the claimed process. The courts have found limitations directed to storing information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “a data attribute extraction unit configured to” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “experiment record information storage unit configured to store experiment record information, in which a machine learning product including the machine learning program and a trained model trained by the training processing of the machine learning program is divided into a plurality of versions and recorded, and an accuracy of an inference obtained by each of the versions of the machine learning product is recorded in association with the version of the machine learning product, the trained model, and the training data” limitation, as discussed above, the additional element of a data storage step is recited at a high level of generality and amounts to extra-solution activity of storing data, i.e. post-solution activity of storing data after or during use in the claimed process. The courts have found limitations directed to storing information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “a processing specification unit configured to” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Accordingly, at Step 2B after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Regarding Claim 2
Step 2A, Prong 1
… generates accuracy contribution information by collecting, for each of the data acquisition conditions, a relationship between a change difference of the machine learning program belonging to the specified machine learning product and the machine learning program. (under the broadest reasonable interpretation, for example, a human can mentally determine a relationship between a change difference and recited in this limitation, and determine a root cause for the change difference)
Step 2A, Prong 2
Regarding the “the processing specification unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “the processing specification unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 3
Step 2A, Prong 1
the conversion condition is defined as a conversion condition for converting the character string or the numerical data into each of the plurality of data acquisition conditions (under the broadest reasonable interpretation, for example, a human can mentally define a conversion condition in a manner that converts character string or numeric data into categories, such as converting numeric time (or time represented in character string format) from 0600-1800 as day and 1801-0559 as night)
… converts the character string or the numerical data into the attribute according to the conversion condition, and extracts the converted attribute as the data acquisition condition from the training data (under the broadest reasonable interpretation, for example, a human can mentally use such conversion condition to mentally convert training data consisting of timesteps to day/night categories as set forth in the previous example)
Step 2A, Prong 2
Regarding the “wherein the training data is implemented by table data including a plurality of columns and rows, and a character string or numerical data is recorded in each row of each column of the table data” limitation, such limitation merely describes the formatting of the training data and therefore such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use. As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Regarding the “the data attribute extraction unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “wherein the training data is implemented by table data including a plurality of columns and rows, and a character string or numerical data is recorded in each row of each column of the table data” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding the “the data attribute extraction unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 4
Step 2A, Prong 1
… estimate a range of the training data in which an inference result obtained by the training processing of the machine learning program using the trained model and the training data varies from normal to abnormal, (under the broadest reasonable interpretation, for example, a human can mentally review training data and estimate a particular range of the training data that will have normal inferences vs. abnormal inferences, for example, in time-series data, estimating that the first 3 samples will be normal)
… estimates, as a lower limit value or an upper limit value of the range of the training data, the inference numerical data used when a result of an inference obtained by the machine learning program becomes abnormal from normal, and generates accuracy variation information including the estimated lower limit value or upper limit value (under the broadest reasonable interpretation, for example, a human can mentally estimate lower or upper limit values, and further generate accuracy variation information, for example by indicating a lower bound where below that bound there is abnormal data, and identifying a potential reason for why such a low value would be abnormal)
Step 2A, Prong 2
Regarding the “range estimation unit sequentially executes inferences performed by the machine learning program using a plurality of pieces of inference numerical data existing over a range wider than a range from a minimum value to a maximum value of the numerical data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “range estimation unit sequentially executes inferences performed by the machine learning program using a plurality of pieces of inference numerical data existing over a range wider than a range from a minimum value to a maximum value of the numerical data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 5
Step 2A, Prong 1
… associate each of the plurality of machine learning products recorded in the experiment record information with each of the plurality of data acquisition conditions extracted by the data attribute extraction unit (under the broadest reasonable interpretation, for example, a human can mentally associate data in the experiment record information with other data)
… associates, with the accuracy variation information generated by the range estimation unit, the data acquisition condition associated with the machine learning product, and associates, with the accuracy contribution information generated by the processing specification unit, the data acquisition condition associated with the machine learning product to generate association information (under the broadest reasonable interpretation, for example, a human can mentally associate data in the experiment record information with other data, such as by mentally associating accuracy information with the data acquisition condition categories to mentally determine links associating such information)
Step 2A, Prong 2
Regarding the “association processing unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “association processing unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 6
Step 2A, Prong 1
…search the association information using information recorded in received input data as a search key (under the broadest reasonable interpretation, for example, a human can mentally use received input data to search the association information)
… extracts an application destination data acquisition condition corresponding to the attribute of the training data from the application destination data when application destination data acquired from a target of technique verification is input as the input data, extracts, as information associated with the extracted application destination data acquisition condition, at least one of the accuracy variation information and the accuracy contribution information from the association information… (under the broadest reasonable interpretation, for example, a human can mentally review and extract information as set forth in this limitation)
Step 2A, Prong 2
Regarding the “search unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “a display unit configured to display the information obtained by the search of the search unit” limitation, such limitation amounts to extra solution activity because it is a mere nominal or tangential addition to the claim, amounting to mere data display (see MPEP 2106.05(g)).
Regarding the “outputs the extracted information to the display unit” limitation, such additional element of a data transmitting step is recited at a high level of generality and amounts to extra-solution activity of transmitting data, i.e. post-solution activity of transmitting data from the claimed process (see MPEP 2106.05(g)).
Step 2B
Regarding the “search unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “a display unit configured to display the information obtained by the search of the search unit” limitation, this limitation amounts to extra solution activity because it is a mere nominal or tangential addition to the claim, amounting to mere data display (see MPEP 2106.05(g)). The courts have similarly found limitations directed to displaying a result, recited at a high level of generality, to be well-understood, routine, and conventional. See (MPEP 2106.05(d)(II), "presenting offers and gathering statistics.", “determining an estimated outcome and setting a price”)
Regarding the “outputs the extracted information to the display unit” limitation, as discussed above, the additional element of a data transmitting step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. post-solution activity of transmitting data from the claimed process. The courts have found limitations directed to transmitting information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding Claim 7
Step 2A, Prong 1
specifies the change difference of the machine learning program belonging to the machine learning product in which the accuracy of the inference is improved, specifies, as the change difference of the machine learning program, a processing parameter or processing belonging to at least one of a training program and an inference program belonging to the machine learning program for each of the data acquisition conditions, and records the specified processing parameter or processing in the accuracy contribution information. (under the broadest reasonable interpretation, for example, a human can mentally review outputs of machine learning data and identify and specify reasons why the model versions have different results, including identifying particular processing attributes leading to the change difference, and can then mentally record the processing parameter by remembering the parameter or writing the parameter on paper)
Step 2A, Prong 2
Regarding the “processing specification unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “processing specification unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 8
Step 2A, Prong 1
…divides the plurality of data acquisition conditions extracted by the data attribute extraction unit into a plurality of groups for each of the data acquisition conditions, refers to the experiment record information based on the data acquisition conditions belonging to each of the groups, specifies, for each of the groups, a machine learning product in which the accuracy of the inference is improved between the old and new versions among the plurality of machine learning products recorded in the experiment record information, and specifies, for each of the groups, a change difference of the machine learning program belonging to the specified machine learning product (under the broadest reasonable interpretation, for example, a human can mentally review data and divide the data into different groups, and then for each grouping with respect to new and old versions of machine learning models, specify a particular reason why there is a difference in accuracy between the older and newer versions)
Step 2A, Prong 2
Regarding the “processing specification unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a generic hardware/software unit. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a generic hardware/software unit). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “processing specification unit” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 9
Step 2A, Prong 1
Claim 9 recites a method that corresponds to the system of claim 1, and therefore the analysis under Step 2A, Prong 1 with respect to claim 1 also applies to this claim 9.
Step 2A, Prong 2
Claim 9 recites a method that corresponds to the system of claim 1, and therefore the analysis under Step 2A, Prong 2 with respect to claim 1 also applies to this claim 9. While claim 9 recites an additional generic computing component (“computer”), such additional generic computing component does not change the analysis under Step 2A, Prong 2. The recited “computer” is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer. Accordingly, the addition of a “computer” does not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Claim 9 recites a method that corresponds to the system of claim 1, and therefore the analysis under Step 2B with respect to claim 1 also applies to this claim 9. While claim 9 recites an additional generic computing component (“computer”), such additional generic computing component does not change the analysis under Step 2B because the limitations merely provide instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Claims 10-15 depend from claim 9 and correspond to the systems of claims 2-7, respectively, and are therefore each rejected for the same reasons explained above with respect to claim 9 and claims 2-7, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 7-11, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over US 20200167645 A1, hereinafter referenced as HUANG, in view of US 20210241241 A1, hereinafter referenced as LOKANTH, and further in view of US 20230205665 A1, hereinafter referenced as GRITTER.
Regarding Claim 1
HUANG teaches:
A machine learning development support system comprising: (HUANG, para. 0005: “An object of the present disclosure is to provide information processing and model training methods, apparatuses, electronic devices, and storage mediums, so as to solve the problem that an electronic device displays a large amount of pop-up information that is not of interest to a user.”;
HUANG, para. 0037: “constructing a pop-up management model based on a deep neural network, wherein a modeling unit of the pop-up management model is information indicating whether pop-up information is of interest to a user, wherein the pop-up information that is of interest to the user is information whose degree of attention is greater than a threshold”;
Examiner’s Note: HUANG teaches providing training methods for deep neural networks)
a data storage unit configured to store training data used in training processing of a machine learning program; (HUANG, para. 0171: “S102: Obtaining a training set, converting pop-up information in the training set into feature vectors, marking the pop-up information in the training set with labels indicating that the pop-up information is of interest to the user or the pop-up information is not of interest to the user.”;
HUANG, para. 0186: “Based on the correspondence between the large amount of pop-up information and the fact whether the user views the pop-up information, the training set is constructed for training the pop-up management model subsequently.”;
HUANG, para. 0189: “In the model training, after the training set is obtained, data mining is performed on the pop-up information in the training set in terms of time and electronic device specifications, to obtain multi-dimensional feature vectors.”;
HUANG, para. 0409: “Corresponding to the embodiment of the model training method, an embodiment of the present application further provides a storage medium. A computer program is stored in the storage medium. The computer program, when executed by a processor, implements the model training method.”
Examiner’s Note: HUANG teaches obtaining and constructing a training set, and then performing data mining on the constructed training set, meaning that the constructed training set must be stored electronically in order for the data mining to happen)
a data attribute extraction unit configured to refer to, based on the training data received from the data storage unit, definition information in which a conversion condition for converting the training data into an attribute thereof is defined, and extract a plurality of data acquisition conditions indicating the attribute from the training data; (HUANG, para. 0190: “In an embodiment of the present application, the pop-up information in the training set may be converted into multi-dimensional feature vectors according to display time, a display delay duration, a display location, and a specification of an electronic device used by a user. The display time is time at which the pop-up information is displayed, which may be divided into working time or rest time, or divided into morning time or afternoon time or evening time. The display delay duration is between viewing the pop-up information and displaying the pop-up information, and the display delay duration may be an average display delay duration for all the pop-up information displayed by an electronic device.”;
HUANG, para. 0200: “Converting the pop-up information in the training set into feature vectors, and marking the pop-up information in the training set with labels having label information corresponding to the pop-up information.”
HUANG, para. 0267: “a converting unit 602, configured for obtaining a training set, converting pop-up information in the training set into feature vectors, and marking the pop-up information in the training set with labels indicating that the pop-up information is of interest to the user or the pop-up information is not of interest to the user”;
Examiner’s Note: HUANG teaches converting training data into feature vectors, where an example is converting a display time into a morning/afternoon/evening timezone category (which is substantially similar to the example in the present application, where a data acquisition condition as illustrated in Fig. 10 includes [time zone: night/morning], which is a feature derived from time data as explained in para. 0083 of the instant specification), and other potential categories (corresponding to recited “data acquisition conditions”) include display delay duration and/or display location as set forth by HUANG; the “definition information” refers to the rules used to convert a time to morning/afternoon/evening feature vectors)
However, HUANG fails to explicitly teach:
an experiment record information storage unit configured to store experiment record information, in which a machine learning product including the machine learning program and a trained model trained by the training processing of the machine learning program is divided into a plurality of versions and recorded, and an accuracy of an inference obtained by each of the versions of the machine learning product is recorded in association with the version of the machine learning product, the trained model, and the training data; and
a processing specification unit configured to refer to the experiment record information for each of the data acquisition conditions extracted by the data attribute extraction unit, and specify, for each of the data acquisition conditions, a machine learning product in which the accuracy of the inference is improved between old and new versions among the versions of the machine learning product recorded in the experiment record information.
However, in a related field of endeavor (machine learning, see para. 0111), LOKANTH teaches and makes obvious:
an experiment record information storage unit configured to store experiment record information, in which a machine learning product including the machine learning program and a trained model trained by the training processing of the machine learning program is divided into a plurality of versions and recorded, and an accuracy of an inference obtained by each of the versions of the machine learning product is recorded in association with the version of the machine learning product, the trained model, and the training data (LOKANTH, para. 0174: “According to another embodiment, multi-tenant support is provided by the immutable and audit compliant record keeping for such transactions. According to such an embodiment, the transactions include data which describes a collection of data descriptors from the following exemplary list: what decision was made by the AI trained model, what version of the AI model made the decision, what decision was made, what collection of training data was utilized to train the AI model, any confidence score or predictive score output by the AI model, what Einstein cloud platform features or GUIs were utilized, and AI intermediate node decision points triggered to lead to the output decision by the Einstein cloud platform.”;
LOKANTH, para. 0190: “According to certain embodiments, the Einstein cloud platform 988 trains a new AI model, registers the version information for the AI model and the training data set with an audit record keeping service pursuant to a request (e.g., regardless of whether the request is an automatic request from a controlling application or a request from an administrator, etc), and then deploys the AI model by transacting a smart contract to execute at the blockchain and enforce decisions and recommendations made by the AI model.”;
Examiner’s Note: LOKANTH teaches keeping detailed records of AI trained model usage, where such records include the decision made by the AI model (the inference made), the version, and a confidence score (corresponding to recited “accuracy of an inference”), together with the training data utilized to train the AI model; the HUANG-LOKANTH combination now modifies the system of HUANG to maintain detailed records of model usage as in LOKANTH)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HUANG and LOKANTH as explained above. As disclosed by LOKANTH, one of ordinary skill would have been motivated to do so in order to maintain an audit trail, “in the event that a customer, the business, administrator, etc., seeks to understand what AI model was utilized to make a decision (e.g., such as a shutting down a compute cluster), as well as precisely what version of that AI model was utilized and further, what training dataset or what version and time-span of training data was utilized in training the AI model, which then led to a particular decision as enforced by the smart contract on the blockchain.” (para. 0187).
However, HUANG and LOKANTH fail to explicitly teach:
a processing specification unit configured to refer to the experiment record information for each of the data acquisition conditions extracted by the data attribute extraction unit, and specify, for each of the data acquisition conditions, a machine learning product in which the accuracy of the inference is improved between old and new versions among the versions of the machine learning product recorded in the experiment record information.
However, in a related field of endeavor (evaluating machine learning model versions, see para. 0004), GRITTER teaches and makes obvious:
a processing specification unit configured to refer to the experiment record information for each of the data acquisition conditions extracted by the data attribute extraction unit, and specify, for each of the data acquisition conditions, a machine learning product in which the accuracy of the inference is improved between old and new versions among the versions of the machine learning product recorded in the experiment record information. (GRITTER, para. 0004: “In brief and at a high level, this disclosure describes, among other things, methods, systems, and computer-readable media for comparatively evaluating distinct versions of a machine-learning data model (hereinafter “model”) for technological performance and/or predictive accuracy, and deploying version(s) having demonstrated improvements to technological performance and/or predictive accuracy relative to other version(s). As will be described, aspects of the invention discussed hereinafter monitor and comparatively evaluate technological performance and/or predictive accuracy by monitoring multiple and varied versions of a model in order to select (e.g., manually or autonomously via a processor without user input) deploy a leading version that is indicated as having the greatest prediction accuracy and/or other indications of superior performance (e.g., metrics, bias, data drift). Prior to deployment, one or more versions of a model can be autonomously (e.g., without user selection, input, and/or intervention) evaluated relative to one or more other (e.g., in use, currently deployed, previously deployed) versions of the model.”;GRITTER, para. 0043: “From the reports 120A, 120B, 120n, the system 102 makes performance determinations 122. The system 102 receives, obtains, retrieves, and/or ingests a baseline file 124 that includes one or more baselines, and one or more thresholds 126. The baseline file 124 and thresholds 126 are used to validate information in the reports 120A, 120B, 120n, and generate version-performance reports 130A, 130B, 130n based on the performance determinations 122. Turning to FIG. 6, for example, box 600 surrounds a portion of computer-executable instructions for generating reports having performance measures, such as reports 120A, 120B, 120n.”;
GRITTER, para. 0044: “Accordingly, the baseline values can be used to evaluate prediction accuracy of each of the model versions 104A, 104B, 104n in comparison to such expected values. Baseline values can further include margins, for example, to determine whether a model version produced a prediction value that is or is not within a predefined buffer range of the baseline expected value for the corresponding variable.”
Examiner’s Note: GRITTER discloses analyzing reporting data to compare the accuracy of model versions to one another; the HUANG-LOKANTH-GRITTER combination now modifies the system of HUANG to use the electronic records of LOKANTH, and now the electronic records of LOKANTH are analyzed using the teachings of GRITTER to determine the model having the highest accuracy to be deployed, where one of ordinary skill would understand that typically newer models based on new training data would be expected to have higher accuracy levels)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HUANG, LOKANTH, and GRITTER as explained above. As disclosed by GRITTER, one of ordinary skill would have been motivated to do so in order to “upgrade, update, and/or replace” a model version with a newer version that has been determined to be better “based on its demonstrated stability and improvement over an existing, in-use data model version.” (para. 0026).
Regarding Claim 2
HUANG, LOKANTH, and GRITTER teach the system of claim 1 as explained above. However, HUANG and LOKANTH fail to explicitly teach:
wherein the processing specification unit generates accuracy contribution information by collecting, for each of the data acquisition conditions, a relationship between a change difference of the machine learning program belonging to the specified machine learning product and the machine learning program.
However, in a related field of endeavor (evaluating machine learning model versions, see para. 0004), GRITTER teaches and makes obvious:
wherein the processing specification unit generates accuracy contribution information by collecting, for each of the data acquisition conditions, a relationship between a change difference of the machine learning program belonging to the specified machine learning product and the machine learning program. (GRITTER, para. 0056: “Pre training bias (evaluates features with actual label) and post training bias (evaluates features with actuals & predictions label values) are supported in some embodiments. For example, once model pipeline data (actual) has been evaluated, the model monitoring system loads a pre-processing file and baseline files into the system and executes the pre-processing algorithm to get the model insight features and actuals. The data may then be analyzed with baseline files using bias & configured metrics to calculate any pre-training bias. In another example, once model pipeline data (actual & predictions) has been evaluated, the model monitoring system loads a pre-processing file and baseline files into the system and executes the pre-processing algorithm to get the model insight features and actuals. The data may then be analyzed with baseline files using bias & configured metrics to calculate any post-training bias.”;
Examiner’s Note: GRITTER teaches calculating a post-training bias (corresponding to recited “relationship between a change difference of the machine learning program belonging to the specified machine learning product and the machine learning program”); the HUANG-LOKANTH-GRITTER combination now calculates a post-training bias (as in GRITTER) for each of the feature categories of HUANG (corresponding to recited “data acquisition conditions”)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HUANG, LOKANTH, and GRITTER as explained above. As disclosed by GRITTER, one of ordinary skill would have been motivated to do so in order to “upgrade, update, and/or replace” a model version with a newer version that has been determined to be better “based on its demonstrated stability and improvement over an existing, in-use data model version.” (para. 0026).
Regarding Claim 3
HUANG, LOKANTH, and GRITTER teach the system of claim 2 as explained above.
HUANG further teaches:
the conversion condition is defined as a conversion condition for converting the character string or the numerical data into each of the plurality of data acquisition conditions, and the data attribute extraction unit converts the character string or the numerical data into the attribute according to the conversion condition, and extracts the converted attribute as the data acquisition condition from the training data. (HUANG, para. 0190: “In an embodiment of the present application, the pop-up information in the training set may be converted into multi-dimensional feature vectors according to display time, a display delay duration, a display location, and a specification of an electronic device used by a user. The display time is time at which the pop-up information is displayed, which may be divided into working time or rest time, or divided into morning time or afternoon time or evening time. The display delay duration is between viewing the pop-up information and displaying the pop-up information, and the display delay duration may be an average display delay duration for all the pop-up information displayed by an electronic device.”;
HUANG, para. 0200: “Converting the pop-up information in the training set into feature vectors, and marking the pop-up information in the training set with labels having label information corresponding to the pop-up information.”
HUANG, para. 0267: “a converting unit 602, configured for obtaining a training set, converting pop-up information in the training set into feature vectors, and marking the pop-up information in the training set with labels indicating that the pop-up information is of interest to the user or the pop-up information is not of interest to the user”;
Examiner’s Note: HUANG teaches converting training data into feature vectors, where an example is converting a display time into a morning/afternoon/evening timezone category (which is substantially similar to the example in the present application, where a data acquisition condition as illustrated in Fig. 10 includes [time zone: night/morning], which is a feature derived from time data as explained in para. 0083 of the instant specification), and other potential categories (corresponding to recited “data acquisition conditions”) include display delay duration and/or display location as set forth by HUANG; the rules used to convert a time to morning/afternoon/evening feature vectors correspond to the recited “conversion condition”)
However, HUANG fails to explicitly teach:
wherein the training data is implemented by table data including a plurality of columns and rows, and a character string or numerical data is recorded in each row of each column of the table data,
However, in a related field of endeavor (machine learning, see para. 0111), LOKANTH teaches and makes obvious:
wherein the training data is implemented by table data including a plurality of columns and rows, and a character string or numerical data is recorded in each row of each column of the table data, (LOKANTH, para. 0246: “Each table generally contains one or more data categories logically arranged as columns or fields in a viewable schema. Each row or record of a table contains an instance of data for each category defined by the fields. For example, a CRM database may include a table that describes a customer with fields for basic contact information such as name, address, phone number, fax number, etc. Another table might describe a purchase order, including fields for information such as customer, product, sale price, date, etc. In some multi-tenant database systems, standard entity tables might be provided for use by all tenants. For CRM database applications, such standard entities might include tables for Account, Contact, Lead, and Opportunity data, each containing pre-defined fields. It is understood that the word “entity” may also be used interchangeably herein with “object” and “table.”;
LOKANTH, para. 0247: “In some multi-tenant database systems, tenants may be allowed to create and store custom objects, or they may be allowed to customize standard entities or objects, for example by creating custom fields for standard objects, including custom index fields. In certain embodiments, for example, all custom entity data rows are stored in a single multi-tenant physical table, which may contain multiple logical tables per organization. It is transparent to customers that their multiple “tables” are in fact stored in one large table or that their data may be stored in the same table as the data of other customers.”;
Examiner’s Note: LOKANTH teaches that data can be stored in tabular format having rows and columns, and one of ordinary skill in the art would understand such fields to be either numerical data or character strings; the HUANG-LOKANTH combination now expressly formats the training data of HUANG into a tabular format as in LOKANTH)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HUANG, LOKANTH, and GRITTER as explained above. As disclosed by LOKANTH, one of ordinary skill would have been motivated to do so in order to maintain an audit trail, “in the event that a customer, the business, administrator, etc., seeks to understand what AI model was utilized to make a decision (e.g., such as a shutting down a compute cluster), as well as precisely what version of that AI model was utilized and further, what training dataset or what version and time-span of training data was utilized in training the AI model, which then led to a particular decision as enforced by the smart contract on the blockchain.” (para. 0187). One of ordinary skill would further have been motivated to do so in order to organize the training data in an easy-to-read, human-readable format for auditing of the training data itself.
Regarding Claim 7
HUANG, LOKANTH, and GRITTER teach the system of claim 2 as explained above. However, HUANG and LOKANTH fail to explicitly teach:
wherein the processing specification unit specifies the change difference of the machine learning program belonging to the machine learning product in which the accuracy of the inference is improved,
specifies, as the change difference of the machine learning program, a processing parameter or processing belonging to at least one of a training program and an inference program belonging to the machine learning program for each of the data acquisition conditions, and
records the specified processing parameter or processing in the accuracy contribution information.
However, in a related field of endeavor (evaluating machine learning model versions, see para. 0004), GRITTER teaches and makes obvious:
wherein the processing specification unit specifies the change difference of the machine learning program belonging to the machine learning product in which the accuracy of the inference is improved, (GRITTER, para. 0039: “The script 114 utilizes the version mapping file 110 to identify and locate specific data points in each model version to be evaluated, in aspects. For example, the version mapping file 110 specifies a plurality of data points or a set of data points in model version 104A to be extracted and another plurality of data points or another set of data points in model version 104B to be extracted, wherein the data points to be extracted in the different versions corresponds the same or similar variable in the model. In simpler terms, the version mapping file 110 is ingested and utilized by the script 114 so that the script can recognize which data points in different versions correspond to the same variables, events, and the like for subsequent comparisons, i.e., enables the script to map between the model versions 104A, 104B, 104n. The data points may correspond to the observed data 106A, 106B, 106n and/or the predictive data 108A, 108B, 108n. In some aspects, the version mapping file 110 is a .json file.”;
Examiner’s Note: GRITTER discloses identifying particular variables or events that lead to changes between different model versions; the HUANG-LOKANTH-GRITTER combination now uses the teachings of GRITTER to identify the differences between model versions with respect to certain variables and events)
specifies, as the change difference of the machine learning program, a processing parameter or processing belonging to at least one of a training program and an inference program belonging to the machine learning program for each of the data acquisition conditions, and (GRITTER, para. 0039: “The script 114 utilizes the version mapping file 110 to identify and locate specific data points in each model version to be evaluated, in aspects. For example, the version mapping file 110 specifies a plurality of data points or a set of data points in model version 104A to be extracted and another plurality of data points or another set of data points in model version 104B to be extracted, wherein the data points to be extracted in the different versions corresponds the same or similar variable in the model. In simpler terms, the version mapping file 110 is ingested and utilized by the script 114 so that the script can recognize which data points in different versions correspond to the same variables, events, and the like for subsequent comparisons, i.e., enables the script to map between the model versions 104A, 104B, 104n. The data points may correspond to the observed data 106A, 106B, 106n and/or the predictive data 108A, 108B, 108n. In some aspects, the version mapping file 110 is a .json file.”;
Examiner’s Note: GRITTER discloses identifying particular variables or events that lead to changes between different model versions; the HUANG-LOKANTH-GRITTER combination now uses the teachings of GRITTER to identify the differences between model versions with respect to certain variables and events (each a corresponding “processing parameter”))
records the specified processing parameter or processing in the accuracy contribution information. (GRITTER, para. 0039: “The script 114 utilizes the version mapping file 110 to identify and locate specific data points in each model version to be evaluated, in aspects. For example, the version mapping file 110 specifies a plurality of data points or a set of data points in model version 104A to be extracted and another plurality of data points or another set of data points in model version 104B to be extracted, wherein the data points to be extracted in the different versions corresponds the same or similar variable in the model. In simpler terms, the version mapping file 110 is ingested and utilized by the script 114 so that the script can recognize which data points in different versions correspond to the same variables, events, and the like for subsequent comparisons, i.e., enables the script to map between the model versions 104A, 104B, 104n. The data points may correspond to the observed data 106A, 106B, 106n and/or the predictive data 108A, 108B, 108n. In some aspects, the version mapping file 110 is a .json file.”;
Examiner’s Note: GRITTER discloses identifying particular variables or events that lead to changes between different model versions; the HUANG-LOKANTH-GRITTER combination now uses the teachings of GRITTER to identify the differences between model versions with respect to certain variables and events (each a corresponding “processing parameter”) and now stores such information in the audit records of LOKANTH)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HUANG, LOKANTH, and GRITTER as explained above. As disclosed by GRITTER, one of ordinary skill would have been motivated to do so in order to “upgrade, update, and/or replace” a model version with a newer version that has been determined to be better “based on its demonstrated stability and improvement over an existing, in-use data model version.” (para. 0026).
Regarding Claim 8
HUANG, LOKANTH, and GRITTER teach the system of claim 1 as explained above. However, HUANG fails to explicitly teach:
wherein the processing specification unit divides the plurality of data acquisition conditions extracted by the data attribute extraction unit into a plurality of groups for each of the data acquisition conditions
refers to the experiment record information based on the data acquisition conditions belonging to each of the groups,
specifies, for each of the groups, a machine learning product in which the accuracy of the inference is improved between the old and new versions among the plurality of machine learning products recorded in the experiment record information, and
specifies, for each of the groups, a change difference of the machine learning program belonging to the specified machine learning product.
However, in a related field of endeavor (machine learning, see para. 0111), LOKANTH teaches and makes obvious:
wherein the processing specification unit divides the plurality of data acquisition conditions extracted by the data attribute extraction unit into a plurality of groups for each of the data acquisition conditions (LOKANTH, para. 0174: “According to another embodiment, multi-tenant support is provided by the immutable and audit compliant record keeping for such transactions. According to such an embodiment, the transactions include data which describes a collection of data descriptors from the following exemplary list: what decision was made by the AI trained model, what version of the AI model made the decision, what decision was made, what collection of training data was utilized to train the AI model, any confidence score or predictive score output by the AI model, what Einstein cloud platform features or GUIs were utilized, and AI intermediate node decision points triggered to lead to the output decision by the Einstein cloud platform.”;
Examiner’s Note: as explained by para. 0105 and illustrated by Fig. 9 of the instant specification, the broadest reasonable interpretation of “divides the plurality of data acquisition conditions extracted by the data attribute extraction unit into a plurality of groups for each of the data acquisition conditions” includes dividing the data acquisition conditions into different groups including (season, time zone, facility name), and the HUANG-LOKANTH combination teaches this by taking the categories of HUANG (such as time zone, display delay duration, and/or display location) and associating each category with other data descriptors as in LOKANTH)
refers to the experiment record information based on the data acquisition conditions belonging to each of the groups, (LOKANTH, para. 0174: “According to another embodiment, multi-tenant support is provided by the immutable and audit compliant record keeping for such transactions. According to such an embodiment, the transactions include data which describes a collection of data descriptors from the following exemplary list: what decision was made by the AI trained model, what version of the AI model made the decision, what decision was made, what collection of training data was utilized to train the AI model, any confidence score or predictive score output by the AI model, what Einstein cloud platform features or GUIs were utilized, and AI intermediate node decision points triggered to lead to the output decision by the Einstein cloud platform.”;
Examiner’s Note: the HUANG-LOKANTH combination teaches this by taking the categories of HUANG (such as time zone, display delay duration, and/or display location) and associating each category with other data descriptors as in LOKANTH, such that the auditable records of LOKANTH now include such information)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HUANG, LOKANTH, and GRITTER as explained above. As disclosed by LOKANTH, one of ordinary skill would have been motivated to do so in order to maintain an audit trail, “in the event that a customer, the business, administrator, etc., seeks to understand what AI model was utilized to make a decision (e.g., such as a shutting down a compute cluster), as well as precisely what version of that AI model was utilized and further, what training dataset or what version and time-span of training data was utilized in training the AI model, which then led to a particular decision as enforced by the smart contract on the blockchain.” (para. 0187).
However, HUANG and LOKANTH fail to explicitly teach:
specifies, for each of the groups, a machine learning product in which the accuracy of the inference is improved between the old and new versions among the plurality of machine learning products recorded in the experiment record information, and (GRITTER, para. 0004: “In brief and at a high level, this disclosure describes, among other things, methods, systems, and computer-readable media for comparatively evaluating distinct versions of a machine-learning data model (hereinafter “model”) for technological performance and/or predictive accuracy, and deploying version(s) having demonstrated improvements to technological performance and/or predictive accuracy relative to other version(s). As will be described, aspects of the invention discussed hereinafter monitor and comparatively evaluate technological performance and/or predictive accuracy by monitoring multiple and varied versions of a model in order to select (e.g., manually or autonomously via a processor without user input) deploy a leading version that is indicated as having the greatest prediction accuracy and/or other indications of superior performance (e.g., metrics, bias, data drift). Prior to deployment, one or more versions of a model can be autonomously (e.g., without user selection, input, and/or intervention) evaluated relative to one or more other (e.g., in use, currently deployed, previously deployed) versions of the model.”;GRITTER, para. 0043: “From the reports 120A, 120B, 120n, the system 102 makes performance determinations 122. The system 102 receives, obtains, retrieves, and/or ingests a baseline file 124 that includes one or more baselines, and one or more thresholds 126. The baseline file 124 and thresholds 126 are used to validate information in the reports 120A, 120B, 120n, and generate version-performance reports 130A, 130B, 130n based on the performance determinations 122. Turning to FIG. 6, for example, box 600 surrounds a portion of computer-executable instructions for generating reports having performance measures, such as reports 120A, 120B, 120n.”;
GRITTER, para. 0044: “Accordingly, the baseline values can be used to evaluate prediction accuracy of each of the model versions 104A, 104B, 104n in comparison to such expected values. Baseline values can further include margins, for example, to determine whether a model version produced a prediction value that is or is not within a predefined buffer range of the baseline expected value for the corresponding variable.”
Examiner’s Note: GRITTER discloses analyzing reporting data to compare the accuracy of model versions to one another; the HUANG-LOKANTH-GRITTER combination now modifies the system of HUANG to use the electronic records of LOKANTH, and now the electronic records of LOKANTH are analyzed using the teachings of GRITTER to determine the model having the highest accuracy to be deployed, where one of ordinary skill would understand that typically newer models based on new training data would be expected to have higher accuracy levels)
specifies, for each of the groups, a change difference of the machine learning program belonging to the specified machine learning product. (GRITTER, para. 0039: “The script 114 utilizes the version mapping file 110 to identify and locate specific data points in each model version to be evaluated, in aspects. For example, the version mapping file 110 specifies a plurality of data points or a set of data points in model version 104A to be extracted and another plurality of data points or another set of data points in model version 104B to be extracted, wherein the data points to be extracted in the different versions corresponds the same or similar variable in the model. In simpler terms, the version mapping file 110 is ingested and utilized by the script 114 so that the script can recognize which data points in different versions correspond to the same variables, events, and the like for subsequent comparisons, i.e., enables the script to map between the model versions 104A, 104B, 104n. The data points may correspond to the observed data 106A, 106B, 106n and/or the predictive data 108A, 108B, 108n. In some aspects, the version mapping file 110 is a .json file.”;
Examiner’s Note: GRITTER discloses identifying particular variables or events that lead to changes between different model versions; the HUANG-LOKANTH-GRITTER combination now uses the teachings of GRITTER to identify the differences between model versions with respect to certain variables and events)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of HUANG, LOKANTH, and GRITTER as explained above. As disclosed by GRITTER, one of ordinary skill would have been motivated to do so in order to “upgrade, update, and/or replace” a model version with a newer version that has been determined to be better “based on its demonstrated stability and improvement over an existing, in-use data model version.” (para. 0026).
Claim 9 recites a method that corresponds to the system of claim 1 and is rejected for the same reasons explained above with respect to claim 1.
Claim 10 depends from claim 9 and recites a method that corresponds to the system of claim 2, and is therefore rejected for the same reasons explained above with respect to claims 2 and 9.
Claim 11 depends from claim 10 and recites a method that corresponds to the system of claim 3, and is therefore rejected for the same reasons explained above with respect to claims 3 and 10.
Claim 15 depends from claim 10 and recites a method that corresponds to the system of claim 7, and is therefore rejected for the same reasons explained above with respect to claims 7 and 10.
Allowable Subject Matter
Claims 4-6 and 12-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, provided that the rejections under 35 U.S.C. 101 are overcome.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 4 would be considered allowable, if rewritten in independent form including all of the limitations of the base claim and any intervening claims and provided that the rejections under 35 U.S.C. 101 are overcome, because none of the references of record either alone or in combination fairly disclose or suggest the combination of limitations specified in claim 4, including at least:
a range estimation unit configured to estimate a range of the training data in which an inference result obtained by the training processing of the machine learning program using the trained model and the training data varies from normal to abnormal,
wherein the range estimation unit
sequentially executes inferences performed by the machine learning program using a plurality of pieces of inference numerical data existing over a range wider than a range from a minimum value to a maximum value of the numerical data,
estimates, as a lower limit value or an upper limit value of the range of the training data, the inference numerical data used when a result of an inference obtained by the machine learning program becomes abnormal from normal, and
generates accuracy variation information including the estimated lower limit value or upper limit value.
The closest prior art of record discloses:
US 20200167645 A1, hereinafter referenced as HUANG, teaches converting training data to feature vectors, for example, converting a time to a morning/afternoon/evening time category. (para. 0190).
US 20210241241 A1, hereinafter referenced as LOKANTH, teaches retaining records for versions of AI models, including information about “what decision was made by the AI trained model, what version of the AI model made the decision, what decision was made, what collection of training data was utilized to train the AI model, any confidence score or predictive score output by the AI model”, etc. (para. 0174).
US 20230205665 A1, hereinafter referenced as GRITTER, teaches comparatively evaluating different machine learning models. (paras. 0004, 0044).
US 20160065604 A1, hereinafter referenced as CHEN, teaches an anomaly detection system that predicts a range of possible, non-anomalous values within training data. (para. 0024).
However, the examiner has found that the distinct feature of the Applicant's claimed invention over the prior art is the explicit claiming of the aforementioned limitations in combination with all the other limitations as specified in claim 4. Moreover, the examiner finds that one of ordinary skill in the art would not have been motivated to combine the prior art of record in the manner specifically recited in claim 4 without the hindsight aid of Applicant’s disclosure. Therefore, claim 4 would be allowed over the prior art if rewritten in independent form including all of the limitations of the base claim and any intervening claims, provided that the rejections under 35 U.S.C. 101 are overcome.
Claims 5-6 depend from claim 4 and would be allowed for the same reasons explained with respect to claim 4 if rewritten in independent form including all of the limitations of the base claim and any intervening claims, provided that the rejections under 35 U.S.C. 101 are overcome.
Claim 12 depends from claim 11 and recites a method that corresponds to the system of claim 4 and would be allowable for the same reasons explained with respect to claim 4 if rewritten in independent form including all of the limitations of the base claim and any intervening claims, provided that the rejections under 35 U.S.C. 101 are overcome.
Claims 13-14 depend from claim 12 and would be allowed for the same reasons explained with respect to claim 12 if rewritten in independent form including all of the limitations of the base claim and any intervening claims, provided that the rejections under 35 U.S.C. 101 are overcome.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20240387041 A1 (Baldauf-Lenschen). “For example, a given type of machine learning model is improved over time (e.g. retrained as new data is available and/or to improve accuracy of the model), and each model has different versions. This can enable identifying which data is generated by older vs. most recent version, can enable reverting to use of older versions of functions, and/or can otherwise enable tracking, storage, and/or usage of multiple function versions.” (para. 0279).
US 20240386317 A1 (Chen). “In some embodiments, as the ML system is building the first machine learning model (e.g., using the configuration and training parameters determined using the techniques disclosed herein) or after the ML system has built the first machine learning model, the ML system may also apply the datasets associated with the first machine learning model to an experiment-based machine learning training process (e.g., the conventional machine learning model configuration and training process discuss herein). For example, by experimenting with different machine learning model types, configuration parameters, and training parameters, and evaluating different versions of the first machine learning model, the ML system may also determine a particular machine learning model type, a particular set of configuration parameters, and a particular set of training parameters most suitable for the first machine learning model (e.g., outputs with the highest score and/or the highest accuracy). Specifically, the ML system may perform the experiment-based machine learning training process asynchronously with respect to building the first machine learning model (e.g., after the ML system builds the first machine learning model by analyzing the characteristics derived from the datasets). Then the ML system may pair the outputs (e.g., the most suitable the machine learning model type and the training parameters tested by the experiment-based machine learning training process) with the data characteristics (e.g., measures) derived from the datasets and store the pairing of the outputs and the data characteristics.” (para. 0028).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL C. LEE/Examiner, Art Unit 2128
Prosecution Insights