Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 4, 8, 11, 13-14, 16, 19, 20, 25, 29, 32, 34-35, 37, 40-41, and 45 have been amended. Claims 2-3, 5-7, 9-10, 12, 15, 17-18, 21, 23-24, 26-28, 30-31, 33, 36, 38-39, 42, 44, and 46-78 have been canceled. Claims 1, 4, 8, 11, 13-14, 16, 19-20, 22, 25, 29, 32, 34-35, 37, 40-41, 43, and 45 are currently pending.
Election/Restrictions
Restriction to one of the following inventions is required under 35 U.S.C. 121:
I. Claims 1, 4, 8, 11, 13-14, 16, 19-20, 22, 25, 29, 32, 34-35, 37, and 40-41, drawn to a system and a method for applying data analytics algorithms to an intermediary output of a machine learning model, and determining a measure of performance in response to determining input values are associated with performance indicators indicating the machine-learning model is performing poorly, classified in G06N20/00.
II. Claims 43 and 45, drawn to a system of determining a likelihood of mistake for a prediction generated by a machine-learning model, and evaluating inputs and respective predictions to determine causes for a high likelihood of mistake, classified in G06N20/00.
The inventions are independent or distinct, each from the other because:
Inventions I and II are directed to related processes. The related inventions are distinct if: (1) the inventions as claimed are either not capable of use together or can have a materially different design, mode of operation, function, or effect; (2) the inventions do not overlap in scope, i.e., are mutually exclusive; and (3) the inventions as claimed are not obvious variants. See MPEP § 806.05(j). In the instant case, the inventions as claimed
(1) have different modes of operations and functions (Group I recites applying data analytics algorithms to an intermediary output of a machine learning model, and determining a measure of performance when the machine-learning model is performing poorly. Group II recites determining a likelihood of mistake for a prediction generated by a machine-learning model, and evaluating inputs and respective predictions to determine causes for a high likelihood of mistake.)
(2) do not overlap in scope (each invention of groups I and II contain features which do not appear in any of the other inventions) and
(3) the inventions of groups I and II are not obvious variants of one another.
Furthermore, the inventions as claimed do not encompass overlapping subject matter and there is nothing of record to show them to be obvious variants.
Restriction for examination purposes as indicated is proper because all the inventions listed in this action are independent or distinct for the reasons given above and there would be a serious search and/or examination burden if restriction were not required because one or more of the following reasons apply:
Groups I and II have different, non-overlapping series of steps which would require a serious search and examination burden.
Applicant is reminded that upon the cancelation of claims to a non-elected invention, the inventorship must be corrected in compliance with 37 CFR 1.48(a) if one or more of the currently named inventors is no longer an inventor of at least one claim remaining in the application. A request to correct inventorship under 37 CFR 1.48(a) must be accompanied by an application data sheet in accordance with 37 CFR 1.76 that identifies each inventor by his or her legal name and by the processing fee required under 37 CFR 1.17(i).
Applicant is advised that the reply to this requirement to be complete must include (i) an election of an invention to be examined even though the requirement may be traversed (37 CFR 1.143) and (ii) identification of the claims encompassing the elected invention.
The election of an invention may be made with or without traverse. To reserve a right to petition, the election must be made with traverse. If the reply does not distinctly and specifically point out supposed errors in the restriction requirement, the election shall be treated as an election without traverse. Traversal must be presented at the time of election in order to be considered timely. Failure to timely traverse the requirement will result in the loss of right to petition under 37 CFR 1.144. If claims are added after the election, applicant must indicate which of these claims are readable upon the elected invention.
Should applicant traverse on the ground that the inventions are not patentably distinct, applicant should submit evidence or identify such evidence now of record showing the inventions to be obvious variants or clearly admit on the record that this is the case. In either instance, if the examiner finds one of the inventions unpatentable over the prior art, the evidence or admission may be used in a rejection under 35 U.S.C. 103 or pre-AIA 35 U.S.C. 103(a) of the other invention.
During a telephone conversation with Ms. Joanna Ma, Reg. No. 72,425 on 20 July 2026, a provisional election was made without traverse to prosecute the invention of Group I, claims 1, 4, 8, 11, 13-14, 16, 19-20, 22, 25, 29, 32, 34-35, 37, and 40-41. Affirmation of this election must be made by applicant in replying to this Office action. Claims 43 and 45 are withdrawn from further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention.
Claim Objections
Claims 8, 13-14, 22, 29, 35, 37 are objected to because of the following informalities: In claim 8, there is a line break after “the” in line 3.
In claim 13, line 1, “where in” should recite “wherein”.
In claim 14, line 3 and in claim 35, line 3, Examiner recommends adding “the” before “substandard characteristics”.
In claim 22, line 10, the limitation “model performing” should recite “model is performing”.
In claim 29, each recitation of inputs value(s) should recite input value(s).
In claim 35, line 4, Examiner recommends adding a comma after “characteristics”.
In claim 37, line 7, “the performance indicator” should recite “the one or more performance indicators”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1, 4, 8, 11, 13-14, 16, 19-20, 22, 25, 29, 32, 34-35, 37, 40-41 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “poorly” in claim 1, lines 12 and 15 is a relative term which renders the claim indefinite. The term “poorly” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “poorly” is a subjective term under MPEP 2173.05(b) subsection IV because the claim and written disclosure do not provide an objective standard for measuring poor performance. It is unclear how to distinguish good/acceptable performance indicators from poor performance indicators. Examiner treats poor performance as any performance having a value that falls below a predetermined threshold.
Claims 4, 8, 11, 13-14, 16, 19-20 are rejected for failing to cure the deficiencies of claim 1.
The relative term “poorly” is recited by claim 4, line 10; claim 14, line 4; claim 19, lines 5 and 8; and claim 20, line 4. These claims are rejected for the same reason as claim 1.
In claim 4, lines 9-10, the limitation “one or more substandard characteristics indicating the machine-learning model is performing poorly” renders the claim indefinite. It is unclear what a “substandard characteristic” means, and it is unclear how it might relate to so-called standard characteristics. Examiner treats a substandard characteristic as an incorrect prediction outcome. In specification paragraph 125, lines 7-10 appear to provide support for this interpretation.
The term “low” in claim 16, line 5 is a relative term which renders the claim indefinite. The term “low” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “low” is an indefinite term of degree under MPEP 2173.05(b) subsection I because the claim and written disclosure do not provide an objective standard for measuring a low measure of confidence for the prediction. It is unclear how to distinguish a low confidence from a high confidence. Examiner treats a low measure of confidence for the prediction as a measure of confidence that falls below a predetermined threshold.
Claim 22 recites the same indefinite limitations as claim 1 and is therefore rejected for at least the same reasons.
Claims 25, 29, 32, 34-35, 37, 40-41 are rejected for failing to cure the deficiencies of claim 22.
The relative term “poorly” is recited by claim 25, line 10; claim 35, line 4; claim 40, lines 5 and 8; and claim 41, line 3. These claims are rejected for the same reason as claim 22.
Claim 25 recites the same indefinite limitations as claim 4 and is therefore rejected for at least the same reasons.
Claim 37 recites the same indefinite limitations as claim 16 and is therefore rejected for at least the same reasons.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 4, 8, 11, 13-14, 16, 19-20, 22, 25, 29, 32, 34-35, 37, 40-41 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1, 4, 8, 11, 13-14, 16, 19-20 recite a system comprising a processor. Claims 22, 25, 29, 32, 34-35, 37, 40-41 recite a method. A system and a method each falls within one of the four statutory categories of patent eligible subject matter.
Claim 1
Step 2A Prong 1: Evaluating a performance of a machine-learning model is an evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Apply one or more data analytics algorithms to one of an intermediary output of the machine-learning model and the set of inputs for associating an input value to each input of the set of inputs is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper. In the instant specification, paragraphs 9-10 discloses data analytics algorithms comprise clustering algorithms which cluster the intermediary output and set of inputs into datapoint clusters corresponding to classes. A person can reasonably cluster data into different datapoint clusters based on similarities in the data.
Evaluate the input values associated with the set of inputs to determine whether one or more input values are associated with one or more performance indicators indicating the machine-learning model is performing poorly is an evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
In response to determining the one or more input values are associated with the one or more performance indicators indicating the machine-learning model is performing poorly, determine a measure of performance for the machine-learning model, the measure of performance being determined based on one or more of the one or more performance indicators and the input values associated with the set of inputs, and the prediction generated for the inputs is a mathematical calculation. In the instant specification, paragraph 130 discloses, “The measure of performance can include a quantitative or qualitative likelihood of a prediction being erroneous”. Determining a quantitative likelihood of an error would amount to calculating a probability.
Generate a recommendation for improving the performance of the machine-learning model based at least on one or more of the one or more performance indicators and the measure of performance is a judgement and opinion mental process which can reasonably be performed in the human mind with the aid of pencil and paper. Specification paragraph 136 discloses recommendations include applying an improved machine-learning model. A person can analyze a model’s prediction performance and recommend changing to an improved model to improve prediction performance. The claim recites an abstract idea.
Step 2A Prong 2: A database having a set of inputs stored thereon and a processor in communication with the database amount to generic computer components for applying the abstract ideas on a generic computer under MPEP 2106.05(f).
Apply the machine-learning model to the set of inputs to generate a prediction for each of the inputs in the set of inputs amounts to invoking computers merely as a tool to perform an existing process under MPEP 2106.05(f).
The additional elements as disclosed above, alone or in combination, do not integrate the abstract ideas into a practical application as they are generic computer functions that are implemented to perform the abstract ideas disclosed above. The claim is directed to an abstract idea.
Step 2B: A database having a set of inputs stored thereon and a processor in communication with the database amount to generic computer components for applying the abstract ideas on a generic computer under MPEP 2106.05(f).
Apply the machine-learning model to the set of inputs to generate a prediction for each of the inputs in the set of inputs amounts to invoking computers merely as a tool to perform an existing process under MPEP 2106.05(f).
The additional elements as disclosed above, in combination with the abstract ideas, are not sufficient to amount to significantly more than the abstract ideas as they are generic computer functions that are implemented to perform the abstract ideas disclosed above. The claim is not patent eligible.
Claim 4 incorporates the rejection of claim 1.
Step 2A Prong 1: The abstract ideas of claim 1 are incorporated. Apply the one or more data analytics algorithms to one of the intermediary output of the machine-learning model and the set of training inputs for associating a training input value to each training input of the set of inputs is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper. Paragraphs 9-10 in the instant specification explains that data analytics algorithms are clustering algorithms. A person can reasonably cluster data into different datapoint clusters based on similarities in the data.
Evaluate the training input values associated with the set of training inputs to determine one or more substandard characteristics indicating the machine-learning model is performing poorly is an evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Determine the one or more performance indicators based at least on the one or more substandard characteristics and the training input values associated with the one or more substandard characteristics is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The processor amounts to a generic computer component for applying the abstract ideas on a generic computer under MPEP 2106.05(f).
Apply the machine-learning model to a set of training inputs to generate a prediction for each of the training inputs in the set of training inputs amounts to invoking computers merely as a tool to perform an existing process under MPEP 2106.05(f). The claim is not patent eligible.
Claim 8 incorporates the rejection of claim 1.
Step 2A Prong 1: The abstract ideas of claim 1 are incorporated. Applying the one or more data analytics algorithms comprises determining, for the intermediary output of the machine-learning model, a plurality of similarity scores, wherein each similarity score indicates a similarity between a first input value of the plurality of input values and a second input value of the plurality of input values based on a comparison metric between the first input value and the second input value is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The claim does not recite any additional elements which, alone or in combination, would integrate the abstract ideas into a practical application or which, in combination with the abstract ideas, would be sufficient to amount to significantly more than the abstract ideas. The claim is not patent eligible.
Claim 11 incorporates the rejection of claim 4.
Step 2A Prong 1: The abstract ideas of claim 4 are incorporated. For the training inputs associated with the subset of training input values associated with the substandard characteristics, determine a plurality of attributes of the training inputs is a judgement mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Determine a subset of shared attributes within the plurality of attributes is a judgement mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Determine the one or more performance indicators based on the determined subset of shared attributes is a judgement mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The processor amounts to a generic computer component for applying the abstract ideas on a generic computer under MPEP 2106.05(f).
Training inputs evaluated by the machine-learning model to generate the prediction amounts to invoking computers merely as a tool to perform an existing process under MPEP 2106.05(f). The claim is not patent eligible.
Claim 13 incorporates the rejection of claim 11.
Step 2A Prong 1: Determine a plurality of attributes of the inputs evaluated by the machine-learning model to generate the prediction is a judgement mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Determine the measure of performance of the machine-learning model based at least on a comparison of the plurality of attributes evaluated by the machine-learning model and the subset of shared attributes is a mathematical calculation, as explained in step 2A prong 1 for claim 1. Comparing the plurality of attributes and the subset of shared attributes is a judgement mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The processor amounts to a generic computer component for applying the abstract ideas on a generic computer under MPEP 2106.05(f). The claim is not patent eligible.
Claim 14 incorporates the rejection of claim 4.
Step 2A Prong 1: Evaluating the plurality of training input values to identify the subset of training input values associated with substandard characteristics comprises identifying one or more prediction outcomes associated with the machine-learning model performing poorly and evaluating the plurality of training input values associated with the one or more prediction outcomes are observation and evaluation mental processes which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The claim does not recite any additional elements which, alone or in combination, would integrate the abstract ideas into a practical application or which, in combination with the abstract ideas, would be sufficient to amount to significantly more than the abstract ideas. The claim is not patent eligible.
Claim 16 incorporates the rejection of claim 4.
Step 2A Prong 1: The abstract ideas of claim 4 are incorporated. Evaluate the prediction to determine a measure of confidence for the prediction for each of the training inputs in the set of training inputs is an evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Evaluate the training input values associated with a low measure of confidence for the prediction to identify the substandard characteristics is an evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Determine the one or more performance indicators based on the identified substandard characteristics associated with the training input values is a judgment and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The processor amounts to a generic computer component for applying the abstract ideas on a generic computer under MPEP 2106.05(f). The claim is not patent eligible.
Claim 19 incorporates the rejection of claim 4.
Step 2A Prong 1: The abstract ideas of claim 4 are incorporated. Determining the one or more performance indicators comprises: evaluating a first plurality of training input values obtained according to a first data analytics algorithm to identify one or more first substandard characteristics of a first subset of input values associated with the machine-learning model performing poorly is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper. Paragraphs 9-10 in the instant specification explains that data analytics algorithms are clustering algorithms. A person can reasonably cluster data into different datapoint clusters based on similarities in the data.
Evaluating a second plurality of training input values obtained according to a second data analytics algorithm to identify one or more second substandard characteristics of a second subset of input values associated with the machine-learning model performing poorly is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Determining the one or more performance indicators based on the first substandard characteristics and the second substandard characteristics is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The claim does not recite any additional elements which, alone or in combination, would integrate the abstract ideas into a practical application or which, in combination with the abstract ideas, would be sufficient to amount to significantly more than the abstract ideas. The claim is not patent eligible.
Claim 20 incorporates the rejection of claim 1.
Step 2A Prong 1: The abstract ideas of claim 1 are incorporated. Determining the one or more performance indicators comprises identifying whether the prediction is associated with a prediction outcome associated with the machine-learning model performing poorly is a judgement and evaluation mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The claim does not recite any additional elements which, alone or in combination, would integrate the abstract ideas into a practical application or which, in combination with the abstract ideas, would be sufficient to amount to significantly more than the abstract ideas. The claim is not patent eligible.
Claims 22, 25, 29, 32, 34-35, 37, 40 are each recites a method which implements the same features as the system of claims 1, 4, 8, 11, 13-14, 16, 19, respectively, and are therefore rejected for at least the same reasons.
Claim 41 incorporates the rejection of claim 22.
Step 2A Prong 1: The abstract ideas of claim 22 are incorporated. Determining the one or more performance indicators comprises identifying one or more prediction outcomes associated with the machine-learning model performing poorly is an observation and judgement mental process which can reasonably be performed in the human mind with the aid of pencil and paper.
Step 2A Prong 2 and Step 2B: The claim does not recite any additional elements which, alone or in combination, would integrate the abstract ideas into a practical application or which, in combination with the abstract ideas, would be sufficient to amount to significantly more than the abstract ideas. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4, 8, 11, 13-14, 16, 20, 22, 25, 29, 32, 34-35, 37, 41 are rejected under 35 U.S.C. 103 as being unpatentable over Sallee et al. (US 20220129712 A1) in view of Sanchez (US 20220383207 A1) and Ghanta et al. (US 20230196101 A1).
Regarding claim 1, Sallee teaches: A system for evaluating a performance of a machine-learning model, the system comprising: a database having a set of inputs stored thereon; and ([0018] and [0073], lines 1-6)
a processor in communication with the database, wherein the processor is operable to: ([0072])
apply the machine-learning model to the set of inputs to generate a prediction for each of the inputs in the set of inputs; ([0047] and [0051]-[0054], where inputs are content 104, and the “machine-learning model” corresponds to a combination of trained ML model 442 and clusterer 444.)
apply one or more data analytics algorithms to one of an intermediary output of the machine-learning model and the set of inputs for associating an input value to each input of the set of inputs; ([0019], lines 1-5, [0020], lines 1-4, [0035], lines 8-12, [0051]-[0054] discloses clustering latent features 450 generated by a trained ML model 442. Since the latent features are based on the set of inputs, the clustering algorithm is applied to the set of inputs. A piece of content 104 is mapped to a data sample in the cluster space, and an “input value” as claimed corresponds to a data sample in the cluster space.)
evaluate the input values associated with the set of inputs to determine whether one or more input values are associated with one or more performance indicators indicating the machine-learning model is performing poorly; ([0050], lines 1-7 and [0051]-[0054] teaches evaluating content 104 to predict classification 446 using cluster data points, and then calculating a loss 448 between the predicted class and a correct class. A data sample in the cluster space that is associated with a different predicted class from the correct class indicates the trained ML model and clusterer are performing poorly and need further training.)
where a large distance measurement indicates the model performs poorly, and the loss is a measure of performance which is based on the data sample and the prediction.)
However, Sallee does not explicitly teach: in response to determining the one or more input values are associated with the one or more performance indicators indicating the machine-learning model is performing poorly, determine a measure of performance for the machine-learning model,
generate a recommendation for improving the performance of the machine-learning model
But Sanchez teaches: in response to determining the one or more input values are associated with the one or more performance indicators indicating the machine-learning model is performing poorly, determine a measure of performance for the machine-learning model, ([0046], [0048]-[0049], [0051] discloses in response to determining an average prediction confidence level for input data 212, comparing machine-predicted data with human-identified data (step 622). An “input value” is data 212, and a poor performance indicator is an average prediction confidence level because it is less than a high prediction confidence level.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have applied Sanchez’s technique to Sallee. A motivation for the combination is to update the machine learning model while limiting the amount of data a human analyzer identifies. (Sanchez, [0048]-[0049])
However, Sallee and Sanchez do not explicitly teach: generate a recommendation for improving the performance of the machine-learning model
But Ghanta teaches: generate a recommendation for improving the performance of the machine-learning model ([0117], lines 1-6, [0119] and [0126] teaches recommending different machine learning algorithms for analyzing the inference data set. An action module may generate a notification or a message that includes a recommendation.)
Sallee teaches training a model to improve an accuracy, and Ghanta teaches changing a machine learning model that is more “suitable” for a data set (i.e., accurate, per [0112]). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have applied Ghanta’s action module which generates a recommendation into the combination of Sallee and Sanchez, wherein the recommendation would include training a neural network. A motivation for the combination is to generate a more suitable machine learning model for a data set. (Ghanta, [0119])
Regarding claim 4, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 1, wherein the processor is operable to:
Sallee teaches: apply the machine-learning model to a set of training inputs to generate a prediction for each of the training inputs in the set of training inputs; ([0052], lines 1-5 and [0053], lines 1-3. The “machine-learning model” corresponds to a combination of trained ML model 442 and clusterer 444.)
apply the one or more data analytics algorithms to one of the intermediary output of the machine-learning model and the set of training inputs for associating a training input value to each training input of the set of inputs; ([0019], lines 1-5, [0020], lines 1-4, [0035], lines 8-12, [0051]-[0054] discloses clustering latent features 450 generated by a trained ML model 442. Since the latent features are based on the set of training inputs, the clustering algorithm is applied to the set of training inputs. A piece of content 104 is mapped to a data sample in the cluster space, and a “ training input value” as claimed corresponds to a data sample in the cluster space.)
evaluate the training input values associated with the set of training inputs to determine one or more substandard characteristics indicating the machine-learning model is performing poorly; and ([0022], lines 5-11, [0050], lines 1-7 and [0051]-[0054] teaches evaluating content 104 to predict classification 446 using cluster data points, and then calculating a loss 448 between the classification and a label 105. A data sample in the cluster space that is associated with a different classification from the label indicates the trained ML model and clusterer are performing poorly and need further training. A “substandard characteristic” is an incorrect prediction.)
determine the one or more performance indicators based at least on the one or more substandard characteristics and the training input values associated with the one or more substandard characteristics. (A performance indicator is a loss 448. [0022], lines 5-11, [0052], lines 5-8, and [0053] teaches determining a loss between predicted classification 446 and label 105.)
Regarding claim 8, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 1,
Sallee teaches: wherein applying the one or more data analytics algorithms comprises determining, for the intermediary output of the machine-learning model, a plurality of similarity scores, wherein each similarity score indicates a similarity between a first input value of the plurality of input values and a second input value of the plurality of input values based on a comparison metric between the first input value and the second input value. ([0048], lines 3-7 and [0054] discloses calculating confidence by measuring a distance between a data sample and a first point that formed a basis for the cluster. A “similarity score” is a confidence that a data sample belongs to a cluster, a “comparison metric” is a distance function, a “first input value” is the first point that formed a basis for the cluster, and a “second input value” is the data sample whose confidence is being determined.)
Regarding claim 11, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 4, wherein the processor is operable to:
Sallee teaches: for the training inputs associated with the subset of training input values associated with the substandard characteristics, determine a plurality of attributes of the training inputs evaluated by the machine-learning model to generate the prediction; ([0048], lines 3-7 and [0053]-[0054] teaches the clusterer determines a plurality of data samples (“attributes of the training inputs”) to generate a classification.)
determine a subset of shared attributes within the plurality of attributes; and ([0020] and [0053] teaches clustering the data samples. A cluster of data samples shares attributes.)
determine the one or more performance indicators based on the determined subset of shared attributes. ([0053] teaches determining a loss for a classification based on its cluster.)
Regarding claim 13, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 11, wherein the processor is operable to:
Sallee teaches: determine a plurality of attributes of the inputs evaluated by the machine-learning model to generate the prediction; and ([0048], lines 3-7 and [0053]-[0054] teaches the clusterer determines a plurality of data samples in the cluster space (“attributes of the inputs”) to generate a classification.)
determine the measure of performance of the machine-learning model based at least on a comparison of the plurality of attributes evaluated by the machine-learning model and the subset of shared attributes. ([0053] teaches determining a loss for a classification which indicates whether a data sample belongs in a cluster. The loss between a classification and a label essentially compares data samples in the cluster space (i.e., “the plurality of attributes”) to the cluster (“subset of shared attributes”), where a lower loss value indicates the data samples are clustered correctly.)
Regarding claim 14, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 4,
Sallee teaches: wherein evaluating the plurality of training input values to identify the subset of training input values associated with substandard characteristics comprises identifying one or more prediction outcomes associated with the machine-learning model performing poorly and evaluating the plurality of training input values associated with the one or more prediction outcomes. ([0022], lines 5-11 and [0052]-[0053] teaches the clusterer determines a classification for data samples in the cluster space (i.e., “evaluating the plurality of training input values”) and is retrained by calculating a loss between a label and classification for all data samples (i.e., “identifying prediction outcomes”).)
Regarding claim 16, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 4, wherein the processor is operable to:
Sallee teaches: evaluate the prediction to determine a measure of confidence for the prediction for each of the training inputs in the set of training inputs; ([0048], lines 3-7 and [0053]-[0054])
evaluate the training input values associated with a low measure of confidence for the prediction to identify the substandard characteristics; and (A “substandard characteristic” is an incorrect prediction. [0048], lines 3-7 and [0052], lines 5-8, [0053]-[0054] teaches determining a loss for all classifications during training. Any classification that is different from the label is identified as a substandard classification.)
determine the one or more performance indicators based on the identified substandard characteristics associated with the training input values. (A performance indicator is a loss. [0053] teaches determining a loss for all classifications during training.)
Regarding claim 20, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 1,
Sallee teaches: wherein determining the one or more performance indicators comprises identifying whether the prediction is associated with a prediction outcome associated with the machine-learning model performing poorly. (This limitation amounts to determining if a model prediction is incorrect. [0052], lines 5-8 and [0053] teaches determining a loss between predicted classification 446 and label 105.)
Claims 22, 25, 29, 32, 34-35, 37 each recites a method which implements the same features as the system of claims 1, 4, 8, 11, 13-14, 16, respectively, and are therefore rejected for at least the same reasons.
Regarding claim 41, the combination of Sallee, Sanchez, and Ghanta teaches: The method of claim 22,
Sallee teaches: wherein determining the one or more performance indicators comprises identifying one or more prediction outcomes associated with the machine-learning model performing poorly. ([0022], lines 5-11 and [0052]-[0053] teaches the clusterer is retrained by calculating a loss between a label and classification for all data samples (i.e., “identifying prediction outcomes”).)
Claims 19 and 40 are rejected under 35 U.S.C. 103 as being unpatentable over Sallee et al. (US 20220129712 A1) in view of Sanchez (US 20220383207 A1), Ghanta et al. (US 20230196101 A1), and O’Connor et al. (US 20220092404 A1).
Regarding claim 19, the combination of Sallee, Sanchez, and Ghanta teaches: The system of claim 4,
Sallee teaches: wherein determining the one or more performance indicators comprises: evaluating a first plurality of training input values obtained according to a first data analytics algorithm to identify one or more first substandard characteristics of a first subset of input values associated with the machine-learning model performing poorly; ([0020] teaches a clustering classification layer (i.e., “a first data analytics algorithm”), and [0052], lines 5-8 and [0053] teaches determining a loss for all classifications during training. Any classification that is different from the label is identified as a substandard classification. In Fig. 4, the clusters 116 show a plurality of data samples. A first plurality of training input values and a first subset of input values corresponds to at least two data samples having incorrect classifications. First substandard characteristics are classifications for these data samples.)
evaluating a second plurality of training input values obtained according to [the first] and ([0052], lines 5-8 and [0053] teaches determining a loss for all classifications during training. Any classification that is different from the label is identified as a substandard classification. A second plurality of training input values and a second subset of input values corresponds to at least two other data samples having incorrect classifications. Second substandard characteristics are classifications for these data samples.)
determining the one or more performance indicators based on the first substandard characteristics and the second substandard characteristics. ([0052], lines 5-8 and [0053] teaches determining a loss for classifications.)
However, Sallee, Sanchez, and Ghanta do not explicitly teach: a second data analytics algorithm
But O’Connor teaches: a first data analytics algorithm and a second data analytics algorithm ([0042]-[0046] teaches training at least a first student neural network 1061 and training a second student neural network 1062.)
Sallee at [0020] teaches the clusterer can include a fully connected NN layer. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated O’Connor’s technique (i.e., directing training data to one of a first machine learning model or a second machine learning model) into the combination of Sallee, Sanchez, and Ghanta. The combination would result in directing latent features to either a first clusterer or a second clusterer for training. A motivation for the combination is to train each clusterer to cluster a specific type of data.
Claim 40 recites a method which implements the same features as the system of claim 19 and is therefore rejected for at least the same reasons.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Asher H. Jablon whose telephone number is (571)270-7648. The examiner can normally be reached Monday - Friday, 9:00 am - 6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at (571)270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.H.J./Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127
Prosecution Insights