DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/28/2022 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
101 Rejection Arguments
Applicant asserts:
Applicant argues, on page 12, that the process of claim 1 cannot be practically performed mentally because training a machine learning model is not an abstract idea because it cannot be practically performed in the human mind. Applicant further points to more detailed steps of the training that make it not practically possible in the human mind.
Examiner response:
Examiner respectfully disagrees. The training step is not considered as judicial exception but considered as an additional element. The more detailed steps regarding how the training is done is not reflected within the language of the claims, so the training step is treated as mere instructions to implement an abstract idea on a computer.
Applicant asserts:
Applicant argues, on page 14, that the independent claim 1 integrates the abstract idea into a practical application. Specifically pointing to the retraining step in the claim and additional steps from the spec that in combination improves the efficiency of the training process and the accuracy of the results. In addition, Applicant states the claimed invention improves the efficiency of the process of retraining based on the production data.
Examiner response:
Examiner respectfully disagrees. The retraining step does not integrate the judicial exception into a practical application because it claimed in a way to recite the additional steps mentioned in the argument to show an improvement in the functioning of a computer, or an improvement to other technology or technical field. In addition, the arguments of improvement does not clearly show how it improves upon existing technology. A similar response to the production dataset is made. What aspect of the sampling step is a improvement upon existing technology.
103 Rejection Arguments
Applicant asserts:
Applicant argues, on page 16, that the prior art does not teach “prioritizing elements of the review dataset for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment, the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that the first element sampled before a second element is given higher priority during review”
Examiner response:
Examiner respectfully disagrees. Jorasch is used to teach the queue as a way to prioritize questions and answers. The questions and answers can be substituted as elements such that the order of the elements were added reflect the priority of the elements.
Claim Rejections - 35 USC § 112b
Claims 1, 7, 9, 15, 20, 22, 24, and 26 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1, 9, and 24 recites the limitation "the subset of elements" in fourth to last line of claim 1. There is insufficient antecedent basis for this limitation in the claim.
Claim 7, 15, and 20 recites the limitation "the review" in line 3 of claim 7. There is insufficient antecedent basis for this limitation in the claim.
The term “higher” in claims 22 and 26 is a relative term which renders the claim indefinite. The term “higher” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term experience is rendered indefinite but the use of “higher”.
The term “less” in claims 22 and 26 is a relative term which renders the claim indefinite. The term “higher” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term experience is rendered indefinite but the use of “less”.
In reference to claims 1, 7, 9, 15, 20, 22, 24, and 26, dependent claims do not cure the deficiencies noted in the rejection of claims 1, 7, 9, 15, 20, 22, 24, and 26. Therefore, these claims are rejected under the same rationale as claim 1, 7, 9, 15, 20, 22, 24, and 26.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 4-7, 9, 12-15, 17, and 19-27 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
In reference to claim 1:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“sampling elements of the production dataset by, repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could identify an element from the production dataset that maximizes a measure of minimum distance between elements of the datasets and add the element to the review dataset.
“prioritizing elements of the review dataset for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment, the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could prioritize elements based on the order in which the elements were added.
“evaluating the machine learning model based on the feedback on the accuracy of the result of execution of the machine learning model for the subset of elements;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could evaluate the machine learning model based on the feedback on the accuracy.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“training a machine learning model using a training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“initializing a review dataset based on elements of the training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g)).
The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“training a machine learning model using a training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“initializing a review dataset based on elements of the training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment;” (well-understood, routine, conventional MPEP 2106.05(d)).
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 4:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
No
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 5:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
No
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 6:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
No
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“wherein the machine learning model is a convolutional neural network configured to process an image and each element of a dataset includes an image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“wherein the machine learning model is a convolutional neural network configured to process an image and each element of a dataset includes an image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 7:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could make a comparison information received in the user feedback with a result of execution of the machine learning model.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 9:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“sampling elements of the production dataset by, repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could identify an element from the production dataset that maximizes a measure of minimum distance between elements of the datasets and add the element to the review dataset.
“prioritizing elements of the review dataset for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment, the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could prioritize elements based on the order in which the elements were added.
“evaluating the machine learning model based on the feedback on the accuracy of the result of execution of the machine learning model for the subset of elements;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could evaluate the machine learning model based on the feedback on the accuracy.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“training a machine learning model using a training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“initializing a review dataset based on elements of the training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g)).The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“training a machine learning model using a training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“initializing a review dataset based on elements of the training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment;” (well-understood, routine, conventional MPEP 2106.05(d)).
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 12:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
No
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 13:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
No
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 14:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
No
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“wherein the machine learning model is a convolutional neural network configured to process an image and each element includes an image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
wherein the machine learning model is a convolutional neural network configured to process an image and each element includes an image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 15:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could make a comparison information received in the user feedback with a result of execution of the machine learning model.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 17:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“sampling elements of the production dataset by, repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could identify an element from the production dataset that maximizes a measure of minimum distance between elements of the datasets and add the element to the review dataset.
“prioritizing elements of the review dataset for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment, the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could prioritize elements based on the order in which the elements were added.
“evaluating the machine learning model based on the feedback on the accuracy of the result of execution of the machine learning model for the subset of elements;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could evaluate the machine learning model based on the feedback on the accuracy.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“A computer system comprising: one or more computer processors; and a non-transitory computer readable storage medium storing instructions that when executed by the one or more computer processors, cause the one or more computer processors to perform steps comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“training a machine learning model using a training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“initializing a review dataset based on elements of the training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g)).
The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“A computer system comprising: one or more computer processors; and a non-transitory computer readable storage medium storing instructions that when executed by the one or more computer processors, cause the one or more computer processors to perform steps comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“training a machine learning model using a training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“initializing a review dataset based on elements of the training dataset;” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment;” (well-understood, routine, conventional MPEP 2106.05(d)).
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 19:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
No
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 20:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could make a comparison information received in the user feedback with a result of execution of the machine learning model.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 21:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element based on the order in which the element was sampled.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could select a user for reviewing the result based on the order in which the element was sampled.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 22:
Claim 22 is directed to a judicial exception from claim(s) depended on and does not recite additional elements that integrate the judicial exception into a practical application and amount to significantly more than the judicial exception.
In reference to claim 23:
Claim 23 is directed to a judicial exception from claim(s) depended on and does not recite additional elements that integrate the judicial exception into a practical application and amount to significantly more than the judicial exception.
In reference to claim 24:
Claim 24 is directed to a judicial exception from claim(s) depended on and does not recite additional elements that integrate the judicial exception into a practical application and amount to significantly more than the judicial exception.
In reference to claim 25:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The non-transitory computer readable storage medium of claim 9, wherein the instructions further cause the one or more processors to perform steps comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element based on the order in which the element was sampled.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could select a user for reviewing the result based on the order in which the element was sampled.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 26:
Claim 26 is directed to a judicial exception from claim(s) depended on and does not recite additional elements that integrate the judicial exception into a practical application and amount to significantly more than the judicial exception.
In reference to claim 27:
Claim 27 is directed to a judicial exception from claim(s) depended on and does not recite additional elements that integrate the judicial exception into a practical application and amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6, 9, 14, 17, 23, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Chen; Xinyu et al; US 20220374274 A1 (hereinafter “Chen”) in view of Richard J. Hathaway; “Maximin Initialization for Cluster Analysis” (hereinafter “Hathaway”) in further view of Jorasch; James et al; US 20210399911 A1 (hereinafter “Jorasch”).
Regarding claim 1, Chen teaches A computer-implemented method comprising: training a machine learning model using a training dataset (Chen Paragraph 0024; " The model build process may include training a selected machine learning model with the training data.");
initializing a review dataset based on elements of the training dataset; (Chen Paragraph 0024; "The feature engineering process may include refining the initial training data (and the validation data) such that the data represents features needed for input to a machine learning model to be trained.");
receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment; (Chen Paragraph 0020; "In some embodiments, a data feed from which production data is to be obtained may be selected, and the production data may be provided as an input to a trained machine learning model.");
sampling elements of the production dataset by, [repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset]; (Chen Paragraph 0031; "a stability of a model may be determined by computing a stability score for the machine learning model based on the production data and the training data." Examiner notes that determining a stability score is sampling elements of the production dataset);
for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment, (Chen Paragraph 0026; “an accuracy of a trained machine learning model may be computed based on production data. Examiner notes that feedback on accuracy of results of execution of the machine learning model (accuracy of a trained machine learning model) is obtained in the production environment (production data))
evaluating the machine learning model based on the feedback on the accuracy of the result of execution of the machine learning model for the subset of elements; (Chen Paragraph 0026; “an accuracy of a trained machine learning model may be computed based on production data. The accuracy of the machine learning model may indicate how well the trained machine learning is able to predict results for production data... However, the accuracy of the trained machine learning model may be determined using the production data, which is obtained from a data feed (e.g., data feed 140), and which may only be available for a limited amount of time (e.g., while in the data stream).” Examiner notes that the machine learning model is evaluated based on the feedback on the accuracy of the result of execution of the machine learning model (The accuracy of the machine learning model may indicate how well the trained machine learning is able to predict results for production data) for the subset of elements (production data))
and responsive to the evaluation of the machine learning model indicating that the performance of the machine learning model is below a threshold value, retraining the machine learning model. (Chen Paragraph 0029; "the threshold accuracy score may be determined based on an accuracy score previously determined for the machine learning model during the training process. If it is determined that the machine learning model satisfies the threshold accuracy condition, then a notification may be generated indicating that the training data used to train the machine learning model is to be updated…the updated training data may be used to re-train the machine learning model" Examiner notes that responsive to the evaluation of the machine learning model indicating the performance of the machine learning model is below a threshold value (if it is determined that the machine learning model satisfies the threshold accuracy condition), retraining the machine learning model (re-train the machine learning model))
Chen does not teach repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset;
However, Hathaway does teach repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset; (Hathaway Page 16 Paragraph 2; "MMI: Maximin Initialization Algorithm Choose The number of clusters c, 1 < c < n; and, Rs x Rs if the available data are object data X, an inner product induced metric d(⋅ , ⋅ ) on. Input Object data X = {x1,...,xn} ⊂ Rs or dissimilarity data D = [dij] ⊂ Rnxn Step 1 Select the indices m1, ..., mc of the c distinguished objects. Select m1 = 1 (Remark: this is the "object seed";m1 = 1 is arbitrary) (If the data are X, calculate dissimilarities { d1,k }, k = 1 to n.) Initialize the minimum distance array… Step 2 Cluster each object in {o1, ..., on} with its nearest distinguished object. Clear the initialization matrix array" Examiner notes that the algorithm shows repeatedly performing/for loop; identifying an element that maximizes a measure of minimum distance; input objects are elements from the production dataset; clustering object is adding the element to the review dataset.)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen and Hathaway. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. One of ordinary skill would have motivation to combine Chen and Hathaway to identify an element that maximizes a measure of minimum distance in an inexpensive manner. “MMI systematically identifies objects that are distributed throughout the data, and uses the identified objects to inexpensively generate an initial partition of the entire data set. We will prove that the MMI partition is exact if the data set consists of compact and separated clusters in the sense of Dunn” (Hathaway Page 14 Paragraph 1).
Chen in view of Hathaway does not teach prioritizing elements of the review dataset for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment,
the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review;
However, Jorasch does teach prioritizing elements of the review dataset [for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment,] (Jorasch Paragraph 2611; "The producer software could display a dynamic queue for questions and answers, showing the order in which individuals ask questions, the priority or importance of their questions, or an ordering created by the meeting owner." Examiner notes that the elements of the review dataset (questions) is prioritized based on an order in which the sample was added using a queue;)
the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review; (Examiner references previous mapping to note the queue prioritizes elements in a first in first out manner meaning first element has higher priority)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, and Jorasch. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. One of ordinary skill would have motivation to combine Chen, Hathaway, and Jorasch to use a queue to show the level of importance and help with organization. “Each queue could also provide visibility to the grouping by role, organization, or level of interest/rating. This reporting could allow improved organization of material and improvement of content delivery and post meeting follow-up.” (Jorasch Paragraph 3172).
Regarding claim 6, Chen teaches The computer-implemented method of claim 1, wherein the machine learning model is a convolutional neural network configured to process an image and each element of a dataset includes an image. (Chen Paragraph 0021; "the model identifier may indicate a type of machine learning model that was obtained from a training environment (e.g., a CNN for computer vision");
Regarding claim 9, Chen teaches A non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising: (Chen Paragraph 0071; “Model execution subsystem 114 may include a set of modules, including a timer 402, a model selector 404, data duplication 406, data distribution 408, other modules, or other components. Each module of model execution subsystem 114 may be implemented by one or more processors executing computer program instructions stored in memory of computer system 102.”);
training a machine learning model using a training dataset (Chen Paragraph 0024; " The model build process may include training a selected machine learning model with the training data.");
initializing a review dataset based on elements of the training dataset; (Chen Paragraph 0024; "The feature engineering process may include refining the initial training data (and the validation data) such that the data represents features needed for input to a machine learning model to be trained.");
receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment; (Chen Paragraph 0020; "In some embodiments, a data feed from which production data is to be obtained may be selected, and the production data may be provided as an input to a trained machine learning model.");
sampling elements of the production dataset by, [repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset]; (Chen Paragraph 0031; "a stability of a model may be determined by computing a stability score for the machine learning model based on the production data and the training data." Examiner notes that determining a stability score is sampling elements of the production dataset);
for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment, (Chen Paragraph 0026; “an accuracy of a trained machine learning model may be computed based on production data. Examiner notes that feedback on accuracy of results of execution of the machine learning model (accuracy of a trained machine learning model) is obtained in the production environment (production data))
evaluating the machine learning model based on the feedback on the accuracy of the result of execution of the machine learning model for the subset of elements; (Chen Paragraph 0026; “an accuracy of a trained machine learning model may be computed based on production data. The accuracy of the machine learning model may indicate how well the trained machine learning is able to predict results for production data... However, the accuracy of the trained machine learning model may be determined using the production data, which is obtained from a data feed (e.g., data feed 140), and which may only be available for a limited amount of time (e.g., while in the data stream).” Examiner notes that the machine learning model is evaluated based on the feedback on the accuracy of the result of execution of the machine learning model (The accuracy of the machine learning model may indicate how well the trained machine learning is able to predict results for production data) for the subset of elements (production data))
and responsive to the evaluation of the machine learning model indicating that the performance of the machine learning model is below a threshold value, retraining the machine learning model. (Chen Paragraph 0029; "the threshold accuracy score may be determined based on an accuracy score previously determined for the machine learning model during the training process. If it is determined that the machine learning model satisfies the threshold accuracy condition, then a notification may be generated indicating that the training data used to train the machine learning model is to be updated…the updated training data may be used to re-train the machine learning model" Examiner notes that responsive to the evaluation of the machine learning model indicating the performance of the machine learning model is below a threshold value (if it is determined that the machine learning model satisfies the threshold accuracy condition), retraining the machine learning model (re-train the machine learning model))
Chen does not teach repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset;
However, Hathaway does teach repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset; (Hathaway Page 16 Paragraph 2; "MMI: Maximin Initialization Algorithm Choose The number of clusters c, 1 < c < n; and, Rs x Rs if the available data are object data X, an inner product induced metric d(⋅ , ⋅ ) on. Input Object data X = {x1,...,xn} ⊂ Rs or dissimilarity data D = [dij] ⊂ Rnxn Step 1 Select the indices m1, ..., mc of the c distinguished objects. Select m1 = 1 (Remark: this is the "object seed";m1 = 1 is arbitrary) (If the data are X, calculate dissimilarities { d1,k }, k = 1 to n.) Initialize the minimum distance array… Step 2 Cluster each object in {o1, ..., on} with its nearest distinguished object. Clear the initialization matrix array" Examiner notes that the algorithm shows repeatedly performing/for loop; identifying an element that maximizes a measure of minimum distance; input objects are elements from the production dataset; clustering object is adding the element to the review dataset.)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen and Hathaway. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. One of ordinary skill would have motivation to combine Chen and Hathaway to identify an element that maximizes a measure of minimum distance in an inexpensive manner. “MMI systematically identifies objects that are distributed throughout the data, and uses the identified objects to inexpensively generate an initial partition of the entire data set. We will prove that the MMI partition is exact if the data set consists of compact and separated clusters in the sense of Dunn” (Hathaway Page 14 Paragraph 1).
Chen in view of Hathaway does not teach prioritizing elements of the review dataset for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment,
the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review;
However, Jorasch does teach prioritizing elements of the review dataset [for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment,] (Jorasch Paragraph 2611; "The producer software could display a dynamic queue for questions and answers, showing the order in which individuals ask questions, the priority or importance of their questions, or an ordering created by the meeting owner." Examiner notes that the elements of the review dataset (questions) is prioritized based on an order in which the sample was added using a queue;)
the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review; (Examiner references previous mapping to note the queue prioritizes elements in a first in first out manner meaning first element has higher priority)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, and Jorasch. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. One of ordinary skill would have motivation to combine Chen, Hathaway, and Jorasch to use a queue to show the level of importance and help with organization. “Each queue could also provide visibility to the grouping by role, organization, or level of interest/rating. This reporting could allow improved organization of material and improvement of content delivery and post meeting follow-up.” (Jorasch Paragraph 3172).
Regarding claim 14, Chen teaches The non-transitory computer readable storage medium of claim 9, wherein the machine learning model is a convolutional neural network configured to process an image and each element includes an image. (Chen Paragraph 0021; "the model identifier may indicate a type of machine learning model that was obtained from a training environment (e.g., a CNN for computer vision");
Regarding claim 17, Chen teaches A computer system comprising: one or more computer processors; and a non-transitory computer readable storage medium storing instructions that when executed by the one or more computer processors, cause the one or more computer processors to perform steps comprising: (Chen Paragraph 0071; “Model execution subsystem 114 may include a set of modules, including a timer 402, a model selector 404, data duplication 406, data distribution 408, other modules, or other components. Each module of model execution subsystem 114 may be implemented by one or more processors executing computer program instructions stored in memory of computer system 102.”);
training a machine learning model using a training dataset (Chen Paragraph 0024; " The model build process may include training a selected machine learning model with the training data.");
initializing a review dataset based on elements of the training dataset; (Chen Paragraph 0024; "The feature engineering process may include refining the initial training data (and the validation data) such that the data represents features needed for input to a machine learning model to be trained.");
receiving a production dataset based on values received from a production environment, wherein the machine learning model is being executed in the production environment; (Chen Paragraph 0020; "In some embodiments, a data feed from which production data is to be obtained may be selected, and the production data may be provided as an input to a trained machine learning model.");
sampling elements of the production dataset by, [repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset]; (Chen Paragraph 0031; "a stability of a model may be determined by computing a stability score for the machine learning model based on the production data and the training data." Examiner notes that determining a stability score is sampling elements of the production dataset);
for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment, (Chen Paragraph 0026; “an accuracy of a trained machine learning model may be computed based on production data. Examiner notes that feedback on accuracy of results of execution of the machine learning model (accuracy of a trained machine learning model) is obtained in the production environment (production data))
evaluating the machine learning model based on the feedback on the accuracy of the result of execution of the machine learning model for the subset of elements; (Chen Paragraph 0026; “an accuracy of a trained machine learning model may be computed based on production data. The accuracy of the machine learning model may indicate how well the trained machine learning is able to predict results for production data... However, the accuracy of the trained machine learning model may be determined using the production data, which is obtained from a data feed (e.g., data feed 140), and which may only be available for a limited amount of time (e.g., while in the data stream).” Examiner notes that the machine learning model is evaluated based on the feedback on the accuracy of the result of execution of the machine learning model (The accuracy of the machine learning model may indicate how well the trained machine learning is able to predict results for production data) for the subset of elements (production data))
and responsive to the evaluation of the machine learning model indicating that the performance of the machine learning model is below a threshold value, retraining the machine learning model. (Chen Paragraph 0029; "the threshold accuracy score may be determined based on an accuracy score previously determined for the machine learning model during the training process. If it is determined that the machine learning model satisfies the threshold accuracy condition, then a notification may be generated indicating that the training data used to train the machine learning model is to be updated…the updated training data may be used to re-train the machine learning model" Examiner notes that responsive to the evaluation of the machine learning model indicating the performance of the machine learning model is below a threshold value (if it is determined that the machine learning model satisfies the threshold accuracy condition), retraining the machine learning model (re-train the machine learning model))
Chen does not teach repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset;
However, Hathaway does teach repeatedly performing: identifying an element from the production dataset that maximizes a measure of minimum distance of the element of the production dataset from elements of the review dataset, and adding the identified element to the review dataset; (Hathaway Page 16 Paragraph 2; "MMI: Maximin Initialization Algorithm Choose The number of clusters c, 1 < c < n; and, Rs x Rs if the available data are object data X, an inner product induced metric d(⋅ , ⋅ ) on. Input Object data X = {x1,...,xn} ⊂ Rs or dissimilarity data D = [dij] ⊂ Rnxn Step 1 Select the indices m1, ..., mc of the c distinguished objects. Select m1 = 1 (Remark: this is the "object seed";m1 = 1 is arbitrary) (If the data are X, calculate dissimilarities { d1,k }, k = 1 to n.) Initialize the minimum distance array… Step 2 Cluster each object in {o1, ..., on} with its nearest distinguished object. Clear the initialization matrix array" Examiner notes that the algorithm shows repeatedly performing/for loop; identifying an element that maximizes a measure of minimum distance; input objects are elements from the production dataset; clustering object is adding the element to the review dataset.)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen and Hathaway. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. One of ordinary skill would have motivation to combine Chen and Hathaway to identify an element that maximizes a measure of minimum distance in an inexpensive manner. “MMI systematically identifies objects that are distributed throughout the data, and uses the identified objects to inexpensively generate an initial partition of the entire data set. We will prove that the MMI partition is exact if the data set consists of compact and separated clusters in the sense of Dunn” (Hathaway Page 14 Paragraph 1).
Chen in view of Hathaway does not teach prioritizing elements of the review dataset for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment,
the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review;
However, Jorasch does teach prioritizing elements of the review dataset [for obtaining feedback on accuracy of results of execution of the machine learning model in the production environment,] (Jorasch Paragraph 2611; "The producer software could display a dynamic queue for questions and answers, showing the order in which individuals ask questions, the priority or importance of their questions, or an ordering created by the meeting owner." Examiner notes that the elements of the review dataset (questions) is prioritized based on an order in which the sample was added using a queue;)
the elements prioritized based on the order in which the elements were added to the review dataset during sampling such that a first element sampled before a second element is given higher priority during review; (Examiner references previous mapping to note the queue prioritizes elements in a first in first out manner meaning first element has higher priority)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, and Jorasch. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. One of ordinary skill would have motivation to combine Chen, Hathaway, and Jorasch to use a queue to show the level of importance and help with organization. “Each queue could also provide visibility to the grouping by role, organization, or level of interest/rating. This reporting could allow improved organization of material and improvement of content delivery and post meeting follow-up.” (Jorasch Paragraph 3172).
Regarding claim 23, Chen teaches The computer-implemented method of claim 1, wherein evaluating the machine learning model is for different types of inputs and evaluation of the machine learning model indicates that the performance of the machine learning model is below threshold for a particular type of input, wherein retraining the machine learning model is performed using training data based on the particular type of input. (Chen Paragraph 0092; “As different machine learning models take, as input, different parameters, one machine learning model performing poorly on the production data may indicate that that machine learning model needs to be rebuilt, retrained, or replaced because of the types of features included in the production data… If a performance score for machine learning model 410a is greater than a performance score for machine learning model 410b, such as by more than a threshold amount… If it is determined that a particular type of feature included in the production data causes problems, and that some machine learning models take, as input, that type of feature, then those machine learning models may be avoided or replaced with other machine learning models that do not take, as input, that type of feature.” Examiner notes that evaluation of the machine learning model is for different types of inputs (types of features) and evaluation of the machine learning model indicates that the performance of the machine learning model is below a threshold for a particular type of input (performance score of machine learning model 410b with respect to the second type of feature performs below the allowed performance threshold), wherein retraining the machine learning model is performed using training data based on the particular type of input (machine learning model is retrained with production data; based on particular type of input not performing well, production data avoids or replaces particular type of input))
Regarding claim 27, Chen teaches The non-transitory computer readable storage medium of claim 9, wherein evaluating the machine learning model is for different types of inputs and evaluation of the machine learning model indicates that the performance of the machine learning model is below threshold for a particular type of input, wherein retraining the machine learning model is performed using training data based on the particular type of input. (Chen Paragraph 0092; “As different machine learning models take, as input, different parameters, one machine learning model performing poorly on the production data may indicate that that machine learning model needs to be rebuilt, retrained, or replaced because of the types of features included in the production data… If a performance score for machine learning model 410a is greater than a performance score for machine learning model 410b, such as by more than a threshold amount… If it is determined that a particular type of feature included in the production data causes problems, and that some machine learning models take, as input, that type of feature, then those machine learning models may be avoided or replaced with other machine learning models that do not take, as input, that type of feature.” Examiner notes that evaluation of the machine learning model is for different types of inputs (types of features) and evaluation of the machine learning model indicates that the performance of the machine learning model is below a threshold for a particular type of input (performance score of machine learning model 410b with respect to the second type of feature performs below the allowed performance threshold), wherein retraining the machine learning model is performed using training data based on the particular type of input (machine learning model is retrained with production data; based on particular type of input not performing well, production data avoids or replaces particular type of input))
Claim(s) 4-5, 7, 12-13, 15, 19-20, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Chen; Xinyu et al; US 20220374274 A1 (hereinafter “Chen”) in view of Richard J. Hathaway; “Maximin Initialization for Cluster Analysis” (hereinafter “Hathaway”) in further view of Jorasch; James et al; US 20210399911 A1 (hereinafter “Jorasch”) in further view of Tang; Zheng et al; US 11816932 B1 (hereinafter “Zheng”)
Regarding claim 4, Chen does not teach The computer-implemented method of claim 1, wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset.
However, Zheng does teach The computer-implemented method of claim 1, wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset. (Zheng Paragraph 22; "The user-recognition system may utilize any type of processing techniques to generate the palm-feature data and may represent the palm of the user depicted in the image data using various types of data structures, such as feature vectors." Examiner notes that the palm feature data of the production/new recognition request and review/existing palm feature data dataset are represented as feature vectors);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 5, Chen does not teach The computer-implemented method of claim 1, wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.
However, Zheng does teach The computer-implemented method of claim 1, wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image. (Zheng Column 3 Line 63; "This palm-feature data may represent biometric characteristics or information that is potentially unique to the palm of the user, such as the pattern of creases in the user's palm, the pattern of veins of the user's palm, the geometry of one or more portions of the user's hand (e.g., finger sizes/shape, palm size/shape, etc.), and/or the like." Zheng Column 34 Line 16; "Differences between the image data and the previously stored images may be assessed. For example, differences in shape, color, relative proportions between features in the images, and so forth" Examiner notes that pattern of creases and veins is one or more local features; shape, color, relative proportions is global features.);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 7, Chen does not teach The computer-implemented method of claim 1, further comprising: comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model.
However, Zheng does teach The computer-implemented method of claim 1, further comprising: comparing information received in the user feedback with a result of execution of the machine learning model to evaluate the machine learning model. (Zheng Column 10 Line 13; " the user-recognition system may perform auditing processes in response to receiving user feedback… the system may perform auditing processes in response to a user being identified more or less than a threshold number of times within a certain amount of time, in response to a large transaction, in response to a transaction associated with a large number of items, in response to learning additional information regarding a user (e.g., that a user was not located at a city or state associated with a facility at which he or she was allegedly identified), or in response to occurrence of any other predefined event. After receiving user feedback (e.g., in the form of a user indicating that he or she objects to a transaction or a match determined by the system), the user-recognition system may perform a higher level of analysis to determine whether image data associated with the transaction was misidentified" Examiner notes that the system higher level of analysis is compared to the review (user feedback) to determine if the transaction was misidentified to audit/evaluate the machine learning model);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 12, Chen does not teach The non-transitory computer readable storage medium of claim 9, wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset.
However, Zheng does teach The non-transitory computer readable storage medium of claim 9, wherein the measure of minimum distance represents a minimum of values representing distances between a feature vector representing an element of the production dataset and a feature vector representing an element of the review dataset. (Zheng Paragraph 22; "The user-recognition system may utilize any type of processing techniques to generate the palm-feature data and may represent the palm of the user depicted in the image data using various types of data structures, such as feature vectors." Examiner notes that the palm feature data of the production/new recognition request and review/existing palm feature data dataset are represented as feature vectors);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 13, Chen does not teach The non-transitory computer readable storage medium of claim 9, wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.
However, Zheng does teach The non-transitory computer readable storage medium of claim 9, wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image. (Zheng Column 3 Line 63; "This palm-feature data may represent biometric characteristics or information that is potentially unique to the palm of the user, such as the pattern of creases in the user's palm, the pattern of veins of the user's palm, the geometry of one or more portions of the user's hand (e.g., finger sizes/shape, palm size/shape, etc.), and/or the like." Zheng Column 34 Line 16; "Differences between the image data and the previously stored images may be assessed. For example, differences in shape, color, relative proportions between features in the images, and so forth" Examiner notes that pattern of creases and veins is one or more local features; shape, color, relative proportions is global features.);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 15, Chen does not teach The non-transitory computer readable storage medium of claim 9, wherein the instructions further cause the computer processor for performs steps comprising: comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model.
However, Zheng does teach The non-transitory computer readable storage medium of claim 9, wherein the instructions further cause the computer processor for performs steps comprising: comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model. (Zheng Column 10 Line 13; " the user-recognition system may perform auditing processes in response to receiving user feedback… the system may perform auditing processes in response to a user being identified more or less than a threshold number of times within a certain amount of time, in response to a large transaction, in response to a transaction associated with a large number of items, in response to learning additional information regarding a user (e.g., that a user was not located at a city or state associated with a facility at which he or she was allegedly identified), or in response to occurrence of any other predefined event. After receiving user feedback (e.g., in the form of a user indicating that he or she objects to a transaction or a match determined by the system), the user-recognition system may perform a higher level of analysis to determine whether image data associated with the transaction was misidentified" Examiner notes that the system higher level of analysis is compared to the review (user feedback) to determine if the transaction was misidentified to audit/evaluate the machine learning model);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 19, Chen does not teach The computer system of claim 17, wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image.
However, Zheng does teach The computer system of claim 17, wherein each element of the production dataset is an image represented as a feature vector, wherein the feature vector includes:(1) one or more global features describing the image, and (2) one or more local features describing a portion of the image. (Zheng Column 3 Line 63; "This palm-feature data may represent biometric characteristics or information that is potentially unique to the palm of the user, such as the pattern of creases in the user's palm, the pattern of veins of the user's palm, the geometry of one or more portions of the user's hand (e.g., finger sizes/shape, palm size/shape, etc.), and/or the like." Zheng Column 34 Line 16; "Differences between the image data and the previously stored images may be assessed. For example, differences in shape, color, relative proportions between features in the images, and so forth" Examiner notes that pattern of creases and veins is one or more local features; shape, color, relative proportions is global features.);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 20, Chen does not teach The computer system of claim 17, wherein the instructions further cause the one or more computer processors to perform steps comprising: comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model;
However, Zheng does teach The computer system of claim 17, wherein the instructions further cause the one or more computer processors to perform steps comprising: comparing information received in the review with a result of execution of the machine learning model to evaluate the machine learning model; (Zheng Column 10 Line 13; " the user-recognition system may perform auditing processes in response to receiving user feedback… the system may perform auditing processes in response to a user being identified more or less than a threshold number of times within a certain amount of time, in response to a large transaction, in response to a transaction associated with a large number of items, in response to learning additional information regarding a user (e.g., that a user was not located at a city or state associated with a facility at which he or she was allegedly identified), or in response to occurrence of any other predefined event. After receiving user feedback (e.g., in the form of a user indicating that he or she objects to a transaction or a match determined by the system), the user-recognition system may perform a higher level of analysis to determine whether image data associated with the transaction was misidentified" Examiner notes that the system higher level of analysis is compared to the review (user feedback) to determine if the transaction was misidentified to audit/evaluate the machine learning model);
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Regarding claim 24, Chen does not teach The computer-implemented method of claim 1, wherein review of an element comprises: sending elements from the subset of elements dataset for presentation via a user interface, the user interface configured to present a result of execution of the machine learning model for each element and receive user feedback indicating accuracy of the result of execution of the machine learning model.
However, Zheng does teach The computer-implemented method of claim 1, wherein review of an element comprises: sending elements from the subset of elements dataset for presentation via a user interface, the user interface configured to present a result of execution of the machine learning model for each element and receive user feedback indicating accuracy of the result of execution of the machine learning model. (Zheng Column 10 Line 14; "the user-recognition system may perform auditing processes in response to receiving user feedback, such as in response to a user indicating that he or she objects to a transaction or a match determined by the system." Examiner notes that the transaction information must be shown to the user for the user to send feedback via interface; the notification of the transaction is a result of execution of the machine learning model/user-recognition system for each element of the review dataset to find a match among all the user profiles; user sending a feedback that objects to a transaction is indicating an accuracy of the result.)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Zheng. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Zheng teaches training a user-recognition system with image data, updating identification data, and notifying user of access attempts. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Zheng to monitor execution of a machine learning model that utilizes image data and update it when needed. “it may be desirable to update identification data over time to maintain or increase a level of accuracy of an identification system.” (Zheng Column 1 Line 16).
Claim(s) 21-22, and 25-26 are rejected under 35 U.S.C. 103 as being unpatentable over Chen; Xinyu et al; US 20220374274 A1 (hereinafter “Chen”) in view of Richard J. Hathaway; “Maximin Initialization for Cluster Analysis” (hereinafter “Hathaway”) in further view of Jorasch; James et al; US 20210399911 A1 (hereinafter “Jorasch”) in view of Lu Sun et al; “Comparing Experts and Novices for AI Data Work: Insights on Allocating Human Intelligence to Design a Conversational Agent” (hereinafter “Lu”).
Regarding claim 21, Chen does not teach The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element based on the order in which the element was sampled.
However, Jorasch does teach [The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element] based on the order in which the element was sampled. (Jorasch Paragraph 2611; "The producer software could display a dynamic queue for questions and answers, showing the order in which individuals ask questions, the priority or importance of their questions, or an ordering created by the meeting owner." Examiner notes that the elements (questions) are organized based on the order in which they were sampled using a queue)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, and Jorasch. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. One of ordinary skill would have motivation to combine Chen, Hathaway, and Jorasch to use a queue to show the level of importance and help with organization. “Each queue could also provide visibility to the grouping by role, organization, or level of interest/rating. This reporting could allow improved organization of material and improvement of content delivery and post meeting follow-up.” (Jorasch Paragraph 3172).
Chen in view of Jorasch does not teach The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element [based on the order in which the element was sampled.]
However, Lu does teach The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element [based on the order in which the element was sampled.] (Lu Page 203 Paragraph 5; “AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” Examiner notes that a user (participant) is selected for reviewing the result of execution of the machine learning model for a particular element (data generation tasks and low level data curation tasks))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Lu. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Lu teaches utilizing experts and novices for AI data work. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Lu to better utilized the personnel cost, expertise, and motivation to efficiently complete data work “a better approach could be to build a hybrid pipeline that mixes the value of both on data work. AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” (Lu Page 203 Paragraph 6).
Regarding claim 22, Chen does not teach The computer-implemented method of claim 1, wherein selecting the user for reviewing the result of execution of the machine learning model comprises, sending elements sampled earlier during the sampling to a user with higher experience and sending elements sampled later during the sampling to a user with less experience.
However, Lu does teach The computer-implemented method of claim 1, wherein selecting the user for reviewing the result of execution of the machine learning model comprises, sending elements sampled earlier during the sampling to a user with higher experience and sending elements sampled later during the sampling to a user with less experience. (Lu Page 203 Paragraph 5; “AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” Examiner notes that elements sampled earlier during the sampling (data generation tasks) is sent to a user with higher experience (experts) and elements sampled later during the sampling (low level data curation tasks) is sent to a user with less experience (crowd workers))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Lu. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Lu teaches utilizing experts and novices for AI data work. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Lu to better utilized the personnel cost, expertise, and motivation to efficiently complete data work “a better approach could be to build a hybrid pipeline that mixes the value of both on data work. AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” (Lu Page 203 Paragraph 6).
Regarding claim 25, Chen does not teach The non-transitory computer readable storage medium of claim 9, wherein the instructions further cause the one or more processors to perform steps comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element based on the order in which the element was sampled.
However, Jorasch does teach [The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element] based on the order in which the element was sampled. (Jorasch Paragraph 2611; "The producer software could display a dynamic queue for questions and answers, showing the order in which individuals ask questions, the priority or importance of their questions, or an ordering created by the meeting owner." Examiner notes that the elements (questions) are organized based on the order in which they were sampled using a queue)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, and Jorasch. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. One of ordinary skill would have motivation to combine Chen, Hathaway, and Jorasch to use a queue to show the level of importance and help with organization. “Each queue could also provide visibility to the grouping by role, organization, or level of interest/rating. This reporting could allow improved organization of material and improvement of content delivery and post meeting follow-up.” (Jorasch Paragraph 3172).
Chen in view of Jorasch does not teach The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element [based on the order in which the element was sampled.]
However, Lu does teach The computer-implemented method of claim 1, further comprising: selecting a user for reviewing the result of execution of the machine learning model for a particular element [based on the order in which the element was sampled.] (Lu Page 203 Paragraph 5; “AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” Examiner notes that a user (participant) is selected for reviewing the result of execution of the machine learning model for a particular element (data generation tasks and low level data curation tasks))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Lu. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Lu teaches utilizing experts and novices for AI data work. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Lu to better utilized the personnel cost, expertise, and motivation to efficiently complete data work “a better approach could be to build a hybrid pipeline that mixes the value of both on data work. AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” (Lu Page 203 Paragraph 6).
Regarding claim 26, Chen does not teach The non-transitory computer readable storage medium of claim 9, wherein selecting the user for reviewing the result of execution of the machine learning model comprises, sending elements sampled earlier during the sampling to a user with higher experience and sending elements sampled later during the sampling to a user with less experience.
However, Lu does teach The non-transitory computer readable storage medium of claim 9, wherein selecting the user for reviewing the result of execution of the machine learning model comprises, sending elements sampled earlier during the sampling to a user with higher experience and sending elements sampled later during the sampling to a user with less experience. (Lu Page 203 Paragraph 5; “AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” Examiner notes that elements sampled earlier during the sampling (data generation tasks) is sent to a user with higher experience (experts) and elements sampled later during the sampling (low level data curation tasks) is sent to a user with less experience (crowd workers))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Hathaway, Jorasch, and Lu. Chen teaches a pipeline for monitoring for performance of a machine learning model. Hathaway teaches a Maximin Initialization Algorithm. Jorasch teaches prioritizing data elements using a queue. Lu teaches utilizing experts and novices for AI data work. One of ordinary skill would have motivation to combine Chen, Hathaway, Jorasch, and Lu to better utilized the personnel cost, expertise, and motivation to efficiently complete data work “a better approach could be to build a hybrid pipeline that mixes the value of both on data work. AI developers should allocate data work judiciously according to participants’ cost, expertise and motivation. For example, experts could be directed to complete data generation tasks, while crowd workers could be directed to focus on low level data curation tasks that experts find tedious.” (Lu Page 203 Paragraph 6).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL DUC TRAN whose telephone number is (571)272-6870. The examiner can normally be reached Mon-Fri 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.D.T./Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147