Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3 September 2025 has been entered.
Claim Rejections - 35 USC § 101
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-18 is/are rejected under 35 U.S.C. 101 because they are directed to abstract ideas without significantly more.
Regarding claims 1-18,
Step 1: With respect to claims 1-6, applying step 1, the preamble of claims 1-6 claims a system, which falls within the statutory category of an apparatus. With respect to claim 7-12, applying step 1, the preamble of claims 7-12 claims a method, which falls within the statutory category of a process. With respect to claim 13-18, applying step 1, the preamble of claims 13-18 claims a non-transitory computer readable medium, which falls within the statutory category of a manufacture.
Regarding claim 1,
Step 2A – Prong One: Claim 1 recites an abstract idea. The limitation of “constructing a mapping graph based on activations inside the reference learning machine in response to the set of labeled data;” is an abstract idea directed to mathematical concepts, for example, the process of constructing a graph based on neuron activation values in response to a set of labeled data inside a reference learning machine involves determining activations inside the learning machine in response to the set of data, which is a mathematical calculation, to create a mapping graph. Additionally, the limitation of “rank a plurality of samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine;” is an abstract idea directed to mathematical concepts, for example, the process of ranking a plurality of samples of the selected set by their potential performance improvement involves generating ranking values representative of a calculation made by the data analyzer 204, where by using mathematical inequalities (greater than, less than) to rank the plurality of samples based on their measured performance improvement, which are mathematical calculations ranked by mathematical relationships. The limitation of “determine a certainty for the plurality of samples of the selected set of unlabeled data; rank the samples of the selected set of unlabeled data based on the determined certainty;” is an abstract idea directed to mathematical concepts, for example, the process of determining certainties for a plurality of samples and then ranking the samples of unlabeled data based on the determined certainty involves a mathematical calculation of the certainty of the sample by the data analyzer, and then using mathematical inequalities to rank the samples based on their numerical certainties, which are mathematical calculations ranked by mathematical relationships.
Step 2A – Prong Two: Claim 1 does not contain any additional elements that would integrate the abstract idea into practical application. The additional element of “a reference learning machine; a set of labeled data; and a computing device comprising a learning machine analyzer configured to: receive the reference learning machine and the set of labeled data as input data samples, and analyze an inner working of the reference learning machine to produce a selected set of unlabeled data by: passing the set of labeled data into the reference learning machine;” is insignificant extra-solution activity that amounts to no more than mere data gathering (See MPEP 2106.05(g)). The additional element of “and retrain the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Step 2B: Claim 1 does not contain any additional elements that would amount to significantly more than the abstract idea. The additional element of “a reference learning machine; a set of labeled data; and a computing device comprising a learning machine analyzer configured to: receive the reference learning machine and the set of labeled data as input data samples, and analyze an inner working of the reference learning machine to produce a selected set of unlabeled data by: passing the set of labeled data into the reference learning machine;” is insignificant extra-solution activity that amounts to no more than mere data gathering (See MPEP 2106.05(g)). Data gathering is considered to be a well-understood, routine conventional activity, as recognized by courts (See MPEP 2106.05(d)(II)). Adding a preliminary step of mere data gathering to a process that only recites selecting a set of unlabeled data does not impose any meaningful limitations to the process of selecting unlabeled data. The additional element of “and retrain the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Regarding claim 7,
Step 2A – Prong One: Claim 7 recites an abstract idea. The limitation of “constructing a mapping graph based on activations inside the reference learning machine in response to the set of labeled data;” is an abstract idea directed to mathematical concepts, for example, the process of constructing a graph based on neuron activation values in response to a set of labeled data inside a reference learning machine involves determining activations inside the learning machine in response to the set of data, which is a mathematical calculation, to create a mapping graph. Additionally, the limitation of “determining a certainty for a plurality of samples of the selected set of unlabeled data; ranking the samples of the selected set of unlabeled data based on the determined certainty;” is an abstract idea directed to mathematical concepts, for example, the process of determining certainties for a plurality of samples and then ranking the samples of unlabeled data based on the determined certainty involves a mathematical calculation of the certainty of the sample by the data analyzer, and then using mathematical inequalities to rank the samples based on their numerical certainties, which are mathematical calculations ranked by mathematical relationships. The limitation of “ranking the samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine;” is an abstract idea directed to mathematical concepts, for example, the process of ranking the samples of the selected set by their potential performance improvement involves generating ranking values representative of a calculation made by the data analyzer 204, where by using mathematical inequalities (greater than, less than) to rank the plurality of samples based on their measured performance improvement, which are mathematical calculations ranked by mathematical relationships.
Step 2A – Prong Two: Claim 7 does not contain any additional elements that would integrate the abstract idea into practical application. The additional element of “receiving a reference learning machine; receiving a set of labeled data as input data samples; analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data, wherein analyzing the inner working of the reference learning machine comprises: passing the set of labeled data into the reference learning machine;” is insignificant extra-solution activity that amounts to no more than mere data gathering (See MPEP 2106.05(g)). The additional element of “and retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty” are mere instructions to apply the abstract ideas on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Step 2B: Claim 7 does not contain any additional elements that would amount to significantly more than the abstract idea. The additional element of “receiving a reference learning machine; receiving a set of labeled data as input data samples; and analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data, wherein analyzing the inner working of the reference learning machine comprises: passing the set of labeled data into the reference learning machine;” is insignificant extra-solution activity that amounts to no more than mere data gathering (See MPEP 2106.05(g)). Data gathering is considered to be a well-understood, routine conventional activity, as recognized by courts (See MPEP 2106.05(d)(II)). Adding a preliminary step of mere data gathering to a process that only recites selecting a set of unlabeled data does not impose any meaningful limitations to the process of selecting unlabeled data. The additional element of “and retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Regarding claim 13,
Step 2A – Prong One: Claim 13 recites an abstract idea. The limitation of “ranking a plurality of samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine;” is an abstract idea directed to mathematical concepts, for example, the process of ranking a plurality of samples of the selected set by their potential performance improvement involves generating ranking values representative of a calculation made by the data analyzer 204, where by using mathematical inequalities (greater than, less than) to rank the plurality of samples based on their measured performance improvement, which are mathematical calculations ranked by mathematical relationships.. The limitation of “determining a certainty for the plurality of samples of the selected set of unlabeled data; ranking the samples of the selected set of unlabeled data based on the determined certainty;” is an abstract idea directed to mathematical concepts, for example, the process of determining certainties for a plurality of samples and then ranking the samples of unlabeled data based on the determined certainty involves a mathematical calculation of the certainty of the sample by the data analyzer, and then using mathematical inequalities to rank the samples based on their numerical certainties, which are mathematical calculations ranked by mathematical relationships..
Step 2A – Prong Two: Claim 13 does not contain any additional elements that would integrate the abstract idea into practical application. The additional element of “receiving a reference learning machine; receiving a set of labeled data as input data samples; analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data, the analyzing the inner working comprising: passing the set of labeled data into the reference learning machine; and extracting activation values from processing layers in the reference learning machine in response to the set of labeled data;” is insignificant extra-solution activity that amounts to no more than mere data gathering (See MPEP 2106.05(g)). The additional element of “and retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Step 2B: Claim 13 does not contain any additional elements that would amount to significantly more than the abstract idea. The additional element of “receiving a reference learning machine; receiving a set of labeled data as input data samples; analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data, the analyzing the inner working comprising: passing the set of labeled data into the reference learning machine; and extracting activation values from processing layers in the reference learning machine in response to the set of labeled data;” is insignificant extra-solution activity that amounts to no more than mere data gathering (See MPEP 2106.05(g)). Data gathering is considered to be a well-understood, routine conventional activity, as recognized by courts (See MPEP 2106.05(d)(11)). Adding a preliminary step of mere data gathering to a process that only recites selecting a set of unlabeled data does not impose any meaningful limitations to the process of selecting unlabeled data. The additional element of “and retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Regarding claim 2,
Step 2A – Prong One: Claim 2 recites an abstract idea. The limitation of “identifies and measures a relation between different input data samples of the set of labeled data and finds pairwise relations to construct a relational graph” is an abstract idea directed to mental processes. For example, a human could use evaluation and observation to identify and measure a relation between different input data samples and find pairwise relations to construct a relational graph.
Step 2A – Prong Two: Claim 2 does not contain any additional elements that would integrate the abstract idea into practical application. The additional element of “the learning machine analyzer” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Step 2B: Claim 2 does not contain any additional elements that would amount to significantly more than the abstract idea. The additional element of “the learning machine analyzer” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Regarding claim 3,
Step 2A – Prong Two: Claim 3 does not contain any additional elements that would integrate the abstract idea into practical application. The additional element of “wherein the relational graph provides a visualization of how much the different input data samples are similar to each other in higher dimensions inside the reference learning machine” is insignificant extra-solution activity that amounts to mere data gathering (See MPEP 2106.05(g)). Adding a step of outputting the generated data does not integrate the abstract idea into practical application.
Step 2B: Claim 3 does not contain any additional elements that would amount to significantly more than the abstract idea. The additional element of “wherein the relational graph provides a visualization of how much the different input data samples are similar to each other in higher dimensions inside the reference learning machine” is insignificant extra-solution activity that amounts to mere data gathering (See MPEP 2106.05(g)). Data gathering is a well-understood, routine conventional activity as recognized by the courts (See MPEP 2106.05(d)(II)). Adding a step of outputting the generated data imposes no meaningful limitations on the claim.
Regarding claim 4,
Step 2A – Prong One: Claim 4 recites an abstract idea. The limitation of “one or more activation vectors extracted from the reference learning machine are processed and project to a second vector which is designed to highlight similarities between the input data samples” is directed to the mental process of human evaluation and judgement, for example, a human could process and project a first activation vector to a second vector designed to highlight similarities between samples with or without the physical aid of a pen and paper.
Step 2A – Prong Two: Claim 4 does not contain any additional elements that would integrate the abstract idea into practical application. The additional element of “the reference learning machine” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Step 2B: Claim 4 does not contain any additional elements that would amount to significantly more than the abstract idea. The additional element of “the reference learning machine” are mere instructions to apply the abstract idea on a generic computer (See MPEP 2106.05(f)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Regarding claim 5,
Step 2A – Prong One: Claim 5 recites an abstract idea. The limitation of “the second vector has a much lower dimension compared to the one or more first activation vectors” is an abstract idea directed to mental processes, for example, a human could use evaluation and judgement to determine that the second vector has a much lower dimension when compared to the one or more first activation vectors.
Step 2A – Prong Two: Claim 5 does not contain any additional elements that would integrate the abstract idea into practical application.
Step 2B: Claim 5 does not contain any additional elements that would amount to significantly more than the abstract idea.
Regarding claim 6,
Step 2A – Prong One: Claim 6 recites an abstract idea. The limitation of “automatically annotate the selected set of unlabeled data” is a mental process of human evaluation and judgement. As stated in the specification of the application, [0036] - “A human user may be asked to annotate the selected portion of unlabeled images.”
Step 2A – Prong Two: Claim 6 does not contain any additional elements that would integrate the abstract idea into practical application. The additional element of “a data annotator” is a mere attempt to generally link the abstract idea to the technological environment of a computer (See MPEP 2106.05(h)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Step 2B: Claim 6 does not contain any additional elements that would amount to significantly more than the abstract idea. The additional element of “a data annotator” is a mere attempt to generally link the abstract idea to the technological environment of a computer (See MPEP 2106.05(h)). The computer is recited at a high level of generality and imposes no meaningful limitations on the claim.
Claims 8-12 and 14-18 incorporate substantively all the limitations of claims 2-6 in a method and computer-readable medium and are rejected under the same rationale.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-4, 7-10, and 13-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US Pub. No. 2018/0285771, hereinafter “Lee”) in view of Zhao M. et al. (NPL: “Automatic Image Annotation via Compact Graph-based Semi-supervised Learning”, hereinafter “Zhao M.”) further in view of Bremer et al. (US Pub. No. 2021/0042330, hereinafter “Bremer”).
Regarding claim 1, Lee teaches a system for selecting unlabeled data for building and improving performance of a learning machine, comprising:
a reference learning machine (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — teaches a reference learning machine (classifier 106));
a set of labeled data (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — teaches a set of labeled data (labeled data 100)); and
a computing device comprising a learning machine analyzer configured to: receive the reference learning machine and the set of labeled data as input data samples (Lee, [0024] — “The labeled data 100, the unlabeled data 102, and the classifier 106 are processed by the semi-supervised learning module 108” — teaches a learning machine analyzer configured to receive the reference learning machine and set of labeled data as input (the semi-supervised learning module 108)), and
analyze an inner working of the reference learning machine to produce a selected set of unlabeled data by: passing the set of labeled data into the reference learning machine (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data by passing the set of labeled data into the reference learning machine); and
based on activations inside the reference learning machine in response to the set of labeled data (Lee, [0046] — “In one embodiment of this invention, the input classifier is a random forest. The random forest consists of a plurality of classification trees, each capable of classifying the unlabeled data 102. The label consensus criteria application module 606 is performed by tallying the classification result of each individual classification tree for an object to be classified.” — teaches using activations (individual classification trees of a random forest) inside the reference learning machine to select or classify unlabeled data in response to the set of labeled data, as in [0043] — “The high confidence data 134 and labeled data 100 is then used to train an updated classifier in the form of a new random forest. This process can repeat itself by replacing the input classifier 500 with the updated classifier 110 until no new high confidence data 134 is produced.” — teaches determining activations inside the reference learning machine in response to the set of labeled data);
determine a certainty for the plurality of samples of the selected set of unlabeled data (Lee, [0047] — “The classification estimation module 702 processes the unlabeled data 102 and the input classifier S00 to produce an estimated classification probability 704. The classification probability criteria application module 706 applies a rule using the estimated classification probability 704 to select unlabeled data 102 as high confidence data 134.” — teaches determining a certainty (confidence score) for a plurality of samples of the selected set of unlabeled data);
Lee fails to explicitly teach constructing a mapping graph, ranking a plurality of samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine; ranking the samples of the selected set of unlabeled data based on the determined certainty; and retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty.
However, analogous to the field of unlabeled data selection and annotation, Zhao M. teaches:
constructing a mapping graph (Zhao M., Figures 1-5 — shows mapping graphs providing visualizations of how much the different samples are similar to each other, and uses these graphs for label propagation)
rank the samples of the selected set of unlabeled data based on the determined certainty (Zhao M., Section 4.6 Paragraph 3 — “We next design an automatic feedback strategy to model the retrieval process. For each query image submitted by the user, the system retrieves and ranks the images in the database. Here, the rank for each image in the database is based on the estimated label information after performing the proposed CGSSL or other state-of-the-art methods. The top images with the highest ranking score are then selected as the feedback images, and their feedback information can be used for re-ranking.” — teaches ranking a plurality of samples on the selected set of unlabeled data by a ranking score (the ranking score being based on the estimated label information, thus based on the certainty of the label)); and
retrain the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for determined certainty (Zhao M., Section 4.6 Paragraph 4- “Then, by taking both the labeled and unlabeled set as inputs, the proposed CGSSL can automatically annotate the labels of the unlabeled set, and the top images (maybe 10, 20, or more images) with the highest values of estimated labels are selected as the feedback images. Next, the users can annotate such feedback images as relevant or irrelevant to the query image. In other words, if a feedback image is judged as relevant, it will be added to the labeled set, making the number of the labeled set increase. The new-formed labeled set combined with the remaining unlabeled set can then be used as inputs for a new-round annotation. The process will be iteratively performed several times until the user’s requirements are satisfied.” — teaches retraining the reference learning machine based on the ranking of the samples of the selected set, where the ranking of the samples is based on the determined certainty).
Therefore, it would have been obvious to a person of ordinary sill in the art, before the effective filing date of the claimed invention, to incorporate the mapping graph, ranking of samples, and retraining of Zhao M. to the reference machine, inner working analysis, and determination of certainty of Lee in order to use a reference learning machine with labeled data to construct graphs used to guide the labeling or selection of unlabeled data. Doing so would capture the manifold structures of datasets (Zhao M., Introduction) and to capture the underlying rules from the labeled data within the classifier to predict the labels from the features of the objects in the unlabeled data (Lee, [0040]).
The combination of Lee and Zhao M. fails to explicitly teach retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty.
However, analogous to the field of the claimed invention, Bremer teaches:
rank a plurality of samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine (Bremer, [0050] – “The selection may for example be performed using a sampling function. Using the sampling function, the current set of unlabeled data points may be ranked. The intermediate subset of unlabeled data points comprises first X ranked data points. The ranking may be performed according to the importance and how beneficial the data points will be to the current trained learning model to increase its accuracy.” – teaches ranking a plurality of samples of the selected set of unlabeled data (current set of unlabeled data points may be ranked) by potential performance improvement of each of the samples to the reference learning machine (according to importance and how beneficial the points will be to trained model to increase accuracy));
retrain the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty (Bremer, [0058] - “These embodiments may enable a multi-layer selection process in which different types of selections are combined in order to obtain a most valuable subset of records for further training the machine learning model. Each of the three selections is based on a different criterion” and in [0050] – “The selection may for example be performed using a sampling function. Using the sampling function, the current set of unlabeled data points may be ranked. The intermediate subset of unlabeled data points comprises first X ranked data points. The ranking may be performed according to the importance and how beneficial the data points will be to the current trained learning model to increase its accuracy” – teaches ranking of the samples of the selected set of unlabeled data for potential performance improvement and other criterion, and in [0089] – “Thus, using steps 203 and 205, further labeled data points are obtained by the present method. This may enable to increase or enhance the current set of labeled data points. For example, the subset of labeled data points may be added to the current set of labeled data points to form the current set of labeled data points for a next iteration of the present method. The current set of labeled data points may be used to repeat steps 201-205. The trained machine learning model that results from a last iteration may be used, in each current iteration of steps 201-205” – teaches retraining the reference learning machine based on the ranking of the samples (repeats training steps 201-205 to repeat training of machine learning model based on the selected data, wherein the selected data was ranked for both potential improvement and other criterion))
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the ranking by potential performance improvement and retraining based on multiple rankings of Bremer to further modify the determination and ranking based on certainty and retraining based on the ranked samples for determined certainty of Lee and Zhao M. in order to rank the samples based on potential performance improvement and determined certainty and to retrain the reference machine learning model based on the ranked samples for both criterion. Doing so would obtain the most valuable subsets of records for training the machine learning model (Bremer, [0058]), resulting in improved training processes and increased accuracy of predictions made by the model (Bremer, [0060]).
Regarding claim 7, Lee teaches a method for selecting unlabeled data for building and improving performance of a learning machine, the method comprising:
receiving a reference learning machine (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — teaches a reference learning machine (classifier 106) and teaches receiving the reference learning machine in [0024] — “The labeled data 100, the unlabeled data 102, and the classifier 106 are processed by the semi-supervised learning module 108”);
receiving a set of labeled data as input data samples (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — teaches labeled data (labeled data 100) and teaches receiving the labeled data in [0024] — “The labeled data 100, the unlabeled data 102, and the classifier 106 are processed by the semi-supervised learning module 108”); and
analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data, wherein analyzing the inner working of the reference learning machine comprises: passing the set of labeled data into the reference learning machine (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data by passing the set of labeled data into the reference learning machine); and
based on activations inside the reference learning machine in response to the set of labeled data (Lee, [0046] — “In one embodiment of this invention, the input classifier is a random forest. The random forest consists of a plurality of classification trees, each capable of classifying the unlabeled data 102. The label consensus criteria application module 606 is performed by tallying the classification result of each individual classification tree for an object to be classified.” — teaches using activations (individual classification trees of a random forest) inside the reference learning machine to select or classify unlabeled data in response to the set of labeled data, as in [0043] — “The high confidence data 134 and labeled data 100 is then used to train an updated classifier in the form of a new random forest. This process can repeat itself by replacing the input classifier 500 with the updated classifier 110 until no new high confidence data 134 is produced.” — teaches determining activations inside the reference learning machine in response to the set of labeled data);
determining a certainty for a plurality of samples of the selected set of unlabeled data (Lee, [0047] — “The classification estimation module 702 processes the unlabeled data 102 and the input classifier S00 to produce an estimated classification probability 704. The classification probability criteria application module 706 applies a rule using the estimated classification probability 704 to select unlabeled data 102 as high confidence data 134.” — teaches determining a certainty (confidence score) for a plurality of samples of the selected set of unlabeled data);
Lee fails to explicitly teach constructing a mapping graph, ranking the samples of the selected set of unlabeled data based on the determined certainty; ranking the samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine; and retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty.
However, analogous to the field of selecting unlabeled data and annotation, Zhao M. teaches:
constructing a mapping graph (Zhao M., Figures 1-5 — shows mapping graphs providing visualizations of how much the different samples are similar to each other, and uses these graphs for label propagation)
ranking the samples of the selected set of unlabeled data based on the determined certainty (Zhao M., Section 4.6 Paragraph 3 — “We next design an automatic feedback strategy to model the retrieval process. For each query image submitted by the user, the system retrieves and ranks the images in the database. Here, the rank for each image in the database is based on the estimated label information after performing the proposed CGSSL or other state-of-the-art methods. The top images with the highest ranking score are then selected as the feedback images, and their feedback information can be used for re-ranking.” — teaches ranking a plurality of samples on the selected set of unlabeled data by a ranking score (the ranking score being based on the estimated label information, thus based on the certainty of the label)); and
retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for determined certainty (Zhao M., Section 4.6 Paragraph 4- “Then, by taking both the labeled and unlabeled set as inputs, the proposed CGSSL can automatically annotate the labels of the unlabeled set, and the top images (maybe 10, 20, or more images) with the highest values of estimated labels are selected as the feedback images. Next, the users can annotate such feedback images as relevant or irrelevant to the query image. In other words, if a feedback image is judged as relevant, it will be added to the labeled set, making the number of the labeled set increase. The new-formed labeled set combined with the remaining unlabeled set can then be used as inputs for a new-round annotation. The process will be iteratively performed several times until the user’s requirements are satisfied.” — teaches retraining the reference learning machine based on the ranking of the samples of the selected set).
Therefore, it would have been obvious to a person of ordinary sill in the art, before the effective filing date of the claimed invention, to incorporate the mapping graph, ranking of samples, and retraining of Zhao M. to the reference machine, inner working analysis, and certainty determination of Lee in order to use a reference learning machine with labeled data to construct graphs used to guide the labeling or selection of unlabeled data, where the data may be ranked according to the certainty scores of Lee. Doing so would capture the manifold structures of datasets (Zhao M., Introduction) and to capture the underlying rules from the labeled data within the classifier to predict the labels from the features of the objects in the unlabeled data (Lee, [0040]).
The combination of Lee and Zhao M. fails to explicitly teach retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty.
However, analogous to the field of the claimed invention, Bremer teaches:
rank a plurality of samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine (Bremer, [0050] – “The selection may for example be performed using a sampling function. Using the sampling function, the current set of unlabeled data points may be ranked. The intermediate subset of unlabeled data points comprises first X ranked data points. The ranking may be performed according to the importance and how beneficial the data points will be to the current trained learning model to increase its accuracy.” – teaches ranking a plurality of samples of the selected set of unlabeled data (current set of unlabeled data points may be ranked) by potential performance improvement of each of the samples to the reference learning machine (according to importance and how beneficial the points will be to trained model to increase accuracy));
retrain the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty (Bremer, [0058] - “These embodiments may enable a multi-layer selection process in which different types of selections are combined in order to obtain a most valuable subset of records for further training the machine learning model. Each of the three selections is based on a different criterion” and in [0050] – “The selection may for example be performed using a sampling function. Using the sampling function, the current set of unlabeled data points may be ranked. The intermediate subset of unlabeled data points comprises first X ranked data points. The ranking may be performed according to the importance and how beneficial the data points will be to the current trained learning model to increase its accuracy” – teaches ranking of the samples of the selected set of unlabeled data for potential performance improvement and other criterion, and in [0089] – “Thus, using steps 203 and 205, further labeled data points are obtained by the present method. This may enable to increase or enhance the current set of labeled data points. For example, the subset of labeled data points may be added to the current set of labeled data points to form the current set of labeled data points for a next iteration of the present method. The current set of labeled data points may be used to repeat steps 201-205. The trained machine learning model that results from a last iteration may be used, in each current iteration of steps 201-205” – teaches retraining the reference learning machine based on the ranking of the samples (repeats training steps 201-205 to repeat training of machine learning model based on the selected data, wherein the selected data was ranked for both potential improvement and other criterion))
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the ranking by potential performance improvement and retraining based on multiple rankings of Bremer to further modify the determination and ranking based on certainty and retraining based on the ranked samples for determined certainty of Lee and Zhao M. in order to rank the samples based on potential performance improvement and determined certainty and to retrain the reference machine learning model based on the ranked samples for both criterion. Doing so would obtain the most valuable subsets of records for training the machine learning model (Bremer, [0058]), resulting in improved training processes and increased accuracy of predictions made by the model (Bremer, [0060]).
Regarding claim 13, Lee teaches a non-transitory computer-readable medium storing instructions, executable by a processor, the instructions comprising instructions for:
receiving a reference learning machine (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — teaches a reference learning machine (classifier 106) and teaches receiving the reference learning machine in [0024] — “The labeled data 100, the unlabeled data 102, and the classifier 106 are processed by the semi-supervised learning module 108”);
receiving a set of labeled data as input data samples (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — teaches labeled data (labeled data 100) and teaches receiving the labeled data in [0024] — “The labeled data 100, the unlabeled data 102, and the classifier 106 are processed by the semi-supervised learning module 108”); and
analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data, the analyzing the inner working comprising: passing the set of labeled data into the reference learning machine (Lee, [0024] — “The initial supervised learning module 104 executed by a computer program of a computerized machine learning tool processes the labeled data 100 to produce a classifier 106.” — analyzing an inner working of the reference learning machine to produce a selected set of unlabeled data by passing the set of labeled data into the reference learning machine); and
extracting activation values from processing layers in the reference learning machine in response to the set of labeled data (Lee, [0046] — “In one embodiment of this invention, the input classifier is a random forest. The random forest consists of a plurality of classification trees, each capable of classifying the unlabeled data 102. The label consensus criteria application module 606 is performed by tallying the classification result of each individual classification tree for an object to be classified.” — teaches extracting activations from processing layers (individual classification trees) inside the reference learning machine (random forest) to select or classify unlabeled data in response to the set of labeled data, as in [0043] — “The high confidence data 134 and labeled data 100 is then used to train an updated classifier in the form of a new random forest. This process can repeat itself by replacing the input classifier 500 with the updated classifier 110 until no new high confidence data 134 is produced.” — teaches determining activations inside the reference learning machine in response to the set of labeled data);
determining a certainty for the plurality of samples of the selected set of unlabeled data (Lee, [0047] — “The classification estimation module 702 processes the unlabeled data 102 and the input classifier S00 to produce an estimated classification probability 704. The classification probability criteria application module 706 applies a rule using the estimated classification probability 704 to select unlabeled data 102 as high confidence data 134.” — teaches determining a certainty (confidence score) for a plurality of samples of the selected set of unlabeled data);
Lee fails to explicitly teach ranking a plurality of samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine; ranking the samples of the selected set of unlabeled data based on the determined certainty; and retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty.
However, analogous to the field of selecting unlabeled data and annotation, Zhao M. teaches:
ranking the samples of the selected set of unlabeled data based on the determined certainty (Zhao M., Section 4.6 Paragraph 3 — “We next design an automatic feedback strategy to model the retrieval process. For each query image submitted by the user, the system retrieves and ranks the images in the database. Here, the rank for each image in the database is based on the estimated label information after performing the proposed CGSSL or other state-of-the-art methods. The top images with the highest ranking score are then selected as the feedback images, and their feedback information can be used for re-ranking.” — teaches ranking a plurality of samples on the selected set of unlabeled data by a ranking score (the ranking score being based on the estimated label information, thus based on the certainty of the label)); and
retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for determined certainty (Zhao M., Section 4.6 Paragraph 4- “Then, by taking both the labeled and unlabeled set as inputs, the proposed CGSSL can automatically annotate the labels of the unlabeled set, and the top images (maybe 10, 20, or more images) with the highest values of estimated labels are selected as the feedback images. Next, the users can annotate such feedback images as relevant or irrelevant to the query image. In other words, if a feedback image is judged as relevant, it will be added to the labeled set, making the number of the labeled set increase. The new-formed labeled set combined with the remaining unlabeled set can then be used as inputs for a new-round annotation. The process will be iteratively performed several times until the user’s requirements are satisfied.” — teaches retraining the reference learning machine based on the ranking of the samples of the selected set).
Therefore, it would have been obvious to a person of ordinary sill in the art, before the effective filing date of the claimed invention, to incorporate the mapping graph, ranking of samples, and retraining of Zhao M. to the reference machine, inner working analysis, and certainty determination of Lee in order to use a reference learning machine with labeled data to construct graphs used to guide the labeling or selection of unlabeled data. Doing so would capture the manifold structures of datasets (Zhao M., Introduction) and to capture the underlying rules from the labeled data within the classifier to predict the labels from the features of the objects in the unlabeled data (Lee, [0040]).
The combination of Lee and Zhao M. fails to explicitly teach retraining the reference learning machine based on the ranking of the samples of the selected set of unlabeled data for both potential performance improvement and determined certainty.
However, analogous to the field of the claimed invention, Bremer teaches:
rank a plurality of samples of the selected set of unlabeled data by potential performance improvement of each of the samples to the reference learning machine (Bremer, [0050] – “The selection may for example be performed using a sampling function. Using the sampling function, the current set of unlabeled data points may be ranked. The intermediate subset of unlabeled data points comprises first X ranked data points. The ranking may be performed according to the importance and how beneficial the data points will be to the current trained learning model to increase its accuracy.” – teaches ranking a plurality of samples of the selected set of unlabeled data (current set of unlabeled data points may be ranked) by potential performance improvement of each of the samples to the reference learning machine (accor