DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 are rejected under 35 U.S.C. 101 because the claimed invention are directed to abstract ideas without significantly more.
Regarding Claim 1:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
determining, by a processor, ranked instances based on the data instances and a machine learning model: - This limitation is directed to the abstract idea of a mental process, as the process of determining ranked instances is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III)).
determining a metric based on the ranked instances: - This limitation is directed to the abstract idea of a mental process, as the process of determining a metric is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
receiving data instances: - This limitation is directed to mere data gathering and outputting. The courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) have recognized mere data gathering and outputting as insignificant extra-solution activity (see MPEP 2106.05(g)(3)) and therefore fails to integrate the exception into a practical application.
and outputting an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the metric: - This limitation is directed to mere data gathering and outputting. The courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) have recognized mere data gathering and outputting as insignificant extra-solution activity (see MPEP 2106.05(g)(3)) and therefore fails to integrate the exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
receiving data instances: - This limitation is directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) (see MPEP 2106.05(d) II).
and outputting an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the metric: - This limitation is directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) (see MPEP 2106.05(d) II).
Regarding Claim 2:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
wherein determining the metric includes determining at least one selected from the group consisting of accuracy and recall: - This limitation is directed to the abstract idea of a mental process, as the process of determining the metric is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 3:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
wherein determining the metric includes generating ground truth ranked instances based on the ranked instances and ground truth instances, wherein the ground truth instances corresponds to error free data: - This limitation is directed to the abstract idea of a mental process, as the process of generating ground truth ranked instances is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 4:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
wherein determining the metric includes generating a first ranking curve based on the metric: - This limitation is directed to the abstract idea of a mental process, as the process of generating a curve is a thought process that can be performed in a human mind or by a human using a pen and paper (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
wherein determining the metric includes determining a first area under curve of the first ranking curve: - This limitation is directed to the abstract idea of a mental process, as the process of determining the area under curve is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 6:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
wherein determining the metric includes generating a second ranking curve based on the ground truth ranked instances: - This limitation is directed to the abstract idea of a mental process, as the process of generating a curve is a thought process that can be performed in a human mind or by a human using a pen and paper (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 7:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
wherein determining the metric includes determining a second area under curve of the second ranking curve: - This limitation is directed to the abstract idea of a mental process, as the process of determining the area under curve is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 8:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
wherein determining the metric includes determining a ranking index based on the first area under curve and the second area under curve, and wherein the one or more parameters of the machine learning model are adjusted based on at least one of the ranking index and the first area under curve: - This limitation is directed to the abstract idea of a mental process, as the process of determining the ranking index is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III)).
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding independent Claim 9, this claim is directed to a system and is rejected on the same basis as independent claim 1 since they are analogous claims.
Regarding dependent Claim 10, this claim is directed to a system and is rejected on the same basis as dependent claim 2 since they are analogous claims.
Regarding dependent Claim 11, this claim is directed to a system and is rejected on the same basis as dependent claim 3 since they are analogous claims.
Regarding dependent Claim 12, this claim is directed to a system and is rejected on the same basis as dependent claim 4 since they are analogous claims.
Regarding dependent Claim 13, this claim is directed to a system and is rejected on the same basis as dependent claim 5 since they are analogous claims.
Regarding dependent Claim 14, this claim is directed to a system and is rejected on the same basis as dependent claim 6 since they are analogous claims.
Regarding dependent Claim 15, this claim is directed to a system and is rejected on the same basis as dependent claim 7 since they are analogous claims.
Regarding dependent Claim 16, this claim is directed to a system and is rejected on the same basis as dependent claim 8 since they are analogous claims.
Regarding Claim 17:
Step 1 - Is the claim directed to a process, a machine, manufacture or composition of matter? - Yes, the claim is directed to a process.
Step 2A - Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon? - Yes, the claim recites the abstract ideas:
determine ranked instances based on the data instances and a machine learning model, determine ground truth ranked instances based on the data instances, ground truth data instances, and the machine learning model, wherein the ground truth data instances are free from errors: - This limitation is directed to the abstract idea of a mental process, as the process of determining ranked instances, ground truth ranked instances, ground truth data instances and the machine learning model is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III));
determine a first metric based on the ranked instances and a second metric based on the ground truth ranked instances: - This limitation is directed to the abstract idea of a mental process, as the process of determining the metrics is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III));
determine a ranking index based on a comparison of the first metric and the second metric: - This limitation is directed to the abstract idea of a mental process, as the process of determining the ranking index is a thought process that can be performed in a human mind by observing, evaluating and judging (concepts performed in the human mind (including an observation, evaluation, judgment, opinion) (see MPEP § 2106.04(a)(2), subsection III));
Step 2A - Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application? - No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
receiving data instances: - This limitation is directed to mere data gathering and outputting. The courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) have recognized mere data gathering and outputting as insignificant extra-solution activity (see MPEP 2106.05(g)(3)) and therefore fails to integrate the exception into a practical application.
and output an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the ranking index: - This limitation is directed to mere data gathering and outputting. The courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) have recognized mere data gathering and outputting as insignificant extra-solution activity (see MPEP 2106.05(g)(3)) and therefore fails to integrate the exception into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception? - No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements:
receiving data instances: - This limitation is directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) (see MPEP 2106.05(d) II);
and output an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the ranking index: - This limitation is directed to receiving or transmitting data over a network. The courts (as per Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) (see MPEP 2106.05(d) II).
Regarding dependent Claim 18, this claim is directed to a non-transitory computer readable medium. Claim 18 is the combination of claims 4 and 6, and is rejected on the same basis as dependent claims 4 and 6 since they are analogous claims.
Regarding dependent Claim 19, this claim is directed to a non-transitory computer readable medium. Claim 19 is the combination of claims 5 and 7, and is rejected on the same basis as dependent claims 5 and 7 since they are analogous claims.
Regarding dependent Claim 20, this claim is directed to a non-transitory computer readable medium and is rejected on the same basis as dependent claim 8 since they are analogous claims.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 9-11 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Revaud et al (US-11521072-B2 - hereinafter Revaud).
Referring to Claim 1, Revaud teaches:
receiving data instances (see Revaud at Column 17 Lines 28-34: “Obtaining the training images may include receiving the training images from a user dealing with the training phase of the CNN or from a memory location (e.g. a local or remote computing system, a local or remote database, a cloud storage or any other memory location known in the field) on which they are stored”. Examiner interprets receiving the training images to be equivalent as the claimed “receiving data instances” );
determining, by a processor, ranked instances based on the data instances and a machine learning model (see Revaud at Column 18 Lines 18-30: “At 1008, a ranking function R is applied to the training images of the batch to sort them based on their similarities. For example, each image of the batch is considered to be an initial query image, and the remaining images of the batch are ranked (or sorted) according to their similarities to the query image. The images of the batch can be ranked by decreasing or by increasing similarity. An example of the ranking obtained in this step for a given query image can be found in FIG. 4 discussed above, at different training stages of the CNN (i.e., at different numbers of iterations of the training method illustrated in FIG. 10)”. Examiner interprets the remaining images in the batch being ranked to be equivalent as the claimed “ranked instances”, each image in the batch and the CNN are interpreted to be equivalent as the claimed “data instances” and “machine learning model” respectively);
determining a metric based on the ranked instances (see Revaud at Column 10 Lines 33-39: “One of the IR metrics, called precision, may refer to the fraction of the images retrieved that are relevant to the user's information need. Another IR metric, called recall, may refer to the fraction of the images relevant (or similar) to the query that are successfully retrieved. Both precision and recall are single-value metrics based on the whole list of images returned by the system”. Examiner interprets the retrieval of precision and recall based on the whole list of images returned by the system to be equivalent as the claimed “determining a metric based on the ranked instances”);
and outputting an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the metric (see Revaud at Column 8 Lines 14-37: “The aim of the training phase may be to modify internal, learnable parameters of the CNN to minimize the result of the loss function. This optimization of the loss function (which converges towards a minimum) is rendered possible by the back-propagation of so called loss gradients, which are obtained from the partial derivatives of the loss function with respect to the learnable parameters of the CNN. These loss gradients are back-propagated to the respective learnable parameters in that they are used to modify (or adapt or update) the learnable parameters to perform better at the next iteration of the training phase. At the next iteration, the CNN may output global feature descriptors of the images computed with the help of the modified learnable parameters, and these global feature descriptors may be better suited for ranking the images, leading to a better evaluation of the ranking obtained by the CNN (i.e., to a lower value of the loss function) and to a smaller adaptation of the learnable parameters at the next back-propagation of the loss gradients, until the loss function converges towards a minimum (e.g., less than a predetermined value) and adaptation of the learnable parameters can be stopped. Once the loss function has converged, the CNN has been successfully trained, and the (trained) CNN can be used to perform image retrieval”. Examiner interprets the modification of the learnable parameters and the outputting of the global feature descriptors to be equivalent as the claimed “outputting an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the metric”).
Referring to Claim 2, Revaud teaches the method of claim 1:
wherein determining the metric includes determining at least one selected from the group consisting of accuracy and recall (see Revaud at Column 10 Lines 33-39: “One of the IR metrics, called precision, may refer to the fraction of the images retrieved that are relevant to the user's information need. Another IR metric, called recall, may refer to the fraction of the images relevant (or similar) to the query that are successfully retrieved. Both precision and recall are single-value metrics based on the whole list of images returned by the system”. Examiner interprets the return of precision and recall by the system to be equivalent as the claimed “determining at least one selected from the group consisting of accuracy and recall”).
Referring to Claim 3, Revaud teaches the method of claim 1:
wherein determining the metric includes generating ground truth ranked instances based on the ranked instances and ground truth instances, wherein the ground truth instances corresponds to error free data (see Revaud at Column 10 Lines 36-50: “ The training images used for the training phase are known, and in particular their respective similarities are known, thereby providing the ground-truth image relevance mentioned above. The ground-truth image relevance includes for example in a set of known binary ground-truth labels (e.g., one label for each pair of images of the batch), each label of a given pair of images informing whether these images are similar (in which case, the label is equal to 1) or not (in which case, the label is equal to 0). For example, the received training images can be one of the datasets of known images mentioned above, such as
PNG
media_image1.png
42
29
media_image1.png
Greyscale
Paris or
PNG
media_image1.png
42
29
media_image1.png
Greyscale
Oxford. In an embodiment, the training images of the batch are high-resolution images of more than 1 Megapixels”. Examiner interprets the labels of the ground truth being ranked 1 and 0 to be equivalent as the claimed “generating ground truth ranked instances based on the ranked instances and ground truth instances” and the respective similarities of the training images are known is interpreted to be equivalent as the claimed “the ground truth instances corresponds to error free data”).
Referring to independent Claim 9, this claim is rejected on the same basis as independent claim 1 since they are analogous claims.
Referring to dependent Claim 10, this claim is rejected on the same basis as dependent claim 2 since they are analogous claims.
Referring to dependent Claim 11, this claim is rejected on the same basis as dependent claim 3 since they are analogous claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 4-7, 12-15, 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Revaud et al (US-11521072-B2 - hereinafter Revaud) in view of Moore et al (JP-7343568-B2 - hereinafter Moore).
Referring to Claim 4, Revaud teaches the method of claim 3.
However, Revaud fails to teach:
wherein determining the metric includes generating a first ranking curve based on the metric.
Moore teaches, in analogous system,
wherein determining the metric includes generating a first ranking curve based on the metric (see Moore at Paragraph 3: “The method further uses the first selected group of hyperparameter values, the second selected group of hyperparameter values, and the dataset to generate a first version of the selected machine learning model. This may include training. Metadata includes information such as the size of the training set, the shape of the dataset, the number of features in the dataset, the proportions of different types of data fields in the dataset, the type of classification problem, and the types of data fields in the dataset. and an indicator of whether the data set follows a certain statistical distribution. The method may include running a second-order machine learning model based on the metadata as input. The secondary machine learning model returns a selection of the first version of the selected machine learning model and returns preferred machine learning hyperparameter values for use with the first version of the selected machine learning model. The one or more performance metrics include accuracy, error, precision, recall, area under a receiver operating characteristic (ROC) curve, and It may include at least one area under a precision recall curve”. Examiner interprets the receiver operating characteristic (ROC) curve and the precision recall curve generated by the first version of the selected machine learning model to be equivalent as the claimed “generating a first ranking curve based on the metric”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Revaud with the above teachings of Moore by ranking data instances to determine when to adjust parameters of a machine learning model, as taught by Revaud, and generating the first ranking curve based on the metric, as taught by Moore. The modification would have been obvious because one of ordinary skill in the art would be motivated to rank plurality of hyperparameters (as suggested by Moore at Paragraph 3: “The machine learning model returns a ranking of the plurality of hyperparameters according to the influence of the selected first version of the machine learning model on the one or more performance metrics”).
Referring to Claim 5, Revaud - Moore teaches the method of claim 4.
However, Revaud fails to teach:
wherein determining the metric includes determining a first area under curve of the first ranking curve. Moore teaches, in analogous system,
wherein determining the metric includes determining a first area under curve of the first ranking curve (see Moore at Paragraph 3: “The method further uses the first selected group of hyperparameter values, the second selected group of hyperparameter values, and the dataset to generate a first version of the selected machine learning model. This may include training. Metadata includes information such as the size of the training set, the shape of the dataset, the number of features in the dataset, the proportions of different types of data fields in the dataset, the type of classification problem, and the types of data fields in the dataset. and an indicator of whether the data set follows a certain statistical distribution. The method may include running a second-order machine learning model based on the metadata as input. The secondary machine learning model returns a selection of the first version of the selected machine learning model and returns preferred machine learning hyperparameter values for use with the first version of the selected machine learning model. . The one or more performance metrics include accuracy, error, precision, recall, area under a receiver operating characteristic (ROC) curve, and It may include at least one area under a precision recall curve”. Examiner interprets the area under the receiver operating characteristic (ROC) curve and the area under the precision recall curve generated by the first version of the selected machine learning model to be equivalent as the claimed “first area under curve of the first ranking curve” ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Revaud with the above teachings of Moore by ranking data instances to determine when to adjust parameters of a machine learning model, as taught by Revaud, and generating the first ranking curve based on the metric, as taught by Moore. The modification would have been obvious because one of ordinary skill in the art would be motivated to rank plurality of hyperparameters (as suggested by Moore at Paragraph 3: “The machine learning model returns a ranking of the plurality of hyperparameters according to the influence of the selected first version of the machine learning model on the one or more performance metrics”).
Referring to Claim 6, Revaud - Moore teaches the method of claim 5.
However, Revaud fails to teach:
wherein determining the metric includes generating a second ranking curve based on the ground truth ranked instances.
Moore teaches, in analogous system,
wherein determining the metric includes generating a second ranking curve based on the ground truth ranked instances (see Moore at Paragraph 3: “The method further uses the first selected group of hyperparameter values, the second selected group of hyperparameter values, and the dataset to generate a first version of the selected machine learning model. This may include training. Metadata includes information such as the size of the training set, the shape of the dataset, the number of features in the dataset, the proportions of different types of data fields in the dataset, the type of classification problem, and the types of data fields in the dataset. and an indicator of whether the data set follows a certain statistical distribution. The method may include running a second-order machine learning model based on the metadata as input. The secondary machine learning model returns a selection of the first version of the selected machine learning model and returns preferred machine learning hyperparameter values for use with the first version of the selected machine learning model. . The one or more performance metrics include accuracy, error, precision, recall, area under a receiver operating characteristic (ROC) curve, and it may include at least one area under a precision recall curve”. Examiner interprets the receiver operating characteristic (ROC) curve and the precision recall curve generated by the secondary machine learning model to be equivalent as the claimed “generating a second ranking curve based on the ground truth ranked instances”, the dataset is interpreted as containing the claimed “ground truth”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Revaud with the above teachings of Moore by ranking data instances to determine when to adjust parameters of a machine learning model, as taught by Revaud, and generating the first ranking curve based on the metric, as taught by Moore. The modification would have been obvious because one of ordinary skill in the art would be motivated to rank plurality of hyperparameters (as suggested by Moore at Paragraph 3: “The machine learning model returns a ranking of the plurality of hyperparameters according to the influence of the selected first version of the machine learning model on the one or more performance metrics”).
Referring to Claim 7, Revaud - Moore teaches the method of claim 6.
However, Revaud fails to teach:
wherein determining the metric includes determining a second area under curve of the second ranking curve.
Moore teaches, in analogous system,
wherein determining the metric includes determining a second area under curve of the second ranking curve (wherein determining the metric includes determining a first area under curve of the first ranking curve (see Moore at Paragraph 3: “The method further uses the first selected group of hyperparameter values, the second selected group of hyperparameter values, and the dataset to generate a first version of the selected machine learning model. This may include training. Metadata includes information such as the size of the training set, the shape of the dataset, the number of features in the dataset, the proportions of different types of data fields in the dataset, the type of classification problem, and the types of data fields in the dataset. and an indicator of whether the data set follows a certain statistical distribution. The method may include running a second-order machine learning model based on the metadata as input. The secondary machine learning model returns a selection of the first version of the selected machine learning model and returns preferred machine learning hyperparameter values for use with the first version of the selected machine learning model. . The one or more performance metrics include accuracy, error, precision, recall, area under a receiver operating characteristic (ROC) curve, and It may include at least one area under a precision recall curve”. Examiner interprets the area under the receiver operating characteristic (ROC) curve and the area under the precision recall curve generated by the secondary machine learning model to be equivalent as the claimed “second area under curve of the second ranking curve”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Revaud with the above teachings of Moore by ranking data instances to determine when to adjust parameters of a machine learning model, as taught by Revaud, and generating the first ranking curve based on the metric, as taught by Moore. The modification would have been obvious because one of ordinary skill in the art would be motivated to rank plurality of hyperparameters (as suggested by Moore at Paragraph 3: “The machine learning model returns a ranking of the plurality of hyperparameters according to the influence of the selected first version of the machine learning model on the one or more performance metrics”).
Referring to dependent Claim 12, this claim is rejected on the same basis as dependent claim 4 since they are analogous claims.
Referring to dependent Claim 13, this claim is rejected on the same basis as dependent claim 5 since they are analogous claims.
Referring to dependent Claim 14, this claim is rejected on the same basis as dependent claim 6 since they are analogous claims.
Referring to dependent Claim 15, this claim is rejected on the same basis as dependent claim 7 since they are analogous claims.
Referring to dependent Claim 18, claim 18 is a combination of claims 4 and 6. Claim 18 is rejected on the same basis as dependent claims 4 and 6 since they are analogous claims.
Referring to dependent Claim 19, claim 18 is a combination of claims 5 and 7. Claim 19 is rejected on the same basis as dependent claims 5 and 7 since they are analogous claims.
Claims 8, 16-17, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Revaud et al (US-11521072-B2 - hereinafter Revaud) in view of Moore et al (JP-7343568-B2 - hereinafter Moore) and in further view of Kartoun et al(US-11742081-B2 - hereinafter Kartoun).
Referring to Claim 8, Revaud - Moore teaches the method of claim 7.
However, Revaud - Moore fails to teach:
wherein determining the metric includes determining a ranking index based on the first area under curve and the second area under curve, and wherein the one or more parameters of the machine learning model are adjusted based on at least one of the ranking index and the first area under curve.
Kartoun teaches, in analogous system,
wherein determining the metric includes determining a ranking index based on the first area under curve and the second area under curve, and wherein the one or more parameters of the machine learning model are adjusted based on at least one of the ranking index and the first area under curve (see Kartoun at Column 2 Lines 14-29: “evaluating performance of the first predictive model against the second predictive model comprises testing each of the first predictive model and the second predictive model using a same model testing set to identify true positives and false positives, and comparing a first area under a first receiver operating curve to a second area under a second receiver operating curve, wherein the first receiver operating curve is based on identified true and false positives of the first predictive model and wherein the second receiver operating curve is based on identified true and false positives of the second predictive model. Thus, embodiments of the present invention verify that a generated predictive model is superior in predictive accuracy. In some embodiments, the second set of additional features is identified based on a statistical significance of each feature satisfying a threshold significance value to predict the outcome”. Examiner interprets the comparison of the first area under a first receiver operating curve to a second area under a second receiver operating curve to determine which model achieves superior predictive accuracy to be equivalent as the claimed “determining a ranking index based on the first area under curve and the second area under curve and wherein the one or more parameters of the machine learning model are adjusted based on at least one of the ranking index and the first area under curve”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Revaud and Moore with the above teachings of Kartoun by receiving and ranking data instances to determine when to adjust parameters of a machine learning model, as taught by Revaud and Moore, and determining a ranking index based on the first area under curve and the second area under curve, as taught by Kartoun. The modification would have been obvious because one of ordinary skill in the art would be motivated to verify that a generated predictive model is superior in predictive accuracy (as suggested by Kartoun at Column 2 Lines 14-29: “evaluating performance of the first predictive model against the second predictive model comprises testing each of the first predictive model and the second predictive model using a same model testing set to identify true positives and false positives, and comparing a first area under a first receiver operating curve to a second area under a second receiver operating curve, wherein the first receiver operating curve is based on identified true and false positives of the first predictive model and wherein the second receiver operating curve is based on identified true and false positives of the second predictive model. Thus, embodiments of the present invention verify that a generated predictive model is superior in predictive accuracy. In some embodiments, the second set of additional features is identified based on a statistical significance of each feature satisfying a threshold significance value to predict the outcome”).
Referring to dependent Claim 16, this claim is rejected on the same basis as dependent claim 8 since they are analogous claims.
Referring to Claim 17, Revaud teaches:
receiving data instances (see Revaud at Column 17 Lines 28-34: “Obtaining the training images may include receiving the training images from a user dealing with the training phase of the CNN or from a memory location (e.g. a local or remote computing system, a local or remote database, a cloud storage or any other memory location known in the field) on which they are stored”. Examiner interprets receiving the training images to be equivalent as the claimed “receiving data instances”);
determine ranked instances based on the data instances and a machine learning model, determine ground truth ranked instances based on the data instances, ground truth data instances, and the machine learning model, wherein the ground truth data instances are free from errors (see Revaud at Column 18 Lines 18-30: “At 1008, a ranking function R is applied to the training images of the batch to sort them based on their similarities. For example, each image of the batch is considered to be an initial query image, and the remaining images of the batch are ranked (or sorted) according to their similarities to the query image. The images of the batch can be ranked by decreasing or by increasing similarity. An example of the ranking obtained in this step for a given query image can be found in FIG. 4 discussed above, at different training stages of the CNN (i.e., at different numbers of iterations of the training method illustrated in FIG. 10)”. Examiner interprets the remaining images in the batch being ranked to be equivalent as the claimed “ranked instances”, each image in the batch and the CNN are interpreted to be equivalent as the claimed “data instances” and “machine learning model” respectively) and further (see Revaud at Column 10 Lines 36-50: “ The training images used for the training phase are known, and in particular their respective similarities are known, thereby providing the ground-truth image relevance mentioned above. The ground-truth image relevance includes for example in a set of known binary ground-truth labels (e.g., one label for each pair of images of the batch), each label of a given pair of images informing whether these images are similar (in which case, the label is equal to 1) or not (in which case, the label is equal to 0). For example, the received training images can be one of the datasets of known images mentioned above, such as
PNG
media_image1.png
42
29
media_image1.png
Greyscale
Paris or
PNG
media_image1.png
42
29
media_image1.png
Greyscale
Oxford. In an embodiment, the training images of the batch are high-resolution images of more than 1 Megapixels”, where Examiner interprets the labels of the ground truth being ranked 1 and 0 to be equivalent as the claimed “determine ground truth ranked instances based on the data instances” and the respective similarities of the training images are known is interpreted to be equivalent as the claimed “the ground truth instances corresponds to error free data”);
and output an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the ranking index (see Revaud at Column 8 Lines 14-37: “The aim of the training phase may be to modify internal, learnable parameters of the CNN to minimize the result of the loss function. This optimization of the loss function (which converges towards a minimum) is rendered possible by the back-propagation of so called loss gradients, which are obtained from the partial derivatives of the loss function with respect to the learnable parameters of the CNN. These loss gradients are back-propagated to the respective learnable parameters in that they are used to modify (or adapt or update) the learnable parameters to perform better at the next iteration of the training phase. At the next iteration, the CNN may output global feature descriptors of the images computed with the help of the modified learnable parameters, and these global feature descriptors may be better suited for ranking the images, leading to a better evaluation of the ranking obtained by the CNN (i.e., to a lower value of the loss function) and to a smaller adaptation of the learnable parameters at the next back-propagation of the loss gradients, until the loss function converges towards a minimum (e.g., less than a predetermined value) and adaptation of the learnable parameters can be stopped. Once the loss function has converged, the CNN has been successfully trained, and the (trained) CNN can be used to perform image retrieval”. Examiner interprets the modification of the learnable parameters and the outputting of the global feature descriptors to be equivalent as the claimed “outputting an adjusted machine learning model generated by adjusting one or more parameters of the machine learning model based on the metric”);
however, Revaud fails to teach:
determine a first metric based on the ranked instances and a second metric based on the ground truth ranked instances;
determine a ranking index based on a comparison of the first metric and the second metric;
Moore teaches, in analogous system,
determine a first metric based on the ranked instances and a second metric based on the ground truth ranked instances (see Moore at Paragraph 3: “The method further uses the first selected group of hyperparameter values, the second selected group of hyperparameter values, and the dataset to generate a first version of the selected machine learning model. This may include training. Metadata includes information such as the size of the training set, the shape of the dataset, the number of features in the dataset, the proportions of different types of data fields in the dataset, the type of classification problem, and the types of data fields in the dataset. and an indicator of whether the data set follows a certain statistical distribution. The method may include running a second-order machine learning model based on the metadata as input. The secondary machine learning model returns a selection of the first version of the selected machine learning model and returns preferred machine learning hyperparameter values for use with the first version of the selected machine learning model. The one or more performance metrics include accuracy, error, precision, recall, area under a receiver operating characteristic (ROC) curve, and It may include at least one area under a precision recall curve”. Examiner interprets the performance metrics (accuracy, error, precision, recall, area under a receiver operating characteristic (ROC) curve, area under a precision recall curve) generated by the first and second version of the model respectively to be equivalent as the claimed “first metric” and “second metric” respectively, the returned preferred machine learning hyperparameter values are interpreted to be equivalent as the claimed “ranked instances” and the metadata used to run the second-order machine learning model is interpreted to be equivalent as the claimed “ground truth ranked instances”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Revaud with the above teachings of Moore by ranking data instances to determine when to adjust parameters of a machine learning model, as taught by Revaud, and determine a first metric based on the ranked instances and a second metric based on the ground truth ranked instances, as taught by Moore. The modification would have been obvious because one of ordinary skill in the art would be motivated to rank plurality of hyperparameters (as suggested by Moore at Paragraph 3: “The machine learning model returns a ranking of the plurality of hyperparameters according to the influence of the selected first version of the machine learning model on the one or more performance metrics”).
Kartoun teaches, in analogous system,
determine a ranking index based on a comparison of the first metric and the second metric (see Kartoun at Column 2 Lines 14-29: “evaluating performance of the first predictive model against the second predictive model comprises testing each of the first predictive model and the second predictive model using a same model testing set to identify true positives and false positives, and comparing a first area under a first receiver operating curve to a second area under a second receiver operating curve, wherein the first receiver operating curve is based on identified true and false positives of the first predictive model and wherein the second receiver operating curve is based on identified true and false positives of the second predictive model. Thus, embodiments of the present invention verify that a generated predictive model is superior in predictive accuracy. In some embodiments, the second set of additional features is identified based on a statistical significance of each feature satisfying a threshold significance value to predict the outcome”. Examiner interprets the comparison of the first area under a first receiver operating curve to a second area under a second receiver operating curve to determine which model achieves superior predictive accuracy to be equivalent as the claimed “determine a ranking index based on a comparison of the first metric and the second metric”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Revaud with the above teachings of Kartoun by receiving and ranking data instances to determine when to adjust parameters of a machine learning model, as taught by Revaud, and determining a ranking index based on the first area under curve and the second area under curve, as taught by Kartoun. The modification would have been obvious because one of ordinary skill in the art would be motivated to verify that a generated predictive model is superior in predictive accuracy (as suggested by Kartoun at Column 2 Lines 14-29: “evaluating performance of the first predictive model against the second predictive model comprises testing each of the first predictive model and the second predictive model using a same model testing set to identify true positives and false positives, and comparing a first area under a first receiver operating curve to a second area under a second receiver operating curve, wherein the first receiver operating curve is based on identified true and false positives of the first predictive model and wherein the second receiver operating curve is based on identified true and false positives of the second predictive model. Thus, embodiments of the present invention verify that a generated predictive model is superior in predictive accuracy. In some embodiments, the second set of additional features is identified based on a statistical significance of each feature satisfying a threshold significance value to predict the outcome”).
Referring to dependent Claim 20, this claim is rejected on the same basis as dependent claim 8 since they are analogous claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AWADAGBE G HOUNTON whose telephone number is (571)270-0670. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AWADAGBE G HOUNTON/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights