DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment and Arguments
Claims 1-34 are pending and are being examined in this application.
Applicant’s arguments with respect to the 103 rejections have been fully considered, but are unpersuasive for at least the following reasons:
Applicant argues that the cited references do not disclose the limitation “generating, by the server, a first usefulness score for the first candidate model and a second usefulness score for the second candidate model using the unlabeled dataset.” In particular, applicant argues that Asher does not disclose using an unlabeled dataset [Remarks, pg. 3].
However, paragraph 31 of Asher discloses “database server 210 may receive a selection of a data set” (emphasis added). Paragraph 38 of Asher further discloses “generate a predictive learning model based in part on the plurality of features” by “evaluat[ing] the plurality of candidate machine learning models. This evaluation may be based in part on a predictive accuracy of each of the machines. The database server 210 may subsequently select the predictive machine learning model based in part on the evaluation (e.g., based on which model is most accurate, or is otherwise best suited for the selected data set or the desired predictive value)” (emphases added).
In other words, each candidate machine learning model makes predictions based on features of the selected data set, which means that at least part of the selected data set requires predicting (i.e., is unlabeled) when it is received from the server. Thus, Asher clearly teaches that the selected dataset received by the server is an unlabeled dataset, and, when the candidate models are evaluated, the evaluation is based on features generated from the selected dataset or best suited to the selected dataset. As such, Asher, in combination with Almasan, clearly teaches “generating, by the server, a first usefulness score for the first candidate model and a second usefulness score for the second candidate model using the unlabeled dataset.” It is noted one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
With respect to applicant’s contention “The above passages from Asher strongly imply that the prediction field is a field of data containing ground-truth values against which predictions can be compared” [Remarks, pg. 5], it is unclear why the exact mechanism for evaluating the candidate models is important as long as the selected dataset is used in some way during the evaluation (already shown above). However, paragraph 36 of Asher (cited by applicant) discloses “evaluate a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features” (emphasis added by applicant). Also cited by applicant, paragraph 33 of Asher discloses “A prediction field may refer to the field of data for which the user wishes the database server 210 to perform a prediction” (emphasis added by applicant).
Based on the above paragraphs cited by applicant, the evaluating is based on statistical relationships between predicted values and features, where the predicted values are for a predictive field that a user wishes to make predictions for. There is no mention of ground-truth values or comparison with existing values of the predictive field, only disclosure of using features to make predictions for the predictive field. Paragraphs 34 and 35 of Asher (not cited by applicant, but very important for context because they are in between paragraphs 33 and 36, which were cited by applicant) provide a detailed example pertaining to generating features and predicting values for fields using the features. In paragraph 34, Asher discloses “a user may indicate which fields include accurate data, and which fields to predict” (emphasis added by examiner). This very clearly indicates that the predictive field is different from fields that already have accurate data. Instead, predictions for the predictive field are used to generate (missing or unlabeled) information desired by the user (e.g., sales leads).
Allowable Subject Matter
Claims 3-13 and 20-30 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 14-19, and 31-34 are rejected under 35 U.S.C. 103 as being unpatentable over Almasan et al. (US Pub. 20230334344) in view of Asher et al. (US Pub. 20190138946).
Referring to claim 1, Alsmasan discloses A method of selecting a target model for an unlabeled dataset of a dataset provider, the target model for generating labels for the unlabeled dataset, the dataset provider being communicatively coupled to a server, the method executable by the server [fig. 6; pars. 13, 22-25, 35, 36, 64, and 73; in response to a search request, a network environment comprising one or more computing devices (e.g., servers) is used to select a machine learning model (e.g., for classifying images) from a plurality of machine learning models provided by members (i.e., users) of a distributed ledger of the network environment], the method comprising:
...acquiring, by the server, a first candidate model from a first model provider and a second candidate model from a second model provider, the first model provider and the second model provider being communicatively coupled to the server, the first candidate model having been trained based on first training data available to the first model provider, and the second candidate model having been trained based on second training data available to the second model provider [pars. 13, 20, 35, 36, 50, 51; the users can register new machine learning models with the distributed ledger, where each machine learning model is trained using training data for its specific use case; when the search request is submitted, the distributed ledger returns a list of one or more machine learning models as search results];
generating, by the server, a first usefulness score for the first candidate model and a second usefulness score for the second candidate model..., the first usefulness score being indicative of likelihood that the first candidate model generates accurate labels...; the second usefulness score being indicative of likelihood that the second candidate model will generate the accurate labels for the unlabeled dataset [pars. 37 and 44; each machine learning model is associated with one or more scores, where the scores can represent various performance metrics that can be used to rank or evaluate the machine learning model (e.g., performance metrics including accuracy of the runtime machine-learning model and/or a measure of any bias by the runtime machine learning model];
...causing, by the server, generation of the labels from the unlabeled dataset using the target model [pars. 35, 36, and 64; the selected machine learning model (e.g., for classifying images) is deployed with an application].
Almasan does not appear to explicitly disclose acquiring, by the server, the unlabeled dataset from the dataset provider; that the first usefulness score and the second usefulness score are generated using the unlabed thled dataset; and selecting, by the server, the first candidate model as the target model using the first usefulness score and the second usefulness score.
However, Asher discloses acquiring, by the server, the unlabeled dataset from the dataset provider; that the first usefulness score and the second usefulness score are generated using the unlabeled dataset; and selecting, by the server, the first candidate model as the target model using the first usefulness score and the second usefulness score [pars. 31 and 38; a database server receives a selection of a data set and selects a predictive learning model based on training a plurality of candidate machine learning models; the predictive learning model is selected based on which model is most accurate or is otherwise best suited for the selected data set].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the scores taught by Almasan so that the accuracy score associated with each machine learning model taught by Almasan so that the accuracy score of each machine learning model is generated using a selected data set and the selected machine learning model is selected based on which machine learning model is most accurate as taught by Asher, with a reasonable expectation of success. The motivation for doing so would have been to selected the machine learning model that is best suited for the selected data set [Asher, par. 38].
Referring to claim 2, Asher discloses The method of claim 1, wherein the generating the first usefulness score comprises: generating, by the server employing the first candidate model, label-wise probabilities for respective input objects from the unlabeled dataset; generating, by the server, scores for respective input objects based on the respective label-wise probabilities, the scores being indicative of how confident the first candidate model is in a given label amongst a plurality of potential labels for the respective input objects; and generating, by the server, the first usefulness score based on a combination of the scores for respective input objects [pars. 33 and 39; the accuracy of each predictive learning model is determined based on whether the predictive learning model can generate accurate predictive scores for a prediction field in records in the data set; statistical analyses are performed on predicted values of the prediction field in each record in the data set; each time a new record is received, the predictive score for the prediction field is updated based on the new record].
Referring to claim 14, Almasan discloses The method of claim 1, wherein labels are classes to classify input objects included in the unlabeled dataset [par. 64; note the image classification].
Referring to claim 15, Almasan discloses The method of claim 1, further comprising, subsequent to acquiring the first and second candidate models, performing a metadata-based selection to identify a subset of potential target models including the first and second candidate models [par. 13; the list of one or more machine learning models is returned as search results according to search criteria].
Referring to claim 16, Almasan discloses The method of claim 1, wherein the causing generation of the labels from the unlabeled dataset using the target model comprises transmitting, by the server, the target model to the dataset provider for generating the labels [pars. 35, 36, and 64; note the deploying of the selected machine learning model (e.g., for classifying images) with the application].
Referring to claim 17, Almasan discloses The method of claim 1, wherein the causing generation of the labels from the unlabeled dataset using the target model comprises generating, by the server, the labels using the target model [pars. 35, 36, and 64; note the deploying of the selected machine learning model (e.g., for classifying images) with the application].
Referring to claim 18, see at least the rejection for claim 1. Almasan further discloses A system for selecting a target model for an unlabeled dataset of a dataset provider, the target model for generating labels for the unlabeled dataset, the system comprising: a server for running a machine learning model (MLM) trading platform, the dataset provider being communicatively coupled to a server, the server being configured to perform the claimed steps [fig. 6; pars. 13, 22-25, 35, 36, 64, and 73; note the network environment comprising the one or more computing devices (e.g., servers) used to select the machine learning model (e.g., for classifying images) from the plurality of machine learning models provided by the members (i.e., users) of the distributed ledger of the network environment].
Referring to claim 19, see the rejection for claim 12.
Referring to claim 31, see the rejection for claim 14.
Referring to claim 32, see the rejection for claim 15.
Referring to claim 33, see the rejection for claim 16.
Referring to claim 34, see the rejection for claim 17.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRACE PARK whose telephone number is (571)270-7727. The examiner can normally be reached M-F 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, TAMARA KYLE can be reached at (571)272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Grace Park/Primary Examiner, Art Unit 2144