Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-8, 11-16, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering).
Regarding claim 1,
Phan teaches the limitation “a mobile device comprising one or more processors” (paragraph 28 “Where an embodiment is described using a mobile device, any type of data storage device suitable for use with the mobile device may provide the data to such embodiment, either locally at the mobile device or over a data network”, and paragraph 54 “Processor set 110 includes one, or more, computer processors of any type now known or to be developed in the future”. Phan discloses a method and system and computer program product of active learning, which can be performed on a mobile device comprising one or more processors.)
Phan teaches or at least suggests the limitation “a non-transitory computer readable medium comprising instructions recorded thereon that when executed by the one or more processors causes operations” (paragraph 51 “A computer program product embodiment (“CPP embodiment” or “CPP”) is a term used in the present disclosure to describe any set of one, or more, storage media (also called “mediums”) collectively included in a set of one, or more, storage devices that collectively include machine readable code corresponding to instructions and/or data for performing computer operations ... A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se” Phan discloses a computer readable storage medium not to be construed as storage in the form of transitory signals per se, and comprising machine readable code corresponding to instructions and/or data for performing computer operations when executed by the one or more processors.)
Phan teaches or at least suggests part of the limitation “obtaining (1) one or more user-defined target parameter values for data labeling, ... (3) a dataset comprising a plurality of unlabeled samples” (paragraph 18 “An embodiment receives a plurality of learning parameters, or sets one or more of the learning parameters to default values ... One learning parameter is a query budget (denoted by B), which is the total number of samples from a dataset that can be selected for labelling ... An embodiment also receives ... an unlabeled dataset” Phan discloses the embodiment of active learning comprises of learning parameter of total number of samples from a dataset that can be selected for labelling, which corresponds to the claimed user-defined target parameter values for data labeling, and an unlabeled dataset, which corresponds to the dataset comprising a plurality of unlabeled samples, as claimed.)
Phan teaches the limitation “selecting, based on the one or more user-defined target parameter values, a first subset of the dataset, wherein the first subset comprises samples of the plurality of unlabeled samples” (paragraph 24 “An embodiment selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset ... An embodiment also subtracts the number of samples in the subset to be labeled from the query budget B” Phan discloses selecting a subset of plurality of samples from the unlabeled dataset based on the budget parameter, which corresponds to the selection of the first unlabeled samples for the first subset based on the one or more user-defined target parameter values, as claimed.)
Phan teaches or at least suggests the limitation “transmitting, to a remote device, a request for labeling the samples of the first subset, wherein the request comprises the samples of the first subset” (paragraph 24 “An embodiment selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset ... thus an embodiment sends the selected samples to be labeled. The labeling is performed by an automated labeling service, a human expert, or another presently available technique.” Phan discloses after the samples are selected from the unlabeled dataset, they are labeled by an automated labeling service or a human expert, which corresponds to the request comprising the samples of the first subset for labeling, as claimed.)
Phan teaches or at least suggests the limitation “receiving, from the remote device, a first training dataset based on the first subset, wherein the first training dataset comprises label data and the samples of the first subset, and wherein the label data indicates a classification for each sample” (paragraph 18 “The labeled dataset includes a label for each piece of data, .... In other words, the label represents a trained model's correct response to the data”, and paragraph 25 “An embodiment receives the now-labeled selected samples and adds them to the set of labeled samples. An embodiment repeats the model training process, including use of the newly-labeled samples, ..., all the available data has been labeled” Phan discloses the embodiment receives the now-labeled selected samples and adds them to the set of labeled samples to obtain an updated labeled training dataset with a label for each piece of data, which corresponds to the receiving a first training dataset based on the first subset comprising of label data and the samples of the first subset, as claimed. Furthermore, the label for each piece of data represents a trained model's correct response to the data, which corresponds to the label data indicates a classification for each sample, as claimed.)
Phan teaches or at least suggests the limitation “training, using the first training dataset, a machine learning model” (paragraph 25 “An embodiment receives the now-labeled selected samples and adds them to the set of labeled samples. An embodiment repeats the model training process, including use of the newly-labeled samples, ..., all the available data has been labeled” Phan discloses the machine learning model repeats the model training process, suggesting that the model perform the model training using the updated labeled training dataset, which corresponds to the claimed training the machine learning model using the first training dataset.)
Phan teaches or at least suggests the limitation “transmitting, to the remote device, a second request for labeling samples of the second subset, wherein the second request comprises samples of the second subset” (paragraph 19 “In each expression, t denotes a number of times unlabeled data has so far been selected”, paragraph 74 “Application 300 repeats the model training process, including use of the newly-labeled samples, until the query budget B has been exhausted, all the available data has been labeled and used to train the model, the model meets a model performance criterion”, and paragraph 78 “In line 9, pseudocode 310 subtracts the number of samples in the subset to be labeled from the query budget B and adds the now-labeled selected samples to the set of labeled samples. In line 10, pseudocode 310 increments t, and in line 11 pseudocode 310 returns to line 3 if query budget B is still greater than zero” Phan discloses the application further repeat the training process. For instance, the number of times unlabeled data has been selected is denoted by t, such that as t increments, more data is selected for labeling and update the training dataset for continuing training the model. One of ordinary skill in the art would have been able to configure an initial t=1 as the first time the first subset of unlabeled data is selected for labeling and training the model corresponding to the disclosure above, and t=2 as the second time the unlabeled data is further selected for labeling and further training the model, thus the whole process of selecting, labeling, and receiving labeled subset of data is repeated for the t=2 time similarly to the process at t=1 time, which corresponds to a second request comprising samples of a second subset for labeling, as claimed. The process to configure the second subset may be configured in view of Meek’s teaching below.)
Phan teaches or at least suggests the limitation “receiving, from the remote device, a second training dataset comprising label data and the samples of the second subset” paragraph 19 “In each expression, t denotes a number of times unlabeled data has so far been selected”, and paragraph 78 “In line 10, pseudocode 310 increments t, and in line 11 pseudocode 310 returns to line 3 if query budget B is still greater than zero” Phan discloses repeating the training process as t increments, such that more unlabeled data is selected and label, and the application receive the second labeled subset of dataset (at t=2) to configure an updated training dataset, which corresponds to the receiving of the second training dataset, as claimed.)
Phan teaches or at least suggests the limitation “updating, using the second training dataset, the machine learning model” (paragraph 74 “Application 300 repeats the model training process, including use of the newly-labeled samples”, and paragraph 78 “In line 10, pseudocode 310 increments t, and in line 11 pseudocode 310 returns to line 3 if query budget B is still greater than zero” Phan discloses repeating the training process as t increments, using the second labeled subset of dataset (at t=2) to configure an updated training dataset to further train the machine learning model.)
Phan does not teach part of the limitation “... (2) a user input indicative of a value added per unit of model performance improvement...”. However, Meek teaches or at least suggests this limitation (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time) ... For example, a decision maker can be asked the question “How long would you be willing to wait to increase the relative accuracy of the learned model by one percent?” If the answer is (e.g.) one hour, then α =0.01 benefit per hour” Meek discloses a learning-curve sampling method to choose an appropriate sample size for training. Meek teaches that α is a relative importance of benefit to runtime and depends on the preferences of the decision maker. Meek further explains that α may be viewed as an incremental-benefit-to-cost ratio having units benefit per time. Under the broadest reasonable interpretation, the incremental unit benefit (e.g., 0.01 benefit) obtained as the model improves through training over time corresponds to the claimed value added. Thus, Meek’s α corresponds to the claimed user input indicative of a value added per unit of model performance improvement, because α reflects the decision maker’s preference regarding how much model’s benefit is valued (e.g., 0.01 benefit) relative to the runtime/resource cost requirement to obtain that improvement.)
Phan does not teach the limitation “generating, using the user input indicative of the value added per unit of model performance improvement, a margin curve of a relationship between resource usage and value added per unit of model performance improvement”. However, Meek teaches or at least suggests this limitation (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time)”, and Page 2 section 1 “Learning-curve sampling methods rely on two basic observations: (1) the computational cost of learning a model increases as a function of the size of the training data, and (2) the performance/accuracy of a model has diminishing improvements as a function of the size of the training data. The curve describing the performance as a function of the sample size of the training data is often called the learning curve ... Thus, a learning curve sampling method monitors the increasing costs and performance as larger and larger amounts of data are used for training, and terminates learning when the increasing costs outweigh the benefit of increasing performance” Meek discloses the user determine the α value, and further generate a learning curve that monitors the relationship between increasing training cost and model performance with the benefit obtained as larger amounts of training data are used. Under the broadest reasonable interpretation, Meek’s learning curve corresponds to the claimed margin curve, because it represents how model-performance benefit changes relative to resource usage as the training sample size increases. Meek’s computational cost/runtime usage corresponds to the resource usage, and Meek’s unit benefit corresponds to the claimed value added, and α reflects the decision maker’s preference regarding how much model’s benefit is valued relative to the runtime/resource cost requirement to obtain that improvement, thus Meek teaches or at least suggest the generating of a margin curve of a relationship between resource usage and value added per unit of model performance improvement, as claimed.)
Phan does not teach the limitation “determining, based on the margin curve, whether an amount of resource usage exceeds an amount of value added”. However, Meek teaches or at least suggests this limitation (Page 2 section 1 “Thus, a learning curve sampling method monitors the increasing costs and performance as larger and larger amounts of data are used for training, and terminates learning when the increasing costs outweigh the benefit of increasing performance” Meek discloses a learning curve represents how model-performance benefit changes relative to resource usage as the training sample size increases. Under the broadest reasonable interpretation, Meek’s determination that the future expected costs outweigh the futured expected benefits corresponds to the determining that the amount of resource usage exceeds an amount of value added, as claimed.)
Phan does not teach the limitation “responsive to determining that the amount of value added does not exceed the amount of resource usage, selecting a second subset of the dataset, wherein a number of samples of the second subset is determined based on the margin curve”. However, Meek teaches or at least suggests this limitation (Page 2 section 2 “As we have discussed, the basic idea of a learning-curve sampling method is to iteratively apply a training algorithm to larger and larger subsets of the data, until the future expected costs outweigh the future expected benefits associated with the training ... Given a data set D, let D1,D2,...,Dn = D denote the sequence of data sets that are examined in the process of finding the appropriate sample size ... The second ingredient is the utility or “goodness” of stopping with data set Di. We decompose this utility into a benefit and cost. Let mi denote the model learned with this data set. A natural measure of cost is proportional to the total time it takes to produce model mi. A natural measure of benefit is proportional to the accuracy of mi on the task to which that model will be applied. Of course, the measure of accuracy will depend on the task at hand” Meek discloses iteratively applying a training algorithm to larger subsets of training data D to find an appropriate sample size, until future expected costs outweigh future expected benefits. Meek further teaches that cost is proportional to the time required to produce the model and benefit is proportional to model accuracy. Under the broadest reasonable interpretation, Meek’s cost/runtime corresponds to resource usage, and Meek’s benefit/accuracy corresponds to value added. Thus, Meek’s determination that expected cost outweigh expected benefits corresponds to determining that value added does not exceed resource usage. A person ordinary skill in the art would have been motivated to use Meek’s learning curve determination to select the number of training samples because doing so avoids unnecessary training of additional samples when the expected performance benefit no longer justifies the resource cost. Therefore, Meek teaches or at least suggests selecting a second subset having a number of samples determined based on the margin curve, as claimed. Such subset obtained based on the learning curve may corresponds to the selection of data subset for labeling at t=2 time by Phan above in view of the combination of Phan in view of Meek.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan with the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek. The motivation to do so is referred to in Meek’s disclosure (page 1 section 1 “In situations where one has access to massive amounts of data, the cost of building a statistical model can be significant if not insurmountable. A common practice is to build the model using a (usually random) sample of the examples or cases in the data. In so doing, however, the choice of the number of cases to use is far from clear. In this paper, we examine the learning-curve sampling method, an algorithmic approach to choosing an appropriate sample size for training”, and page 21 section 8 “Finally, our approach of using computationally efficient abbreviated training methods for determining the appropriate number of training cases can be—in principle—applied to various iterative training methods” Meek discloses the benefit of the proposed method in situations where one has access to massive amounts of data and the cost of building a statistical model can be significant if not insurmountable. The approach by Meek is computationally efficient for determining the appropriate number of training cases and can be applied to various iterative training method. Given that Phan discloses the active learning method that continue to receive data for labeling and training, but the cost of labeling is high (Phan at paragraph 2 “Active learning reduces the size of the training set needed to perform supervised learning, which is especially beneficial when the unlabeled data is abundant but the cost of labeling data is high”). Thus, the teaching by Meek can be applied to cure the deficiency of the cost of labeling data is high by Phan. A person ordinary skill in the art would have been motivated to use Meek’s learning curve determination to select an appropriate number of training samples for labeling in view of Phan’s active learning paradigm and therefore improve the active learning machine learning model.)
Regarding claim 2 which recites a method, the applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps to the system of claim 1, thus the claim is rejected under similar rationale.
Regarding claim 3 depends on claim 2, thus the rejection of claim 2 is incorporated. The applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps, thus the claim is rejected under similar rationale.
Regarding claim 4 depends on claim 3, thus the rejection of claim 3 is incorporated.
Meek teaches or at least suggests the limitation “responsive to determining that the amount of value added exceeds the amount of resource usage, determining completion of training for the machine learning model” (Page 3 section 2 equation 3 “The second ingredient is the utility or “goodness” of stopping with data set Di. We decompose this utility into a benefit and cost. Let mi denote the model learned with this data set. A natural measure of cost is proportional to the total time it takes to produce model mi. A natural measure of benefit is proportional to the accuracy of mi on the task to which that model will be applied ... We then choose the alternative among these two that maximizes our expected utility. If we choose alternative 1, we indeed stop, returning model mi and its accuracy score. If we choose alternative 2, we evaluate another decision problem” Meek discloses that the utility or “goodness” of stopping with a dataset Di is decomposed into a benefit and cost, where the cost is proportional to the time required to produce the model and the benefit is proportional to the accuracy of the model. Meek further discloses choosing the alternative that maximizes expected utility, and if alternative 1 is chosen, the system stops and returns the model and its accuracy score. Under the broadest reasonable interpretation, Meek’s benefit/accuracy corresponds to the claimed value added, and Meek’s cost/time corresponds to the claimed resource usage. Because Meek’s utility is based on benefit minus cost according to equation 3, Meek teaches or at least suggests determining whether the model performance benefits exceed the training/resource cost. Thus, Meek’s stopping and returning of the model teaches or at least suggests determining completion of training for the model based on the value added-resource usage comparison. A person ordinary skill in the art would have been motivated to configure the active learning process by Phan to stop training when the expected model-performance benefit is sufficient relative to the additional labeling/training cost, because continuing to label and train additional samples after the desired cost-benefit condition is satisfied would unnecessarily consume computation time and labeling effort.)
Phan teaches the limitation “generating, for display on a user interface, a notification of completion of training of the machine learning model” (paragraph 63 “For example, in a hypothetical case where computer 101 is designed to provide a recommendation to an end user, this recommendation would typically be communicated from network module 115 of computer 101 through WAN 102 to EUD 103. In this way, EUD 103 can display, or otherwise present, the recommendation to an end user”, and paragraph 74 “Application 300 repeats the model training process, until the query budget B has been exhausted, all the available data has been labeled and used to train the model, the model meets a model performance criterion” Phan discloses repeating the model training process until the model meets a model performance criterion or all data the available data has been labeled and used to train the model, suggesting that the model complete its training. Phan further discloses communicating information to end user device regarding the application, wherein the device can display or otherwise present the information to the user. Because Phan teaches that training is repeated until completion and displaying information to the user, a person ordinary skill in the art would have been motivated to generate and display the notification indicating completion of the training when the model reaches it performance criterion or otherwise ends. Doing so would merely provide the user with the status and result of the training process and would predictably improve user’s experience in utilizing the machine learning model. Therefore, Phan teaches or at least suggests the claimed limitation.)
Regarding claim 6, depends on claim 3, thus the rejection of claim 3 is incorporated.
Meek teaches or at least suggests the limitation “responsive to determining that the amount of value added exceeds the amount of resource usage, determining completion of training for the machine learning model” (Page 3 section 2 equation 3 “The second ingredient is the utility or “goodness” of stopping with data set Di. We decompose this utility into a benefit and cost. Let mi denote the model learned with this data set. A natural measure of cost is proportional to the total time it takes to produce model mi. A natural measure of benefit is proportional to the accuracy of mi on the task to which that model will be applied ... We then choose the alternative among these two that maximizes our expected utility. If we choose alternative 1, we indeed stop, returning model mi and its accuracy score. If we choose alternative 2, we evaluate another decision problem” Meek discloses that the utility or “goodness” of stopping with a dataset Di is decomposed into a benefit and cost, where the cost is proportional to the time required to produce the model and the benefit is proportional to the accuracy of the model. Meek further discloses choosing the alternative that maximizes expected utility, and if alternative 1 is chosen, the system stops and returns the model and its accuracy score. Under the broadest reasonable interpretation, Meek’s benefit/accuracy corresponds to the claimed value added, and Meek’s cost/time corresponds to the claimed resource usage. Because Meek’s utility is based on benefit minus cost according to equation 3, Meek teaches or at least suggests determining whether the model performance benefits exceed the training/resource cost. Thus, Meek’s stopping and returning of the model teaches or at least suggests determining completion of training for the model based on the value added-resource usage comparison. A person ordinary skill in the art would have been motivated to configure the active learning process by Phan to stop training when the expected model-performance benefit is sufficient relative to the additional labeling/training cost, because continuing to label and train additional samples after the desired cost-benefit condition is satisfied would unnecessarily consume computation time and labeling effort.)
Phan teaches or at least suggests the limitation “receiving, from a remote device, one or more unseen samples” (paragraph 25 “Once a model training ending condition is satisfied, an embodiment uses the now-trained model to perform the function for which the model was trained. For example, if the model was being trained to identify and classify anomalies in medical images, an embodiment uses the now-trained model to identify and classify anomalies in new medical images that were not part of the original training data” Phan discloses once the training ending condition is satisfied, the model is employed to perform the function for which the model was trained such as to classify new image samples, thus suggest the model receive new unseen image sample that were not part of the original training data.)
Phan teaches or at least suggests the limitation “generating one or more classifications for the one or more unseen samples using the machine learning model” (paragraph 25 “Once a model training ending condition is satisfied, an embodiment uses the now-trained model to perform the function for which the model was trained. For example, if the model was being trained to identify and classify anomalies in medical images, an embodiment uses the now-trained model to identify and classify anomalies in new medical images that were not part of the original training data” Phan discloses once the training ending condition is satisfied, the model is employed to perform the function for which the model was trained, which is to classify new image samples.)
Phan teaches or at least suggests the limitation “transmitting, to a remote device, the one or more classifications” (paragraph 25 “Once a model training ending condition is satisfied, an embodiment uses the now-trained model to perform the function for which the model was trained. For example, if the model was being trained to identify and classify anomalies in medical images, an embodiment uses the now-trained model to identify and classify anomalies in new medical images that were not part of the original training data”, and paragraph 63 “End user device (EUD) 103 is any computer system that is used and controlled by an end user ... For example, in a hypothetical case where computer 101 is designed to provide a recommendation to an end user,”. Phan discloses once the training ending condition is satisfied, the model is employed to perform the function for which the model was trained, which is to classify new image samples and Phan further teaches end user device to receive recommendation, therefore teaches or at least suggests that the classification result is transmitted as recommendation to end user device to user.)
Regarding claim 7, depends on claim 3, thus the rejection of claim 3 is incorporated.
Phan teaches or at least suggests the limitation “determining, based on unlabeled samples of the dataset, a measure of uncertainty corresponding to each sample, wherein the measure is indicative of a confidence of the machine learning model in classifying each sample” (paragraph 22 “Once an embodiment has completed the M training epochs, an embodiment scores data in the unlabeled dataset. One embodiment scores all of the data in the unlabeled dataset, so that the next batch of data will be selected from as much data as possible. One embodiment computes an uncertainty score”, and paragraph 72 “Module 220 computes the uncertainty score as a weighted sum of all i, in which each element in the sum is an upper bound of the loss incurred on an unlabeled data point divided by the number of data points belonging to a particular class of data points i” Phan discloses computing an uncertainty score for selecting subset of data for labeling, which corresponds to the claimed measure of uncertainty under the broadest reasonable interpretation. Phan further teaches a loss incurred on an unlabeled data point divided by the number of data points to compute the uncertainty score, which suggest that the uncertainty score is indicative of the model’s confidence in classifying a label for the sample because loss/uncertainty reflects how unsure the model is regarding the sample. Thus, Phan teaches or at least suggests the limitation.)
Phan teaches or at least suggests the limitation “identifying the samples of the plurality of unlabeled samples having a threshold measure of uncertainty” (paragraph 73 “Module 220 selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset. One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset” Phan teaches that module computes and uncertainty score and an overall score based on the uncertainty score, diversity score, and class-imbalance score. Phan then selects, for labeling, the samples having the lowest scores from the unlabeled dataset. Because the overall score used for selection includes the uncertainty score as a component, selecting the lowest-scored samples teaches or at least suggests identifying samples that satisfy a threshold measure based at least in part on uncertainty. Under the broadest reasonable interpretation, the score cutoff corresponding to the selected samples operates as the claimed threshold measure of uncertainty, because sample meeting the cutoff are identified for labeling while samples outside the cutoff are not selected.)
Regarding claim 8, depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches or at least suggests the limitation “The method of claim 2, wherein selecting a first subset of the dataset comprises selecting samples based on the one or more user-defined target parameter values” (paragraph 18 “An embodiment receives a plurality of learning parameters, or sets one or more of the learning parameters to default values. The learning parameters control training of a model using active learning. One learning parameter is a query budget (denoted by B), which is the total number of samples from a dataset that can be selected for labelling during the active learning process.” Phan discloses setting a plurality of learning parameters to default values, such as the budget parameter to control the number of samples from a dataset that can be selected for labelling. A person ordinary skill in the art would have been motivate to set a parameter indicating a budget for selecting sample for labeling, thus Phan’s parameter corresponding to the claimed user-defined target parameter values for selecting the first subset of the dataset.)
Regarding claim 11, depends on claim 2, thus the rejection of claim 2 is incorporated.
Meek teaches or at least suggests the limitation “receiving a user selection of a user-defined valuation of the model performance improvement” (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time)” Meek discloses that α depends on the preferences of the decision maker and represents the incremental-benefit-to-cost-ratio. Under the broadest reasonable interpretation, this corresponds to receiving a user selection of a user-defined valuation of model performance improvement because the decision maker provides the preference used to value model-performance benefit relative to the runtime/cost of training the machine learning model.)
Meek teaches or at least suggests the limitation “determining, based on the user-defined valuation of the model performance improvement, the value added per unit of model performance improvement” (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time) ... For example, a decision maker can be asked the question “How long would you be willing to wait to increase the relative accuracy of the learned model by one percent?” If the answer is (e.g.) one hour, then α =0.01 benefit per hour”. Meek discloses that α is determined from the decision maker’s preference and should be assessed on a problem-by-problem basis. In other word, the decision maker first determines how long they would wait for a one percent accuracy improvement (user-defined valuation of the model performance improvement), then the unit benefit is obtained (value added), thus Meek teaches or at least suggest the claimed limitation.)
Regarding claim 12, depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches or at least suggests the limitation “receiving a user selection of a target number of samples to be labeled from the plurality of unlabeled samples” (paragraph 17 “uses a parametric function of a feature extractor parameter and a discriminator parameter to score a plurality of samples of a dataset of unlabeled data, selects, for labeling, a subset of the scored plurality of samples” Phan discloses select, for labeling, a subset of the scored plurality of samples, which corresponds to the user selection of a target number of samples to be labeled from the plurality of unlabeled samples, as claimed.)
Phan teaches or at least suggests the limitation “determining, based on the target number of samples to be labeled from the plurality of unlabeled samples, a threshold number of samples for labeling” (paragraph 73 “Module 220 selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset. One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset” Phan discloses selects, for labeling, a subset of the scored plurality of samples based on a score, such that selecting the lowest-scored samples teaches or at least suggests a threshold number of samples for labeling. Under the broadest reasonable interpretation, the score cutoff corresponding to the selected samples operates as the claimed threshold, because sample meeting the cutoff are identified for labeling while samples outside the cutoff are not selected.)
Regarding claim 13, depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches or at least suggests the limitation “receiving a user selection of an upper limit of total cost associated with labeling the plurality of unlabeled samples” (paragraph 73 “One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset. Module 220 also subtracts the number of samples in the subset to be labeled from the query budget B. If the query budget is greater than zero, there are samples remaining in the query budget, and thus module 220 sends the selected samples to be labeled.” Phan discloses a query budget which represents the number of samples that can be selected for labeling. Under the broadest reasonable interpretation, the query budget corresponds to the claimed upper limit of total cost associated with labeling the plurality of unlabeled samples, because labeling cost increases with the number of samples selected for labeling, and limiting the number of selected samples limits the total labeling cost.)
Phan teaches or at least suggests the limitation “determining, based on the upper limit of total cost associated with labeling, a maximum number of samples to select for labeling” (paragraph 73 “One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset. Module 220 also subtracts the number of samples in the subset to be labeled from the query budget B. If the query budget is greater than zero, there are samples remaining in the query budget, and thus module 220 sends the selected samples to be labeled.” Phan discloses subtracts the number of samples in the subset to be labeled from the query budget B, such that if the query budget is greater than zero, there are samples remaining in the query budget. Under the broadest reasonable interpretation, the query budget corresponds to the claimed upper limit of total cost associated with labeling the plurality of unlabeled samples, because labeling cost increases with the number of samples selected for labeling, and limiting the number of selected samples limits the total labeling cost. Thus, selecting all samples from the query budget teaches or at least suggests of selecting the maximum number of samples for labeling based on the upper limit, as claimed.)
Regarding claim 14 depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches the limitation “receiving a user selection of target model complexity and a target model performance” (paragraph 69 “Application 200 receives a plurality of learning parameters, or sets one or more of the learning parameters to default values. The learning parameters control training of a model using active learning. One learning parameter is a query budget (denoted by B), which is the total number of samples from a dataset that can be selected for labelling during the active learning process. Two other learning parameters are the number of model training cycles, or epochs, to be used in training the model (denoted by M) and a learning rate” , and paragraph 74 “Application 300 repeats the model training process, including use of the newly-labeled samples, until the query budget B has been exhausted, all the available data has been labeled and used to train the model, the model meets a model performance criterion (e.g., the model classifies more than a threshold percentage of samples correctly, or the model classifies more than a threshold percentage of a particular class of samples correctly)” Phan discloses application receives a plurality of parameters such as query budget, number of model training cycles, and learning rates, which corresponds to the user selection of target model complexity because these parameters control the extent and configuration of model training. Phan’s model performance criterion corresponds to the claimed target model performance because it defines a target condition for when the trained model perform sufficiently, such as classifying more than a threshold percentage of samples correctly. Thus, Phan teaches or at least suggests the claim.)
Meek teaches or at least suggests the limitation “determining, based on the user selection, a value added per unit of model performance improvement” (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time) ... For example, a decision maker can be asked the question “How long would you be willing to wait to increase the relative accuracy of the learned model by one percent?” If the answer is (e.g.) one hour, then α =0.01 benefit per hour” Meek teaches that α is a relative importance of benefit to runtime and depends on the preferences of the decision maker. Meek further explains that α may be viewed as an incremental-benefit-to-cost ratio having units benefit per time. Under the broadest reasonable interpretation, the incremental unit benefit (e.g., 0.01 benefit) obtained as the model improves through training over time corresponds to the claimed value added. Thus, Meek’s α corresponds to the claimed user input indicative of a value added per unit of model performance improvement, because α reflects the decision maker’s preference regarding how much model’s benefit is valued (e.g., 0.01 benefit) relative to the runtime/resource cost requirement to obtain that improvement.)
Regarding claim 15, which recites a computer program product, the applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps to the system of claim 1, thus the claim is rejected under similar rationale.
Regarding claim 16 depends on claim 15, thus the rejection of claim 15 is incorporated. The applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps to the system of claim 1, thus the claim is rejected under similar rationale.
Regarding claim 17 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim 4 above, because the claim recites similar limitations and processing steps to the system of claim 4, thus the claim is rejected under similar rationale.
Regarding claim 19 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim above, because the claim 6 recites similar limitations and processing steps to the system of claim 6, thus the claim is rejected under similar rationale.
Regarding claim 20 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim above, because the claim 7 recites similar limitations and processing steps to the system of claim 7, thus the claim is rejected under similar rationale.
Claims 5, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering), further in view of Chen et.al (US 11704577 B1)
Regarding claim 5 depends on claim 3, thus the rejection of claim 3 is incorporated.
Meek teaches or at least suggests the limitation “responsive to determining that the amount of value added exceeds the amount of resource usage, determining completion of training for the machine learning model” (Page 3 section 2 equation 3 “The second ingredient is the utility or “goodness” of stopping with data set Di. We decompose this utility into a benefit and cost. Let mi denote the model learned with this data set. A natural measure of cost is proportional to the total time it takes to produce model mi. A natural measure of benefit is proportional to the accuracy of mi on the task to which that model will be applied ... We then choose the alternative among these two that maximizes our expected utility. If we choose alternative 1, we indeed stop, returning model mi and its accuracy score. If we choose alternative 2, we evaluate another decision problem” Meek discloses that the utility or “goodness” of stopping with a dataset Di is decomposed into a benefit and cost, where the cost is proportional to the time required to produce the model and the benefit is proportional to the accuracy of the model. Meek further discloses choosing the alternative that maximizes expected utility, and if alternative 1 is chosen, the system stops and returns the model and its accuracy score. Under the broadest reasonable interpretation, Meek’s benefit/accuracy corresponds to the claimed value added, and Meek’s cost/time corresponds to the claimed resource usage. Because Meek’s utility is based on benefit minus cost according to equation 3, Meek teaches or at least suggests determining whether the model performance benefits exceed the training/resource cost. Thus, Meek’s stopping and returning of the model teaches or at least suggests determining completion of training for the model based on the value added-resource usage comparison. A person ordinary skill in the art would have been motivated to configure the active learning process by Phan to stop training when the expected model-performance benefit is sufficient relative to the additional labeling/training cost, because continuing to label and train additional samples after the desired cost-benefit condition is satisfied would unnecessarily consume computation time and labeling effort.)
Phan/Meek does not teach the limitation “generating one or more data files comprising parameters of the machine learning model in a standardized format” However, Chen teaches or at least suggests the limitation (Column 2 lines 12-20“According to some embodiments, a unified inference framework for heterogenous edge devices is provided that can accept machine learning (ML) models from a user in any of multiple formats (as generated by multiple different frameworks), convert and optimize these ML models for use by heterogeneous “edge” devices having heterogeneous computing resources, and deploy these ML models for use in one or more edge devices of one or more different types”, and Column 6 lines 45-51 “... using a conversion library/module that identifies certain values (e.g., weights) in model files generated by a first framework and inserts them in a different format (or location) within files adherent to a different framework or format (e.g., a different framework's format, a standardized “generic” format such as the Open Neural Network eXchange format “ONNX”, etc.)” Chen discloses a unified inference framework for heterogenous edge devices that can accept machine learning (ML) models from a user in any of multiple formats, and convert and optimize these ML models for use by heterogeneous “edge” devices. The framework use a conversion library/module that can identifies certain values (e.g., weights) in model files generated and convert them to a standardized “generic” format, which corresponds to the claimed generating one or more data files of the machine learning model in a standardized format, and the data files can be the budget parameter as disclosed by Phan in view of the combination below.)
Phan/Meek does not teach the limitation “transmitting, to a remote device, the one or more data files” However, Chen teaches or at least suggests the limitation (Column lines Column 2 lines 12-20 “According to some embodiments, a unified inference framework for heterogenous edge devices is provided that can accept machine learning (ML) models from a user in any of multiple formats (as generated by multiple different frameworks), convert and optimize these ML models for use by heterogeneous “edge” devices having heterogeneous computing resources, and deploy these ML models for use in one or more edge devices of one or more different types” Chen discloses converting and optimizing the ML models in any format into a standardized format, which can be deployed for use in one or more edge devices, suggesting the transmitting of the one or more data files, as claimed because the ML models having their data being formatted is deployed for use in one or more edge devices.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan, and the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek with the teaching of a unified inference framework for heterogenous edge devices that can accept machine learning (ML) models from a user in any of multiple formats, and convert and optimize these ML models for use by heterogeneous “edge” devices by Chen. The motivation to do so is referred to in Chen’s disclosure (Column 3 lines 20-26 “Embodiments disclosed herein allow users to implement machine learning at the edge in a manageable and sustainable way. Further, some embodiments can provide a flywheel for ML on connected devices, allowing users to deploy models across potentially millions of devices and continually improve them. Data can be collected in a secure manner from devices and used to train new models, which can be re-deployed to devices. This in turn generates new data for additional cycles of re-training and re-deployment, where each cycle can increase availability of the system and improve device experiences” Chen discloses the benefit of implementing machine learning at the edge in a manageable and sustainable way, which allow users to deploy models across potentially millions of devices and continually improve them, and further allow new data for additional cycles of re-training and re-deployment, which increase availability of the system and improve device experiences. Therefore, the teaching by Phan/Meek can be further improved by combining with the teaching by Chen to format any information regarding the training of the active machine learning model into a standardized format for use by multiple devices and users, thereby continually improve the model. One of ordinary skill in the art would have been motivated to format regarding parameters as disclosed by Phan above such as the budget parameter to ensure that when the model is utilized across other edge devices, the model would behave appropriately given the amount of data at each edge devices.)
Regarding claim 18 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim above, because the claim 5 recites similar limitations and processing steps to the system of claim 5, thus the claim is rejected under similar rationale.
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering), further in view of Rink et.al (US 20210287119 A1)
Regarding claim 9 depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan/Meek does not teach the limitation “The method of claim 2, wherein the one or more user-defined target parameter values correspond to target parameters indicative of a threshold for bias and variance for the machine learning model” However, Rink teaches or at least suggests the part of the limitation (paragraph 55 “At 218, the processor may transform the raw data set 204 into a transformed data set based on pre-processing thresholds 216 ... To illustrate, the thresholds may be for identifying whether data entries may be subject to disparate impact beyond a threshold value. In some examples, thresholds may be for identifying skewness or probability/imbalance in received data sets”, and paragraph 56 “In some examples, operations associated with pre-processing thresholds 216 may include operations for identifying data bias based on statistical bias (e.g., skew, kurtosis, variance), minority class recognition, disparity impact identification, or sensitive attribute identification” Rink discloses systems and methods for mitigation bias in machine learning model output. Within the disclosure Rink discloses pre-processing thresholds for identifying whether data entries are suitable to transform into data set. Pre-processing thresholds may include operations for identifying data bias based on statistical bias (e.g., skew, kurtosis, variance). Under the broadest reasonable interpretation, -processing thresholds correspond to the claimed target parameter values because they define threshold conditions used to identify bias and variance related issues in the data use for generating or refining the ML model. Thus, Rink teaches or at least suggest the claimed limitation.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan, and the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek with the teaching of pre-processing threshold for bias and variance by Rink. The motivation to do so is referred to in Rink’s disclosure (paragraph 71 “By pre-processing the raw data set 204 prior to providing the data set as an input for model generation or to the first in-processing stage 230, potential biases within the raw data set 204 may be identified or reduced, thereby reducing data entries that may be associated with skewed data or other unintended data artifacts prior to generating or refining a machine learning model” Rink discloses the improvement in using the preprocessing threshold for identifying or reducing potential biases within the raw data thereby reducing data entries that may be associated with skewed data or other unintended data artifacts. A person ordinary skill in the art would have been motivated to use such bias/variance threshold as parameters in Phan’s active learning process to control the quality of training data and reduce bias or variance before or during model training, thereby improving the resulting model.)
Claims 10 is rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering), further in view of Sengupta et.al (US 20230072171 A1)
Regarding claim 10 depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches a part of the limitation “The method of claim 2, ... wherein the user input indicative of a value added per unit of model performance improvement comprises a cost associated with labeling a sample of the plurality of unlabeled samples” (paragraph 73 “Module 220 selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset. One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset. Module 220 also subtracts the number of samples in the subset to be labeled from the query budget B” Phan discloses selecting for labeling a subset of samples and subtracting the number of samples in the subset to be labeled from the query budget. Under the broadest reasonable interpretation, the query budget corresponds to a labeling cost-parameter because it limits how many samples may be selected for labeling, and each selected samples consumes part of the budget. Thus, Phan teaches or at least suggests the cost associated with labeling a sample of the plurality of unlabeled samples as claimed.)
Phan/Meek does not teach a part of the limitation “The method of claim 2, wherein the unit of model performance improvement comprises a unit of improvement in precision, recall, or F1-score ...” However, Sengupta teaches the part of the limitation (paragraph 58 “The method may include calculating precision and recall for the trained model based on these prediction scores output by the trained model (e.g., performing operations 506, 508, and 510). Precision and recall are indicators of a machine learning model's performance”, and paragraph 60 “The method may include using the true class of the selection of the unlabeled input dataset (which is unknown to the machine learning model) and the prediction scores as inputs to calculate precision and recall for the trained machine learning model” Sengupta discloses a system and method for training and refining machine learning models. Within the disclosure, Sengupta discloses calculating precision and recall for the trained model as precision and recall are indicators of a machine learning model's performance, which corresponds to the claimed model performance improvement comprises unit of improvement in precision, recall, or F1-score.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan, and the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek with the teaching of calculating precision and recall as indicators of a machine learning model's performance by Sengupta. The motivation to do so is referred to in Sengupta‘s disclosure (paragraph 5 “However, real world data may contain intents the training data does not include. Accordingly, these models do not perform well at identifying these unknown intents”, and paragraph 7 “Precision and recall are often in tension. Improving precision may reduce recall, and improving recall may reduce precision. Thus, tuning a machine learning model can be complicated, as one improvement can undo another. The disclosed system and method overcome this issue by applying an optimization technique (e.g., multi-objective optimization) to optimize precision and recall to find optimal threshold values for prediction values” Sengupta discloses the system and method applying an optimization technique (e.g., multi-objective optimization) to optimize precision and recall in producing the result of machine learning model, which help tuning the trained machine learning model, wherein a machine learning model can use precision and recall value to indicate its improvement over performing its learning. One of ordinary skill in the art would have been motivated to combine the teaching of Phan/Meek with the teaching of precision and recall as the performance indicator for performance improvement because the combination would have predictably allowed the system to measure model-performance improvement while determining whether labeling/training improves the model sufficiently.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY TU DIEP whose telephone number is (703)756-1738. The examiner can normally be reached M-F 8-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY T DIEP/ Examiner, Art Unit 2123
/ALEXEY SHMATOV/ Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights