Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The arguments and amendments filed 08/21/2026 have been entered. Claims 1-8, 10-20 remain pending in the application.
Applicant’s argument, with respect to claim rejections of claims 1-20 under 35 U.S.C 103 filed 08/21/2026 have been considered and they are not persuasive. Therefore, the previous rejections as set forth in the previous office action will be maintained.
Applicant argues that the combination of Phan and Meek fails to teach or suggest the amended limitations. First, Applicant argues that Meek’s α represents the relative importance of benefit to runtime, having units of benefit per time, rather than the claimed “user input indicative of a value per unit of model performance improvement.” Second, Applicant argues that Meek’s learning curve merely represents model performance as a function of training-sample size and is not generated using the claimed user-provided value, and therefore does not teach the claimed relationship between resource usage and value per unit of model-performance improvement. Third, Applicant argues that Phan’s query budget is a predetermined number of samples and that Meek likewise uses predetermined training-set sizes, such that neither Phan nor Meek teaches or suggests selecting a second subset comprising a number of samples identified from the margin curve as a number at which the labeling cost associated with labeling additional samples exceeds a corresponding predicted value of model-performance improvement.
Regarding the first argument, Applicant’s characterization of Meek’s α does not account for the decision-maker input from which α is determined. Meek teaches that α depends on the preferences of the decision maker and provides an example of asking the decision maker how long he or she would be willing to wait to increase the relative accuracy of the learned model by one percent. Meek further provides that, where the decision maker is willing to wait one hour for such an increase, α = 0.01 benefit per hour. Thus, Meek’s decision-maker input indicates the amount of considered resource worthwhile for obtaining an increase in model accuracy, and α represents the relationship between such model-performance benefit and resource expenditure. Accordingly, although Meek expresses this relationship in terms of benefit and runtime, the decision-maker input used to determine α teaches or at least suggests the claimed “user input indicative of a value per unit of model performance improvement.”
Regarding the second argument, Applicant’s characterization of Meek’s learning curve as merely model performance versus sample size does not account for Meek’s decision-theoretic use of the learning-curve information. Meek teaches that its learning-curve sampling method progressively evaluates model performance using increasing amounts of training data while considering the corresponding computational cost. Meek further discloses that the decision maker’s preference, represented by α, is used to determine whether the expected benefit of additional model training justifies the corresponding runtime cost. When further training is justified, the learning-curve sampling process continues and additional learning-curve points may be obtained; otherwise, the sampling process terminates. Thus, the decision maker’s preference influences whether additional learning-curve points are generated and, consequently, the extent to which the learning curve is developed. Accordingly, Meek teaches or at least suggests using the decision-maker input as part of the learning-curve generation process, corresponding to the claimed generating, using the user input, a margin curve. Furthermore, in view of Phan’s active-learning process, in which selected samples are obtained for labeling and subsequently used for model training, it would have been obvious to apply Meek’s learning-curve sampling method to determine whether additional labeling and training are justified by the expected model-performance improvement relative to resource usage. Accordingly, Meek teaches or at least suggests generating of the learning curve comprises using the decision maker’s preference, and the combined teachings of Phan and Meek teach or at least suggest the claimed margin-curve relationship between resource usage, value associated with model-performance improvement, and further data selection for labeling.
Regarding the third argument, Applicant’s assertion that Meek merely relies upon predetermined sample sizes does not account for Meek’s express disclosure of adaptive sample-size selection. Meek explains that the sequence of data sets may either be fixed in advance or chosen adaptively as the data sets are examined. For an adaptively selected sequence, Meek teaches using learning-curve points obtained from D1 through Di, together with a model of the shape of the learning curve, to estimate future points along the curve. Meek expressly states that these estimates can be used to expand the alternatives in the sequential decision problem to include “a choice about the number of cases in Di+1.” Thus, Meek does not merely teach deciding whether to stop at a predetermined sample size; Meek expressly suggests using estimated future points from the learning curve to determine how many cases should be included in the next data set, Di+1. Meek further evaluates the expected incremental benefit and expected incremental cost associated with proceeding from D1 to Di and provides that additional subsets should not be considered when the expected incremental-benefit-to-cost ratio falls below α. Meek additionally explains that, because successive sample sizes may not be closely spaced, its criterion may result in stopping at Di+1 even though the expected utility at Di is slightly greater, i.e., the criterion may stop “one step late” Thus, Meek teaches or at least suggests both using estimated future points along the cost-benefit relationship to determine the number of cases in Di+1 and identifying a subsequent sample size at which the additional expected cost is no longer justified by the corresponding expected benefit. When these teachings are applied to Phan’s iterative active-learning paradigm, wherein the subsequent subset comprises additional samples selected for labeling and subsequent model training, the combination suggests determining the number of samples in the subsequent subset based on the cost-benefit relationship reflected by the curve, including at a point where the resource cost associated with obtaining and using the additional labeled samples exceeds the corresponding expected benefit from further model-performance improvement. Accordingly, Phan in view of Meek teaches or at least suggests the claimed second subset comprising a number of samples identified from the margin curve as a number at which labeling cost associated with labeling additional samples exceeds a corresponding predicted value of model-performance improvement.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-7, 11-16, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering).
Regarding claim 1,
Phan teaches the limitation “a mobile device comprising one or more processors” (paragraph 28 “Where an embodiment is described using a mobile device, any type of data storage device suitable for use with the mobile device may provide the data to such embodiment, either locally at the mobile device or over a data network”, and paragraph 54 “Processor set 110 includes one, or more, computer processors of any type now known or to be developed in the future”. Phan discloses a method and system and computer program product of active learning, which can be performed on a mobile device comprising one or more processors.)
Phan teaches or at least suggests the limitation “one or more a non-transitory computer readable media comprising instructions recorded thereon that when executed by the one or more processors causes operations” (paragraph 51 “A computer program product embodiment (“CPP embodiment” or “CPP”) is a term used in the present disclosure to describe any set of one, or more, storage media (also called “mediums”) collectively included in a set of one, or more, storage devices that collectively include machine readable code corresponding to instructions and/or data for performing computer operations ... A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se” Phan discloses a computer readable storage medium not to be construed as storage in the form of transitory signals per se, and comprising machine readable code corresponding to instructions and/or data for performing computer operations when executed by the one or more processors.)
Phan teaches or at least suggests part of the limitation “obtaining (1) one or more user-defined target parameter values for data labeling, ... (3) a dataset comprising a plurality of unlabeled samples” (paragraph 18 “An embodiment receives a plurality of learning parameters, or sets one or more of the learning parameters to default values ... One learning parameter is a query budget (denoted by B), which is the total number of samples from a dataset that can be selected for labelling ... An embodiment also receives ... an unlabeled dataset” Phan discloses the embodiment of active learning comprises of learning parameter of total number of samples from a dataset that can be selected for labelling, which corresponds to the claimed user-defined target parameter values for data labeling, and an unlabeled dataset, which corresponds to the dataset comprising a plurality of unlabeled samples, as claimed.)
Phan teaches the limitation “selecting, based on the one or more user-defined target parameter values, a first subset of the dataset comprising first samples of the plurality of unlabeled samples” (paragraph 24 “An embodiment selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset ... An embodiment also subtracts the number of samples in the subset to be labeled from the query budget B” Phan discloses selecting a subset of plurality of samples from the unlabeled dataset based on the budget parameter, which corresponds to the selection of the first unlabeled samples for the first subset based on the one or more user-defined target parameter values, as claimed.)
Phan teaches or at least suggests the limitation “transmitting, to a remote device, a first request to label the first samples of the first subset, wherein the request comprises the samples of the first subset” (paragraph 24 “An embodiment selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset ... thus an embodiment sends the selected samples to be labeled. The labeling is performed by an automated labeling service, a human expert, or another presently available technique.” Phan discloses after the samples are selected from the unlabeled dataset, they are labeled by an automated labeling service or a human expert, which corresponds to the request comprising the samples of the first subset for labeling, as claimed.)
Phan teaches or at least suggests the limitation “receiving, from the remote device, a first comprising first label data indicating a classification for each of the first samples of the first subset and the first samples of the first subset” (paragraph 18 “The labeled dataset includes a label for each piece of data, .... In other words, the label represents a trained model's correct response to the data”, and paragraph 25 “An embodiment receives the now-labeled selected samples and adds them to the set of labeled samples. An embodiment repeats the model training process, including use of the newly-labeled samples, ..., all the available data has been labeled” Phan discloses the embodiment receives the now-labeled selected samples and adds them to the set of labeled samples to obtain an updated labeled training dataset with a label for each piece of data, which corresponds to the receiving a first training dataset based on the first subset comprising of label data and the samples of the first subset, as claimed. Furthermore, the label for each piece of data represents a trained model's correct response to the data, which corresponds to the label data indicates a classification for each sample, as claimed.)
Phan teaches or at least suggests a part of the limitation “… training, using the first training dataset, a machine learning model, to obtain a first trained machine learning model” (paragraph 25 “An embodiment receives the now-labeled selected samples and adds them to the set of labeled samples. An embodiment repeats the model training process, including use of the newly-labeled samples, ..., all the available data has been labeled” Phan discloses the machine learning model repeats the model training process, suggesting that the model perform the model training using the updated labeled training dataset, which corresponds to the claimed training the machine learning model using the first training dataset.)
Phan teaches or at least suggests the limitation “transmitting, to the remote device, a second request to label samples of the second subset, the second request comprising the second samples of the second subset” (paragraph 19 “In each expression, t denotes a number of times unlabeled data has so far been selected”, paragraph 74 “Application 300 repeats the model training process, including use of the newly-labeled samples, until the query budget B has been exhausted, all the available data has been labeled and used to train the model, the model meets a model performance criterion”, and paragraph 78 “In line 9, pseudocode 310 subtracts the number of samples in the subset to be labeled from the query budget B and adds the now-labeled selected samples to the set of labeled samples. In line 10, pseudocode 310 increments t, and in line 11 pseudocode 310 returns to line 3 if query budget B is still greater than zero” Phan discloses the application further repeat the training process. For instance, the number of times unlabeled data has been selected is denoted by t, such that as t increments, more data is selected for labeling and update the training dataset for continuing training the model. One of ordinary skill in the art would have been able to configure an initial t=1 as the first time the first subset of unlabeled data is selected for labeling and training the model corresponding to the disclosure above, and t=2 as the second time the unlabeled data is further selected for labeling and further training the model, thus the whole process of selecting, labeling, and receiving labeled subset of data is repeated for the t=2 time similarly to the process at t=1 time, which corresponds to a second request comprising samples of a second subset for labeling, as claimed. The process to configure the second subset may be configured in view of Meek’s teaching below.)
Phan teaches or at least suggests the limitation “receiving, from the remote device, a second training dataset based on the second subset comprising second label data indicating a classification for each of the second samples of the second subset and the second samples of the second subset” (paragraph 19 “In each expression, t denotes a number of times unlabeled data has so far been selected”, and paragraph 78 “In line 10, pseudocode 310 increments t, and in line 11 pseudocode 310 returns to line 3 if query budget B is still greater than zero” Phan discloses repeating the training process as t increments, such that more unlabeled data is selected and label, and the application receive the second labeled subset of dataset (at t=2) to configure an updated training dataset, which corresponds to the receiving of the second training dataset, as claimed.)
Phan teaches or at least suggests the limitation “updating, using the second training dataset, the machine learning model” (paragraph 74 “Application 300 repeats the model training process, including use of the newly-labeled samples”, and paragraph 78 “In line 10, pseudocode 310 increments t, and in line 11 pseudocode 310 returns to line 3 if query budget B is still greater than zero” Phan discloses repeating the training process as t increments, using the second labeled subset of dataset (at t=2) to configure an updated training dataset to further train the machine learning model.)
Phan does not teach “determining an amount of resource usage associated with obtaining the first label data …” However, Phan in view Meek teaches or at least suggests this limitation (page 3 section 2 “When we examine data set Di, we apply some training method to it, gather information about that application—for example, the accuracy of the resulting model and the time it took to learn that model”, page 4 section 2 “To scale cost, we simply write cost(mi)=α·runtimei ... where runtimei is total time required to produce model mi”. Meek discloses applying a training method to a dataset and gathering information regarding the training process, including the time taken to learn the resulting model. Meek further determines the cost of producing the model based on the total runtime required for training. Thus, Meek’s determination of training runtime corresponds to determining an amount of computational resource usage associated with training the machine learning model. In view of Phan’s previously discussed process of obtaining labeled samples and training the model using the labeled samples, Phan in view of Meek teaches or at least suggests determining an amount of resource usage associated with obtaining the first label data and training the machine learning model, as claimed.)
Phan does not teach part of the limitation “... (2) a user input indicative of a value per unit of model performance improvement...”. However, Phan in view Meek teaches or at least suggests this limitation (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time) ... For example, a decision maker can be asked the question “How long would you be willing to wait to increase the relative accuracy of the learned model by one percent?” If the answer is (e.g.) one hour, then α =0.01 benefit per hour” Meek discloses that α depends on the preferences of the decision maker and provides an example in which the decision maker is asked how long he or she would be willing to wait for a one-percent improvement in model accuracy. Meek further discloses that a willingness to wait one hour for such an improvement results in α = 0.01 benefit per hour. Thus, the decision maker’s expressed preference indicates how much time the decision maker considers worthwhile for obtaining a unit of model-performance improvement, which corresponds to the claimed user input indicative of a value per unit of model performance improvement, as claimed.)
Phan does not teach the limitation “generating, using the user input indicative of the value per unit of model performance improvement, a margin curve of a relationship between resource usage and value per unit of model performance improvement”. However, Phan in view Meek teaches or at least suggests this limitation (Figure 1 “A hypothetical plot of expected benefit versus expected run time”, page 2 section 1 “Learning-curve sampling methods rely on two basic observations: (1) the computational cost of learning a model increases as a function of the size of the training data, and (2) the performance/accuracy of a model has diminishing improvements as a function of the size of the training data. The curve describing the performance as a function of the sample size of the training data is often called the learning curve ... Thus, a learning curve sampling method monitors the increasing costs and performance as larger and larger amounts of data are used for training, and terminates learning when the increasing costs outweigh the benefit of increasing performance”, page 5 section 2 “α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time), below which additional subsets of data should not be considered. We refer to α as our stopping threshold and Equation 4 as our stopping criterion”, page 5 section 2 “Now, consider the difference between the height of the curve and the height of a line with slope α through the origin (also shown in the figure) as a function of expected run time. This difference represents expected utility as a function of expected run time”, and page 6 section 2 “according to the MEU principle, we should stop at a stage corresponding to point B in the figure. Just beyond this point, the ratio of expected incremental benefit to expected incremental run time falls below α”. Meek discloses that its learning-curve sampling method monitors increasing computational costs and model-performance improvements as larger amounts of training data are used, and that the decision-maker preference α, as discussed above, serves as a stopping threshold for determining whether additional training subsets should be considered. Since additional training may produce further learning-curve points, α influences whether the learning-curve sampling process continues or terminates, thereby affecting the continued generation of the learning curve. Under the broadest reasonable interpretation consistent with Applicant’s specification, the claim does not specify how the user input must be used in generating the margin curve. A person of ordinary skill in the art would have understood that using α to determine whether additional learning-curve points should be obtained allows further training only when the expected model-performance benefit justifies the associated computational cost, thereby avoiding unnecessary resource usage. Thus, Meek teaches or at least suggests using the decision-maker input in generating the learning curve, corresponding to the claimed generating of margin curve suing the user input.)
Phan does not teach the limitation “responsive to determining, based on the margin curve, the amount of resource usage exceeds a predicted value, selecting a second subset of the dataset, selecting a second subset of the dataset comprising a number of samples identified from the margin curve as a number at which labeling cost associated with labeling additional samples exceeds a corresponding predicted value of model performance improvement”. However, Phan in view Meek teaches or at least suggests this limitation (Page 2 section 2 “As we have discussed, the basic idea of a learning-curve sampling method is to iteratively apply a training algorithm to larger and larger subsets of the data, until the future expected costs outweigh the future expected benefits associated with the training ... Given a data set D, let D1,D2,...,Dn = D denote the sequence of data sets that are examined in the process of finding the appropriate sample size ... The second ingredient is the utility or “goodness” of stopping with data set Di. We decompose this utility into a benefit and cost. Let mi denote the model learned with this data set. A natural measure of cost is proportional to the total time it takes to produce model mi. A natural measure of benefit is proportional to the accuracy of mi on the task to which that model will be applied. Of course, the measure of accuracy will depend on the task at hand”, and page 6 section 2 “according to the MEU principle, we should stop at a stage corresponding to point B in the figure. Just beyond this point, the ratio of expected incremental benefit to expected incremental run time falls below α ... For example, suppose points A and C in the figure correspond to stages Di and Di+1. In this case, although our criterion yields a stopping point at stage i + 1, the expected utility of stopping at stage i is slightly greater. In general, we may stop one step late, but never early—that is, our stopping criterion is conservative.” Meek discloses identifying an appropriate sample size from its learning-curve sampling method based on the expected costs and benefits of further model training. Meek’s runtime corresponds to the claimed resource usage, while its expected benefit based on model accuracy corresponds to the claimed predicted value of model-performance improvement. Meek further discloses identifying a stopping stage at which the expected incremental benefit relative to runtime falls below the decision-maker’s preference α, indicating that the expected incremental cost of further training outweighs the corresponding expected incremental benefit. Meek also discloses that its stopping criterion may yield a stopping stage at i+1, even though stopping at stage i would provide slightly greater expected utility, thereby suggesting selection of a dataset at a stage where further resource usage is no longer justified by the expected benefit. In view of Phan’s selection of additional samples for labeling and subsequent model training, corresponding to the claimed second subset, a person of ordinary skill in the art would have found it obvious to apply Meek’s learning-curve-based stopping criterion to Phan’s active-learning process because obtaining additional labeled samples provides training data for improving model performance. Accordingly, the combination would have suggested using Meek’s identified stopping stage at i+1, where the expected incremental cost of further training outweighs the corresponding expected incremental benefit, to determine the number of samples to include in Phan’s second subset by considering the cost of labeling and using additional samples. Thus, applying Meek’s stopping criterion to Phan’s labeling process would have suggested selecting the second subset at a stage corresponding to Meek’s one-step-late stopping stage, where further resource usage is no longer justified by the expected incremental benefit, thereby suggesting selection of a number of samples at which the expected labeling cost exceeds the corresponding expected benefit of further model training, as claimed.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan with the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek. The motivation to do so is referred to in Meek’s disclosure (page 1 section 1 “In situations where one has access to massive amounts of data, the cost of building a statistical model can be significant if not insurmountable. A common practice is to build the model using a (usually random) sample of the examples or cases in the data. In so doing, however, the choice of the number of cases to use is far from clear. In this paper, we examine the learning-curve sampling method, an algorithmic approach to choosing an appropriate sample size for training”, and page 21 section 8 “Finally, our approach of using computationally efficient abbreviated training methods for determining the appropriate number of training cases can be—in principle—applied to various iterative training methods” Meek discloses the benefit of the proposed method in situations where one has access to massive amounts of data and the cost of building a statistical model can be significant if not insurmountable. The approach by Meek is computationally efficient for determining the appropriate number of training cases and can be applied to various iterative training method. Given that Phan discloses the active learning method that continue to receive data for labeling and training, but the cost of labeling is high (Phan at paragraph 2 “Active learning reduces the size of the training set needed to perform supervised learning, which is especially beneficial when the unlabeled data is abundant but the cost of labeling data is high”). Thus, the teaching by Meek can be applied to cure the deficiency of the cost of labeling data is high by Phan. A person ordinary skill in the art would have been motivated to use Meek’s learning curve determination to select an appropriate number of training samples for labeling in view of Phan’s active learning system and therefore improve the active learning machine learning model.)
Regarding claim 2 which recites a method, the applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps to the system of claim 1, thus the claim is rejected under similar rationale.
Regarding claim 3 depends on claim 2, thus the rejection of claim 2 is incorporated. The applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps, thus the claim is rejected under similar rationale.
Regarding claim 4 depends on claim 3, thus the rejection of claim 3 is incorporated.
Phan teaches or at least suggests the limitation “determining whether training for the machine learning model is complete or incomplete” (paragraph 71 “Once module 210 has completed the M training epochs”, and paragraph 77 “Once pseudocode 310 has completed the M training epochs” Phan discloses the module and pseudocode command complete the training epoch, which teaches or at least suggest the determination by the pseudocodes/module of whether the machine learning model complete the training, thereby teaches or at least suggests determining whether training for the machine learning model is complete or incomplete, as claimed.)
Phan teaches the limitation “generating, for display on a user interface, a notification of completion of training of the machine learning model” (paragraph 63 “For example, in a hypothetical case where computer 101 is designed to provide a recommendation to an end user, this recommendation would typically be communicated from network module 115 of computer 101 through WAN 102 to EUD 103. In this way, EUD 103 can display, or otherwise present, the recommendation to an end user”, and paragraph 74 “Application 300 repeats the model training process, until the query budget B has been exhausted, all the available data has been labeled and used to train the model, the model meets a model performance criterion” Phan discloses repeating the model training process until the model meets a model performance criterion or all data the available data has been labeled and used to train the model, suggesting that the model complete its training. Phan further discloses communicating information to end user device regarding the application, wherein the device can display or otherwise present the information to the user. Because Phan teaches that training is repeated until completion and displaying information to the user, a person ordinary skill in the art would have been motivated to generate and display the notification indicating completion of the training when the model reaches it performance criterion or otherwise ends. Doing so would merely provide the user with the status and result of the training process and would predictably improve user’s experience in utilizing the machine learning model. Therefore, Phan teaches or at least suggests the claimed limitation.)
Regarding claim 6, depends on claim 3, thus the rejection of claim 3 is incorporated.
Phan teaches or at least suggests the limitation “determining whether training for the machine learning model is complete or incomplete” (paragraph 71 “Once module 210 has completed the M training epochs”, and paragraph 77 “Once pseudocode 310 has completed the M training epochs” Phan discloses the module and pseudocode command complete the training epoch, which teaches or at least suggest the determination by the pseudocodes/module of whether the machine learning model complete the training, thereby teaches or at least suggests determining whether training for the machine learning model is complete or incomplete, as claimed.)
Phan teaches or at least suggests the limitation “receiving, from the remote device, one or more unseen samples” (paragraph 25 “Once a model training ending condition is satisfied, an embodiment uses the now-trained model to perform the function for which the model was trained. For example, if the model was being trained to identify and classify anomalies in medical images, an embodiment uses the now-trained model to identify and classify anomalies in new medical images that were not part of the original training data” Phan discloses once the training ending condition is satisfied, the model is employed to perform the function for which the model was trained such as to classify new image samples, thus suggest the model receive new unseen image sample that were not part of the original training data.)
Phan teaches or at least suggests the limitation “generating one or more classifications for the one or more unseen samples using the machine learning model” (paragraph 25 “Once a model training ending condition is satisfied, an embodiment uses the now-trained model to perform the function for which the model was trained. For example, if the model was being trained to identify and classify anomalies in medical images, an embodiment uses the now-trained model to identify and classify anomalies in new medical images that were not part of the original training data” Phan discloses once the training ending condition is satisfied, the model is employed to perform the function for which the model was trained, which is to classify new image samples.)
Phan teaches or at least suggests the limitation “transmitting, to the remote device, the one or more classifications” (paragraph 25 “Once a model training ending condition is satisfied, an embodiment uses the now-trained model to perform the function for which the model was trained. For example, if the model was being trained to identify and classify anomalies in medical images, an embodiment uses the now-trained model to identify and classify anomalies in new medical images that were not part of the original training data”, and paragraph 63 “End user device (EUD) 103 is any computer system that is used and controlled by an end user ... For example, in a hypothetical case where computer 101 is designed to provide a recommendation to an end user,”. Phan discloses once the training ending condition is satisfied, the model is employed to perform the function for which the model was trained, which is to classify new image samples and Phan further teaches end user device to receive recommendation, therefore teaches or at least suggests that the classification result is transmitted as recommendation to end user device to user.)
Regarding claim 7, depends on claim 3, thus the rejection of claim 3 is incorporated.
Phan teaches or at least suggests the limitation “determining, based on unlabeled samples of the dataset, a measure of uncertainty corresponding to each unlabeled sample indicative of a confidence of the machine learning model in classifying each sample” (paragraph 22 “Once an embodiment has completed the M training epochs, an embodiment scores data in the unlabeled dataset. One embodiment scores all of the data in the unlabeled dataset, so that the next batch of data will be selected from as much data as possible. One embodiment computes an uncertainty score”, and paragraph 72 “Module 220 computes the uncertainty score as a weighted sum of all i, in which each element in the sum is an upper bound of the loss incurred on an unlabeled data point divided by the number of data points belonging to a particular class of data points i” Phan discloses computing an uncertainty score for selecting subset of data for labeling, which corresponds to the claimed measure of uncertainty under the broadest reasonable interpretation. Phan further teaches a loss incurred on an unlabeled data point divided by the number of data points to compute the uncertainty score, which suggest that the uncertainty score is indicative of the model’s confidence in classifying a label for the sample because loss/uncertainty reflects how unsure the model is regarding the sample. Thus, Phan teaches or at least suggests the limitation.)
Phan teaches or at least suggests the limitation “identifying one or more samples of the unlabeled samples having a threshold measure of uncertainty” (paragraph 73 “Module 220 selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset. One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset” Phan teaches that module computes and uncertainty score and an overall score based on the uncertainty score, diversity score, and class-imbalance score. Phan then selects, for labeling, the samples having the lowest scores from the unlabeled dataset. Because the overall score used for selection includes the uncertainty score as a component, selecting the lowest-scored samples teaches or at least suggests identifying samples that satisfy a threshold measure based at least in part on uncertainty. Under the broadest reasonable interpretation, the score cutoff corresponding to the selected samples operates as the claimed threshold measure of uncertainty, because sample meeting the cutoff are identified for labeling while samples outside the cutoff are not selected.)
Regarding claim 11, depends on claim 2, thus the rejection of claim 2 is incorporated.
Meek teaches or at least suggests the limitation “receiving a user selection of a user-defined valuation of the model performance improvement” (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time)” Meek discloses that α depends on the preferences of the decision maker and provides an example in which the decision maker expresses how long he or she would be willing to wait for an improvement in model accuracy. Under the broadest reasonable interpretation, the decision maker’s expressed preference teaches or at least suggests to the claimed user selection of a user-defined valuation of model-performance improvement, because it specifies how much runtime the decision maker considers worthwhile for obtaining an improvement in model performance.)
Meek teaches or at least suggests the limitation “determining, based on the user-defined valuation of the model performance improvement, the value per unit of model performance improvement” (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time) ... For example, a decision maker can be asked the question “How long would you be willing to wait to increase the relative accuracy of the learned model by one percent?” If the answer is (e.g.) one hour, then α =0.01 benefit per hour”. Meek further discloses determining α based on the decision maker’s expressed preference, providing an example in which a willingness to wait one hour for a one-percent improvement in model accuracy results in α = 0.01 benefit per hour. Thus, Meek teaches or at least suggests determining a value representing the benefit of model-performance improvement relative to runtime based on the decision maker’s valuation, which teaches or at least suggests the claimed determining, based on the user-defined valuation, the value per unit of model-performance improvement.)
Regarding claim 12, depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches or at least suggests the limitation “receiving a user selection of a target number of samples to be labeled from the plurality of unlabeled samples” (paragraph 17 “uses a parametric function of a feature extractor parameter and a discriminator parameter to score a plurality of samples of a dataset of unlabeled data, selects, for labeling, a subset of the scored plurality of samples” Phan discloses select, for labeling, a subset of the scored plurality of samples, which corresponds to the user selection of a target number of samples to be labeled from the plurality of unlabeled samples, as claimed.)
Phan teaches or at least suggests the limitation “determining, based on the target number of samples to be labeled from the plurality of unlabeled samples, a threshold number of samples for labeling” (paragraph 73 “Module 220 selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset. One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset” Phan discloses selects, for labeling, a subset of the scored plurality of samples based on a score, such that selecting the lowest-scored samples teaches or at least suggests a threshold number of samples for labeling. Under the broadest reasonable interpretation, the score cutoff corresponding to the selected samples operates as the claimed threshold, because sample meeting the cutoff are identified for labeling while samples outside the cutoff are not selected.)
Regarding claim 13, depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches or at least suggests the limitation “receiving a user selection of an upper limit of total cost associated with labeling the plurality of unlabeled samples” (paragraph 73 “One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset. Module 220 also subtracts the number of samples in the subset to be labeled from the query budget B. If the query budget is greater than zero, there are samples remaining in the query budget, and thus module 220 sends the selected samples to be labeled.” Phan discloses a query budget which represents the number of samples that can be selected for labeling. Under the broadest reasonable interpretation, the query budget corresponds to the claimed upper limit of total cost associated with labeling the plurality of unlabeled samples, because labeling cost increases with the number of samples selected for labeling, and limiting the number of selected samples limits the total labeling cost.)
Phan teaches or at least suggests the limitation “determining, based on the upper limit of total cost associated with labeling, a maximum number of samples to select for labeling” (paragraph 73 “One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset. Module 220 also subtracts the number of samples in the subset to be labeled from the query budget B. If the query budget is greater than zero, there are samples remaining in the query budget, and thus module 220 sends the selected samples to be labeled.” Phan discloses subtracts the number of samples in the subset to be labeled from the query budget B, such that if the query budget is greater than zero, there are samples remaining in the query budget. Under the broadest reasonable interpretation, the query budget corresponds to the claimed upper limit of total cost associated with labeling the plurality of unlabeled samples, because labeling cost increases with the number of samples selected for labeling, and limiting the number of selected samples limits the total labeling cost. Thus, selecting all samples from the query budget teaches or at least suggests of selecting the maximum number of samples for labeling based on the upper limit, as claimed.)
Regarding claim 14 depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches the limitation “receiving a user selection of target model complexity and a target model performance” (paragraph 69 “Application 200 receives a plurality of learning parameters, or sets one or more of the learning parameters to default values. The learning parameters control training of a model using active learning. One learning parameter is a query budget (denoted by B), which is the total number of samples from a dataset that can be selected for labelling during the active learning process. Two other learning parameters are the number of model training cycles, or epochs, to be used in training the model (denoted by M) and a learning rate” , and paragraph 74 “Application 300 repeats the model training process, including use of the newly-labeled samples, until the query budget B has been exhausted, all the available data has been labeled and used to train the model, the model meets a model performance criterion (e.g., the model classifies more than a threshold percentage of samples correctly, or the model classifies more than a threshold percentage of a particular class of samples correctly)” Phan discloses application receives a plurality of parameters such as query budget, number of model training cycles, and learning rates, which corresponds to the user selection of target model complexity because these parameters control the extent and configuration of model training. Phan’s model performance criterion corresponds to the claimed target model performance because it defines a target condition for when the trained model perform sufficiently, such as classifying more than a threshold percentage of samples correctly. Thus, Phan teaches or at least suggests the claim.)
Meek teaches or at least suggests the limitation “determining, based on the user selection, a value added per unit of model performance improvement” (Page 4-5 section 2 “α is the relative importance of benefit to run time. This quantity, which depends on the preferences of the decision maker—the person who is controlling the execution of the algorithm—should be assessed on a problem-by-problem basis ... α can be viewed as the value of this incremental-benefit-to-cost ratio (having units benefit per time) ... For example, a decision maker can be asked the question “How long would you be willing to wait to increase the relative accuracy of the learned model by one percent?” If the answer is (e.g.) one hour, then α =0.01 benefit per hour” Meek teaches that α is a relative importance of benefit to runtime and depends on the preferences of the decision maker. Meek further explains that α may be viewed as an incremental-benefit-to-cost ratio having units benefit per time. Under the broadest reasonable interpretation, the incremental unit benefit (e.g., 0.01 benefit) obtained as the model improves through training over time corresponds to the claimed value added. Thus, Meek’s α corresponds to the claimed user input indicative of a value added per unit of model performance improvement, because α reflects the decision maker’s preference regarding how much model’s benefit is valued (e.g., 0.01 benefit) relative to the runtime/resource cost requirement to obtain that improvement.)
Regarding claim 15, which recites a computer program product, the applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps to the system of claim 1, thus the claim is rejected under similar rationale.
Regarding claim 16 depends on claim 15, thus the rejection of claim 15 is incorporated. The applicant is further directed to the rejection of claim 1 above, because the claim recites similar limitations and processing steps to the system of claim 1, thus the claim is rejected under similar rationale.
Regarding claim 17 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim 4 above, because the claim recites similar limitations and processing steps to the system of claim 4, thus the claim is rejected under similar rationale.
Regarding claim 19 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim above, because the claim 6 recites similar limitations and processing steps to the system of claim 6, thus the claim is rejected under similar rationale.
Regarding claim 20 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim above, because the claim 7 recites similar limitations and processing steps to the system of claim 7, thus the claim is rejected under similar rationale.
Claims 5, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering), further in view of Chen et.al (US 11704577 B1)
Regarding claim 5 depends on claim 3, thus the rejection of claim 3 is incorporated.
Phan teaches or at least suggests the limitation “determining whether training for the machine learning model is complete or incomplete” (paragraph 71 “Once module 210 has completed the M training epochs”, and paragraph 77 “Once pseudocode 310 has completed the M training epochs” Phan discloses the module and pseudocode command complete the training epoch, which teaches or at least suggest the determination by the pseudocodes/module of whether the machine learning model complete the training, thereby teaches or at least suggests determining whether training for the machine learning model is complete or incomplete, as claimed.)
Phan/Meek does not teach the limitation “generating one or more data files comprising parameters of the machine learning model in a standardized format” However, Chen teaches or at least suggests the limitation (Column 2 lines 12-20“According to some embodiments, a unified inference framework for heterogenous edge devices is provided that can accept machine learning (ML) models from a user in any of multiple formats (as generated by multiple different frameworks), convert and optimize these ML models for use by heterogeneous “edge” devices having heterogeneous computing resources, and deploy these ML models for use in one or more edge devices of one or more different types”, and Column 6 lines 45-51 “... using a conversion library/module that identifies certain values (e.g., weights) in model files generated by a first framework and inserts them in a different format (or location) within files adherent to a different framework or format (e.g., a different framework's format, a standardized “generic” format such as the Open Neural Network eXchange format “ONNX”, etc.)” Chen discloses a unified inference framework for heterogenous edge devices that can accept machine learning (ML) models from a user in any of multiple formats, and convert and optimize these ML models for use by heterogeneous “edge” devices. The framework use a conversion library/module that can identifies certain values (e.g., weights) in model files generated and convert them to a standardized “generic” format, which corresponds to the claimed generating one or more data files of the machine learning model in a standardized format, and the data files can be the budget parameter as disclosed by Phan in view of the combination below.)
Phan/Meek does not teach the limitation “transmitting, to a remote device, the one or more data files” However, Chen teaches or at least suggests the limitation (Column lines Column 2 lines 12-20 “According to some embodiments, a unified inference framework for heterogenous edge devices is provided that can accept machine learning (ML) models from a user in any of multiple formats (as generated by multiple different frameworks), convert and optimize these ML models for use by heterogeneous “edge” devices having heterogeneous computing resources, and deploy these ML models for use in one or more edge devices of one or more different types” Chen discloses converting and optimizing the ML models in any format into a standardized format, which can be deployed for use in one or more edge devices, suggesting the transmitting of the one or more data files, as claimed because the ML models having their data being formatted is deployed for use in one or more edge devices.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan, and the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek with the teaching of a unified inference framework for heterogenous edge devices that can accept machine learning (ML) models from a user in any of multiple formats, and convert and optimize these ML models for use by heterogeneous “edge” devices by Chen. The motivation to do so is referred to in Chen’s disclosure (Column 3 lines 20-26 “Embodiments disclosed herein allow users to implement machine learning at the edge in a manageable and sustainable way. Further, some embodiments can provide a flywheel for ML on connected devices, allowing users to deploy models across potentially millions of devices and continually improve them. Data can be collected in a secure manner from devices and used to train new models, which can be re-deployed to devices. This in turn generates new data for additional cycles of re-training and re-deployment, where each cycle can increase availability of the system and improve device experiences” Chen discloses the benefit of implementing machine learning at the edge in a manageable and sustainable way, which allow users to deploy models across potentially millions of devices and continually improve them, and further allow new data for additional cycles of re-training and re-deployment, which increase availability of the system and improve device experiences. Therefore, the teaching by Phan/Meek can be further improved by combining with the teaching by Chen to format any information regarding the training of the active machine learning model into a standardized format for use by multiple devices and users, thereby continually improve the model. One of ordinary skill in the art would have been motivated to format regarding parameters as disclosed by Phan above such as the budget parameter to ensure that when the model is utilized across other edge devices, the model would behave appropriately given the amount of data at each edge devices.)
Regarding claim 18 depends on claim 16, thus the rejection of claim 16 is incorporated. The applicant is further directed to the rejection of claim above, because the claim 5 recites similar limitations and processing steps to the system of claim 5, thus the claim is rejected under similar rationale.
Claims 8 is rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering), further in view of Rink et.al (US 20210287119 A1)
Regarding claim 8 depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan/Meek does not teach the limitation “selecting samples based on the one or more user-defined target parameter values, wherein the one or more user-defined target parameter values correspond to target parameters indicative of a threshold for bias and variance for the machine learning model” However, Rink teaches or at least suggests the limitation (paragraph 55 “At 218, the processor may transform the raw data set 204 into a transformed data set based on pre-processing thresholds 216 ... To illustrate, the thresholds may be for identifying whether data entries may be subject to disparate impact beyond a threshold value. In some examples, thresholds may be for identifying skewness or probability/imbalance in received data sets”, and paragraph 56 “In some examples, operations associated with pre-processing thresholds 216 may include operations for identifying data bias based on statistical bias (e.g., skew, kurtosis, variance), minority class recognition, disparity impact identification, or sensitive attribute identification” Rink discloses systems and methods for mitigation bias in machine learning model output. Within the disclosure Rink discloses pre-processing thresholds for identifying whether data entries are suitable to transform into data set. Pre-processing thresholds may include operations for identifying data bias based on statistical bias (e.g., skew, kurtosis, variance). Under the broadest reasonable interpretation, processing thresholds correspond to the claimed target parameter values because they define threshold conditions used to identify bias and variance related issues in the data used for generating or refining the ML model. Thus, Rink teaches or at least suggests the claimed limitation.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan, and the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek with the teaching of pre-processing threshold for bias and variance by Rink. The motivation to do so is referred to in Rink’s disclosure (paragraph 71 “By pre-processing the raw data set 204 prior to providing the data set as an input for model generation or to the first in-processing stage 230, potential biases within the raw data set 204 may be identified or reduced, thereby reducing data entries that may be associated with skewed data or other unintended data artifacts prior to generating or refining a machine learning model” Rink discloses the improvement in using the preprocessing threshold for identifying or reducing potential biases within the raw data thereby reducing data entries that may be associated with skewed data or other unintended data artifacts. A person ordinary skill in the art would have been motivated to use such bias/variance threshold as parameters in Phan’s active learning process to control the quality of training data and reduce bias or variance before or during model training, thereby improving the resulting model.)
Claims 10 is rejected under 35 U.S.C. 103 as being unpatentable over Phan et.al (US 20240169253 A1) in view of Meek et.al (NPL: The Learning-Curve Sampling Method Applied to Model-Based Clustering), further in view of Sengupta et.al (US 20230072171 A1)
Regarding claim 10 depends on claim 2, thus the rejection of claim 2 is incorporated.
Phan teaches a part of the limitation “The method of claim 2, ... wherein the user input indicative of a value added per unit of model performance improvement comprises a cost associated with labeling a sample of the plurality of unlabeled samples” (paragraph 73 “Module 220 selects, for labeling, a subset of the scored plurality of samples from the unlabeled dataset. One implementation of module 220 selects, as the subset to be labeled, the |B| samples having the lowest scores (the lowest score is considered the best score) from the unlabeled dataset. Module 220 also subtracts the number of samples in the subset to be labeled from the query budget B” Phan discloses selecting for labeling a subset of samples and subtracting the number of samples in the subset to be labeled from the query budget. Under the broadest reasonable interpretation, the query budget corresponds to a labeling cost-parameter because it limits how many samples may be selected for labeling, and each selected samples consumes part of the budget. Thus, Phan teaches or at least suggests the cost associated with labeling a sample of the plurality of unlabeled samples as claimed.)
Phan/Meek does not teach a part of the limitation “The method of claim 2, wherein the unit of model performance improvement comprises a unit of improvement in precision, recall, or F1-score ...” However, Sengupta teaches the part of the limitation (paragraph 58 “The method may include calculating precision and recall for the trained model based on these prediction scores output by the trained model (e.g., performing operations 506, 508, and 510). Precision and recall are indicators of a machine learning model's performance”, and paragraph 60 “The method may include using the true class of the selection of the unlabeled input dataset (which is unknown to the machine learning model) and the prediction scores as inputs to calculate precision and recall for the trained machine learning model” Sengupta discloses a system and method for training and refining machine learning models. Within the disclosure, Sengupta discloses calculating precision and recall for the trained model as precision and recall are indicators of a machine learning model's performance, which corresponds to the claimed model performance improvement comprises unit of improvement in precision, recall, or F1-score.)
Before the effective filing date, it would have been obvious to a person ordinary skill in the art to combine the teaching of the method, system and computer program product of active learning, which can be performed on a mobile device by Phan, and the teaching of the learning-curve sampling method to choose an appropriate sample size for training by Meek with the teaching of calculating precision and recall as indicators of a machine learning model's performance by Sengupta. The motivation to do so is referred to in Sengupta‘s disclosure (paragraph 5 “However, real world data may contain intents the training data does not include. Accordingly, these models do not perform well at identifying these unknown intents”, and paragraph 7 “Precision and recall are often in tension. Improving precision may reduce recall, and improving recall may reduce precision. Thus, tuning a machine learning model can be complicated, as one improvement can undo another. The disclosed system and method overcome this issue by applying an optimization technique (e.g., multi-objective optimization) to optimize precision and recall to find optimal threshold values for prediction values” Sengupta discloses the system and method applying an optimization technique (e.g., multi-objective optimization) to optimize precision and recall in producing the result of machine learning model, which help tuning the trained machine learning model, wherein a machine learning model can use precision and recall value to indicate its improvement over performing its learning. One of ordinary skill in the art would have been motivated to combine the teaching of Phan/Meek with the teaching of precision and recall as the performance indicator for performance improvement because the combination would have predictably allowed the system to measure model-performance improvement while determining whether labeling/training improves the model sufficiently.)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY TU DIEP whose telephone number is (703)756-1738. The examiner can normally be reached M-F 8-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY T DIEP/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights