DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/22/2024 .The submission is in compliance with the provisions of 37 CFR1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Allowable Subject Matter
Claim 11 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 101 as set forth in this Office action and rewritten in independent form including all the limitations of base claim and any intervening claims.
The following is an examiner’s statement of reasons for indicating allowable subject matter:
Regarding claim 11:
The prior art of record teaches the following limitations
As to Claim 11.
Prechelt et. al., “Early Stopping | but when?” (“Prechelt” ) teaches “ wherein the second threshold difference is based on an average validation loss” (Prechelt “[pg-1] Typically, the generalization error is estimated by a validation error, i.e., the average error on a validation set is a fixed set of examples not from the training set. Stop training as soon as the error on the validation set is higher than it was the last time it was checked .Then Etr(t), the training set error (for short: training error), is the average error per example over the training set, measured after epoch t. Eva(t), the validation error, is the corresponding error on the validation set and is used by the stopping criterion. Ete(t), the test error, is the corresponding error on the test set; it is not known to the training algorithm but estimates the generalization error and thus benchmarks the quality of the network resulting from training.
In Prechlet, as noted above, Eva(t) is defined as an average per example error on validation set(i.e average validation loss) and used in stop criterion(i.e. second threshold difference)
Regarding the limitations:
Baylor et.al “TFX: A TensorFlow-Based Production-Scale Machine Learning Platform”(“ Baylor”) teaches “and wherein the step of comparing the first model before the further training to the updated first model produced by the further training comprises” (Baylor,[pg-1392] “Once a model is launched to production and is continuously being updated, automated validation is used to ensure that the updated models are good. We validate that a model is safe to serve with a simple canary process. We evaluate prediction quality by comparing the model quality against a fixed threshold as well as against a baseline model (e.g., the current production model).”
Baylor teaching of model being launched and having continuous update is interpreted as first model and updated first model , using automated validation and comparing the model with baseline model is interpreted as the step of comparing the first model before further training and the model produced after training, corresponding to the claim.
Regarding the limitation:
Jain et. al “Checkpoints in Deep Learning” (“jain”) accessed from <<www.indusmic.com/post/checkpoints-in-deep-learning >> teaches “calculating an average validation loss Vmodeli for the first model before the further training, on a validation dataset. “( jain [ pg-2]Given above is a block diagram explaining the generation of checkpoints during training. Firstly, inside the neural network model the layers as required by the model are applied and then the initial checkpoint file is saved in the model directory as model.ckpt file. After that the main training begins. The training occurs on no of steps as provided by the coder and its roughly calculates loss after every 100 steps. When the loss is minimum as the model feels as if overfitting will occur beyond this point, the training stops and evaluation is performed on the validation dataset.
PNG
media_image1.png
360
624
media_image1.png
Greyscale
In this, if we are saving model at 49 epoch then its validation accuracy at that point is 0.6487 which is less than that at epoch 46 (where it is 0.6728). So instead of saving model at epoch 49 we must save our model at epoch 46 i.e., at an epoch where our model performs the best. “).
Average validation loss is interpreted as Jain, disclosure of average precision calculation along with validation accuracy and evaluation being performed on the validation dataset is interpreted as before the further training on a validation dataset. Additionally, first model, Under BRI is interpreted as network model , and calculating the average validation loss of the network model is further interpreted as calculating the average validation loss of the first model.
calculating an average validation loss Vupdatedmodeli for the first updated model produced by the further training, on the validation dataset; “( jain [ pg-3] After the creation and validation, the checkpoint_path is saved inside model.ckpt at a destination specified by the coder. Training resumes again from the step it was left and continues until the next checkpoint occurs or until the training steps end.”)
Calculating an average validation loss of updated model is interpreted as Jain’s disclosure of resuming training again from the step it was left.(see above)
Wang et. al., “FBUNet: Full Convolutional Network Based on Fusion Block Architecture for Biomedical Image Segmentation” (“Wang”) teaches “calculating an average validation loss between the first model and the updated first model according to: (Vupdatedmodell - Vmodell)/ (Vupdatedmodell + Vmodell).” (Wang “[pg-194] The accurate blood vessel segmentation results of the classic U-Net, RU-Net, the FBUNet-1 and FBUNet-2 models are shown in Table 4. Figure 13 shows the performance of the four models by the example outputs and the thresh-old of outputs is 0.5. For further useful analysis, we show the progress of average validation loss JS and DC of each epoch in Fig. 14. The experimental results illustrate that the FBUNet-2 model exhibits better performance than the other models.
Wang disclosure of showing the progress of average validation loss is interpreted as the teaching of calculating average validation, and experimental results of the performance on one model exhibiting a better performance than the other model is interpreted as the teaching of first and updated first model i.e corresponding to the claim. Additionally, Wang at pg-194, section 4 discloses computation of average validation loss of two models.
However, the combination of Prechlet, Baylor and Jain fails to teach or suggest the limitation
“Calculating an average validation loss between the first model and the updated first model according to: (Vupdatedmodell - Vmodell)/ (Vupdatedmodell + Vmodell)”
The specific mathematical formula where the average validation loss between the two model is calculated as the difference in average validation loss of updated first model in ratio with sum of average validation loss of the updated first model and the average validation loss first model before the further training is not suggested by the combination of closest art, taken alone or in combination with the remining features and elements of the claim invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-15 is rejected under 35 USC § 101 because claimed invention is directed to the abstract idea without significantly more.
As to claim 1:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Claim 1 is a method claim, therefore it falls under one of four categories of statutory subject matter.
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “determining a performance measure for the first model” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
The limitation “if the performance measure indicates a performance below a threshold performance level, triggering an upgrade process wherein the upgrade process comprises performing further training on the first model to produce an updated first model” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
The limitation “wherein the further training is performed using an active learning process wherein training data for the further training is selected from a pool of unlabeled data samples” ” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The limitation “A computer implemented method of managing a first model that was trained using a first machine learning process and is deployed and used to label medical data, the method comprising” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
The limitation “according to the active learning process and sent to a labeler to obtain ground truth labels for use in the further training” is an additional element that amounts to adding insignificant extra-solution activity of mere data output and mere data gathering to the judicial exception. The limitation is reciting sending unlabeled data to obtain ground truth label. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The limitation “A computer implemented method of managing a first model that was trained using a first machine learning process and is deployed and used to label medical data, the method comprising” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Furthermore, the additional element is directed to a method Performed by computational unit, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP §§ 2106.04(d), 2106.05(f)(2).
The limitation “according to the active learning process and sent to a labeler to obtain ground truth labels for use in the further training” is an additional element that amounts to adding insignificant extra-solution activity of mere data output and mere data gathering to the judicial exception. The limitation is reciting sending unlabeled data to obtain ground truth label which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
Therefore, in examining elements as recited by the limitations individually and as an ordered combination, as a whole the independent claim limitations do not recite what the courts have identified as “significantly more”.
As to claim 2
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “wherein the performance measure is a measure of accuracy of the first model and/or a measure of user satisfaction of the first model.” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
As to claim 3
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “wherein the active learning process comprises selecting training data from the pool of unlabeled data samples that:when passed through the first model, result in outputs that are predicted with low confidence when compared to a threshold confidence level” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
The limitation “or when passed through the first model result in outputs that are different to the outputs of a second model trained using a second machine learning process, the second model having different hyper-parameters to the first model, and wherein the outputs are different by more than a first threshold difference” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
As to claim 4
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “wherein the step of performing further training on the first model to produce an updated first model comprises: compressing the unlabeled data samples using a compression process; reconstructing the compressed unlabeled data samples using a decompression process; determining a reconstruction error for the reconstructed unlabeled data samples;” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
The limitation “and filtering the pool of unlabeled data samples to remove unlabeled data samples that have a reconstruction error higher than a first reconstruction error threshold.” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
As to claim 5
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “wherein the step of performing further training on the first model to produce an updated first model comprises: filtering the pool of unlabeled data samples to remove unlabeled data samples that do not have parameters corresponding to the input and/or output parameters of the first model” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
The limitation “and/or filtering the pool of unlabeled data samples to remove unlabeled data samples that correspond to a different patient demographic to the patient demographic covered by the first model.” is an abstract idea of a mental process.(i.e. evaluation) See MPEP § 2106.04(a)(2)(III).
As to claim 6
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The limitation “wherein the labeler comprises a plurality of clinicians and the active learning process comprises obtaining annotations from the plurality of clinicians for the selected unlabeled data samples;” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. (i.e., data gathering). See MPEP §§ 2106.04(d), 2106.05(g).
The limitation “and using a weighted combination of the annotations obtained from the plurality of clinicians as the ground truth labels for the selected unlabeled data samples.” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The limitation “wherein the labeler comprises a plurality of clinicians and the active learning process comprises obtaining annotations from the plurality of clinicians for the selected unlabeled data samples;” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. i.e., data gathering , which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP §§ 2106.04(d), 2106.05(f)(2).
The limitation “and using a weighted combination of the annotations obtained from the plurality of clinicians as the ground truth labels for the selected unlabeled data samples.” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception or merely uses a computer in its ordinary capacity as a tool to perform an existing process , which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
As to claim 7
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The limitation “wherein the weights for the weighted combination are set according to a level of experience of each of the plurality of clinicians and/or a level of consistency with which each of the plurality of clinicians provides the annotations” ” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
As to claim 8
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “wherein the first model is accredited by a medical body as complying with a medical standard” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
As to claim 9
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “following the further training, comparing the first model as it was before the further training to the updated first model produced by the further training; and if the comparison indicates a difference greater than a second threshold difference, flagging the updated first model for compliance testing against the medical standard. ” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
As to claim 10
The limitation “following the further training, comparing the first model before the further training to the updated first model produced by the further training; and if the comparison indicates a difference less than the second threshold difference, deploying the updated first model for use in labelling subsequent medical data.” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
As to claim 11
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “wherein the second threshold difference is based on an average validation loss and wherein the step of comparing the first model before the further training to the updated first model produced by the further training comprises” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
The limitation “calculating an average validation loss Vmodeli for the first model before the further training, on a validation dataset; calculating an average validation loss Vupdated modeli for the updated first model produced by the further training, on the validation dataset” is an abstract idea of mathematical relationship and as directed to “a mathematical relationship is a relationship between variables or numbers. A mathematical relationship may be expressed in words or using mathematical symbols. See MPEP § 2106.04(a)(2)(I)(A).
The limitation “calculating an average validation loss between the first model and the updated first model according to: (Vupdatedmodell - Vmodell)/ (Vupdatedmodell + Vmodell). ” is an abstract idea of mathematical relationship and as directed to “a mathematical relationship is a relationship between variables or numbers. A mathematical relationship may be expressed in words or using mathematical symbols. See MPEP § 2106.04(a)(2)(I)(A).
As to claim 12
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
The limitation “repeating steps i) and ii) in an iterative manner.” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
As to claim 13
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The limitation “ wherein the first model is for use in diagnosing abnormalities in Computed Tomography, Magnetic Resonance Imaging, or Ultrasound images or wherein the first model is for use in detecting events in Electrocardiogram, Electroencephalogram, or Electrooculogram measurements. ” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The limitation “ wherein the first model is for use in diagnosing abnormalities in Computed Tomography, Magnetic Resonance Imaging, or Ultrasound images or wherein the first model is for use in detecting events in Electrocardiogram, Electroencephalogram, or Electrooculogram measurements. ” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception or merely uses a computer in its ordinary capacity as a tool to perform an existing process , which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
As to claim 14
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Claim 14 is drawn to a storage medium consisting of Hardware components. i.e (article of manufacture), therefore claim 12 falls under one of four categories of statutory subject matter.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The limitation “A system for managing a first model that was trained using a machine learning process and is deployed and used to label medical data, the system comprising” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
The limitation “a memory comprising instruction data representing a set of instructions; and a processor configured to communicate with the memory and to execute the set of instructions, wherein the set of instructions, when executed by the processor, cause the processor to:” is additional element is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2)
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The limitation “A system for managing a first model that was trained using a machine learning process and is deployed and used to label medical data, the system comprising” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP §§ 2106.04(d), 2106.05(f)(2
The limitation “a memory comprising instruction data representing a set of instructions; and a processor configured to communicate with the memory and to execute the set of instructions, wherein the set of instructions, when executed by the processor, cause the processor to: “is the additional claim elements of one or more computing devices; a memory coupled to at least one of the computing devices; a stored instructions stored in the memory and executed by at least one of the computing devices are not sufficient to amount to significantly more than the judicial exception since these additional claim elements are recited at a high level of generality (i.e. using a generic processor and generic memory.
And for all other claim elements of claim 14 they are rejected using the PEG analysis of claim 1 since they are analogous claims.
As to claim 15
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Claim 15 is a product claim, therefore it falls under one of four categories of statutory subject matter.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
The limitation “a non-transitory computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform” is additional element is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2)
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
The limitation “A computer program product comprising a non-transitory computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform” is the additional claim elements of one or more computing devices; a memory coupled to at least one of the computing devices; a stored instructions stored in the memory and executed by at least one of the computing devices are not sufficient to amount to significantly more than the judicial exception since these additional claim elements are recited at a high level of generality (i.e. using a generic processor and generic memory.
And the method of claim 1 as mentioned is claim 15 is rejected using the same PEG analysis of claim 1.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103, which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claims 1-3, 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over Wubbels et al. Grant Publication No. US 10599984 B1 (“Wubbels”) view of Szanto et. al., Pre-Grant No.US 20200286002 A1 “” (“Szanto”).
As to Claim 1
Wubbels teaches “A computer implemented method of managing a first model that was trained using a first machine learning process and is deployed and used to label medical data, the method comprising”( Wubbels “paragraph [38] The training module 410 can train the machine learning model 420 [of managing a first model that was trained using a first machine learning process and is deployed]to diagnose various diseases such as cancer, diabetic retinopathy, hemorrhage, etc., from a variety of medical imagery. [and used to label medical data, the method comprising] [12] FIG. 11 shows an example system to (1) monitor the performance of a deployed, machine learning model . [and is deployed]
2. determining a performance measure for the first model; and (Wubbels,” [abs] Validation processes presented here assess a performance [determining a performance measure] of the machine learning model [for the first model; and] post-deployment. In post-deployment validation, the validation process monitoring can include: (1) monitoring to ensure a model performs as well as a reference member such as another machine learning model, and (2) monitoring to detect anomalies in data.
Wubbels does not explicitly teach:
if the performance measure indicates a performance below a threshold performance level
triggering an upgrade process wherein the upgrade process comprises performing further training on the first model to produce an updated first model,
wherein further training is performed using an active learning process
wherein training data for the further training is selected from a pool of unlabeled data samples, according to the active learning process
Szanto teaches “if the performance measure indicates a performance below a threshold performance level “(Szanto [0003] gathering a known dataset for which one has already identified what the model is going to try to identify/label/tag/organize; (ii) running the model on the known dataset to see what results it produces; (iii) improving the model by updating it in accordance with the mistakes that it made in step (ii), and (iv) repeating steps (ii)-(iii) until the model's accuracy [if the performance measure indicates] does not improve further by a specified threshold. [below a threshold level]
2. triggering an upgrade process wherein the upgrade process comprises performing further training on the first model to produce an updated first model, (Szanto “paragraph [0003] At the most basic level, the process of training a machine learning model can include: (i) gathering a known dataset for which one has already identified what the model is going to try to identify/label/tag/organize; (ii) running the model on the known dataset to see what results it produces; (iii) improving the model by updating it in accordance with the mistakes that it made in step (ii), and (iv) repeating steps (ii)-(iii) [triggering an upgrade process wherein the upgrade process comprises performing further training on the first model to produce an updated first model] until the model's accuracy does not improve further by a specified threshold.”).
3. wherein further training is performed using an active learning process (Szanto “paragraph [0037] FIG. 2B is a flowchart of an example process 220 for dynamically updating a machine learning model such as a text classifier and including active learning.
[0045] Finally, process 300 can include selecting documents 322 for further training [wherein the further training is performed] using an active learning engine [using an active learning process] as described above with reference to FIG. 1
Examiner notes: Also see, fig 3, block 322.
4. wherein training data for the further training is selected from a pool of unlabeled data samples, according to the active learning process (Szanto paragraph 0046, 0047 “[0046] “One arena in which active learning has potential is in radiographic diagnosis. […] Assume there are many other x-rays in the database that are true positives or true negatives, but those x-rays are unlabeled with respect to the disease. One can train a first model[wherein training data for the further training] using the small amount of disease-specific data, then one can use the first model and the active learning approach [according to the active learning process] described herein to select those unlabeled x-rays that [is selected from a pool of unlabeled data sample], when combined with a label that describes whether the x-ray corresponds to a positive or negative diagnosis of the disease, will give the model more predictive power. […] [0047] Active learning is perhaps most useful when there is a confluence of: (i) little initial labeled data; (ii) large initial unlabeled data; and (iii) high human expense in getting all the unlabeled data labeled. Active learning aims to choose the smallest number of unlabeled examples to present to the expensive human expert such that the model performs at an acceptable level.
Examiner notes: Under BRI, In light of specification (pg-11, line30-35 “The pool of unlabeled data samples may be retrieved from a hospital or hospital(s) or any other database of relevant data [….]” ). The unlabeled x-rays used in active learning process is interpreted as training data for further training from hospital/hospital(s).
5. and sent to a labeler to obtain ground truth labels for use in the further training.(Szanto “paragraph [0026] The document sampling engine 106 forwards the unlabeled data samples to an annotation platform 110. [and sent to a labeler to] The annotation platform can be an annotation engine that automatically annotates the sampled unlabeled data […][0027] The annotation platform 110 uses the annotations to create labeled data[to obtain ground truth labels for] and forwards the labeled data 112 to a concept labeling engine 114.[…]. [0028] The concept labeling engine 114 can use the labeled data to train, [use in the further training]and provide as a service, a machine learning model such as a text classifier.”).
Examiner notes: Also see paragraph 0034.
Szanto and Wubbels are related to the same field of endeavor(i.e. Machine learning] . In view of the teachings of Wubbels it would have been obvious for a person of ordinary skill in the art to apply the teachings of Szanto to Wubbels before the effective filing date of the claimed invention in order to improve the performance of models and use it in the medical field. Furthermore, one would be motivated to apply the teachings to improve accuracy and eliminate human error while making model updated more efficient. (Szanto “Paragraph [0013] Such systems and methods can eliminate human error, free up engineering resources, reduce the amount of human expert-generated data required to produce and maintain a machine learning model, improve text classification models over time and automatically.”
As to claim 14
Wubbels teaches “A system for managing a first model that was trained using a machine learning process and is deployed and used to label medical data, the system comprising”
( Wubbels “paragraph [81] FIG. 11 shows an example system [A system for managing] to perform two phases: phase (1) to monitor the performance [a first model that was trained] of a deployed, machine learning model, [using a machine learning process and is deployed ]and phase (2) to detect an anomaly associated with an input. [100] To perform phase (1) above, a processor, in step 1400, can generate a diagnostic result by using the machine learning model on one or more medical images. [and used to label medical data, the system comprising]”).
2. a memory comprising instruction data representing a set of instructions and a processor configured to communicate with the memory and to execute the set of instructions, wherein the set of instructions, when executed by the processor, cause the processor to. ( Wubbels “paragraph [125] The computer programs typically comprise one or more instructions (e.g., instructions 1604, 1608, 1628) set at various times in various memory [a memory comprising instruction data representing a set of instructions] and storage devices in a computing device. [To communicate with the memory ]When read and executed by the one or more computer processors 1602 [and a processor configured to and to execute the set of instructions,], the instruction(s) cause the processing system 1600 to perform operations to execute elements. [wherein the set of instructions, when executed by the processor, cause the processor to]
And for all the other limitation of Claim 12, it is rejected under same basis as Claim 1. As the Claim are Analogous.
Wubbels and Szanto are combinable for the same rationale as set forth above with respect to Claim 1.
As to claim 15
Wubbels teaches “A non-transitory computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform. . ( Wubbels “paragraph [125] the computer programs typically comprise one or more instructions (e.g., instructions 1604, 1608, 1628) set at various times in various memory [A non-transitory computer readable medium, the computer readable medium having computer readable code embodied therein,] and storage devices in a computing device. When read and executed by the one or more computer processors 1602 the instruction(s) cause the processing system 1600[ the computer readable code being configured such that, on execution by a suitable computer or processor], to perform operations to execute elements. [on execution by a suitable computer or processor, the computer or processor is caused to perform]
the method as claimed in claim 1.
See the reasoning of Claim 1 mentioned above:
Wubbels and Szanto are combinable for the same rationale as set forth above with respect to Claim 1.
As to Claim 2
Wubbels in view of Szanto teaches the method of claim 1.
Wubbels further teaches “wherein the performance measure is a measure of accuracy of the first model [
(Wubbels paragraph [50] The training module 410 can train the multiple models on the same inputs and measure the performance of the multiple models [performance measure] at the end of the training. [...] For example, the substantially optimal machine learning models can be selected as the machine learning models having accuracy above a minimum threshold such as 60% [measure is a measure of accuracy of the first model] and having a latency below a maximum threshold such as 5 hours.
Examiner notes: Under BRI, And/or denotes or. For the purpose of this examination and/or is interpreted as OR.
Wubbels and Szanto are combinable for the same rationale as set forth above with respect to Claim 1.
As to claim 3
Wubbels , as modified by Szanto, teaches the method of claim 1.
Szanto further teaches “wherein the active learning process comprises selecting training data from the pool of unlabeled data samples that” (Szanto paragraph 0046 ,0047“[0046] One arena in which active learning has potential is in radiographic diagnosis. […] Assume there are many other x-rays in the database that are true positives or true negatives, but those x-rays are unlabeled with respect to the disease. One can train a first model using the small amount of disease-specific data, then one can use the first model and the active learning approach [wherein the active learning process ] described herein to select those unlabeled x-rays that[comprises selecting training data from the pool of unlabeled data samples that:], when combined with a label that describes whether the x-ray corresponds to a positive or negative diagnosis of the disease, will give the model more predictive power. […] [0047] Active learning is perhaps most useful when there is a confluence of: (i) little initial labeled data; (ii) large initial unlabeled data; and (iii) high human expense in getting all the unlabeled data labeled. Active learning aims to choose the smallest number of unlabeled examples to present to the expensive human expert such that the model performs at an acceptable level.
When passed through the first model, results in outputs that are predicted with low confidence when compared to a threshold confidence level
(Szanto “paragraph [0034] This determination 128 is based on several factors, including whether the article at hand is very different from any article in the current training set and whether the model is [When passed through the first model] particularly uncertain about the proper tag for that article as reflected by an uncertainty metric.[ results in outputs that are predicted with low confidence] If so determined, that document is fed back into the annotation process, associating it with an expert label, […] In one implementation, a concept label confidence score can have a range of from 0 to 1 with a score of 1 reflecting highest confidence and the system can set a threshold metric, e.g., a confidence score of 0.7 as a threshold [when compared to a threshold confidence level.] for attaching a concept label to a document.
Wubbels and Szanto are combinable for the same rationale as set forth above with respect to Claim 1.
As to Claim 12
Wubbels in view of Szanto teaches the method of claim 1.
Szanto further teaches “ repeating steps i) and ii) in an iterative manner.”
(Szanto paragraph” [0003] At the most basic level, the process of training a machine learning model can include: (i) gathering a known dataset for which one has already identified what the model is going to try to identify/label/tag/organize; (ii) running the model on the known dataset to see what results it produces; (iii) improving the model by updating it in accordance with the mistakes that it made in step (ii), and (iv) repeating steps (ii)-(iii) [repeating steps i) until the model's accuracy does not improve further by a specified threshold. [and ii) in an iterative manner.] “).
Wubbels and Szanto are combinable for the same rationale as set forth above with respect to Claim 1.
As to Claim 13
Wubbels in view of Szanto teaches the method of claim 1.
Wubbels teaches “wherein the first model is for use in diagnosing abnormalities in Computed Tomography, Magnetic Resonance Imaging, or ultrasound images, Electrocardiogram, Electroencephalogram, or Electrooculogram measurements.” (Wubbels “paragraph [98] In phase (1), a processor can monitor the quality of a deployed machine learning model [wherein the first model is for use in] by sampling, and overreading a subset of inputs (i.e., images) received, and can compute metrics to evaluate the model performances and compare them with benchmark metrics.[…] [100] To perform phase (1) above, a processor, in step 1400, can generate a diagnostic result [is for use in diagnosing abnormalities] by using the machine learning model [wherein the first model] on one or more medical images. The medical image can be a fundus image, an MRI image, [Magnetic Resonance Imaging] an X-ray, an ultrasound, [ultrasound images] etc. In step 1410, the processor can request from one or more reference members one or more diagnostic results. The reference members receive substantially identical medical images and provide the diagnostic results. For example, multiple ophthalmologists can be consulted for diagnosis of retinal images.
Examiner notes: Model is for use in diagnosing abnormalities is interpreted as Wubbels disclosure of generating diagnostic result of MRI image ultrasound images.
Wubbels and Szanto are combinable for the same rationale as set forth above with respect to Claim 1.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over (“Wubbels”) in view of (“Szanto”) and in view of Vaidya et.al., Pre-Grant No. US 20200380531A1 (“Vaidya”)
As to claim 4
Wubbels in view of Szanto, teaches the method of claim 1.
Szanto further teaches wherein the step of performing further training on the first model to produce an updated first model comprises , (Szanto “paragraph [0003] At the most basic level, the process of training a machine learning model can include: (i) gathering a known dataset for which one has already identified what the model is going to try to identify/label/tag/organize; (ii) running the model on the known dataset to see what results it produces; (iii) improving the model by updating it in accordance with the mistakes that it made in step (ii), and (iv) repeating steps (ii)-(iii) [wherein the step of performing further training on the first model to produce an updated first model comprises] until the model's accuracy does not improve further by a specified threshold. [0034] If so determined, that document is fed back into the annotation process, associating it with an expert label, adding it to labeled data, and possibly retraining the current model.
2. filtering the pool of unlabeled data samples to remove unlabeled data samples
(Szanto [0022] The document sampling engine 106 can use more than one method to determine which unlabeled data [unlabeled data] to forward [filtering the pool to remove unlabeled data samples]on to an annotation platform 110.
Wubbels in view of Szanto does not teach:
compressing unlabeled data samples using a compression process.
reconstructing the compressed unlabeled data samples using a decompression process.
determining a reconstruction error for the reconstructed unlabeled data samples.
and filtering the pool of unlabeled data samples to remove unlabeled data samples
that have a reconstruction error higher than a first reconstruction error threshold
Vaidya teaches “compressing unlabeled data samples using a compression process; US 20200380531 A1(Vaidya [0057] The task of the encoder 202 is to compress the input 208 to the embedded vector space 206, while the task of the decoder 204 is to decompress the embedded vector back to the dimension of the original input. [0059] Instead, the autoencoder [compressing using a compression process ]decoder is trained using a dataset (e.g., training data 116 in FIG. (1) that is unlabeled, [unlabeled data samples]
2. reconstructing the compressed [unlabeled data samples] using a decompression process; (Vaidya[0057] The task of the encoder 202 is to compress the input 208 to the embedded vector space 206, while the task of the decoder 204 is to decompress the embedded vector back to the dimension of the original input. The decoder 204 operates to decompress [using a decompression process;] the compressed input in the embedded space 206 to output a reconstructed version of the input. [reconstructing the compressed unlabeled data samples]
Examiner notes: Also see Vaidya paragraph [0057]; discloses unlabeled data samples.
3. determining a reconstruction error for the reconstructed unlabeled data samples.
( Vaidya [0057] In some embodiments, the reconstruction error is based on the reconstruction error [determining a reconstruction error for the] between each of the features in the input (i.e., features 1, 2, . . . , m) and each of the corresponding features in the output (i.e., reconstructed features 1, 2, . . . , m). In some embodiments, the reconstruction error is calculated as the mean squared error (“MSE”) among the feature-specific reconstruction errors.
Examiner notes: Under BRI, in light of specification, (pg-13, line 11-12] the unlabeled data samples may be compressed using a compression process, (e.g. such as an autoencoder); the compressed unlabeled data samples may then be reconstructed using a decompression process (e.g. reverse autoencoder). So, the operation done by the encoder is interpreted as compressing the labeled data while operation done by decoder is interpreted as reconstructing the compressed unlabeled data samples using a decompression process. Also see -Vaidya paragraph [0057]; discloses unlabeled data samples.
4. and filtering the pool of unlabeled data samples to remove unlabeled data samples
that have a reconstruction error higher than a first reconstruction error threshold.
(Vaidya [0078]At block 120, the platform provides reporting based on the reconstruction errors. In some embodiments, the reporting comprises providing features and the reconstruction errors associated with the features [that have a reconstruction error], for example, using one or more user interfaces. In some embodiments, the platform determines whether a feature-specific reconstruction error exceeds a threshold. In accordance with a determination that the feature-specific reconstruction error exceeds the threshold, [higher than a first reconstruction error threshold.] the platform reports the reconstruction error and the associated feature. In some embodiments, the platform determines whether a total reconstruction error exceeds a threshold.
Vaidya and Wubbels are related to the same field of endeavor (i.e. Machie learning). In view of the teachings of Wubbels it would have been obvious for a person of ordinary skill in the art to apply the teachings of Vaidya to Wubbels before the effective filing date of the claimed invention in order to reduce cost while detecting anomaly early. Thus, early detection of anomaly and filtering unwanted data increases data quality and improves decision making process in the various domains. (Vaidya Paragraph [0005] Accordingly, there is a need for an accurate, efficient, and cost-effective platform for detecting abnormal entities and activities that goes beyond preprogrammed rulesets and is not susceptible to biases.
Claims 5 is rejected under 35 U.S.C. 103 as being unpatentable over (“Wubbels”) in view of (“Szanto”) and in view of Weese et.al EP 3686805 A1 (“Weese”)
As to Claim 5
Wubbels in view of Szanto, teaches the method of claim 1.
Szanto teaches “wherein the step of performing further training on the first model to produce an updated first model comprises” (Szanto paragraph [0037] FIG. 2B is a flowchart of an example process 220 for dynamically updating a machine learning model such as a text classifier and including active learning.
2. filtering the pool of unlabeled data samples to remove unlabeled data samples(Szanto [0022] The document sampling engine 106 can use more than one method to determine which unlabeled data [unlabeled data] to forward [filtering the pool to remove unlabeled data samples]on to an annotation platform 110.
Wubbels teaches “[and/or filtering the pool of unlabeled data samples to remove unlabeled data samples]” that correspond to a different patient demographic to the patient demographic covered by the first model. (Wubbels (45) In one or more examples, the validator module 400 can select a dimension to use in comparing the machine learning model 420 with the reference member 430. For example, the dimension can be a gender of a subject used to generate the input, an age of the subject, a race of the subject, an ethnicity of the subject, or a type of device used to obtain the input.(104) To determine whether the cause of the anomaly is due to a change in the input, or due to inaccuracy of the machine learning model [covered by the first model.], the processor can monitor a dimension associated with a process generating the medical image to obtain a dimension value. (104)If the anomaly temporally corresponds to the change in a dimension, then, the change in the dimension needs to be investigated [correspond to a different patient demographic to the] first as the likely cause of the anomaly. The dimension can include an attribute associated with the medical image, an […] the attribute associated with the subject can include the subject's age, gender, ethnicity, race, health history, current health condition, etc. [to the patient demographic covered] […] (112)[…] For example, the dimension can be age of the subjects in the month of July, and the age of the subjects in the month of August.
Examiner notes: Under BRI, Change in Dimension is interpreted as different patient demographic to the patient demographic covered. As, Dimension can be gender , age ethnicity which are interpreted as patient demographic.
Wubbels in view of Szanto does not teach:
that do not have parameters corresponding to the input and/or output parameters of the first model
Wesse teaches “that do not have parameters corresponding to the input and/or output parameters of the first model”. (Wesse, Col 24, lines 30-35 ] for example, the input dataset may comprise records of the MIMIC database of ICU encounters. Conformance checking CCH may allow us to assess, detect, and/or flag for instance pediatric patient data that is erroneously fed to a model [first model] that was trained only on adults, or combinations of input parameters that were never or rarely seen in the training dataset. [that do not have parameters corresponding to the input and]
Examiner notes: Unseen parameters on training set is interpreted as parameters not corresponding to input.
Weese and Wubbels are related to the same field of endeavor(i.e. Machine learning) . In view of the teachings of Wubbels it would have been obvious for a person of ordinary skill in the art to apply the teachings of Weese to Wubbels before the effective filing date of the claimed invention in order to optimize the performance of models when fed with data of different categories.(mainly medical data). (Weese col 5 23-30 The above measures involve a trained model that is trained on a training dataset. The training dataset, sometimes also called training data, may comprise a plurality of training records, which are sometimes also called training examples. The techniques described are applicable to a wide range of training datasets, for
example, comprising medical images, patient records, or non-medical data such as measurements of the performance of a machine that is to be optimized, financial information, etcetera.”
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over (“Wubbels”) in view of (“Szanto”) and in view of Welikala et. al., “Automated Detection and Classification of
Oral Lesions Using Deep Learning for Early Detection of Oral Cancer” (“Welikala”)
As to Claim 6
Wubbels in view of Szanto, teaches the method of claim 1.
Szanto further teaches “wherein the labeler comprises a plurality of clinicians and the active learning process comprises”(Szanto [0046] One can train a first model using the small amount of disease-specific data, then one can use the first model and the active learning approach, [and the active learning process comprises:]described herein to select those unlabeled x-rays that, when combined with a label that describes whether the x-ray corresponds to a positive or negative diagnosis of the disease, will give the model more predictive power. Of course, doing so can mean asking expensive (human) MD radiologists [labeler comprises from the plurality of clinicians] what the diagnosis is for the selected unlabeled images that the system is going to put into the set of labeled training examples, so the system has to be economical with the number of examples it asks to be labeled.
2. obtaining annotations from the plurality of clinicians for the selected unlabeled data samples. (Szanto [0046] One can train a first model using the small amount of disease-specific data, then one can use the first model and the active learning approach described herein to select those unlabeled x-rays that, when combined with a label that describes whether the x-ray corresponds to a positive or negative diagnosis of the disease, will give the model more predictive power. Of course, doing so can mean asking expensive (human) MD radiologists [from the plurality of clinicians] what the diagnosis is for the selected unlabeled images[for the selected unlabeled data samples] that the system is going to put into the set of labeled training examples, so the system has to be economical with the number of examples it asks to be labeled.
Wubbels in view of Szanto does not teach:
and using a weighted combination of the annotations obtained from the plurality of clinicians as the ground truth labels for the selected unlabeled data samples.
Welikala teaches “and using a weighted combination of the annotations obtained from the plurality of clinicians as the ground truth labels for the selected unlabeled data samples. (Welikala [abs] As part of the MeMoSA R (Mobile Mouth Screening Anywhere) project, images are currently in the process of being gathered from clinical experts [from the plurality of clinicians as] from across the world, who have been provided with an annotation tool to produce rich labels. A novel strategy to combine bounding box annotations from multiple clinicians is provided in this paper. [Pg-3] […] all our clinicians were of senior level, we opted not to take a weighted voting approach (e.g. based on the number of years of experience) [and using a weighted combination of the annotations obtained] and instead we simply used majority voting which assumed all clinicians were equally valued [...] [pg-4] We propose a strategy to combine annotations from multiple clinicians into a composite annotation, focusing on location and the referral decision label of the bounding boxes.
Under the BRI, in light of specification ([pg-15] The plurality of clinicians might have different levels of experience. Thus, the weights for the weighted combination can be set according to a level of experience of each of the plurality of clinicians and/or a level of consistency with which each of the plurality of clinicians provides the annotations.”
Welikala and Wubbels are related to the same field of endeavor . (i.e. Machine learning) In view of the teachings of Wubbels it would have been obvious for a person of ordinary skill in the art to apply the teachings of Welikala to Wubbels before the effective filing date of the claimed invention in order to effectively and accurately diagnose the medical disease early. (Welikala [pg-14]
Another contribution of this paper is the novel application of deep learning-based object detection to tackle oral lesion detection and classification for the early detection of oral cancer. Whilst a similar framework has been applied to detect cold sores and canker sores [20], we attempt the much more challenging task of detecting OPMDs and oral cancer
As to Claim 7
Wubbels in view of Szanto, teaches the method of claim 6
Wubbels in view of Szanto, does not teach.
Welikala teaches “wherein the weights for the weighted combination are set according to a level of experience of each of the plurality of clinicians” (Welikala [abs] A novel strategy to combine bounding box annotations from multiple clinicians is provided in this paper. [Pg-3] […]As such,using a combination of their annotations would be more reliable and stable. The combination of conventional annotations could be achieved with a variety of schemes [35], among which the ``voting policy’ was the most common and simplest, and had been demonstrated to be as effective as morecomplicated strategies [36]. The majority vote would need to be adapted to this task (detailed below) As all our clinicians [the plurality of clinicians] were of senior level, we opted not to take a weighted voting approach (e.g. based on the number of years of experience) [wherein the weights for the weighted combination are set according to a level of experience of each] and instead we simply used majority voting which assumed all clinicians were equally valued [...]
2. and/or a level of consistency with which each of the plurality of clinicians provides the annotations. (Welikala [abs] propose a strategy to combine annotations from multiple clinicians into a composite annotation, [with each plurality of clinicians provides the annotations].focusing on location and the referral decision label of the bounding boxes. This strategy was applied to each image and handled the inconsistencies discussed so far for our dataset. [and/or a level of consistency with which]
Wubbels , Szanto and Welikala are combinable for the same rationale as set forth above with respect to Claim 6.
Claims 8 is rejected under 35 U.S.C. 103 as being unpatentable over (“Wubbels”) in view of (“Szanto”) and in view of Long et. al., Pre- Grant Publication No. US 20200320349 A1 (“long”)
As to Claim 8
Wubbels in view of Szanto, teaches the method of claim 1.
Wubbels in view of Szanto does not teach.
wherein the first model is accredited by a medical body as complying with a medical standard.
Long teaches “wherein the first model is accredited by a medical body as complying with a medical standard.” (Long “paragraph [0006] In some embodiments, updates to the software program are controlled by a regulatory policy. For example, in one implementation, the software program can be or include a medical device software product, wherein clinical use of the medical device software product requires clearance and approval by a regulatory agency (e.g., the Food and Drug Administration (FDA) [is accredited by a medical body as] in accordance with define rules and regulations). [as complying with a medical standard] In accordance with these embodiments, the request can further comprise compliance information declaring that the model [wherein the first model is] development data complies with the regulatory policy and/or documenting compliance with the regulatory policy in association with generation and utilization of the model development data.
Examiner notes: Under the BRI, In light of Specification (pg-8) “The first model may have been accredited by a medical body as complying with a medical standard. For example, the first model may be accredited by the FDA or another standards agency.” Long disclosure of FDA approval is interpreted as accredited by a medical body as complying with a medical standard.
Long and Wubbels are related to the same field of endeavor. (i.e. Machine learning) In view of the teachings of Wubbels it would have been obvious for a person of ordinary skill in the art to apply the teachings of Long to Wubbels before the effective filing date of the claimed invention to use the medical models in clinical trials or at hospital for medical diagnostics.( Long, paragraph[0006] For example, in one implementation, the software program can be or include a medical device software product, wherein clinical use of the medical device software product requires clearance and approval by a regulatory agency (e.g., the Food and Drug Administration (FDA) in accordance with define rules and regulations.
Claims 9-10 is rejected under 35 U.S.C. 103 as being unpatentable over (“Wubbels”) in view of (“Szanto”) and in view (“long”) and in further view of Baylor et.al “TFX: A TensorFlow-Based Production-Scale Machine Learning Platform”(“ Baylor”)
As to Claim 9
Wubbels in view of Szanto and in view of long teaches the method of claim 8.
Wubbels in view of Szanto and in view of long does not teach
Following further training, comparing the first model as it was before the further training to the updated first model produced by the further training.
and if the comparison indicates a difference greater than a second threshold difference, flagging the updated first model for compliance testing against the medical standard.
Baylor teaches Following further training, comparing the first model as it was before the further training to the updated first model produced by the further training.(Baylor,[pg-1392] “Once a model is launched to production and is continuously being updated, [Following further training,] automated validation is used to ensure that the updated models are good. We validate that a model is safe to serve with a simple canary process. We evaluate prediction quality by comparing the model quality against a fixed threshold as well as against a baseline model (e.g., the current production model).” [comparing the first model as it was before the further training to the updated first model produced by the further training]
and if the comparison indicates a difference greater than a second threshold difference, flagging the updated first model for compliance testing against the medical standard.
(Baylor,[pg-1392] “We evaluate prediction quality by comparing the model quality against a fixed threshold as well as against a baseline model (e.g., the current production model). [and if the comparison indicates a difference greater than a second threshold difference] Any new model [the updated first model] failing any of these checks is not pushed to serving, and product teams are alerted.”[ flagging the updated first model]
for compliance testing against the medical standard: Recall the model being tested against medical standard in long paragraph [0006].
Baylor and Wubbels are related to the same field of endeavor(i.e. Machine learning] . In view of the teachings of Wubbels it would have been obvious for a person of ordinary skill in the art to apply the teachings of Baylor to Wubbels before the effective filing date of the claimed invention in order to validate the model for use in real world problems like medical diagnostics and others with Transparency to user about the decision.(i.e Trust and Compliance).( Baylor [pg-1395]While TFX is general-purpose and already supports a variety of model and data types, it must be flexible and accommodate new innovations from the machine learning community.[…] Furthermore, as machine learning becomes more prevalent, there is a strong need for understandability where a model can explain its decision and actions to users. We believe the lessons we learned from deploying TFX provide a basis for building an interactive platform that provides deeper insights to users.”)
As to Claim 10
Wubbels in view of Szanto and in view of long and in further view of Baylor teaches the method of claim 9.
Baylor teaches Following further training, comparing the first model as it was before the further training to the updated first model produced by the further training.(Baylor,[pg-1392] “Once a model is launched to production and is continuously being updated, [Following further training,] automated validation is used to ensure that the updated models are good. We validate that a model is safe to serve with a simple canary process. We evaluate prediction quality by comparing the model quality against a fixed threshold as well as against a baseline model (e.g., the current production model).” [comparing the first model as it was before the further training to the updated first model produced by the further training]
Szanto teaches “and if the comparison indicates a difference less than the second threshold difference,” (Szanto “paragraph[0003] (iii) until the model's accuracy does not improve further by a specified threshold.
deploying the updated first model for use in labelling subsequent medical data.
Szanto [0043] The process 300 can produce a set of models and deploy the best model 318. [ deploying the updated first model for use in] The process can curate a gold standard dataset to use as a final validator of model performance. [0046] One arena in which active learning has potential is in radiographic diagnosis. For example, active learning can be used if one is building a model to detect a rare disease on the basis of x-rays and one has access to millions of x-rays in a database […] [One can train a first model using the small amount of disease-specific data, then one can use the first model and the active learning approach described herein to select those unlabeled x-rays that, when combined with a label that describes whether the x-ray corresponds to a positive or negative diagnosis of the disease,. [labelling subsequent medical data.]
Wubbels , Szanto, Long and Baylor is Combinable for the same rationale as set forth above with respect to Claim 9.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIROJ KOIRALA whose telephone number is (571)270-0748. The examiner can normally be reached Monday -Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MICHAEL HUNTLEY can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.K./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
