DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The claims will be read under the broadest reasonable interpretation standard outlined in
MPEP § 2111.01.
Claim Objections
Claim 1 is objected to for the following informalities: “label” should be changed to “labeling” to maintain consistency with the recitation of a method and overall grammatical correctness with respect to “comprising”.
Claim 3 is objected to for the following informalities: “mix examples from the multiple mini-set of images” reads as a grammatical error for “mix examples from the multiple mini-sets of images”
Claim 8 is objected to for the following informalities: “The computer-implemented method of any one of claims 1” reads as a grammatical error.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 3-4 and 11-15 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
With respect to claim 3, claim 1 recites “splitting, by the data processing system, the first annotated training set of images into mini-sets of images each mini-set representing a distinct modeling subtask and comprising a limited number of examples”. However, claim 3 refers to an instance “when only one mini-set of images is available”. This is logically inconsistent, as claim 1 requires the annotated training set to split into “mini-sets” of images, and further requires “using the mini-sets” in the plural as well. As such, the examiner understands this recitation to have been in error. In the interest of compact prosecution, the examiner will assume the general intention was for claim 3 to refer to instances where the larger training pools exist in the singular/multiple.
With respect to claim 4, the claim recites “per target class” without antecedent basis.
With respect to claim 11-15, the claims recite “the updated machine learning model” without antecedent basis.
With respect to claim 13, it further recites “receiving, by the data processing system, a new image” without antecedent basis. However, claim 1 already recites a plurality of new images, including “the new images”. It is unclear whether the new image of claim 13 is of the first class (“the new images” of the training), or a separate post-training image to be evaluated. The examiner will interpret both readings as applicable to the prior art.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5-7, 11, 13, and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Oquab et. al Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks in view of Finn et. al Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (Hereinafter, “Finn”) and Su et. al Active Adversarial Domain Adaptation (Hereinafter, “Su”)
With respect to claim 1, Oquab teaches:
A computer-implemented method ([4]), comprising:
obtaining, at a data processing system, a first annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof some or all regions or objects with the images ([Abstract] “In this work we show how image representations learned with CNNs on large-scale annotated datasets can be efficiently transferred to other visual recognition tasks with limited amount of training data; [1] “This situation has changed with the appearance of large-scale ImageNet dataset [9] and the rise of GPU computing…; Fig. 2, “First, the network is trained on the source task (ImageNet classification, top row) with a large amount of available, labelled images.”) wherein the first annotated training set of images are in a first image domain (Fig. 2; [1] “For example, while objects are typically centered in Caltech256 and ImageNet datasets, other datasets such as Pascal VOC and LabelMe are more likely to contain objects embedded in a scene (see Figure 3). Differences in viewpoints, scene context, “background” (negative class) and other factors, inevitably affect recognition performance when training and testing across different domains”; [2] “Transfer learning aims to transfer knowledge between related source and target domains..”; [3] “However, this is difficult, as the labels and the distribution of images (type of objects, typical viewpoints, imaging conditions, etc.) in the source and the target datasets can be very different, as illustrated in Fig. 3; [3.1]; [3.2] “Each image typically contains one object centered and occupying significant portion of the image with limited background clutter as illustrated in Figure 3 (left). The network is trained to predict the ImageNet object class label given the entire image as input…”; [4]; Fig. 2; Fig. 3)
PNG
media_image1.png
255
442
media_image1.png
Greyscale
training, by the data processing system, the machine learning algorithm in a first phase [3] “The key idea of this work is that the internal layers of the CNN can act as a generic extractor of mid-level image representation, which can be pre-trained on one dataset (the source task, here ImageNet) and then re-used on other target tasks (here object and action classification in Pascal VOC), as illustrated in Figure 2; [3]; Fig. 2; Fig. 3; [3.1] “The parameters of layers C1…C5, FC6 and FC7 are first trained on the source task, then transferred to the target task and kept fixed.”; [4]; [4], “To understand the influence of this overlap on our results, we have pre-trained the network on a source task data formed by 1,000 ImageNet classes selected, this time, at random among all the 22,000 available ImageNet classes”)
[3]; [4]; [4] “We next apply our method to the Pascal VOC 2012 objection classification task…we compare these results to the performance of an identical network directly trained on the Pascal VOC 2012 training data (NO PRETRAIN) without using any external data from ImageNet”), wherein the second annotated training set of images are in a second image domain ([2]; [3], “The key idea of this work is that the internal layers of the CNN can act as a generic extractor of mid-level image representation, which can be pre-trained on one dataset (the source task, here ImageNet) and then re-used on other target tasks (here object and action classification in Pascal VOC)”; [3]; Fig. 2, “Pre-trained parameters of the internal layers of the network (C1-FC7) are then transferred to the target tasks (Pascal VOC object or action classification, bottom row). To compensate for different image statistics (type of objects, typical viewpoints, imaging conditions) of the source and target data we add an adaptation layer (fully connected layers FCa and Fcb) and train them on the labelled data of the target task.”; [4], “The overall performance has decreased slightly, indicating that the overlap between classes in the source and target domains may have a positive effect on the transfer”; [4]; Fig. 2; Fig. 3; Fig. 4)
training, by the data processing system, the preconditioned machine learning model in a second phase using the second annotated training set of images to generate a target machine learning model configured to detect, characterize, classify, or a combination thereof some or all regions or objects within the new images, wherein a number of classes being trained on in the first phase are a part of or match the number of classes being trained on in the second phase (Fig. 2, “…we add an adaptation layer (fully connected layers FCa and FCb) and train them on the labelled data of the target task”; Fig. 2; [3]; [3.1] “The parameters of layers C1…C5, FC6 and FC7 are first trained on the source task, then transferred to the target task and kept fixed. Only the adaptation layer is trained on the target task training data as described next”; [4]; Fig. 2) wherein a number of classes being trained on in the first phase are a part of or match the number of classes being trained on in the second phase ([4]; [4] “Our source, ILSVRC-2012 dataset contains target-related object classes, in particular, 59 species of birds and 120 breeds of dogs related to the bird and dog classes of Pascal VOC. To understand the influence of this overlap on our results…”)
Oquab does not explicitly teach:
splitting, by the data processing system, the first annotated training set of images into mini-sets of images each mini-set representing a distinct modeling subtask and comprising a limited number of examples
[training…the machine learning algorithm] using the mini-sets of images [to generate a preconditioned machine learning model…]
label, by the data processing system, a limited number of images from a target dataset to generate a second annotated training set of images
However, Finn, in the same fields of endeavor of machine learning training and image analysis, teaches:
A computer-implemented method ([5] “All of the experiments were performed using TensorFlow…The code is available online”), comprising:
obtaining, at a data processing system, a first annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof some or all regions or objects within the images ([5.2] “The Omniglot dataset consists of 20 instances of 1623 characters from 50 different alphabets…The MiniImagenet dataset was proposed by Ravi & Larochelle (2017), and involves 64 training classes, 12 validation classes, and 24 test classes”),
splitting, by the data processing system, the first annotated training set of images into mini-sets of images each mini-set representing a distinct modeling subtask and comprising a limited number of examples ([2]; [3]; [3.1], Algorithm 2, “MAML for Few-Shot Supervised Learning…”; [5.2] “The problem of N-way classification is set up as follows: select N unseen classes, provide the model with K different instances of each of the N classes, and evaluate the model’s ability to classify new instances within the N classes”)
training, by the data processing system, the machine learning algorithm in a first phase using the mini-sets of images to generate a preconditioned machine learning model configured to detect, characterize, classify, or a combination thereof some or all regions or objects within new images ([2.2], Algorithm 1; [Abstract] “In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model to be easy to fine-tune”)
training, by the data processing system, the preconditioned machine learning model in a second phase ([3.1]) [2.2]; [3.1]; [5.2] “The problem of N-way classification is set up as follows: select N unseen classes, provide the model with K different instances of each of the N classes, and evaluate the model’s ability to classify new instances within the N classes”)
PNG
media_image2.png
232
408
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab to include the limitations of task splitting, as taught by Finn. Finn teaches rapid adaptation using mini-task structuring, which can be used as a pre-training stage to predictable success within the image domain source sets of Oquab (ImageNet/ILSVRC-2012). The systems readily integrate, as following weight adjustment, the underlying architecture of Oquab could otherwise proceed without compromise. Doing so would improve Oquab’s ability to more quickly fine-tune in areas of scarce target domain data, such as in rare pathology. Disclosure of Finn provides further evidence of compatibility with a system such as Oquab ([Abstract], “We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent…”; [1] “…and it can be readily combined with fully connected convolutional, or recurrent neural networks”; [1] “simply fine-tuning the parameters slightly (e.g. by primarily modifying the top layer weights in a feedforward model) can produce good results”…).
Oquab/Finn does not explicitly teach:
label, by the data processing system, a limited number of images from a target dataset to generate a second annotated training set of images
However, Su, in the same field of endeavor of machine learning and image analysis, teaches:
A computer-implemented method ([1]), comprising:
obtaining, at a data processing system, a first annotated training set of images for training for training a machine learning algorithm to detect, characterize, classify, or a combination thereof some or all regions or objects within the images, wherein the first annotated training set of images are in a first image domain ([3.1]; [4]; [5])
training, by the data processing system, the machine learning algorithm in a first phase using [2]; [3]; Fig. 2)
label, by the data processing system, a limited number of images from a target dataset to generate a second annotated training set of images for training a machine learning algorithm to detect, characterize, classify, or a combination thereof some or all regions or objects within the images wherein the second annotated training set of images are in a second image domain ([3]; [3.2], Algorithm 1; [4]; [5]; discussing image domains KITTI and Cityscapes; Fig. 5)
PNG
media_image3.png
365
531
media_image3.png
Greyscale
training, by the data processing system, the preconditioned machine learning model in a second phase using the second annotated training set of images to generate a target machine learning model configured to detect, characterize, classify, or a combination thereof some or all regions or objects within the new images ([3]; [3.3], Algorithm 1; [4]), wherein a number of classes being trained on in the first phase are a part of or match the number of classes being trained on in the second phase ([3]; [3.1]; [3.4]; [4])
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn to include the limitations of labeling to generate the “second annotated training set of images”, as taught by Su. Doing so would increase the flexibility of the technique in operating where data is unlabeled, such as in niche contexts like rare disease pathology. The systems readily integrate, as like Oquab, Su also seeks to solve the challenge of varying image domains. The specific means by which the second annotated training set of images is generated is also minor in the overall scheme of the claim, since any remaining unlabeled data is not explicitly employed to any further effect.
With respect to claim 3, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1, wherein the splitting comprises:
when only one mini-set of images is available (Finn, [5.2]; Finn, Table 1), for each distinct modeling subtask (Finn, [2.1]), select a subset of classes to be a part of or match the number of classes being trained on in the second phase and select the limited number of examples based on the selected subset of classes (Finn, [5.2]; Finn, [3.1], Algorithm 2)
MPEP § 2111.04(II) – Contingent Limitations)
With respect to claim 5, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1, wherein the first phase further comprises:
an inner learning-loop, wherein the machine learning algorithm updates model weights or parameters on one subtask with a predefined or flexible number of epochs for initializing the preconditioned machine learning model for adaption to the target dataset (Finn, [2.2]; Algorithm 1), which generates a loss for a validation set of images after model update, denoted as L-subtask-i for an ith-subtask (Finn, [3]; Finn, [3.1]; Finn, [3.1], Algorithm 2; sampling separate validation set for task Ti, and evaluating post-update task loss using post-update parameters)
an outer learning-loop, wherein an objective is to search for a set of model initializations that generates the preconditioned machine learning model when used for updating all subtasks, each with only the limited number of examples, by finding a model initialization that minimizes a sum of all loss, denoted as summing up L-subtask-i, with i ranging from 1 to a number of subtasks calculated from validation sets of images of the subtasks with respect to the model initialization (Finn, [2.2]; Finn, [2.2] “The intuition behind this approach is that some internal representations are more transferrable than others…we will aim to find model parameters that are sensitive to changes in the task, such that small changes in the parameters on the loss function of any task drawn from p(T), when altered in the direction of the gradient of that loss…In effect, our proposed method aims to optimize the model parameters such that one or a small number of gradient steps on a new task will produce maximally effective behavior on that task”; [2.2],
PNG
media_image4.png
43
370
media_image4.png
Greyscale
; Finn, [2.2], Algorithm 1, line 8; Finn, [3]; Finn, [3.1]; Finn, [3.1], Algorithm 2, line 10; Finn, [2.2], Equation 2)
With respect to claim 6, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1
wherein the training of the second phase comprises performing iterative operations to learn a set of parameters to detect, characterize, classify, or a combination thereof some or all regions or objects within the second annotated training set of images that maximizes or minimizes a cost function (Oquab [2], “We pre-train the convolutional layers of CNNs on a large-scale supervised task”; Oquab, [3.2]; Oquab, [3.2], “Dealing with background…re-weighting its gradients during training…mini-batches for the stochastic gradient descent training”; Oquab, [4]; Finn, [2.2], Algorithm 1; Finn, [2.2]; Finn, [3.1]; Su, [3]; Su, [4])
wherein each iteration involves finding the set of parameters for the preconditioned machine learning model so that a value of the cost function using the set of parameters is larger or smaller than a value of the cost function using another set of parameters in a previous iteration (Finn, [2.1]; Finn, [2.2]; Finn, [2.2], Equation 1, Algorithm 1; Finn, [3.1], Algorithm 2; Oquab, [4], “The initial learning rates are set to 0.01 and the network is trained until the cross-entropy is stabilized…learning rates are then divided by 10 and the training procedure repeats”; Su, [3.1]; Su, [4])
wherein the cost function is constructed to measure a difference between predictions made for some or all the regions or the objects using the preconditioned machine learning model and ground truth labels provided for the second annotated training set of images (Finn, [2.2]; Finn, [3.1]; Finn, Equations 2-3; Oquab, [3.2]; Oquab, [3.2], “To label patches in training images we measure the overlap between the bounding box of a patch P and ground truth bounding boxes B of annotated objects…”; Oquab, [4]; Oquab, Fig. 4; Su, [3.1]; Su, [4])
PNG
media_image5.png
350
389
media_image5.png
Greyscale
With respect to claim 7, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1
wherein the training of the second phase comprises performing iterative operations to learn a set of parameters to detect, characterize, classify, or a combination thereof some or all regions or objects within the second annotated training set of images that maximizes or minimizes a cost function (Finn, [2.2]; Finn, [2.2], “updated parameter vector…is computed using one or more gradient descent updates; Finn, [3.1]; Finn, [3.1], Algorithm 2, lines 6-8; Oquab, [3.1]; Oquab, [3.1] “The parameters of layers C1...C5, FC6 and FC7 are first trained on the source task, then transferred to the target task and kept fixed. Only the adaptation layer is trained on the target task training data as described next”; Oquab, [3.2]; Oquab, [3.2], “resampled training set is then used to form mini-batches for the stochastic gradient descent training”; Su, [3]; Su, [4])
wherein each iteration involves finding the set of parameters for the preconditioned machine learning model so that a value of the cost function using the set of parameters is larger or smaller than a value of the cost function using another set of parameters in a previous iteration (Finn, [2.2]; Finn, [3.1]; Finn, [3.1], Algorithm 2; Finn, [5.1] “MAML continues to improve with additional gradient steps…achieving a loss that is substantially lower…”; Finn, Fig. 3; Oquab, [3.2], “This resampled training set is then used to form mini-batches for the stochastic gradient descent training. This is implemented by sampling a random 10% of the training background patches”; Oquab, [3.2]; Oquab, [4] “Training convolutional networks”; Oquab, [4]; Su, [3.1]; Su, [4])
PNG
media_image6.png
230
392
media_image6.png
Greyscale
wherein the cost function is constructed to measure a difference between predictions made for some or all the regions or the objects using the preconditioned machine learning model and ground truth labels provided for the second annotated training set of images (Finn, [2.2]; Finn, [3.1], Equations 2-3; Finn, [3.2], Algorithm 2; Oquab, [3.2]; Oquab, Fig. 4; Oquab, [4]; Su, [3.1]; Su, [4])
With respect to claim 11, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1, further comprising providing the target machine learning model and/or the updated machine learning model (Oquab, [5] “The code of our method is available at…”)
With respect to claim 13, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1, further comprising:
receiving, by the data processing system, a new image (Finn, [2.1] “At the end of meta-training, new tasks are sampled from p(T), and meta-performance is measured by the model’s performance after learning from K samples. Generally, tasks used for meta-testing are held out during meta-training.”; Finn, [2.1]; Finn, [5.2], “The problem of N-way classification is set up as follows: select N unseen classes, provide the model with K different instances of each of the N classes, and evaluate the model’s ability to classify new in stances within the N classes. For Omniglot, we randomly select 1200 characters for training, irrespective of alphabet, and use the remaining for testing”; Finn, [5.2]; Oquab, [3.3] “At test time we apply the network to each of the (approximately) 500 overlapping multi-scale patches extracted from the test image”; Oquab, [3.3]; Oquab [4]; Su, [4]; Su, [5])
inputting the new image into the target machine learning model or the updated machine learning model (Finn, [2.1]; Finn, [5.2]; Oquab, [3.3]; Oquab, [4]; Su, [4]; Su, [5])
detecting, characterizing, classifying, or a combination thereof, by the target machine learning model or the updated machine learning model, some or all regions or objects within the new images (Finn, [2.1]; Finn, [5.2]; Oquab, [3.3]; Oquab, [4] “For qualitative assessment of localization results, we compute an output map for each category by averaging the scores of all the testing patches covering a given pixel of the test image. Examples of such output maps are given in Figures 1 and 5…”; Oquab, [4]; Su, [4]; Su, [5])
outputting, by the target machine learning model or the updated machine learning model, an inference based on the detecting, characterizing, classifying, or a combination thereof (Finn, [2.1]; Finn, [2.1], “The model f is then improved by considering how the test error on new data from qi changes with respect to the parameters”; Finn, [5.2]; Oquab, [3.3]; Oquab, [4]; Su, [4]; Su, [5])
With respect to claim 16, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1, wherein the training the machine learning algorithm comprises implementing meta-learning principles to enable the first phase to use the limited number of examples to generate the preconditioned machine learning models (Oquab, Fig. 2; Oquab, Fig. 2, “First, the network is trained on the source task (ImageNet classification, top row) with a large amount of available labelled images. Pre-trained parameters of the internal layers of the network (C1-FC7) are then transferred to the target tasks (Pascal VOC object or action classification, bottom row). To compensate for the different image statistics (type of objects, typical viewpoints, imaging conditions) of the source and target data we add an adaptation layer (fully connected layers FCa and FCb) and train them on the labelled data of the target task”; Oquab, [3.2], “This resampled training set is then used to form mini-batches for the stochastic gradient descent train ing. This is implemented by sampling a random 10% of the training background patches”; Oquab, [3.2]; Finn, [Abstract] “We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning…In effect, our method trains the model to be easy to fine-tune”; Finn, [2.1. Meta-Learning Problem Set-up]; Fig. 1; Finn, [2.2]; Finn, [3.1])
With respect to claim 17, Oquab/Finn/Su teaches:
The computer-implemented method of claim 1, wherein the training the preconditioned machine learning model comprises implementing meta-learning principles to enable the second phase to use the limited number of images to generate the target machine learning model (Finn, [Abstract]; Finn, [2.1]; Finn, [2.2])
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Oquab/Finn/Su in view of Hosseini et. al Atlas of Digital Pathology: A Generalized Hierarchical Histological Tissue Type-Annotated Database for Deep Learning (Hereinafter, “Hosseini”)
With respect to claim 2, Oquab/Finn/Su teaches the method of claim 1.
Oquab/Finn/Su does not explicitly teach the further limitations of claim 2
However, Hosseini, in the same field of endeavor of digital pathology training, teaches:
wherein the first annotated training set of images are digital pathology images comprising one or more types of cells ([Abstract]; [1]; Fig. 1)
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su to include the limitations of annotated digital pathology images. Doing so would increase the ability of the Oquab/Finn/Su to specifically recognize digital pathology for the purposes of diagnosis. The systems readily integrate, as the method of Oquab/Finn/Su otherwise is adapted to classify a broad variety of images subject to training.
Claims 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Oquab/Finn/Su in view of Jha et. al (US 20220309670 A1) (Hereinafter, “Jha”) and Lange A Continual Learning Survey: Defying Forgetting in Classification Tasks (Hereinafter, “Lange”)
With respect to claim 8, Oquab/Finn/Su teaches the method of claim 8.
Oquab/Finn/Su does not explicitly teach the further limitations of claim 8
However, Jha, in the same field of endeavor of machine learning and image analysis, teaches:
The computer-implemented method of any of claims 1, further comprising:
identifying a digital pathology scenario ([0039]; [0045] “The segmentation system can be configured to segment images into any appropriate set of tissue classes…”; [0057]-[0059] “As another example, the user interface 612 may enable the user to change the region of the image labelled as having a particular tissue class”; Fig. 8, 802-804)
selecting an adaptive continual learning method for updating the target machine learning model given the digital pathology scenario ([0048] “In this manner, rather than being trained once on a static and limited set of training data (as in some conventional systems), the machine learning model continually learns and adapts its parameter values based on the feedback being provided by the user through the edited segmentation”; [0057]-[0058]; [0042]; [0063])
updating the target machine learning model based on the adaptive continual learning method to generate an updated machine learning model ([0048]; [0062]; [0075])
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su to include the limitations of adaptive continual learning, as taught by Jha. Doing so would ensure the model continues to refine past initial training. The systems readily integrate, as Jha specifically involves adapting a machine-learning model to various digital pathology contexts, even as training data is not available ([0046] “However, it may be difficult to train a machine learning model to achieve a level of accuracy that would be considered acceptable for certain practical applications, e.g., identifying cancerous tissue in biopsy samples. In particular, the microscopic appearance of tissue can be highly complex and variable due to factors that are both intrinsic to the tissue (e.g., the type and stage of the disease present in tissue) and extrinsic to the tissue (e.g., how the microscope is calibrated and the procedure used to stain the tissue). This makes it hard to aggregate a set of labeled training data (i.e., for training a machine learning model) that is sufficiently large to capture the full scope of possible variations in the microscopic appearance of tissue”)
In the narrow sense, Jha does not “identify”/“select” a “scenario”/“method”, as it uses one technique for achieving adaptive continual learning. There is no explicit proof that a user of Jha considered from multiple options prior to choosing to (continual[ly]) train its artificial intelligence in the ultimate fashion of Jha. However, Lange, in the same field of endeavor of continual learning teaches:
[identifying multiple scenarios, then choosing one] ([2]; [2] “Task incremental learning defines {Y(t)} ̸ = {Y(t+1)}, which additionally requires task label t to indicate the isolated output nodes Y(t) for current task t…Our experiments focus on the task incremental learning setting for a sequence of classification tasks”)
[selecting from methods] ([6]; [6.3] “In the following, we discuss results of model choice”; Table 4; Table 5)
The examiner notes that under the plain language of claim 8, it is not required that a machine be configured to utilize multiple “scenario[s]” and “method[s]”. It is sufficient that during the performance of the method (which under a “continual learning” approach may be perpetual) the user of the system is aware (or becomes aware) that these alternative scenarios/methods exist, such that a decision to continue the loop under any given methodology becomes a conscious operational choice (to identify/select a certain scenario/method, in favor of others). Accordingly, it would have been obvious to a person of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su/Jha to include the limitations of multiple scenario and method consideration, as taught by Lange. Doing so would facilitate potential further modifications to suit changing user needs, circumstances, and efficiency thresholds.
The systems readily integrate, as the adaptive continual learning as recited constitutes a post-processing step that is otherwise detached from the central processing of claim 1. The exact adaptive continual learning scenario/method chosen by Oquab/Finn/Su/Jha/Lange, under BRI, may appropriately also read as a user preference amongst options. As the language claims the choice of options rather than the execution of each possible option, a person of ordinary skill in the art need not also know the precise details of how to actually change a model to a different scheme – only that other similar schemes for continual learning exist in an identified and finite form (regardless of whether these options are actually pursued in the end) Such knowledge is valuable in its own right, for the advantage of further experimentation. See Rationale E, MPEP § 2141(III), “Obvious to try”.
With respect to claim 9, Oquab/Finn/Su/Jha/Lange teaches:
The computer-implemented method of claim 8, wherein the digital pathology scenario is a data incremental scenario (Jha, [0007]; Jha, [0047]-[0049]; Jha, [0059] “Generally, the training data 616 stores multiple training examples (i.e., that each specify a respective image and an edited segmentation), and may be continually augmented over time as users generate edited segmentations of new images”; Lange, [2]) , a domain incremental scenario (Lange, [2]), class incremental scenario (Lange, [2]), or a task incremental scenario (Lange, [2])
With respect to claim 10, Oquab/Finn/Su/Jha/Lange teaches:
The computer implemented method of claim 8, wherein the adaptive continual learning method is selected from the group comprising: Elastic Weight Consolidation (EWC) (Lange, [5.2]), Learning without Forgetting (LWF) (Lange, References, [58]), Incremental Learner and Representation Learning (iCaRL) (Lange, [3.1]), Continual Prototype Evaluation (CoPE) (Lange, References, [33]; noting that Continual Prototype Evolution (CoPE) and its requisite paper are synonymously referred to as CoPE (Continual Prototype Evaluation) as evidenced by Munagala et. al CLActive: Episodic Memories for Rapid Active Learning, [2.2]; Jain et. al Instance-Conditional Timescales of Decay for Non-Stationary Learning, [4.3]), A-GEM (Lange, [3.1]), and a parameter isolation method (Lange, [3]).
The examiner further notes that under the language “comprising”, the ultimate selection made could involve a technique outside of the specific recitals of claim 10, as relevant to Jha, [0048]. Similarly, as a potentially user-driven method operating under a “select[ion]”, there is no requirement that a single working embodiment be configured to operate under every disclosed member from the group. See, MPEP § 2111.04(II) – Contingent Limitations. It is sufficient that a person of ordinary skill in the art actively considers the advantages of these sub-methods during the training process for the advantage of potentially modifying the system – even if said person is not actually aware of the details on how it would be achieved. Lange sufficiently outlines the various advantages of different continual learning methods and the general ability to choose amongst them, which a person of ordinary skill in the art could further research in making any potential modifications to the perpetual post-training process, or otherwise simply consider in such a fashion that their continual training with a given scenario/method would constitute a conscious choice with agency.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Oquab/Finn/Su in view of Hosseini and Bera et. al Artificial intelligence in digital pathology – new tools for diagnosis and precision oncology (Hereinafter, “Bera”)
With respect to claim 12, Oquab/Finn/Su teaches the method of claim 11
Oquab/Finn/Su does not explicitly teach the further limitations of claim 12
However, Hosseini teaches:
wherein the providing comprises deploying the target machine learning model and/or the updated machine learning model in [Abstract] “We demonstrate the quality of our image labels through pathologist consultation and by training three state-of-the art neural networks on tissue type classification”; [1]; Fig. 1)
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su to include the limitations of annotated digital pathology images. Doing so would increase the ability of the Oquab/Finn/Su to specifically recognize digital pathology for the purposes of diagnosis. The systems readily integrate, as the method of Oquab/Finn/Su otherwise is adapted to classify a broad variety of images subject to training.
Oquab/Finn/Su/Hosseini does not explicitly teach:
in a digital pathology system
However, Bera, in the same field of endeavor of artificial intelligence and digital pathology, teaches:
in a digital pathology system ([Abstract]; Table 1, “Large amounts of well-annotated data with several exemplars from the different target categories or classes of interest are often required”)
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su to include the limitations
Claims 14 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Oquab/Finn/Su in view of Lee et. al (US 20240136068 A1) (Hereinafter, “Lee”)
With respect to claim 14, Oquab/Finn/Su teaches the method of claim 13
Oquab/Finn/Su does not explicitly teach the further limitations of claim 14.
However, Lee, in the same field of endeavor of machine learning and image analysis, teaches:
determining, by a user ([0041] “A user of the present disclosure may be a medical expert, such as, a physician, a nurse, a clinical pathologist, a sonographer, a medical imaging professional, as well as an ordinary person such as a patient or a guardian, but is not limited to these examples”; [0048]; [0084]-[0086] “Referring to FIG. 8 , the prediction device 10 may provide a interface screen 21 for inputting a medical image that is a source of medical prediction to a user terminal 20. The user terminal 20 may display the interface screen 21 for inputting medical image and transmit a medical image selected by the user to the prediction device 10. The user terminal 20 may receive a selection of mammograms of the patient”; [0056] “The prediction device 10 may extract the risk factors that influenced the prediction result in order of contribution and provide the extracted risk factors in the patient report. This allows the patient or doctor to understand the primary risk factor influencing the disease risk, whether it be age, family history, breast density, or lesion information detected in the image, enabling them to establish an appropriate treatment plan”; [0093]), a diagnosis of a subject associated with the new image, wherein the diagnosis is determined based on the inference output by the target machine learning model or the updated machine learning model ([0056]; [0088] “The user terminal 20 may provide a patient report written based on the medical prediction from the prediction device 10 on the screen 23. The patient report may include a disease risk (for example, breast cancer risk) that is predictable from the input images. The disease risk may be represented as a risk score… Further, the patient report may further include a prediction of the likelihood of a cancer diagnosis based on whether or not the recommended scan is performed”; [0093]; Fig. 6)
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su to include the limitations of inference and diagnosis, as taught by Lee. Doing so would transfer the broad advantages of Oqaub/Finn’s classification techniques into the specific field of medical diagnosis – where fine-tuning and precision are of major importance. The systems readily integrate, as Oquab/Finn/Su is configured to classify images, which could necessarily include the classification of a pathology or otherwise.
With respect to claim 18, the claim is parallel to claim 1, but recited in the form of a system. Lee teaches:
one or more data processors ([0039]; [0095])
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations ([0039]; [0095]-[0098]) comprising: [the method of claim 1]
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su to include the limitations of hardware implementation via a non-transitory computer readable storage medium, as taught by Lee. Doing so would allow efficient execution of the method across multiple devices. The systems readily integrate, as the methods of Oquab/Finn/Su are already available as code (Oquab, [5])
With respect to claim 19, the claim is parallel to claim 1, but recited in the form of a computer-program product. Lee teaches:
A computer-program product ([0039] “The computer program may be downloaded through a network or sold as a product”) tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform operations ([0039] “A computer program may include instructions that cause a processor to execute the operations of the present disclosure and may be stored on a non-transitory computer readable storage medium”) comprising: [the method of claim 1]
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su to include the limitations of computer program product implementation, as taught by Lee. Doing so would allow efficient distribution of the method across multiple devices. The systems readily integrate, as the methods of Oquab/Finn/Su are already available as code (Oquab, [5])
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Oquab/Finn/Su/Lee in view of Xie et. al (US 20220230310 A1) (Hereinafter, “Xie”)
With respect to claim 15, Oquab/Finn/Su/Lee teaches:
The computer-implemented method of claim 14, further comprising:
[0091] “The patient report may include personalized treatment plans, recommended scans, recommended treatments, and the like. For example, in case of a patient with a predicted ovarian cancer risk score higher than a reference, the patient report may recommend an ovarian cancer scan with a statement like, “Your risk score of 0.9 indicates a higher likelihood of developing ovarian cancer, placing you in the high-risk group. We recommend getting an additional ovarian cancer scan”)
Oquab/Finn/Su/Lee does not explicitly teach:
administering, by the user
However, Xie, in the same field of endeavor of medical diagnosis, teaches:
administering, by the user ([0030])
It would have been obvious to one of ordinary skill in the art as of the effective filing date of the claimed invention to modify Oquab/Finn/Su/Lee to include the limitations of treatment administration by the user. Doing so would provide a practical application of the information which Oquab/Finn/Su/Lee is already configured to generate. The systems readily integrate, as the user of Oquab/Finn/Su/Lee should also reasonably be capable of administering treatment (Lee, [0041]).
Allowable Subject Matter
Claim 4 stands rejected under 35 U.S.C. § 112(b), but otherwise overcomes the prior art.
With respect to claim 4, modification would involve a non-trivial incorporation of prototype representation through vector averaging. Claim 4 recites using “unlabeled” data as more than a byproduct, to achieve its result. In contrast, claim 1 generates an annotated dataset (rather than using a pre-labeled dataset), but ultimately uses the “second annotated training set” in line with the similar technique of Oquab. Claim 4 however, turns a leftover unlabeled portion of the target dataset into an integrated component of the second phase. Su sufficiently bridges any gaps in the context of claim 1, however further drawing away from the teachings of the base labeled techniques would constitute impermissible hindsight.
Additional References
Additionally cited references (see attached PTO-892) otherwise not relied upon above have been made of record in view of the manner in which they evidence the general state of the art.
The examiner notes in particular Snell et. al Prototypical networks for few-shot learning, teaching subject matter in line with claim 4. While not obvious to combine under the current claim structure, the reference has been provided for the Applicant’s consideration.
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NOAH WILLIAM BOYAR whose telephone number is (571)272-8392. The examiner can normally be reached 8:30 – 5:00 EST, Monday – Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at https://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NOAH W BOYAR/Examiner, Art Unit 2669
/CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
