DETAILED ACTION
Claims 1-20 are currently pending.
The objection to the title of the invention is maintained.
The previous rejections to claims 1-20 under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, are withdrawn due to Applicant’s amendment.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/18/25 has been entered.
Response to Arguments
Applicant's arguments filed 11/18/25 have been fully considered but they are not persuasive.
The Applicant argues on pages 12-13 of the response in essence that: The '733 application discloses that the "at the beginning of every epoch t . . . [as a result of] the part reassignment, the centroid approximation by the nearest tiles, as well as the MSE loss for pushing the feature of these tiles to their centroids, . . . important features [will be weighed] more than others." See '733 application, paras. [0025]-[0027]. As such, Applicant argues that the '733 application complies with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. 119(e)
The '733 application fails to provide adequate support for “updating, by the computing system using the loss metric, at least one of the first plurality of weights in the tile encoder, the plurality of centroids of the clusterer, or the second plurality of weights in the aggregator based on the comparison”. Although the '733 application specifies that “the MSE loss for pushing the feature of these tiles to their centroids, will weigh the important features more than others”, the '733 application does not specify that this results in updating weights in the tile encoder, centroids of the clusterer, or weights in the aggregator.
The Applicant argues on page 15 of the response in essence that: Further, neither Madabhushi nor the other cited references disclose each aspect of the claims, including "updating, by the computing system using the loss metric, at least one of the first plurality of weights in the tile encoder, the plurality of centroids of the clusterer, and the second plurality of weights in the aggregator based on the comparison." (emphasis added)
Madabhushi discloses minimizing the softmax loss function may include searching for a weight vector that minimizes the softmax loss function (paragraph 59). The claim limitation specifies updating “at least one of the first plurality of weights in the tile encoder, the plurality of centroids of the clusterer, and the second plurality of weights in the aggregator” (emphasis added). Reciting “at least one of” before listing elements places the listing in alternative form. Thus, Madabhushi disclosure of updating the weight vector reads on the claim limitation.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested:
SYSTEMS AND METHODS FOR END-TO-END PART LEARNING FOR IDENTIFYING CANCER
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. 119(e) as follows:
The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original nonprovisional application or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 U.S.C. 112(a) or the first paragraph of pre-AIA 35 U.S.C. 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994).
The disclosure of the prior-filed application, Application No. 63/033,733, fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph for one or more claims of this application. For example, Application No. 63/033,733 fails to provide adequate support for the limitations of “determining, by the computing system, a loss metric based on a comparison between the second category determined by the classification model with the first category of the label of the training dataset” and “updating, by the computing system using the loss metric, at least one of the first plurality of weights in the tile encoder, the plurality of centroids of the clusterer, or the second plurality of weights in the aggregator based on the comparison”. Accordingly, claims 1-20 are not entitled to the benefit of the prior application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Rejections Using Prior Art That Can Possibly Be Excepted under 102(b)(1)(A)
Claims 1-4 and 6-20 are rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. US Publication 2017/0161891 (hereafter “Madabhushi”) and Muhammad et al. Unsupervised Subtyping of Cholangiocarcinoma Using a Deep Clustering Convolutional Autoencoder. Paper in International Conference on Medical Image Computing and Computer-Assisted Intervention Pages 604-612 (hereafter “Muhammad”).
Referring to claim 1, Madabhushi discloses a method of training models to classifying biomedical images, comprising:
identifying, by a computing system, a training dataset comprising (i) a first plurality of tiles of a biomedical image of a sample and (ii) a label identifying a first category for the sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles);
applying, by the computing system, the first plurality of tiles to a classification model, the classification model comprising:
a tile encoder having a first plurality of weights to generate, based on the first plurality of tiles, a plurality of feature vectors defined in a feature space, wherein a number of feature vectors corresponds to a number of the first plurality of tiles (paragraph 55, First layer 220 comprises a convolutional layer 224 followed by a pooling layer 226. Convolutional layer 224 applies a 2D convolution of the input image 210 with a kernel of eight pixels by eight pixels to produce a feature map. Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224);
a clusterer to select a subset of feature vectors from the plurality of feature vectors from the tile encoder (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224); and
an aggregator having a second plurality of weights to determine, based on the subset of feature vectors from the clusterer, a second category for the sample (paragraph 55, The output of pooling layer 226 is fed to fully connected layer 240. Fully connected layer 240 outputs to a final classification layer 250. Final classification layer 250 is a softmax classifier comprising an invasive tissue output 260 and a non-invasive tissue output 270. Invasive tissue output 260 and non-invasive tissue output 270 are activated by a logistic regression function);
determining, by the computing system, a loss metric based on a comparison between the second category determined by the classification model with the first category of the label of the training dataset (paragraph 59, Method 300 further includes, at 350 minimizing a softmax loss function. Example methods and apparatus may minimize the softmax loss function using a stochastic gradient descent);
updating, by the computing system using the loss metric, at least one of the first plurality of weights in the tile encoder, the plurality of centroids of the clusterer, and the second plurality of weights in the aggregator based on the comparison (paragraph 59, Minimizing the softmax loss function may include searching for a weight vector that minimizes the softmax loss function); and
storing, by the computing system, in one or more data structures, the first plurality of weights in the tile encoder, and the second plurality of weights of the aggregator (paragraph 55, FIG. 2 illustrates in greater detail an example CNN architecture 200 that may be implemented with example methods and apparatus described herein).
Madabhushi does not disclose expressly a clusterer to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space.
Muhammad discloses a clusterer to select a subset of feature vectors from the plurality of feature vectors from the tile encoder based on a plurality of centroids defined in the feature space, wherein:
the plurality of centroids define a set of regions within the feature space, each region of the set of regions corresponding to a portion of the feature space,
the set of regions are defined based on distances about the plurality of centroids in the feature space,
the clusterer assigns the plurality of feature vectors to the set of regions based on the distances to the plurality of centroids within the feature space, and
the subset of feature vectors includes include a at least one feature vector from each of set of the set of regions (page 608, In the case of the first training epoch, centroid locations are randomly assigned upon initialization. A training epoch is defined by the forward-passing of all mini-batches once through the network. After θ have been updated, all samples are reassigned to the nearest centroid using Eq. 3. Finally, all centroid locations are updated using Eq. 4 and used in the calculations of the next training epoch).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space. The motivation for doing so would have been to use a fast and reliable manner of segmentation in order to analyze the cell structure that has the capacity to analyze any number of clusters without bias introduced by training. Therefore, it would have been obvious to combine Muhammad with Madabhushi to obtain the invention as specified in claim 1.
Referring to claim 2, Madabhushi discloses wherein applying further comprises applying the first plurality of tiles from the biomedical image of a plurality of biomedical images to the classification model, the clusterer of the classification model to identify the plurality of feature vectors from which to select the subset of feature vectors based on the biomedical image (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claim 3, Madabhushi discloses wherein applying further comprises applying the first plurality of tiles to the classification model, the clusterer of the classification model to select a subset of tiles in the biomedical image based on the subset of feature vectors (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claim 4, Madabhushi discloses wherein applying further comprises applying the first plurality of tiles to the classification model, the aggregator of the classification model to determine a plurality of confidence scores for a subset of tiles corresponding to the subset of feature vectors (paragraph 45, Method 100 also includes, at 152, receiving, from the automated classifier, a prediction probability. The prediction probability is based, at least in part, on the sampling subset. The prediction probability indicates the probability of invasive pathology at a location in the WSI occupied by a tile. In one embodiment, the prediction probability may be within, for instance, a range of 1 for invasive tissue, to 0 for non-invasive tissue. In one embodiment, a prediction probability in the range (0.4, 0.6) indicates a threshold uncertainty as to whether the tissue at the location occupied by the tile is invasive or non-invasive).
Referring to claim 5, Madabhushi discloses wherein identifying further comprises identifying the training dataset comprising a plurality of labels for a corresponding first plurality of categories for the sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles), and
wherein applying further comprises applying the first plurality of tiles to the classification model, the aggregator of the classification model to determine a second plurality of categories for the sample (paragraph 65, Method 600 also includes, at 640, receiving, from the automated classifier, a classification of the sample set of tiles).
Referring to claim 7, Muhammad discloses wherein updating further comprises determining the plurality of centroids based on a second plurality of feature vectors generated by the tile encoder, subsequent to updating of the first plurality of weights (page 608, In the case of the first training epoch, centroid locations are randomly assigned upon initialization. A training epoch is defined by the forward-passing of all mini-batches once through the network. After θ have been updated, all samples are reassigned to the nearest centroid using Eq. 3. Finally, all centroid locations are updated using Eq. 4 and used in the calculations of the next training epoch).
Referring to claims 8 and 15, Madabhushi discloses a method of classifying biomedical images, comprising:
identifying, by a computing system, a first plurality of tiles from a first biomedical image of a first sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles);
determining, by the computing system, a first category for the first sample by applying the first plurality of tiles to a classification model, the classification model trained using a training dataset having a plurality of examples each including (i) a second plurality of tiles of a second biomedical image of a second sample and (ii) a label identifying a second category for the sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles), the classification model comprising:
a tile encoder having a first plurality of weights to determine, based on the first plurality of tiles, a corresponding plurality of feature vectors in a feature space (paragraph 55, First layer 220 comprises a convolutional layer 224 followed by a pooling layer 226. Convolutional layer 224 applies a 2D convolution of the input image 210 with a kernel of eight pixels by eight pixels to produce a feature map. Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224);
a clusterer to select a subset of feature vectors from the plurality of feature vectors (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224); and
an aggregator having a second plurality of weights to generate, based on the subset of feature vectors, the first category for the sample,
wherein the aggregator is updated based on a comparison, the comparison having been determined based on a loss metric between the second category and the first category (paragraph 59, Minimizing the softmax loss function may include searching for a weight vector that minimizes the softmax loss function); and
storing, by the computing system, an association between the first category and the first biomedical image (paragraph 55, The output of pooling layer 226 is fed to fully connected layer 240. Fully connected layer 240 outputs to a final classification layer 250. Final classification layer 250 is a softmax classifier comprising an invasive tissue output 260 and a non-invasive tissue output 270. Invasive tissue output 260 and non-invasive tissue output 270 are activated by a logistic regression function).
Madabhushi does not disclose expressly a clusterer to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space, or wherein the tile encoder, the clusterer, and the aggregator are updated based on a comparison.
Muhammad discloses a clusterer to select a subset of feature vectors from the plurality of feature vectors from the tile encoder based on a plurality of centroids defined in the feature space, wherein the plurality of centroids define a set of regions within the feature space, each region of the set of regions corresponding to a portion of the feature space, wherein the set of regions are defined based on distances about the plurality of centroids in the feature space, wherein the clusterer assigns the plurality of feature vectors to the set of regions based on the distances to the plurality of centroids within the feature space, wherein the subset of feature vectors includes include a at least one feature vector from each of set of regions (page 608, In the case of the first training epoch, centroid locations are randomly assigned upon initialization. A training epoch is defined by the forward-passing of all mini-batches once through the network. After θ have been updated, all samples are reassigned to the nearest centroid using Eq. 3. Finally, all centroid locations are updated using Eq. 4 and used in the calculations of the next training epoch); and
wherein the tile encoder, the clusterer, and the aggregator are updated based on a comparison, the comparison having been determined based on a loss metric between the second category and the first category (page 606, Mean-squared-error loss (MSE) is commonly used to optimize such a model, updating model weights (θ) relative to the error between the original (input, xi) image and the reconstruction (output, x’i) image in a set of N images. Each mini-batch is forwarded through the model and network weights are respectively updated. At the end of an epoch, defined by the forward-passing of all mini-batches, cluster assignments are updated by Eq. 3, given the new embedding space).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space, and to update the tile encoder, the clusterer, and the aggregator. The motivation for doing so would have been to use a fast and reliable manner of segmentation in order to analyze the cell structure that has the capacity to analyze any number of clusters without bias introduced by training. Updating the tile encoder, the clusterer, and the aggregator would improve the result of the machine learning model. Therefore, it would have been obvious to combine Muhammad with Madabhushi to obtain the invention as specified in claims 8 and 15.
Referring to claims 9 and 16, Madabhushi discloses providing, by the computing system, the association between the first category and the first biomedical image (paragraph 65, Method 600 also includes, at 640, receiving, from the automated classifier, a classification of the sample set of tiles).
Referring to claims 10 and 17, Madabhushi discloses determining a plurality of confidence scores for a subset of tiles corresponding to the subset of feature vectors (paragraph 45, Method 100 also includes, at 152, receiving, from the automated classifier, a prediction probability. The prediction probability is based, at least in part, on the sampling subset. The prediction probability indicates the probability of invasive pathology at a location in the WSI occupied by a tile. In one embodiment, the prediction probability may be within, for instance, a range of 1 for invasive tissue, to 0 for non-invasive tissue. In one embodiment, a prediction probability in the range (0.4, 0.6) indicates a threshold uncertainty as to whether the tissue at the location occupied by the tile is invasive or non-invasive).
Referring to claims 11 and 18, Madabhushi discloses determining a second plurality of categories for the sample (paragraph 19, The training exemplars may be annotated by an expert pathologist. Example methods and apparatus may extract tiles of a fixed size from annotated invasive tissue regions and annotated non-invasive tissue regions).
Referring to claims 12 and 19, Madabhushi discloses selecting a subset of tiles in the biomedical image based on the subset of feature vectors (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claims 13 and 20, Madabhushi discloses selecting the first plurality of tiles from a second plurality of tiles of the first biomedical image paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claim 14, Madabhushi discloses obtaining, by the computing system, the first biomedical image of the first sample via a histological image preparer (paragraph 62, Method 600 includes, at 610, accessing a radiological image of a region of tissue demonstrating invasive pathology. The radiological image may be a digitized WSI of a hematoxylin and eosin (H&E) stained histopathology slide of a region of tissue demonstrating cancerous pathology).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. US Publication 2017/0161891 and Muhammad as applied to claim 1 above, and further in view of Backhus et al. US Publication 2019/0286980 (hereafter “Backhus”).
Referring to claim 6, Madabhushi discloses determining, by the computing system, a loss metric based on comparison among the plurality of feature vectors, and
wherein updating further comprises updating, using the loss metric, at least one of the first plurality of weights in the tile encoder or the plurality of centroids of the clusterer within the feature space (paragraph 59, Method 300 further includes, at 350 minimizing a softmax loss function. Example methods and apparatus may minimize the softmax loss function using a stochastic gradient descent).
Muhammad discloses the plurality of centroids in the feature space (page 608, In the case of the first training epoch, centroid locations are randomly assigned upon initialization. A training epoch is defined by the forward-passing of all mini-batches once through the network. After θ have been updated, all samples are reassigned to the nearest centroid using Eq. 3. Finally, all centroid locations are updated using Eq. 4 and used in the calculations of the next training epoch).
Madabhushi and Muhammad do not disclose expressly determining a second loss metric.
Backhus discloses determining, by the computing system, a second loss metric based on comparison among the plurality of feature vectors and the plurality of centroids in the feature space, and
wherein updating further comprises updating, using the second loss metric, at least one of the first plurality of weights in the tile encoder or the plurality of centroids of the clusterer within the feature space (paragraph 7, The second loss function is a linear combination of the classification error and a clustering error. The clustering error aggregates a deviation between each of the kernels and one of the C centroids nearest to the respective kernel).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to determine a second loss metric. The motivation for doing so would have been to improve the performance of the classifier. Therefore, it would have been obvious to combine Backhus with Madabhushi and Muhammad to obtain the invention as specified in claim 6.
Rejections Not Using Prior Art That Can Possibly Be Excepted under 102(b)(1)(A)
Claims 1-4 and 6-20 are additionally rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. US Publication 2017/0161891 and Xu et al. Deep Learning of Feature Representation with Multiple Instance Learning for Medical Image Analysis. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing. Pages 1626-1630 (hereafter “Xu”).
Referring to claim 1, Madabhushi discloses a method of training models to classifying biomedical images, comprising:
identifying, by a computing system, a training dataset comprising (i) a first plurality of tiles of a biomedical image of a sample and (ii) a label identifying a first category for the sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles);
applying, by the computing system, the first plurality of tiles to a classification model, the classification model comprising:
a tile encoder having a first plurality of weights to generate, based on the first plurality of tiles, a plurality of feature vectors defined in a feature space, wherein a number of feature vectors corresponds to a number of the first plurality of tiles (paragraph 55, First layer 220 comprises a convolutional layer 224 followed by a pooling layer 226. Convolutional layer 224 applies a 2D convolution of the input image 210 with a kernel of eight pixels by eight pixels to produce a feature map. Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224);
a clusterer to select a subset of feature vectors from the plurality of feature vectors from the tile encoder (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224); and
an aggregator having a second plurality of weights to determine, based on the subset of feature vectors from the clusterer, a second category for the sample (paragraph 55, The output of pooling layer 226 is fed to fully connected layer 240. Fully connected layer 240 outputs to a final classification layer 250. Final classification layer 250 is a softmax classifier comprising an invasive tissue output 260 and a non-invasive tissue output 270. Invasive tissue output 260 and non-invasive tissue output 270 are activated by a logistic regression function);
determining, by the computing system, a loss metric based on a comparison between the second category determined by the classification model with the first category of the label of the training dataset (paragraph 59, Method 300 further includes, at 350 minimizing a softmax loss function. Example methods and apparatus may minimize the softmax loss function using a stochastic gradient descent);
updating, by the computing system using the loss metric, at least one of the first plurality of weights in the tile encoder, the plurality of centroids of the clusterer, and the second plurality of weights in the aggregator based on the comparison (paragraph 59, Minimizing the softmax loss function may include searching for a weight vector that minimizes the softmax loss function); and
storing, by the computing system, in one or more data structures, the first plurality of weights in the tile encoder, and the second plurality of weights of the aggregator (paragraph 55, FIG. 2 illustrates in greater detail an example CNN architecture 200 that may be implemented with example methods and apparatus described herein).
Madabhushi does not disclose expressly a clusterer to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space.
Xu discloses a clusterer to select a subset of feature vectors from the plurality of feature vectors from the tile encoder based on a plurality of centroids defined in the feature space, wherein:
the plurality of centroids define a set of regions within the feature space, each region of the set of regions corresponding to a portion of the feature space,
the set of regions are defined based on distances about the plurality of centroids in the feature space,
the clusterer assigns the plurality of feature vectors to the set of regions based on the distances to the plurality of centroids within the feature space, and
the subset of feature vectors includes include a at least one feature vector from each of the set of regions (page 1628, A centroid is also a vector in R3d2 and the centroids are the ”most common rfs” in all the patches of all the images. We randomly extract n rfs from the image set and form a set of vectors P, and then run the K-means algorithm to generate k centroids C1, ..., Ck. The K-means algorithm contains t iterations. In each iteration, we find the closest centroid measured in Euclidean distance for each rf in P, and assign the rf to the centroid. Then, for each centroid Ci, we take all rfs that assigned to this centroid in the current iteration, and modify the centroid into a new one C i which is the mean of all these rfs. After running such iteration for t rounds, the set of centroids converges to describe the most common rfs of P);
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space. The motivation for doing so would have been to use a fast and reliable manner of segmentation in order to analyze the cell structure. Therefore, it would have been obvious to combine Muhammad with Xu to obtain the invention as specified in claim 1.
Referring to claim 2, Madabhushi discloses wherein applying further comprises applying the first plurality of tiles from the biomedical image of a plurality of biomedical images to the classification model, the clusterer of the classification model to identify the plurality of feature vectors from which to select the subset of feature vectors based on the biomedical image (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claim 3, Madabhushi discloses wherein applying further comprises applying the first plurality of tiles to the classification model, the clusterer of the classification model to select a subset of tiles in the biomedical image based on the subset of feature vectors (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claim 4, Madabhushi discloses wherein applying further comprises applying the first plurality of tiles to the classification model, the aggregator of the classification model to determine a plurality of confidence scores for a subset of tiles corresponding to the subset of feature vectors (paragraph 45, Method 100 also includes, at 152, receiving, from the automated classifier, a prediction probability. The prediction probability is based, at least in part, on the sampling subset. The prediction probability indicates the probability of invasive pathology at a location in the WSI occupied by a tile. In one embodiment, the prediction probability may be within, for instance, a range of 1 for invasive tissue, to 0 for non-invasive tissue. In one embodiment, a prediction probability in the range (0.4, 0.6) indicates a threshold uncertainty as to whether the tissue at the location occupied by the tile is invasive or non-invasive).
Referring to claim 5, Madabhushi discloses wherein identifying further comprises identifying the training dataset comprising a plurality of labels for a corresponding first plurality of categories for the sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles), and
wherein applying further comprises applying the first plurality of tiles to the classification model, the aggregator of the classification model to determine a second plurality of categories for the sample (paragraph 65, Method 600 also includes, at 640, receiving, from the automated classifier, a classification of the sample set of tiles).
Referring to claim 7, Xu discloses wherein updating further comprises determining the plurality of centroids based on a second plurality of feature vectors generated by the tile encoder, subsequent to updating of the first plurality of weights (page 1628, A centroid is also a vector in R3d2 and the centroids are the ”most common rfs” in all the patches of all the images. We randomly extract n rfs from the image set and form a set of vectors P, and then run the K-means algorithm to generate k centroids C1, ..., Ck. The K-means algorithm contains t iterations. In each iteration, we find the closest centroid measured in Euclidean distance for each rf in P, and assign the rf to the centroid. Then, for each centroid Ci, we take all rfs that assigned to this centroid in the current iteration, and modify the centroid into a new one C i which is the mean of all these rfs. After running such iteration for t rounds, the set of centroids converges to describe the most common rfs of P).
Referring to claims 8 and 15, Madabhushi discloses a method of classifying biomedical images, comprising:
identifying, by a computing system, a first plurality of tiles from a first biomedical image of a first sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles);
determining, by the computing system, a first category for the first sample by applying the first plurality of tiles to a classification model, the classification model trained using a training dataset having a plurality of examples each including (i) a second plurality of tiles of a second biomedical image of a second sample and (ii) a label identifying a second category for the sample (paragraph 56, Method 300 includes, at 310, accessing a set of digitized WSIs of tissue demonstrating invasive pathology and non-invasive pathology. The set of digitized WSIs may be annotated by an expert pathologist. A member of the set of digitized WSIs includes a set of tiles), the classification model comprising:
a tile encoder having a first plurality of weights to determine, based on the first plurality of tiles, a plurality of feature vectors in a feature space (paragraph 55, First layer 220 comprises a convolutional layer 224 followed by a pooling layer 226. Convolutional layer 224 applies a 2D convolution of the input image 210 with a kernel of eight pixels by eight pixels to produce a feature map. Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224);
a clusterer to select a subset of feature vectors from the plurality of feature vectors (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224); and
an aggregator having a second plurality of weights to generate, based on the subset of feature vectors, the first category for the sample, storing, by the computing system, an association between the first category and the first biomedical image (paragraph 55, The output of pooling layer 226 is fed to fully connected layer 240. Fully connected layer 240 outputs to a final classification layer 250. Final classification layer 250 is a softmax classifier comprising an invasive tissue output 260 and a non-invasive tissue output 270. Invasive tissue output 260 and non-invasive tissue output 270 are activated by a logistic regression function).
Madabhushi does not disclose expressly a clusterer to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space, or wherein the tile encoder, the clusterer, and the aggregator are updated based on a comparison.
Xu discloses a clusterer to select a subset of feature vectors from the plurality of feature vectors from the tile encoder based on a plurality of centroids defined in the feature space, wherein:
the plurality of centroids define a set of regions within the feature space, each region of the set of regions corresponding to a portion of the feature space,
the set of regions are defined based on distances about the plurality of centroids in the feature space,
the clusterer assigns the plurality of feature vectors to the set of regions based on the distances to the plurality of centroids within the feature space,
the subset of feature vectors includes include a at least one feature vector from each of the set of regions (page 1628, A centroid is also a vector in R3d2 and the centroids are the ”most common rfs” in all the patches of all the images. We randomly extract n rfs from the image set and form a set of vectors P, and then run the K-means algorithm to generate k centroids C1, ..., Ck. The K-means algorithm contains t iterations. In each iteration, we find the closest centroid measured in Euclidean distance for each rf in P, and assign the rf to the centroid. Then, for each centroid Ci, we take all rfs that assigned to this centroid in the current iteration, and modify the centroid into a new one C i which is the mean of all these rfs. After running such iteration for t rounds, the set of centroids converges to describe the most common rfs of P);
wherein the tile encoder, the clusterer, and the aggregator are updated based on a comparison, the comparison having been determined based on a loss metric between the second category and the first category (page 1628, The bag is labeled positive if and only if there is at least one positive instance in the bag, i.e. some part of the image, but maybe not the whole image are cancer tissue. Thus we can formulate a binary MIL model which optimizes the loss func tion of bag classification, while the bag classifier is a softmax of the instance classifiers. Using the gradient descent algorithm, we update h(x) by h(x) ← h(x)+αh(x), where α is the coefficient which is obtained by line searching to minimize the loss function).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to select a subset of feature vectors from the plurality of feature vectors based on a plurality of centroids defined in the feature space, and to update the tile encoder, the clusterer, and the aggregator. The motivation for doing so would have been to use a fast and reliable manner of segmentation in order to analyze the cell structure. Updating the tile encoder, the clusterer, and the aggregator would improve the result of the machine learning model. Therefore, it would have been obvious to combine Xu with Madabhushi to obtain the invention as specified in claims 8 and 15.
Referring to claims 9 and 16, Madabhushi discloses providing, by the computing system, the association between the first category and the first biomedical image (paragraph 65, Method 600 also includes, at 640, receiving, from the automated classifier, a classification of the sample set of tiles).
Referring to claims 10 and 17, Madabhushi discloses determining a plurality of confidence scores for a subset of tiles corresponding to the subset of feature vectors (paragraph 45, Method 100 also includes, at 152, receiving, from the automated classifier, a prediction probability. The prediction probability is based, at least in part, on the sampling subset. The prediction probability indicates the probability of invasive pathology at a location in the WSI occupied by a tile. In one embodiment, the prediction probability may be within, for instance, a range of 1 for invasive tissue, to 0 for non-invasive tissue. In one embodiment, a prediction probability in the range (0.4, 0.6) indicates a threshold uncertainty as to whether the tissue at the location occupied by the tile is invasive or non-invasive).
Referring to claims 11 and 18, Madabhushi discloses determining a second plurality of categories for the sample (paragraph 19, The training exemplars may be annotated by an expert pathologist. Example methods and apparatus may extract tiles of a fixed size from annotated invasive tissue regions and annotated non-invasive tissue regions).
Referring to claims 12 and 19, Madabhushi discloses selecting a subset of tiles in the biomedical image based on the subset of feature vectors (paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claims 13 and 20, Madabhushi discloses selecting the first plurality of tiles from a second plurality of tiles of the first biomedical image paragraph 55, Pooling layer 226, which may also be referred to as a subsampling layer, applies a spatial L2-pooling function without overlapping, using a pooling kernel of two pixels by two pixels, to a feature map obtained from convolutional layer 224).
Referring to claim 14, Madabhushi discloses obtaining, by the computing system, the first biomedical image of the first sample via a histological image preparer (paragraph 62, Method 600 includes, at 610, accessing a radiological image of a region of tissue demonstrating invasive pathology. The radiological image may be a digitized WSI of a hematoxylin and eosin (H&E) stained histopathology slide of a region of tissue demonstrating cancerous pathology).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Madabhushi et al. US Publication 2017/0161891 and Xu as applied to claim 1 above, and further in view of Backhus et al. US Publication 2019/0286980 (hereafter “Backhus”).
Referring to claim 6, Madabhushi discloses determining, by the computing system, a loss metric based on comparison among the plurality of feature vectors, and
wherein updating further comprises updating, using the loss metric, at least one of the first plurality of weights in the tile encoder or the plurality of centroids of the clusterer within the feature space (paragraph 59, Method 300 further includes, at 350 minimizing a softmax loss function. Example methods and apparatus may minimize the softmax loss function using a stochastic gradient descent).
Xu discloses the plurality of centroids in the feature space (page 1628, A centroid is also a vector in R3d2 and the centroids are the ”most common rfs” in all the patches of all the images. We randomly extract n rfs from the image set and form a set of vectors P, and then run the K-means algorithm to generate k centroids C1, ..., Ck. The K-means algorithm contains t iterations. In each iteration, we find the closest centroid measured in Euclidean distance for each rf in P, and assign the rf to the centroid. Then, for each centroid Ci, we take all rfs that assigned to this centroid in the current iteration, and modify the centroid into a new one C i which is the mean of all these rfs. After running such iteration for t rounds, the set of centroids converges to describe the most common rfs of P).
Madabhushi and Xu do not disclose expressly determining a second loss metric.
Backhus discloses determining, by the computing system, a second loss metric based on comparison among the plurality of feature vectors and the plurality of centroids in the feature space, and
wherein updating further comprises updating, using the second loss metric, at least one of the first plurality of weights in the tile encoder or the plurality of centroids of the clusterer within the feature space (paragraph 7, The second loss function is a linear combination of the classification error and a clustering error. The clustering error aggregates a deviation between each of the kernels and one of the C centroids nearest to the respective kernel).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to determine a second loss metric. The motivation for doing so would have been to improve the performance of the classifier. Therefore, it would have been obvious to combine Backhus with Madabhushi and Xu to obtain the invention as specified in claim 6.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PETER K HUNTSINGER whose telephone number is (571)272-7435. The examiner can normally be reached Monday - Friday 8:30 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benny Q Tieu can be reached at 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER K HUNTSINGER/Primary Examiner, Art Unit 2682
Prosecution Insights