DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/26/2026 has been entered.
Response to Amendments/Arguments
Applicant has not presented new amendments or arguments since those filed after final on 04/24/2026. The amendments were previously entered by the Examiner with an explanation of how the previous art would still read on the claims was appended to the Advisory Action filed 05/04/2026. Since no new amendments or arguments have been presented, the Examiner is maintaining his arguments against the current version of the claims as appended to the Advisory Action with more detailed rejections below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 27,29-36, 38, 40, and 52 is/are rejected under 35 U.S.C. 103 as being unpatentable over You et al, “Real-time intraoperative diagnosis by deep neural network driven multiphoton virtual histology” (published npj Precision Oncology, Vol 3, Article Number 33, December 17, 2019) in view of Tellez et al, “Gigapixel Whole-Slide Image Classification Using Unsupervised Image Compression And Contrastive Training” (published April 2018, cited on the IDS filed 06/22/2023) as further evidenced by Tellez et al, “Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks” (published March 2018, cited on the IDS filed 06/22/2023, hereafter referred to as Tellez 2).
Regarding claim 27, You teaches a method of determining a plurality of regions of interest in an input histology image (see You Abstract), the method comprising:
receiving an input histology image; tiling the input histology image into a set of tiles (see You Figure 2, input image which is tiled and caption);
for each tile, extracting a feature of that tile by applying a feature extractor images (see Figure 2, DNN and page 5, first column, “In search of an intuitive understanding of the image features used by the trained DNN, we first extracted for each sample tile the neuron activity profile in the penultimate layer of the network. This 512-dimensional vector acts as input to the final neuron that makes the decision to classify the image as cancer or normal, and may thus be considered as a compact representation of the image that captures its salient features for discerning its class”), the feature extractor trained with a machine learning algorithm using a training set of histology images (see caption for Figure 2);
clustering the extracted features to assign each of the set of tiles to one of a plurality of regions of interest for each tile (see Figure 5 and page 5, first column, “This allows us to visualize the collection of images on a ‘canvas’ where images are clustered by their mutual similarity as defined by the DNN. We can see in the resulting plot (Fig. 5) that the DNN tends to cluster tiles with similar optical signatures and shapes”); and
outputting the plurality of regions of interest (see Figure 3 and page 3, first column, “The DNN predicts a cancer versus normal probability score for each tile, allowing us to create a heatmap that highlights regions likely to be cancerous in each image (Fig 3a, b). It is to be noted that a significant portion of breast tissue is adipocytes”);
wherein the feature extractor is trained by:
receiving the training set of histology images, wherein each image in the training set of histology images is annotation-free (see caption for Figure 2 and page 6, second column “The input to the algorithm was the raw four-channel virtual slides (multiphoton images) together with the labels generated by the lipid segmentation network and the labels of being diagnosed as cancer or normal (generated by pathologist)” wherein the only labels are whether the training image does or does not contain cancerous cells. As noted in the arguments presented in the Advisory Action, Applicant in the originally filed Specification does not consider such labels to be “annotations” as paragraph [0041] of the present application distinguishes annotations from “slide-level labels” while paragraph [0035] specifically lists a binary label of containing metastasis i.e. cancerous cells as a slide-level label);
tiling the training set of histology images into a set of tiles (see Figure 2 as applied for training images).
You does not expressively teach
the feature extractor trained with an unsupervised machine learning algorithm;
wherein the feature extractor is trained by:
performing data augmentation on the set of tiles to generate at least two batches of tiles, wherein each batch of tiles includes randomly augmented views of the original set of tiles;
extracting a first set of features from the first batch of tiles by applying the first batch of tiles to the feature extractor;
extracting a second set of features from the second batch of tiles by applying the second batch of tiles to the feature extractor; and
training the feature extractor using a contrastive loss between pairs of the first set of features and the second set of features to bring matching pairs of tiles closer and different pairs of tiles further apart.
However, Tellez in a similar invention in the same field of endeavor teaches a method involving a trained feature extractor (see Tellez section 2.1, “We extracted relevant information from tissue images using a CNN-based encoder. This network mapped tissue patches into embedding vectors”) configured to be trained a set of tiles (see section 3, “We trained instances of the five different encoders explained in Sec. 2.1 using a patch size of 128x128 px extracted at 0.5 um/px resolution”) from histology images (see Abstract) which are annotation-free (see Tellez section 3, first paragraph “We divided the set of slides into training (180), validation (90) and test (128). Each slide is associated with a binary label indicating the presence of tumor metastasis” and explanation given above) as taught in You wherein
the trained feature extractor [is] trained with an unsupervised machine learning algorithm using a set of training images (see Abstract);
wherein the feature extractor is trained by:
performing data augmentation on the set of tiles (see section 2.1, “We investigated the effectiveness of several types of encoders trained in an unsupervised manner, using tissue patches that were heavily augmented with the data augmentation routines detailed in [1]”, wherein [1] is Tellez 2) to generate at least two batches of tiles (see section 2.1, “We created an artificial training dataset consisting of pairs of tissue patches representing either the same or different tissue morphology. Positive pairs consisted of patches extracted from the exact same WSI location (although different augmentation). Negative pairs consisted of patches from: a) different WSI locations, and b) neighbor locations but non-overlapping tissue”), wherein each batch of tiles includes randomly augmented views of the original set of tiles (see Tellez 2, page 2129, “We used this annotated set of samples to train CNN2 to distinguish PHH3 candidates among mitotic and non-mitotic patches. During training, we randomly applied several techniques to augment the data and prevent overfitting, namely: rotations, vertical and horizontal mirroring, elastic deformation [32], Gaussian blurring, and translations”);
extracting a first set of features from the first batch of tiles by applying the first batch of tiles to the feature extractor; extracting a second set of features from the second batch of tiles by applying the second batch of tiles to the feature extractor (see section 2.1, “We extracted relevant information from tissue images using a CNN-based encoder. This network mapped tissue patches into embedding vectors”. See also Figure 2 and arguments noted above); and
training the feature extractor using a contrastive loss between pairs of the first set of features and the second set of features to bring matching pairs of tiles closer and different pairs of tiles further apart (see section 2.1, “Third, we proposed and trained a novel contrastive encoding scheme…A model composed of two encoders sharing weights, followed by a feature-wise concatenation operation and an MLP, was trained to distinguish between the two classes (see Fig. 2). Because two same patches present the same tissue morphology with heavily altered appearance, the encoder learns to extract high-level semantic features instead of low-level pixel ones, an advantage over encoders based on reconstruction error”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of using an unsupervised algorithm as taught in Tellez with the system taught in You, the motivation being to save processing resources by not using a supervisor in the training process.
Regarding claim 40¸ the claim recites a non-transitory machine-readable medium with a memory storing code instructions which, when executed by a processor, cause the processor to perform operations as those recited in claim 27 which You in view of Tellez and Tellez 2 further teaches (see You Figure 1, computer, which is well-known to contain instructions in a memory for a processor).
Regarding claim 29, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, and further teaches wherein the input histology image and the training set of images are from the same domain (see Tellez section 3, “We used Camelyon16 data [4] to train and evaluate our methodology. We divided the set of slides into training (180), validation (90) and test (128). Each slide is associated with a binary label indicating the presence of tumor metastasis” wherein the [4] citation on page 3 shows the training images are for breast cancer and You Abstract).
Regarding claim 30, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, but does not expressively teach wherein the clustering is a K-Means clustering.
However, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the clustering of You in view of Tellez and Tellez 2 with the K-Means clustering claimed to yield the predictable results of successfully separating the tiles appropriately.
Regarding claim 31, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, and further teaches wherein the input histology image is a whole slide image (see You caption for Figure 2 and Tellez Abstract).
Regarding claim 32, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, and further teaches wherein the input histology image is derived from a patient tissue sample (see You caption for Figure 1).
Regarding claim 33, You in view of Tellez and Tellez 2 teaches all the limitations of claim 32, and further teaches wherein the patient tissue sample is known or suspected to contain a tumor (see You Abstract).
Regarding claim 34, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, but does not expressively teach wherein the unsupervised machine learning algorithm is a self-supervised machine learning algorithm.
However, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the unsupervised machine learning algorithm of You in view of Tellez and Tellez 2 with a self-supervised machine learning algorithm as claimed to yield the predictable results of successfully training and using the feature extractor.
Regarding claim 35, You in view of Tellez teaches all the limitations of claim 27, and further teaches wherein the unsupervised machine learning algorithm is a contrastive loss machine learning algorithm (see Tellez section 2.1, “Third, we proposed and trained a novel contrastive encoding scheme…A model composed of two encoders sharing weights, followed by a feature-wise concatenation operation and an MLP, was trained to distinguish between the two classes (see Fig. 2). Because two same patches present the same tissue morphology with heavily altered appearance, the encoder learns to extract high-level semantic features instead of low-level pixel ones, an advantage over encoders based on reconstruction error”)
You in view of Tellez and Tellez 2 does not expressively teach the algorithm include[es] one of Momentum Contrast or Momentum Contrast v2.
However, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the algorithm of You in view of Tellez and Tellez 2 with those claimed to yield the predictable results of successfully training and using the feature extractor.
Regarding claim 36, You in view of Tellez teaches all the limitations of claim 27, but does not expressively teach wherein the trained feature extractor is a ResNet type of feature extractor.
However, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the trained feature extractor of You in view of Tellez and Tellez 2 with that claimed to yield the predictable results of successfully training and using the feature extractor.
Regarding claim 38, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, and further teaches annotating at least one cluster of extracted features (see You Figure 5 and caption).
Regarding claim 52, You teaches a system for determining a plurality of regions of interest in an input histology image, comprising:
an image processor within a processing device (see You Figure 1, computer), the image processor configured to receive an input histology image and tile the input histology image into a set of tiles (see Figure 2, input image which is tiled and caption);
a feature extractor for extracting features from each tile (see Figure 2, DNN and page 5, first column, “In search of an intuitive understanding of the image features used by the trained DNN, we first extracted for each sample tile the neuron activity profile in the penultimate layer of the network. This 512-dimensional vector acts as input to the final neuron that makes the decision to classify the image as cancer or normal, and may thus be considered as a compact representation of the image that captures its salient features for discerning its class”), the feature extractor trained with a machine learning algorithm using a training set of histology images (see caption for Figure 2);
a clustering module within the processing device, the clustering module configured to cluster the extracted features to assign each tile to one of a plurality of regions of interest for each tile (see Figure 5 and page 5, first column, “This allows us to visualize the collection of images on a ‘canvas’ where images are clustered by their mutual similarity as defined by the DNN. We can see in the resulting plot (Fig. 5) that the DNN tends to cluster tiles with similar optical signatures and shapes”, wherein the DNN also acts as the clustering module); and
an output device to output the plurality of regions of interest (see Figure 3 and page 3, first column, “The DNN predicts a cancer versus normal probability score for each tile, allowing us to create a heatmap that highlights regions likely to be cancerous in each image (Fig 3a, b). It is to be noted that a significant portion of breast tissue is adipocytes”);
wherein the feature extractor is trained by:
receiving the training set of histology images, wherein each image in the training set of histology images is annotation-free (see caption for Figure 2 and page 6, second column “The input to the algorithm was the raw four-channel virtual slides (multiphoton images) together with the labels generated by the lipid segmentation network and the labels of being diagnosed as cancer or normal (generated by pathologist)” wherein the only labels are whether the training image does or does not contain cancerous cells. As noted in the arguments presented in the Advisory Action, Applicant in the originally filed Specification does not consider such labels to be “annotations” as paragraph [0041] of the present application distinguishes annotations from “slide-level labels” while paragraph [0035] specifically lists a binary label of containing metastasis i.e. cancerous cells as a slide-level label);
tiling the training set of histology images into a set of tiles (see Figure 2 as applied for training images).
You does not expressively teach
the feature extractor trained with an unsupervised machine learning algorithm;
wherein the feature extractor is trained by:
performing data augmentation on the set of tiles to generate at least two batches of tiles, wherein each batch of tiles includes randomly augmented views of the original set of tiles;
extracting a first set of features from the first batch of tiles by applying the first batch of tiles to the feature extractor;
extracting a second set of features from the second batch of tiles by applying the second batch of tiles to the feature extractor; and
training the feature extractor using a contrastive loss between pairs of the first set of features and the second set of features to bring matching pairs of tiles closer and different pairs of tiles further apart.
However, Tellez in a similar invention in the same field of endeavor teaches a method involving a trained feature extractor (see Tellez section 2.1, “We extracted relevant information from tissue images using a CNN-based encoder. This network mapped tissue patches into embedding vectors”) configured to be trained a set of tiles (see section 3, “We trained instances of the five different encoders explained in Sec. 2.1 using a patch size of 128x128 px extracted at 0.5 um/px resolution”) from histology images (see Abstract) which are annotation-free (see Tellez section 3, first paragraph “We divided the set of slides into training (180), validation (90) and test (128). Each slide is associated with a binary label indicating the presence of tumor metastasis” and explanation given above) as taught in You wherein
the trained feature extractor [is] trained with an unsupervised machine learning algorithm using a set of training images (see Abstract);
wherein the feature extractor is trained by:
performing data augmentation on the set of tiles (see section 2.1, “We investigated the effectiveness of several types of encoders trained in an unsupervised manner, using tissue patches that were heavily augmented with the data augmentation routines detailed in [1]”, wherein [1] is Tellez 2) to generate at least two batches of tiles (see section 2.1, “We created an artificial training dataset consisting of pairs of tissue patches representing either the same or different tissue morphology. Positive pairs consisted of patches extracted from the exact same WSI location (although different augmentation). Negative pairs consisted of patches from: a) different WSI locations, and b) neighbor locations but non-overlapping tissue”), wherein each batch of tiles includes randomly augmented views of the original set of tiles (see Tellez 2, page 2129, “We used this annotated set of samples to train CNN2 to distinguish PHH3 candidates among mitotic and non-mitotic patches. During training, we randomly applied several techniques to augment the data and prevent overfitting, namely: rotations, vertical and horizontal mirroring, elastic deformation [32], Gaussian blurring, and translations”);
extracting a first set of features from the first batch of tiles by applying the first batch of tiles to the feature extractor; extracting a second set of features from the second batch of tiles by applying the second batch of tiles to the feature extractor (see section 2.1, “We extracted relevant information from tissue images using a CNN-based encoder. This network mapped tissue patches into embedding vectors”. See also Figure 2 and arguments noted above); and
training the feature extractor using a contrastive loss between pairs of the first set of features and the second set of features to bring matching pairs of tiles closer and different pairs of tiles further apart (see section 2.1, “Third, we proposed and trained a novel contrastive encoding scheme…A model composed of two encoders sharing weights, followed by a feature-wise concatenation operation and an MLP, was trained to distinguish between the two classes (see Fig. 2). Because two same patches present the same tissue morphology with heavily altered appearance, the encoder learns to extract high-level semantic features instead of low-level pixel ones, an advantage over encoders based on reconstruction error”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of using an unsupervised algorithm as taught in Tellez with the system taught in You, the motivation being to save processing resources by not using a supervisor in the training process.
Claim(s) 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over You et al, “Real-time intraoperative diagnosis by deep neural network driven multiphoton virtual histology” (published npj Precision Oncology, Vol 3, Article Number 33, December 17, 2019) in view of Tellez et al, “Gigapixel Whole-Slide Image Classification Using Unsupervised Image Compression And Contrastive Training” (published April 2018, cited on the IDS filed 06/22/2023) as further evidenced by Tellez et al, “Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks” (published March 2018, cited on the IDS filed 06/22/2023, hereafter referred to as Tellez 2) and Hall et al, U.S. Publication No. 2018/0180590.
Regarding claim 37, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, but does not expressively teach removing background segments from the input histology image.
However, Hall in a similar invention in the same field of endeavor teaches a method of analyzing an input histology image (see Hall paragraph [0049]) as taught in You in view of Tellez and Tellez 2 further comprising
removing background segments from the input histology image (see paragraph [0049]).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of removing background from a histology image as taught in Hall with the method taught in You in view of Tellez and Tellez 2, the motivation being to save processing resources by not analyzing background.
Claim(s) 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over You et al, “Real-time intraoperative diagnosis by deep neural network driven multiphoton virtual histology” (published npj Precision Oncology, Vol 3, Article Number 33, December 17, 2019) in view of Tellez et al, “Gigapixel Whole-Slide Image Classification Using Unsupervised Image Compression And Contrastive Training” (published April 2018, cited on the IDS filed 06/22/2023) as further evidenced by Tellez et al, “Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks” (published March 2018, cited on the IDS filed 06/22/2023, hereafter referred to as Tellez 2) and Naylor et al, “PREDICTING RESIDUAL CANCER BURDEN IN A TRIPLE NEGATIVE BREAST CANCER COHORT” (published in 2019 IEEE 16th International Symposium on Biomedical Imaging, pages 933-937, April 2019).
Regarding claim 39, You in view of Tellez and Tellez 2 teaches all the limitations of claim 27, but does not expressively teach quantifying the input histology image by a level of expression of a plurality of clusters.
However, Naylor in a similar invention in the same field of endeavor teaches a method comprising tiling an input histology image (see Naylor Abstract) into a plurality of tiles, extracting features from the plurality of times (see section 4.2, “This mapping can be divided into 3 steps: 1) finding tissue areas in the WSI, 2) overlaying a grid on this tissue area and 3) encoding each tile of size 224 224 to a vector), and clustering the extracted features (see section 4.2.2, “2. cluster-based down sampling: we first cluster all feature vectors from one patient into ni 40 clusters and then sample the same (small) number of feature vectors from each cluster so that the amount of feature vectors is constant across patients) as taught in You in view of Tellez and Tellez 2 further comprising
quantifying the input histology image by a level of expression of a plurality of clusters (see section 4.2.2, “Once each tile is clustered, we thus represent a WSI by the percentage of patches belonging to each of the k clusters. Hence, we represent a patient’s biopsy by a vector z(i) of size k”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of quantifying input histology images based on clusters as taught in Naylor with the method taught in You in view of Tellez, the motivation being to more easily analyze the results of the clustering.
Conclusion
The Examiner attempted twice to contact an attorney of record to see if further amendments or arguments were to be filed with the RCE but was unable to reach them in a timely fashion.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CASEY L KRETZER whose telephone number is (571)272-5639. The examiner can normally be reached M-F 10:00-7:00 PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Payne can be reached at (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CASEY L KRETZER/Primary Examiner, Art Unit 2635
Prosecution Insights