DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2/25/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 11-13, 15 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shao et al ("Transmil: Transformer based correlated multiple instance learning for whole slide image classification." Advances in neural information processing systems 34 (2021): pages 1-12, retrieved from the Internet on 6/10/2026) in view of Yip et al (US20220101519).
Regarding claim 1, Shao teaches a computer-implemented method, the method comprising:
providing a first machine learning model (section 3.1, binary MIL classification), wherein the first machine learning model is configured to receive an image and assign the image to one of two classes (equation 1 in section 3.3, the bag-level label is Yi; step 5 in Algorithm 2 in section 3.3);
providing a pre-trained second machine learning model, wherein the second machine learning model is configured and trained to generate a patch embedding based on a patch of an image (ResNet50 in fig. 3; section 4, the feature of each patch is embedded in a 1024-dimensional vector by a ResNet50 model pre-trained on ImageNet); and
receiving training images, each training image being assigned to one of the at least two classes (section 3.2, Given a set of bags, and each bag contains
multiple instances and a corresponding label Yi);
for each training image:
generating a plurality of patches based on the training image (caption of fig. 3, Each WSI is cropped into patches);
generating a patch embedding for each patch of the plurality of patches using the second machine learning model (caption of fig. 3, embedded in feature vectors by ResNet50);
generating a region comprising a number of patches (step 2 in Algorithm 3; section 3.3, PPEG); and
generating a regional embedding for each region based on patch embeddings of patches comprised by the region (steps 2-4 in Algorithm 3);
training the first machine learning model using the training images (steps 2-5 in Algorithm 2; section 4, the training step, cross-entropy loss was adopted, and the Lookahead optimizer was employed), wherein the training comprises:
receiving a training image (fig. 3);
selecting a number of patches from the training image (caption of fig. 3, Each WSI is cropped into patches. It would be obvious to select a number of patches for processing);
generating a patch embedding for each selected patch (caption of fig. 3, embedded in feature vectors by ResNet50);
generating a global embedding (output in Algorithm 2) based on the patch embeddings of the selected patches (input in Algorithm 2) and the regional embeddings of regions comprising the selected patches (steps 2-4 in Algorithm 2);
assigning the global embedding to one of the at least two classes (step 5 in Algorithm 2);
computing a loss based on a difference between the class to which the global embedding is assigned and the class to which the training image is assigned (section 4, In the training step, cross-entropy loss was adopted); and
modifying parameters of the first machine learning model based on the computed loss (section 4, the Lookahead optimizer was employed with a learning rate of 2e-4 and weight decay of 1e-5); and
storing the trained first machine learning model (it would be necessary to store the model for future use), and/or using the trained first machine learning model and the second machine learning model to classify one or more new images (section 4, In the inference step, the softmax is used to normalize the predicted scores for each class).
Shao fails to teach generating a multitude of regions, each region comprising a number of patches.
However Yip teaches generating a multitude of regions, each region comprising a number of patches, for use in a classifier (para. [0019]).
Therefore taking the combined teachings of Shao and Yip as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Yip into the method of Shao. The motivation to combine Yip and Shao would be to accurately capture structural and local histology of various diseases (para. [0087] of Yip).
Regarding claim 2, the modified method of Shao teaches a method wherein each region contains a number of patches in the range from 10 to 1000 (step 3 in Algorithm 3 of Shao, k=3, 5, 7 would create patches between 10 and 1000).
Regarding claim 11, the modified invention of Shao teaches a method according wherein one class of the at least two classes comprises images showing tissue in which a specific gene mutation is present (para. [0020], [0180], [0383] of Yip), preferably a mutation affecting one or more of the following genes: HER2, TOP2A, HER3, EGFR, P53, MET, ALK, FLT3, AXL, FLT4, DDR2, EGFR (para. [0180] of Yip), HER4, EML4-ALK, IGF1R, EPHA1, INSR, EPHA2, IRR, EPHA3, KIT, EPHA4, LTK, EPHA5, MER, EPHA6, MET, EPHA7, MUSK, EPHA8, NPM1-ALK, EPHB1, PDGFRα, EPHB2, PDGFRβ, EPHB3, RET, EPHB4, RON, FGFR1, ROS, FGFR2, TIE2, FGFR3, TRKA, FGFR4, TRKB, FLT1, TRKC, ATM, BRCA1, BRCA2, BRCA3, CCND1, E-Cadherin, ERBB2, ETV6, FGFR1, HRAS, KRAS (para. [0180] of Yip), NRAS, NTRK3, p53, PTEN, BCL2, BRD4, CCND1, CDKN1A, CDKN2A, CTNNB1, HES1, MAP2, MEN1, NF1, NOTCH1, NUT, RAF, SDHD, VEGFA, APC, MSH6, AXIN2, MYH, BMPRIA, p53, DCC, PMS2, KRAS2, PTEN, MLH1, SMAD4, MSH2, STK11, MSH6, PTEN, CCND1, RASSF1A, CDKN2A, RB1, EGFR, RET, EML4, ROS1, KRAS2, TP53, MYC, Axin1, MALAT1, b-catenin, p16 INK4A, c-ERBB-2, p53, CTNNB1, RB1, Cyclin D1, SMAD2, EGFR, SMAD4, IGFR2, TCF1, KRAS, Alpha, PRCC, ASPSCR1, PSF, CLTC, TFE3, p54nrb/NONO, TFEB, AKAP10, NTRK1, AKAP9, RET, BRAF, TFG, ELE1, TPM3, H4/D10S170, TPR, AKT2, MDM2, BCL2, MYC, BRCA1, NCOA4, CDKN2A, p53, ERBB2, PIK3CA, GATA4, RB, HRAS, RET, KRAS, RNASET2, AR, KLK3, BRCA2, MYC, CDKNIB, NKX3.1, EZH2, p53, GSTP1, CDH11, COL12A1, CNBP, OMD, COLIA1, THRAP3, COL4A5, USP6.
Regarding claim 12, the modified method of Yip teaches a method wherein each training image is a medical image, preferably a whole slide image (abstract of Shao; para. [0010] of Yip), most preferably a histopathological image of tissue from a patient stained with hematoxylin and eosin (para. [0009]-[0010] of Yip).
Regarding claim 13, the modified method of Shao teaches a method further comprising:
receiving a new image (para. [0020] of Yip);
generating a plurality of patches based on the new image (para. [0023] of Yip);
inputting the patches into the trained first machine learning model (para. [0025] of Yip);
receiving a classification result from the trained first machine learning model (para. [0025] of Yip); and
outputting the classification result (para. [0021] and [0049] of Yip).
Regarding claim 15, the claim recites similar subject matter as claim 1 and is rejected for the same reasons as stated above.
Regarding claim 16, the claim recites similar subject matter as claim 1 and is rejected for the same reasons as stated above.
Claim(s) 3-7, 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shao et al ("Transmil: Transformer based correlated multiple instance learning for whole slide image classification." Advances in neural information processing systems 34 (2021): pages 1-12, retrieved from the Internet on 6/10/2026) and Yip et al (US20220101519) in view of Konstantinov et al ("Multi-attention multiple instance learning." Neural Computing and Applications 34.16 (4/20/2022): pages 14029-14051, retrieved from the Internet on 6/10/2026).
Regarding claim 3, the modified method of Shao fails to teach a method wherein the patch embeddings generated by the second machine learning model are generated in advance of the training of the first machine learning model and stored in a data memory.
However Konstantinov teaches wherein patch embeddings generated by a second machine learning model are generated in advance of training of a first machine learning model and stored in a data memory (fig. 1; page 14036 right side second paragraph).
Therefore taking the combined teachings of Shao and Yip with Konstantinov as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Konstantinov into the method of Shao and Yip. The motivation to combine Konstantinov, Yip and Shao would be to increase the generalization ability of the model and allow possible training on small datasets (page 14036 left side first paragraph of Konstantinov).
Regarding claim 4, the modified method of Shao fails to teach a method wherein the regional embeddings are generated by the pre-trained second machine learning model, preferably in advance of the training of the first machine learning model.
However Konstantinov teaches wherein the regional embeddings are generated by a pre-trained second machine learning model (page 14033 left side last paragraph, All patches are fed to the input of a trained neural network (encoder or feature extractor) for computing their embedding; page 14034 right side first paragraph).
Therefore taking the combined teachings of Shao and Yip with Konstantinov as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Konstantinov into the method of Shao and Yip. The motivation to combine Konstantinov, Yip and Shao would be to increase the generalization ability of the model and allow possible training on small datasets (page 14036 left side first paragraph of Konstantinov).
Regarding claim 5, the modified method of Shao fails to teach a method wherein the regional embeddings are generated by the first machine learning model.
However Konstantinov teaches wherein regional embeddings are generated by a first machine learning model (page 14033 right side last paragraph to page 14034 right side first paragraph, Bi).
Therefore taking the combined teachings of Shao and Yip with Konstantinov as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Konstantinov into the method of Shao and Yip. The motivation to combine Konstantinov, Yip and Shao would be to increase the generalization ability of the model and allow possible training on small datasets (page 14036 left side first paragraph of Konstantinov).
Regarding claim 6, the modified method of Shao fails to teach a method further comprising:
generating, for each training image, a feature vector representing all regions of the training image based on the regional embeddings of the training image; and
generating the global embedding based on the patch embeddings of the selected patches and the regional embeddings of regions comprising the selected patches and the feature vector representing all regions of the training image.
However Konstantinov teaches generating, for each training image, a feature vector representing all regions of the training image based on the regional embeddings of the training image (equation 13; page 14035 left side last paragraph to right side first paragraph); and
generating a global embedding (equation 15) based on the patch embeddings of the selected patches (equation 12, Fi) and the regional embeddings of regions comprising the selected patches (equation 12, Bi) and the feature vector representing all regions of the training image (equation 13).
Therefore taking the combined teachings of Shao and Yip with Konstantinov as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Konstantinov into the method of Shao and Yip. The motivation to combine Konstantinov, Yip and Shao would be to increase the generalization ability of the model and allow possible training on small datasets (page 14036 left side first paragraph of Konstantinov).
Regarding claim 7, the modified method of Shao teaches a method according wherein pre-training of the second machine learning model comprises:
receiving training images, each training image being assigned to one of at least two classes (section 3.2 of Shao, Given a set of bags, and each bag contains multiple instances and a corresponding label Yi. It would be obvious to apply the steps to the second machine learning model);
selecting a number of patches and optionally neighbor patches from the training image (caption of fig. 3 of Shao, Each WSI is cropped into patches. It would be obvious to select a number of patches for processing and apply the steps to the second machine learning model);
generating a patch embedding for each selected patch and optionally for each neighbor patch (caption of fig. 3 of Shao, embedded in feature vectors by ResNet50), and
modify parameters of the second machine learning model (section 4 of Shao, the Lookahead optimizer was employed with a learning rate of 2e-4 and weight decay of 1e-5) based on a computed loss (section 4 of Shao, In the training step, cross-entropy loss was adopted); and
storing the trained second machine learning model (it would be necessary to store the model for future use), and/or using the trained second machine learning model in the training of the first machine learning model.
The modified method of Shao fails to teach generating a joint feature vector based on the patch embeddings of all selected patches and optionally all neighbor patches;
assigning the joint feature vector to one of the at least two classes;
computing a loss based on a difference between the class to which the joint feature vector is assigned and the class to which the training image is assigned;
modify parameters of the second machine learning model based on the computed loss.
However Konstantinov teaches generating a joint feature vector based on the patch embeddings of all selected patches and optionally all neighbor patches (page 14034 right side first paragraph, Fi);
assigning the joint feature vector to one of the at least two classes (page 14035 right side second paragraph, Z);
computing a loss based on a difference between the class to which the joint feature vector is assigned and the class to which the training image is assigned (page 14036 right side second paragraph, The loss function is determined based on the
obtained estimate and the whole image label);
modify parameters of the second machine learning model based on the computed loss (page 14036 right side second paragraph, To update the
neural network weights, values of partial derivatives of the loss function are determined for each trained parameter using an automatic differentiation algorithm. Then the training parameter values are updated in a standard way).
Therefore taking the combined teachings of Shao and Yip with Konstantinov as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Konstantinov into the method of Shao and Yip. The motivation to combine Konstantinov, Yip and Shao would be to increase the generalization ability of the model and allow possible training on small datasets (page 14036 left side first paragraph of Konstantinov).
Regarding claim 9, the modified method of Shao teaches a method wherein the classes when pre-training the second machine learning model differ from the classes when training the first machine learning model (section 4 of Shao, To demonstrate the superior performance of the proposed TransMIL, various experiments were conducted over three public datasets: CAMELYON16, The Caner Genome Atlas (TCGA) non-small cell lung cancer (NSCLC), as well as the TCGA renal cell carcinoma (RCC). The feature of each patch is embedded in a 1024-dimensional vector by a ResNet50 model pre-trained on ImageNet).
Regarding claim 10, the modified method of Shao fails to teach a method wherein the pre-trained second machine learning model is an encoder of an autoencoder, the autoencoder being trained to generate a compressed representation of an image and reconstructing the image for the compressed representation.
However Konstantinov teaches an autoencoder (page 14039 left side second paragraph) being trained to generate a compressed representation of an image and reconstructing the image for the compressed representation (page 14036 left side first paragraph; page 14038 right side second paragraph).
Therefore taking the combined teachings of Shao and Yip with Konstantinov as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Konstantinov into the method of Shao and Yip. The motivation to combine Konstantinov, Yip and Shao would be to increase the generalization ability of the model and allow possible training on small datasets (page 14036 left side first paragraph of Konstantinov).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shao et al ("Transmil: Transformer based correlated multiple instance learning for whole slide image classification." Advances in neural information processing systems 34 (2021): pages 1-12, retrieved from the Internet on 6/10/2026), Yip et al (US20220101519) and Konstantinov et al ("Multi-attention multiple instance learning." Neural Computing and Applications 34.16 (4/20/2022): pages 14029-14051, retrieved from the Internet on 6/10/2026) in view of Li et al ("Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021. Pages 14318-14328, retrieved from the Internet on 6/11/2026).
Regarding claim 8, the modified method of Shao fails to teach a method wherein the classes when pre-training the second machine learning model match the classes when training the first machine learning model.
However Li teaches wherein classes when pre-training a second machine learning model match the classes when training a first machine learning model (Table 3; page 14323 right side last paragraph; page 14324 right side first paragraph).
Therefore taking the combined teachings of Shao, Yip and Konstantinov with Li as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Li into the method of Shao, Konstantinov and Yip. The motivation to combine Konstantinov, Li, Yip and Shao would be to improve the accuracy of classification and localization (abstract of Li).
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shao et al ("Transmil: Transformer based correlated multiple instance learning for whole slide image classification." Advances in neural information processing systems 34 (2021): pages 1-12, retrieved from the Internet on 6/10/2026) and Yip et al (US20220101519) in view of Schmitz et al (WO2020/229152A1).
Regarding claim 14, the modified method of Shao teaches a method wherein the first machine learning model is trained (steps 2-5 in Algorithm 2 of Shao).
The modified method of Shao fails to teach wherein the trained first machine learning model is used to assign histopathological images of tissues from patients to one of at least two classes, wherein one class comprises images showing tissue in which a NTRK or BRAF gene mutation is present.
However Schmitz teaches wherein a machine learning model is used (page 2 lines 17-19) to assign histopathological images of tissues from patients (page 6 lines 4-9) to one of at least two classes (page 2 lines 21-23), wherein one class comprises images showing tissue in which a NTRK or BRAF gene mutation is present (page 5 lines 38-40).
Therefore taking the combined teachings of Shao and Yip with Schmitz as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Schmitz into the method of Shao and Yip. The motivation to combine Schmitz, Yip and Shao would be to generate predictions that are often just as accurate as verification experiments (page 8 lines 34-36 of Schmitz).
Related Art
Arnold et al (US20220207730) – see para. [0025], [0099], [0105], [0135]
Park et al (US20210334994) – see para. [0002], [0028], [0031], [0048]
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEON VIET Q NGUYEN whose telephone number is (571)270-1185. The examiner can normally be reached Mon-Fri 11AM-7PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEON VIET Q NGUYEN/Primary Examiner, Art Unit 2663
Prosecution Insights