DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Claims 11-18 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected inventions, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 06/19/2026.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/25/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim 2 is objected to because of the following informalities: The preamble of claim 2 recites “The method of claim 1, wherein step (a) includes,” but there is no “step (a)” recited in claim 1. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1-2, Claim 1 describes that the “WSI data comprise whole-slide images of a histopathology sample” in (a) which are then input into the machine learning model in (c). In contrast, claim 2 recites “storing the embedded instance data as WSI data for inputting to the machine learning model." Accordingly, it is unclear if the WSI data is the actual whole slide images or embedded instance data generated based on these actual whole slide images. This also renders it unclear what is actually being input into the machine learning model of claim 1. Additionally, this creates antecedence issues, particularly in claim 2, where “WSI data” is used to refer to two different things in the first and last limitations of the claim. Overall, in view of the Specification and under broadest reasonable interpretation, the claims are being interpreted such that there is an initial set of whole slide images, which are either input directly into the machine learning model of claim 1(c) or processed prior to their input.
Claims 3-10 are rejected as dependent on above claims.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Coudray (US20200184643A1).
Regarding claim 1, Coudray teaches “ A method for predicting gene profile data from a whole-slide image using a computer system , (Coudray, Paragraph 6, “In this study, a deep convolutional neural network (inception v3) was trained on whole-slide images obtained from The Cancer Genome Atlas to accurately and automatically classify them into LUAD, LUSC or normal lung tissue. The performance of our method is comparable to that of pathologists, with an average area under the curve (AUC) of 0.97. The described system and processes were validated on independent datasets of frozen tissues, formalin-fixed paraffin-embedded tissues and biopsies. Furthermore, we trained the network to predict the ten most commonly mutated genes in LUAD. We found that mutations (in the examples, STK11, EGFR, FAT1, SETBP1, KRAS and TP53) can be predicted from pathology images, with AUCs from 0.733 to 0.856 as measured on a held-out population. These findings suggest that deep-learning models can assist pathologists in the detection of cancer subtype or gene mutations.”; Paragraph 11, “At least one aspect of the present disclosure is directed to a computer-implemented machine for identifying tumors. The computer-implemented machine includes a processor. The computer-implemented machine includes a tangible computer-readable medium operatively connected to the processor and including computer code. The computer code is configured to receive image data regarding a region of interest. The computer code is configured to tile images into sets of windows of between 75×75 μm to 1000×1000 μm. The computer code is configured to analyze each of the windows with a machine learning system to identify phenotypic information for each of the windows. The computer code is configured to predict genotypic information for each window from the phenotypic information. The computer code is configured to generate a heatmap for the region of interest based on the predicted genotypic information for each window.”)
The method comprising: (a) accessing whole-slide image (WSI) data with the computer system, wherein the WSI data comprise whole-slide images of a histopathology sample; (Coudray, Paragraph 31, “Embodiments described herein relate generally to classification and mutation prediction from histopathology images using deep learning. In particular, the examples described herein utilize lung cancer as an example, in particular adenocarcinoma (LUND) and squamous cell carcinoma (LUSC) as two subtypes of lung cancer to identify and distinguish between.; Paragraph 40, “With regard to the experiments, TCGA dataset characteristics and the overall computational strategy are summarized in FIGS. 1A-D (Methods). We used 1,634 whole-slide images from the Genomic Data Commons database: 1,176 tumor tissues and 459 normal tissues (FIG. 1A). The 1,634 whole-slide images were split into three sets: training, validation and testing (FIG. 1B). Importantly, this ensures that our model is never trained and tested on tiles (see below) obtained from the same tumor sample. Because the sizes of the whole-slide images are too large to be used as direct input to a neural network (FIG. 1C), the network was instead trained, validated and tested using 512×512 pixel tiles, obtained from non-overlapping ‘patches’ of the whole-slide images. This resulted in tens to thousands of tiles per slide, depending on the original size (FIG. 1D).”)
(b) accessing a machine learning model with the computer system, wherein the machine learning model has been trained on training data to predict gene profile data and to classify whole-slide images; (Coudray, Paragraph 6, “In this study, a deep convolutional neural network (inception v3) was trained on whole-slide images obtained from The Cancer Genome Atlas to accurately and automatically classify them into LUAD, LUSC or normal lung tissue. The performance of our method is comparable to that of pathologists, with an average area under the curve (AUC) of 0.97. The described system and processes were validated on independent datasets of frozen tissues, formalin-fixed paraffin-embedded tissues and biopsies. Furthermore, we trained the network to predict the ten most commonly mutated genes in LUAD. We found that mutations (in the examples, STK11, EGFR, FAT1, SETBP1, KRAS and TP53) can be predicted from pathology images, with AUCs from 0.733 to 0.856 as measured on a held-out population. These findings suggest that deep-learning models can assist pathologists in the detection of cancer subtype or gene mutations.”)
(c) inputting the WSI data to the machine learning model using the computer system, generating as outputs gene profile data and classified WSI data, wherein the gene profile data are indicative of a predicted gene profile for the histopathology sample and the classified WSI data are indicative of a classification of the whole-slide images of the histopathology sample as one of different disease classifications; (Coudray, Paragraph 6, “In this study, a deep convolutional neural network (inception v3) was trained on whole-slide images obtained from The Cancer Genome Atlas to accurately and automatically classify them into LUAD, LUSC or normal lung tissue. The performance of our method is comparable to that of pathologists, with an average area under the curve (AUC) of 0.97. The described system and processes were validated on independent datasets of frozen tissues, formalin-fixed paraffin-embedded tissues and biopsies. Furthermore, we trained the network to predict the ten most commonly mutated genes in LUAD. We found that mutations (in the examples, STK11, EGFR, FAT1, SETBP1, KRAS and TP53) can be predicted from pathology images, with AUCs from 0.733 to 0.856 as measured on a held-out population. These findings suggest that deep-learning models can assist pathologists in the detection of cancer subtype or gene mutations.”)
“and (d) outputting the gene profile data and classified WSI data with the computer system.” (Coudray, Figure 5 top right and bottom left. Also see Figures 10A (gene profile data output) and 13C-D (classified WSI data output).)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Coudray in view of Klaiman (US20250046454A1).
Regarding claim 6, Coudray teaches “The method of claim 1,”
Coudray does not expressly disclose, “wherein the machine learning model comprises a transformer encoder model.”
Klaiman teaches a whole slide image analysis model comprising a transformer encoder model (Klaiman, Paragraph 32, “The term “transformer layer” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a layer of a deep learning model that adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data. For example, the transformer layer may be based on a vision transformer model. A vision transformer model is an image classification model based on the transformer encoder architecture and using embeddings of image patches as inputs. In the vision transformer model, an image may be split into patches, then the patches are flattened, projected into lower dimensional embeddings, add-ed to positional embeddings and fed to a transformer encoder network. The output of the transformer encoder may be used as an input to a Multi-Layer Perceptron (MLP) head to generate the final prediction. The MLP head may comprise a set of linear transformation layers. The transformer encoder may comprise n encoders. Each encoder may comprise a multi-headed attention layer, normalization layers and an MLP layer. The term “Multi-Layer Perceptron neural network” or “MLP neural network” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a class of feedforward artificial neural networks. Residual skip connexions may be additionally used between the encoder's sublayers to enable the interaction between different level representations and prevent the vanishing gradient problem. Multi-headed attention may be based on running the self-attention mechanism multiple times. With respect to further design of the Multi-headed attention reference can be made to Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . & Po-losukhin, I. (2017), “Attention is all you need in Advances in neural information processing systems”, pp. 5998-6008. Self attention is a mechanism allowing to learn the relationships between the different inputs and to take these relationships into consideration during model training. Using a vision transformer as a secondary network to aggregate different modality embeddings is a new way of using this category of models. The present invention proposes to find relevant relationships between the different modalities and use it as an additional information during training.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use a transformer encoder model, as taught by Klaiman, for the machine learning model of Coudray.
The motivation for doing so would have been because the transformer encoder model includes self-attention, enabling better scalability and a wider receptive field for better handling of long range dependencies compared to the CNN disclosed by Coudray. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Coudray with the above teaching of Klaiman to fully disclose, “wherein the machine learning model comprises a transformer encoder model.”
Regarding claim 7, Coudray in view of Klaiman teaches “The method of claim 6,”
“wherein the transformer encoder model implements an attention mechanism.” (Klaiman, Paragraph 32, “The term “transformer layer” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a layer of a deep learning model that adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data. For example, the transformer layer may be based on a vision transformer model. A vision transformer model is an image classification model based on the transformer encoder architecture and using embeddings of image patches as inputs. In the vision transformer model, an image may be split into patches, then the patches are flattened, projected into lower dimensional embeddings, add-ed to positional embeddings and fed to a transformer encoder network. The output of the transformer encoder may be used as an input to a Multi-Layer Perceptron (MLP) head to generate the final prediction. The MLP head may comprise a set of linear transformation layers. The transformer encoder may comprise n encoders. Each encoder may comprise a multi-headed attention layer, normalization layers and an MLP layer. The term “Multi-Layer Perceptron neural network” or “MLP neural network” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a class of feedforward artificial neural networks. Residual skip connexions may be additionally used between the encoder's sublayers to enable the interaction between different level representations and prevent the vanishing gradient problem. Multi-headed attention may be based on running the self-attention mechanism multiple times. With respect to further design of the Multi-headed attention reference can be made to Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . & Po-losukhin, I. (2017), “Attention is all you need in Advances in neural information processing systems”, pp. 5998-6008. Self attention is a mechanism allowing to learn the relationships between the different inputs and to take these relationships into consideration during model training. Using a vision transformer as a secondary network to aggregate different modality embeddings is a new way of using this category of models. The present invention proposes to find relevant relationships between the different modalities and use it as an additional information during training.” Note that this was incorporated with motivation and rationale in the rejection of claim 6.)
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Coudray in view of Klaiman further in view of Khan (Multi-Task Learning with Transformers: Transformers with Multiple Prediction Heads).
Regarding claim 8, Coudray in view of Klaiman teaches “The method of claim 6,”
While Coudray in view of Klaiman disclose dual output of the gene profile data and the classified WSI data (see claim 1 rejection), Coudray in view of Klaiman do not expressly disclose that the transformer encoder model has a respective head for each data output.
Khan discloses transformer encoder models with respective prediction heads for outputting data of respective tasks, (Khan, Paragraphs 2 and 6 and Figure 1 reproduced below, “In this article, I will give present two different approaches to employing multi-task learning on your own custom datasets. The first approach is based on adding additional prediction heads to the BERT model and extending the implementation of the transformers package. You can find the implementation of both approaches on my Github as well.”; “Now comes the most detail of our implementation, here we borrow the implementation of BERTForSequenceClassification class and extend it to have more than one prediction head and route the batch of data to its task-specific prediction head, the encoder representations are identical for all tasks. I achieved this by adding “**kwargs” to __init__ method of the class and then passed “task_labels_map” as a keyword argument containing num labels for each task, and then I initialize the classification heads based on this mapping.”)
PNG
media_image1.png
318
386
media_image1.png
Greyscale
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use respective prediction task heads, as taught by Khan, for the output of the gene profile data and the classified WSI data by the transformer encoder model of Coudray in view of Klaiman.
The motivation for doing so would have been because, as described in Paragraph 1 of Khan, “multi-task learning can offer advantages like improved data efficiency, reduced overfitting through shared representations, and fast learning by leveraging auxiliary information”. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Coudray in view of Klaiman with the above teaching of Khan to fully disclose, “wherein the transformer encoder model has a first output head to output the gene profile data and a second output head to output the classified WSI data.”
Claim(s) 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Coudray in view of Comiter (WO 2023091970 A1).
Regarding claim 9, Coudray teaches “The method of claim 1,”
Coudray does not expressly disclose, “wherein the gene profile data comprise transcriptomic data.”
Comiter teaches using machine learning for inferring transcriptomic data (RNA-seq) from histology images (Comiter, Paragraph 607, “Given a training dataset of tissue samples, each with a corresponding tiled H&E image and sc/snRNA-seq data, SCHAF learns to infer an entire sc/snRNA-seq dataset from a histology image. SCHAF, if necessary, first integrates all training histology images into a common discrepancy -free domain via image normalization as described above, and then tiles the histology images (as described above).)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate the SCHAF model of Comiter, for generating transcriptomic data, into the gene prediction step of Coudray.
The motivation for doing so would have been to collect additional data covering a wider range of cell information. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Coudray with the above teaching of Comiter to fully disclose, “wherein the gene profile data comprise transcriptomic data.”
Regarding claim 10, Coudray in view of Comiter teaches “The method of claim 9,”
“wherein the transcriptomic data comprise RNA sequence (RNA-seq) data.” (Comiter, Paragraph 607, “Given a training dataset of tissue samples, each with a corresponding tiled H&E image and sc/snRNA-seq data, SCHAF learns to infer an entire sc/snRNA-seq dataset from a histology image. SCHAF, if necessary, first integrates all training histology images into a common discrepancy -free domain via image normalization as described above, and then tiles the histology images (as described above). Note that this was incorporated with motivation and rationale in the rejection of claim 9.)
Allowable Subject Matter
Claims 2-5 are objected to as being dependent upon a rejected base claim and rejected under 35 USC 112(b), but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and amended to overcome the 35 USC 112(b) rejection.
The following is a statement of reasons for the indication of allowable subject matter: With respect to claim 2 and dependent claims 3-5, in addition to other limitations in the claims the Prior Art of Record fails to teach, disclose or render obvious the applicant' s invention as claimed, in particular:
Claim 2 recites, “The method of claim 1, wherein step (a) includes:
generating WSI patch data by extracting patches from whole-slide images in the WSI data;
generating embedded WSI patch data by accessing a trained neural network and inputting the WSI patch data to the trained neural network, generating an output as the embedded WSI patch data;
forming embedded instance data from the embedded WSI patch data, wherein the embedded instance data comprises bags of instances in the embedded instance data; and
storing the embedded instance data as WSI data for inputting to the machine learning model.”
Coudray teaches classification, based on histopathology slides, of lung cancer type and gene mutations. Comiter teaches prediction of RNA-seq data using machine learning-based image analysis of histology images. Khan teaches transformer encoder architectures and codes for performing multi-task classifications. The closest secondary references to claim 2 are Ocampo (US20260120864A1) and Klaiman. Firstly, note that Coudray does generate image patches (see Coudray, Paragraph 40). Ocampo generates embeddings of image patches using a trained network (see Ocampo, Paragraph 157). Additionally, Ocampo discloses multi-instance learning (MIL) for image patch labelling in Paragraph 185, which amounts to the forming of instance data comprising bags of instances. However, Ocampo does not expressly disclose using the MIL on the embedded patch data to generate embedded instance data, and also does not expressly disclose using the MIL output as an input to other machine learning model for performing classification. Rather, the MIL of Ocampo appears to directly perform labeling. Klaiman, in Figure 2 element 126 and Paragraph 105, describes the use of a trained model that transforms WSI tiles into embeddings, then inputs these tile embeddings into a multi-instance learning (MIL) model (which amounts to the forming of instance data comprising bags of instances) to generate patient level embeddings (embedded instance data). Next, the resulting embedded instance data (patient level embeddings) are passed to an aggregation model along with other embeddings to generate patient predictions. This embedding-to-aggregation model pipeline is distinct from the classification pipeline of Coudray which uses the WSI patches for classifications without an aggregation step preceding a final patient prediction. Additionally, the machine learning model of Coudray is not configured to receive these patient level embeddings. Therefore, it would not be obvious to generate the embedded instance data and store the embedded instance data as the WSI data as the input to the machine learning model. Accordingly, none of the references expressly disclose the limitations of claim 2.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON JOSEPH SORRIN whose telephone number is (703)756-1565. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON JOSEPH SORRIN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights