DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Response to Arguments
The amendment to the claims filed June 1, 2026 has been entered. The disposition of the outstanding rejections and objections is as follows. The claim objections to claims 18 and 19 are withdrawn. The rejection under 35 U.S.C. 112(b) is withdrawn. The rejections under 35 U.S.C. 101 are maintained. The nonstatutory double patenting rejections are maintained. The rejections under 35 U .S.C. 103 are maintained as recast below to address the amended claim language.
The applicant argues the amended "such that" clause differs from Desjardins in that the amendment does not reflect an improvement to how the models operate (no reduced storage, no mitigation of catastrophic forgetting, no reduced complexity); training a respective model per biomarker cluster if anything increases storage and complexity. The identified benefit is an improved diagnostic result, which under Desjardins is not enough. To the extent the specification touts a training-efficiency advantage of weakly supervised learning (¶ 356), that advantage is not recited in the pending claims.
Reciting which conventional architectures (FCN, ResNet, Inception; specification ¶¶ 10-11, 348) are trained on which data is the application of established machine learning to a field of use, which Recentive found ineligible. Performing the task faster than a human does not confer eligibility.
Coudray's "multitask rather than single-task" statement contrasts multi-label with single-label output; it does not criticize or discourage an ensemble of separate per-biomarker models, and Coudray's "each individual classifier" language points toward, not away from, the claimed arrangement (MPEP 2145(X), 2123).
Even accepting that Coudray does not itself partition the dataset per biomarker, Couture and Faust each train a separate classifier per biomarker from H&E, and the one-versus-rest reorganization is obvious under KSR; the limitation is met under 35 U.S.C. 103.
Priority
Priority is not perfected. Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. 35 U.S.C. 119(e) as follows:
The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original nonprovisional application or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 U.S.C. 112(a) or the first paragraph of pre-AIA 35 U.S.C. 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994).
The disclosure of at least the two prior-filed applications, Application No. 62/671300 and 62/824039, fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph for one or more claims of this application. The independent claims of the instant application requires training biomarkers classification models using “a molecular training dataset” that includes “molecular data based on sequencing” and is “clustered by biomarker.” Processing segmented tile images by predicting a biomarker classification and prediction a tissue classification for each tile, and then determining the overall presence of biomarkers based on both predictions. Neither of the reviewed provisionals (62/671300 and 62/824039) teaches these specific elements.
Provisional 62/671300 describes an adversarial framework using a dual-classifier (Figure 3) and demonstrates that its output image embeddings naturally group by MSI statuses (Figures 8A and 8B). However, it fails to support the independent claims, which requires training models using a “molecular training dataset” that includes “molecular data based on sequencing” and includes “molecular data subsets clustered by biomarkers.” The provisional only teaches clustering as a mathematical visualization of the model’s output embeddings, rather than requiring pre-clustered, sequence-based input dataset to train the models. Since it is missing this information, it cannot provide an effective filling date.
The burden rests on the applicant to establish this support. Transco Products Inc. v. Performance Contracting, Inc., 38 F.3d 551 (Fed. Cir. 1994); PowerOasis, Inc. v. T-Mobile USA, Inc., 522 F.3d 1299, 1305 (Fed. Cir. 2008).
Applicant's reply did not provide the requested mapping. Each reference applied below was published or issued before March 25, 2020 and qualifies as prior art under 35 U.S.C. 102(a)(1). The burden to establish entitlement to an earlier effective filing date rests with the Applicant. Transco Products Inc. v. Performance Contracting, Inc., 38 F.3d 551 (Fed. Cir. 1994); PowerOasis, Inc. v. T-Mobile USA, Inc., 522 F.3d 1299, 1305 (Fed. Cir. 2008).
The pending claims are accorded an effective filing date of March 25, 2020. Application 18/433,240 is the last in an unbroken chain of continuations of U.S. application Ser. No. 16/830,186, filed March 25, 2020 (issued as U.S. Patent No. 10,957,041 B2), the earliest-filed application whose disclosure provides 35 U.S.C. 112(a) support for the pending claims; that application discloses and claims generating "for each of the one or more molecular data subsets, one of the trained biomarker classification models, based on the plurality of digital images of the H&E stained training slides," corresponding to the amended "such that" clause. Application 16/830,186 is a continuation-in-part of Ser. Nos. 16/732,242 and 16/412,362 and claims benefit of Provisional Nos. 62/671,300, 62/787,047, 62/824,039, 62/889,521, and 62/983,524; the molecular-training limitations were introduced by the continuation-in-part and are not supported under 35 U.S.C. 112(a) in the pre-CIP applications or the provisionals (e.g., 62/671,300 is directed to "predicting microsatellite instability from histopathology slide images" and discloses none of the molecular-dataset, sequencing, clustering, or per-subset-model limitations).
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
A request to hold the rejection in abeyance is not a complete reply; no terminal disclaimer has been filed, so the rejections are maintained.
Claims 1-19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-30 of U.S. Patent No. 10,957,041 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because the ‘240 application claims recite the same fundamental process of using a deep learning framework to predict biomarker and tissue classification from tiled H&E slide images to determine biomarker presence, differing only in that the ‘240 application omits narrowing limitations. The amended "such that'" clause does not render the claims patentably distinct: the '041 patent itself claims generating "for each of the one or more molecular data subsets, one of the trained biomarker classification models, based on the plurality of digital images of the H&E stained training slides."
Claims 1-19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-30 of U.S. Patent No. 11,610,307 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because the ‘240 application claims recite the same fundamental process of using a deep learning framework to predict biomarker and tissue classification from tiled H&E slide images to determine biomarker presence, differing only in that the ‘240 claims being a broader genus that encompasses the more specific species claimed in the ‘307 patent. The '307 patent claims generating "for each of the one or more molecular data subsets, a trained image-based biomarker classifier model, based on the plurality of digital images of the H&E stained training slides."
Claims 1-19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-28 of U.S. Patent No. US 11,682,098 B2. Although the claims at issue are not identical, they are not patentably distinct from each other the ‘240 application claims recite the same fundamental process of using a deep learning framework to predict biomarker and tissue classification from tiled H&E slide images to determine biomarker presence, with the ‘098 patent claims adding details of multi-tile adjacency analysis and 3D probability array generation. The '098 patent likewise claims generating "for each of the one or more molecular data subsets, a trained image-based biomarker classifier model, based on the plurality of digital images of the H&E stained training slides."
Claim Rejections - 35 USC § 101 – Maintained
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. § 101 because the claimed invention is directed
to an abstract idea without significantly more. The amendment adding the "such that" per-biomarker training clause docs not overcome the rejection, for the reasons set forth below
and in the Response to Arguments.
The claims recite using trained deep learning/biomarker classification models to predict biomarker classifications and tissue classifications for tile images, and determining a predicted presence of biomarkers which constitutes mathematical concepts (ML model inference) and mental processes (evaluating classification results to determine biomarker presence). The additional elements of generic computer components (processors, memory, network), processing segmented tile images, and transmitting predictions do not integrate the judicial exception into a practical application, as they amount to mere instructions to apply the abstract idea on a computer, insignificant extra-solution activity, and field-of-use limitations. At Step 2B,
these additional elements are well-understood, routine, and conventional as evidenced by
the specification's description of generic computing hardware and known image processing techniques (spec ¶¶ 10-11 , 392-395) and the prior art of record (Chukka ¶ 10; Coudray p.
1560). The amended "such that" clause recites how the models are trained, a characteristic of the mathematical model and its training data; it further defines the mathematical concept and does not remove the claim from the abstract-idea grouping or integrate the exception. Applicant's Ex parte Desjardins and Recentive arguments are addressed in the Response to Arguments above.
STEP 1: Statutory Category
Claim
Category
Pass?
1 (+ deps 2-17)
Machine (system)
Yes
18
Manufacture (CRM)
Yes
19
Process (method)
Yes
STEP 2A, PRONG 1: Judicial Exception?
Yes – Abstract Idea (Mathematical Concepts + Mental Processes)
The independent claims recite:
"predicting a respective biomarker classification for each tile image using one
or more biomarker classification models." This is a mathematical concept. The
biomarker classification models perform mathematical operations (neural network
computations) to generate classification predictions. Per the TC2100 guidance and
July 2024 SME Examples 47-49, ML model inference constitutes mathematical
calculations.
"predicting a respective tissue classification for each tile image using one or more trained deep learning classifier models." Same analysis as above, applying trained DL models to classify tiles is a mathematical calculation.
"determine, based on (i) and (ii), a predicted presence of one or more
biomarkers in the target tissue." This is both a mathematical concept (aggregating classification predictions) and a mental process (a pathologist could, in principle, evaluate tissue and biomarker classifications to determine biomarker presence, the specification at [0004]-[0005] describes pathologists performing this evaluation manually).
Training data description of a molecular training dataset with sequencing data
clustered by biomarker. This describes characteristics of training data, which is a data characteristic that further defines the mathematical model but does not add a non-abstract limitation.
The claims recite mathematical concepts (ML model inference/classification) and mental
processes (determining biomarker presence from classification results) . These are considered together as a single abstract idea. The amended "such that" clause recites how the models are trained, that is, a characteristic of the mathematical model and of its training data. Describing what data a model is trained on, and that a respective model is trained per data subset, further defines the mathematical concept; it does not remove the claim from the abstract-idea grouping. (Step 2A, Prong 1: YES)
STEP 2A, PRONG 2: Practical Application?
No, the claims do not integrate the abstract idea into a practical application.
Additional elements beyond the judicial exception:
"one or more processors; an electronic network; one or more memories" (claim 1) Generic computer components recited at a high level of generality. These are mere instructions to "apply it" on a computer. See MPEP § 2106.05(f).
"process a plurality of segmented tile images each corresponding to a different
respective portion of the digital image." Data gathering/pre-processing. This is insignificant extra solution activity (data characterization). See MPEP § 2106.05(g).
"transmit, via the electronic network, the predicted presence of the one or more biomarkers" Insignificant extra-solution activity (mere data output/transmission). See MPEP§ 2106.05(g).
"Hematoxylin and Eosin-stained slide of a target tissue." Field of use limitation (medical imaging/histopathology). See MPEP § 2106.05(h).
Training data characteristics such as the molecular dataset corresponding to training tissue samples, including molecular sequencing data, clustered by biomarker. These
describe the type of training data used, which is a data characteristic of the mathematical model, not a practical application. This limits the field of use of the abstract idea.
The amended per-biomarker training clause is not an additional element that
integrates the exception; it is part of the recited mathematical concept. Considered as a whole,
the claims do not recite an improvement to the functioning of a computer or to any other
technology. The specification frames the advance as an improved diagnostic result. An improved result obtained by applying conventional models to a new data set is not an improvement to the operation of the computer or model.
No improvement to computer functionality is claimed . The specification describes the
improvement as better biomarker detection results from H&E images (avoiding IHC
staining), which is an improvement to results, not to the functioning of a computer or other
technology. See Intellectual Ventures I LLC v. Capital One Bank, 792 F.3d 1363 (Fed. Cir.
2015).
No particular machine beyond generic processors/memory.
No transformation of a physical article. The claims do not treat a patient, control a medical device, or interface with physical pathology equipment in a meaningful way.
(Step 2A, Prong 2: NO, not integrated into practical application. Claims are directed to the abstract idea.)
STEP 2B: Inventive Concept?
No, the additional elements do not provide an inventive concept.
Generic computer components (processors, memory, network) are well-understood, routine, and conventional (WURC). See MPEP § 2106.05(d); Berkheimer evidence: specification at ¶¶ 392-395 describes generic processor implementations.
Transmitting data over a network is WURC. See OIP Techs., Inc. v. Amazon.com, 788
F.3d 1359 (Fed. Cir. 2015).
Processing tile images from a digital image is WURC in the histopathology field. The
specification at ¶¶ 10-11 acknowledges that tiling/segmentation of WSls and use of CNNs/FCNs for image classification are known.
The specific training data characteristics do not supply an inventive concept at Step
2B because they describe what data the mathematical model was trained on, not an
unconventional technical step.
(Step 2B: NO - no inventive concept.)
Dependent claims: Most dependent claims add further details of the mathematical process (claims 3-8, 10-16) or data characteristics (claim 9, 11) that remain within the abstract idea. Claim 17 (adding pathology slide scanner system and receiving image from it) adds a particular data gathering device, but this is likely WURC extra-solution activity given that digital slide scanners are conventional in histopathology.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 9, and 12-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chukka et al. (US 2016/0042511 A1 – hereinafter “Chukka”) in view of Coudray et al. (Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning – hereinafter “Coudray”), and further in view of Couture et al. (Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype – hereinafter “Couture”) and Faust et al. (WO 2019/084697 A1 – hereinafter “Faust”).
Claim 1.
A computing system for identifying biomarkers in a digital image of a Hematoxylin and Eosin-stained slide of a target tissue (Chukka teaches a computing system for identifying biomarkers in a slide stained with "hematoxylin (blue stain)" (Chukka, ¶ [0024]).), comprising:
one or more processors;
an electronic network (Chukka, ¶¶ [0024]-[0033]); and
one or more memories having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to (Chukka discloses the hardware components, stating "analyzing system 100 includes an imaging apparatus 120 and a computer system 110" that are interconnected by a "network 130" (Chukka, ¶ [0027]), where the computer system contains processors and memory executing instructions (Chukka, ¶ [0029]).):
process a plurality of segmented tile images each corresponding to a different respective portion of the digital image (Chukka teaches processing a plurality of segmented tile images, stating, "The tissue region is then segmented into tiles and classification and nuclei counting algorithms are performed with respect to each tile" (Chukka, ¶ [0010]). Chukka does not explicitly teach processing the tiles "using a deep learning framework.")
(i) predicting a respective biomarker classification for each segmented tile image using one or more biomarker classification models (Chukka teaches predicting classifications for the extracted tiles and objects, including utilizing trained object classifiers to determine "gene status in breast carcinomas," which constitutes a biomarker classification model (Chukka, ¶ [0023]). Chukka teaches processing a plurality of segmented tile images, stating, "The tissue region is then segmented into tiles and classification and nuclei counting algorithms are performed with respect to each tile" (Chukka, ¶ [0010]).),
, and
(ii) predicting a respective tissue classification for each segmented tile image using one or
more trained deep learning classifier models (Chukka teaches predicting tissue classifications for each tile image by assigning a probability of the extracted feature belonging to a "tissue, stromal, or lymphatic region" (Chukka, ¶ [0036]). As established below, Coudray teaches implementing these classifiers using deep learning models);
determine, based on (i) and (ii), a predicted presence of one or more biomarkers in the target tissue (Chukka teaches determining the presence of biomarkers based on the individual
classifications, disclosing obtaining a composite score for the slide based at least in part
on the characteristics of the extracted objects" (Chukka, ¶ [0010]); and
transmit, via the electronic network, the predicted presence of the one or more biomarkers (Chukka teaches transmitting the predicted results over the electronic network, stating that
the data "In addition or alternatively, a score and record of the analysis performed for the tissue in the slide can be transmitted over a computer communication link (e.g., the Internet) to a remote computer system for viewing, storage, or analysis " (Chukka, ¶ [0052]).).
Chukka does not explicitly teach processing the tiles "using a deep learning framework." Also, Chukka does not explicitly teach “training dataset that (a) corresponds to a plurality of training tissue samples, (b) includes molecular data based on sequencing of a corresponding sample associated with each training tissue sample, and (c) includes a plurality of molecular data subsets clustered by biomarker.” However, Coudray teaches utilizing a deep learning framework to process segmented tile images corresponding to portions of a digital whole-slide image. Coudray states, "the network was instead trained, validated and tested using 512x512 pixel tiles, obtained from nonoverlapping 'patches' of the whole-slide images" (Coudray, Page 1560, left column) and processed using a "deep convolutional neural network (inception v3)" (Coudray, Page 1559, Abstract). Coudray also teaches (i) predicting a respective biomarker classification for each tile image using one or more biomarker classification models. Coudray teaches predicting a biomarker classification for each tile image, stating, "we trained the network to predict the ten most commonly mutated genes in LUAD" (Coudray, Page 1559, Abstract). Coudray applies these biomarker classification models to each individual tile, explaining that "Each mutation classification was treated as a binary classification, and our formulation allowed multiple mutations to be assigned to a single tile" (Coudray, Page 1568, left column) and "the per-tile classification results were aggregated on a per-slide basis" (Coudray, Page 1560, right column). Wherein the one or more biomarker classification models are trained using a molecular training dataset that (a) corresponds to a plurality of training tissue samples, (Coudray teaches training the models using a molecular training dataset that corresponds to a plurality of training tissue samples and includes molecular data based on sequencing.). (b) includes molecular data based on sequencing of a corresponding sample associated with each training tissue sample, (Coudray states, "gene mutation data for matched patient samples were downloaded from TCGA" (Coudray, Page 1562, right column) and "data from the TCGA dataset used for training were identified with the next-generation sequencing (NGS) tools lllumina HiSeq 2000 or Genome Analyzer II" (Coudray, Page 1564, right column) and (c) includes a plurality of molecular data subsets clustered by biomarker, and (Coudray teaches this limitation by grouping or partitioning the molecular dataset into distinct subsets based on the presence of the 10 most common gene mutations (biomarkers). Coudray teaches, “To make sure the training and test sets contained enough images from the mutated genes, we only selected those which were mutated in at least 10% of the available tumors” (Coudray, Page 1562, right column). Coudray structures this training dataset into biomarker-specific subsets to train the classifiers, stating, “fully trained this network … using only LUAD whole-slide images … each cell associated to a mutation and set to 1 or 0 depending on the presence or absence of the mutation. Only the most commonly mutated genes were used … leading to a training set of 223,185 tiles.”) (ii) predicting a respective tissue classification for each tile image using one or more trained deep learning classifier models. (Coudray teaches predicting a respective tissue classification for each tile using the trained deep learning models, stating: "we trained inception v3 to recognize tumor versus normal" (Coudray, Page 1560) and "we trained the network on a direct three-way classification into the three types of images (normal, LUAD, LUSC)" (Coudray, Page 1560). Coudray explicitly performs this via per tile classification, noting "it takes ~20s to calculate per-tile classification probabilities on 500 tiles" (Coudray, Page 1561, left column)). determine, based on (i) and (ii), a predicted presence of one or more biomarkers in the target tissue; and (Coudray teaches determining the predicted presence of the biomarker in the target tissue based on the tile classifications, explicitly stating that "the per-tile classification results were aggregated on a per-slide basis ... thus generating a per-slide classification" (Coudray, Page 1560, right column)).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to modify the automated digital slide analysis system of Chukka with the deep convolutional neural network framework of Coudray. A person of ordinary skill in the art would be motivated to integrate Coudray's deep learning architecture into Chukka's automated scanning and computer network system to advance beyond traditional image feature extraction, thereby improving the accuracy of tissue classification (e.g., distinguishing between tumor subtypes like LUAD and LUSC). Furthermore, the artisan would be strongly motivated to utilize Coudray's method of training the deep learning framework on sequencing-derived molecular datasets clustered by biomarker. Combining these teachings predictably allows the computing system to not only classify standard optical tissue morphologies but also accurately predict actionable molecular biomarkers (such as specific gene mutations) directly from standard H&E slide tiles, avoiding the need for separate costly NGS assays and streamlining the diagnostic workflow.
As to the amended "such that" per-biomarker separate-model clause: Coudray trains a single inception-v3 with a shared backbone and a multi-label output rather than a separate model per biomarker (Coudray: "multitask classification rather than a single-task classification ... Each mutation classification was treated as a binary classification" (p. 1568); "we optimized the average of the cross entropy of each individual classifier" (p. 1568)). Couture trains a separate image-based classifier for each of a plurality of biomarkers from H&E (Couture: "create image-based classifiers for tumor grade, ER status, P AM50 intrinsic subtype, histologic subtype, and risk of recurrence score (ROR-PT)" (Abstract); "A new classifier must be trained" per target (p. 5)), and Faust trains a separate deep convolutional neural network fine-tuned per classification target from H&E, including molecular biomarkers (Faust: "Each convolutional neural network may thus be optimized or fine-tuned for different classification purposes" (¶ 106); "retrain the final layers of the image-based VGG 19 neural network ... transfer learning" (¶ 213 ); IDH mutant/wildtype and "1 p/19g co-deletions" (¶ 103)).
It would have been obvious to train Coudray's deep biomarker model separately for each biomarker cluster, as Couture and Faust do, providing a respective model per biomarker trained on that biomarker's H&E images, with the predictable benefit of modular per-biomarker prediction and per-biomarker training-set tuning, and a reasonable expectation of success. In the alternative, the clause would have been obvious over Chukka in view of Coudray under KSR, instantiating Coudray's per-biomarker "individual classifier" as separate one-versus-rest models (MPEP 2143(E), (B)).
Claims 18 and 19.
The combination of Chukka, Coudray, Couture, and Faust teaches a non-transitory computer-readable medium comprising a set of computer executable instructions that, when executed by one or more processors, cause a computer to (Chukka, ¶¶ [0024]-[0033]) …
The remaining limitations correspond to claim 1 and are taught by Chukka in view of Coudray, Couture, and Faust for the reasons given for claim 1, applied mutatis mutandis.
Claim 2.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 1, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
separate the digital image into the plurality of segmented tile images by processing the digital image using at least one of (i) a tiling mask or (ii) a trained multiple instance learning controller (“The component 112 can use a grid pattern to tile the portion of the slide corresponding to the tissue data” (Chukka, ¶ [0044]) Applying a grid pattern to a digital image to segment it into discrete sub-regions (tiles) constitutes applying a tiling mask. IPR2026-00185, Exhibit 1002, Dr. Papanikolopoulos, ¶¶ [0069]-[0071].).
Claim 9.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 1, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: for each tile image in the plurality of tile images: infer a class status of the tile image; and discard, when the class status of the tile image does not correspond to a desired class, the tile image (Chukka teaches inferring the class of the region/tile and discarding it if it does not correspond to the desired class. Chukka states, “an approximate region segmentation process can be performed based at least in part on, for example, an identification of dense lymphocyte clusters. Accordingly, detected lymphocyte regions can be discarded while searching for, as an example, blue-stained tumor nuclei” ¶[0045].).
Claim 12.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 1, wherein the deep learning framework includes at least one of a multi-scale deep learning framework or a single-scale deep learning framework (Coudray trains the neural network using fixed, uniform "512x512 pixel tiles” (Coudray, Page 1560, left column) at a single magnification, this is a single-scale framework.).
Claim 13.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 12, wherein the single-scale deep learning framework is a convolution neural network having a ResNet configuration or an Inception configuration ("we trained inception v3 to recognize tumor versus normal" (Coudray, Page 1560)).
Claim 14.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 1, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: for each tile image in the plurality of tile images: process the tile image using a biomarker classification model trained to predict a different respective biomarker classification; and determine, based on the predicted biomarkers of the tile image, a predicted presence of one or more biomarkers in the target tissue (Coudray applies these biomarker classification models to each individual tile, explaining that "Each mutation classification was treated as a binary classification, and our formulation allowed multiple mutations to be assigned to a single tile" (Coudray, Page 1568, left column) and "the per-tile classification results were aggregated on a per-slide basis" (Coudray, Page 1560, right column)); and generate a report containing the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers (Chukka teaches "a score and record of the analysis performed for the tissue in the slide can be transmitted " (Chukka, ¶ [0052]) and “an overlay image is produced to label features of interest in the image of a specimen from a subject” (Chukka, ¶ [0010]).).
Claim 15.
Chukka, Coudray, Couture, and Faust teaches the computing system of claim 14, wherein the digital overlay includes an overlay element identifying tumor content of the digital image or tumor percentage of the digital image (Chukka teaches generating a digital overlay based on the percentage of tumor classification “estimating the percentage of positively-stained (e.g., brown-colored) nuclear objects to the total number of positively-stained and negatively-stained ( e.g., blue-colored)” ¶¶ [0003]-[0004]. It would be obvious that an overlay image designed to visualize a slide scored by tumor percentage would include an element identifying that percentage.).
Claim 16.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 1, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: for each molecular data subset in the one or more molecular data subsets: receive a plurality of digital images of Hematoxylin and Eosin-stained training slides of training tissue samples corresponding to the respective different biomarker of the molecular data subset in an image-based biomarker prediction system having one or more processors (Coudray teaches, “To make sure the training and test sets contained enough images from the mutated genes, we only selected those which were mutated in at least 10% of the available tumors” (Coudray, Page 1562, right column) and received images corresponding to these biomarkers “fully trained this network … using only LUAD whole-slide images … each cell associated to a mutation and set to 1 or 0 depending on the presence or absence of the mutation.” (Coudray, Page 1568, right column)); and generate one of the trained biomarker classification models, based on the plurality of digital images of the Hematoxylin and Eosin-stained training slides (Coudray teaches generating the trained biomarker classification models based on these digital images, stating, "we trained the network to predict the ten most commonly mutated genes in LUAD" (Coudray, Page 1559, Abstract).).
Claim 17.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 1, wherein the computing system further comprises: a pathology slide scanner system; and the one or more memories have stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: receive, via the electronic network, the digital image from the pathology slide scanner system (Chukka states, "analyzing system 100 includes an imaging apparatus 120 and a computer system 110" and the “images are sent to a computer system 110 either through a direct connection or via a network 130” (Chukka, ¶ [0027]), where the computer system contains processors and memory executing instructions (Chukka, ¶ [0029])).
Claims 3, 4, 7, 8, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Chukka in view of Coudray, Couture, and Faust as applied to claim 1 above, and further in view of Pan et al. (Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks – hereinafter “Pan”).
Claim 3.
The combination of Chukka, Coudray, Couture, and Faust teaches the computing system of claim 1, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to:
receive, at the deep learning framework, at least one training Hematoxylin and Eosin stained
slide image having a respective label corresponding to a respective biomarker (Coudray teaches receiving training H&E slide images having labels corresponding to respective biomarkers. Coudray states that for the training datasets, "each cell [was] associated to a mutation and set to 1 or 0 depending on the presence or absence of the mutation" (Coudray, Page 1567) and "we trained the network to predict the ten most commonly mutated genes" (Coudray, Page 1559).);
classify the Hematoxylin and Eosin-stained slide image using tile-based tissue classification analysis (Coudray teaches classifying the slide image using a tile-based tissue classification analysis. Coudray states, "we trained inception v3 to recognize tumor versus normal. .. the per-tile classification results were aggregated on a per-slide basis" (Coudray, Page 1560).);
Neither Chukka nor Coudray explicitly teach “analyzing the Hematoxylin and Eosin-stained slide image using a pixel-based cell segmentation.” However, Pan, in the same field of endeavor of pathology image analysis, teaches analyzing the Hematoxylin and Eosin-stained slide image using a pixel-based cell segmentation. Specifically, Pan discloses utilizing a neural network approach for "regression of a density map to robustly detect the nuclei," explicitly operating at the pixel level by predicting density values continuously across the image space to segment individual cells (Pan, Page 1, Abstract).
It would have been obvious to a person of ordinary skill in the art to integrate Pan's pixel-based Fully Convolutional Network into the Chukka and Coudray tile-based architecture. A person of ordinary skill in the art would be motivated to do so to resolve common pathology "data complexity (cell overlapping, inhomogeneous intensities, background clutters and image artifacts)" (Pan, Page 1, Abstract) by utilizing Pan's density map regression provides highly accurate, pixel-level cell isolation, yielding cleaner cellular inputs for deep learning biomarker prediction models.
Claim 4.
The combination of Chukka, Coudray, Couture, Faust, and Pan teaches the computing system of claim 3, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: identify a plurality of cells within the plurality of tile images using a trained cell segmentation model by: applying each of the plurality of segmented tile images to a cell segmentation model and, for each segmented tile image, assigning a cell classification to one or more pixels within the segmented tile image (Pan teaches assigning values to individual pixels within a tile, stating "The typical application of the convolution network is the classification task, where the output of the image is a class label. However, in many visual tasks, especially in biomedical image processing, the expected output should include localization. That is to say, a class label should be assigned to each pixel." (Pan, p. 7, § 3.2.1)).
It would have been obvious to a person of ordinary skill in that art to incorporate Pan’s pixel-level classification to Chukka’s and Coudray’s system to provide precise, robust, localization and boundaries for individual cells within complex, overlapping pathology images.
Claim 7.
The combination of Chukka, Coudray, Couture, Faust, and Pan teaches the computing system of claim 3, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: classify the Hematoxylin and Eosin-stained image using tile-based biomarker classification analysis (Coudray teaches classifying the H&E stained slide using a tile-based biomarker classification system, "the per-tile classification results were aggregated on a per-slide basis" (Coudray, Page 1560, right column).).
Claim 8.
The combination of Chukka, Coudray, Couture, Faust, and Pan teaches the computing system of claim 3, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: generate one or both of (i) the trained biomarker classification models, and (ii) the trained deep learning classifier models (Coudray teaches generating both the trained biomarker classification models and the trained deep learning classifier models, "we trained the network to predict the ten most commonly mutated genes in LUAD" (Coudray, Page 1559, Abstract) and "we trained inception v3 to recognize tumor versus normal" (Coudray, Page 1560).).
Claim 10.
The combination of Chukka, Coudray, Couture, Faust, and Pan teaches the computing system of claim 1, wherein at least one of the trained deep learning classifier models is a tile-resolution Fully Convolutional Network (FCN) classification model (Pan explicitly teaches applying "a novel multi-scale fully convolutional neural networks approach" to image patches (tiles) to detect nuclei (Pan, Page 1, Abstract).).
The rationale provided for the rejection of claim(s) 3 is applicable to claim 10, mutatis mutandis.
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Chukka, Coudray, Couture, Faust, and Pan as applied to claim 1 above, and further in view of Jones et al. (Voronoi-Based Segmentation of Cells on Image Manifolds– hereinafter “Jones”).
Claim 5.
The combination of Chukka, Coudray, Couture, Faust, and Pan teaches the computing system of claim 4, the one or more memories having stored thereon further instructions that, when executed by the one or more processors, cause the computing system to: assign the cell classification to one or more pixels within the tile image by: identifying the one or more pixels as a cell interior, (Chukka discloses an adaptive thresholding technique “to distinguish between darker objects ( e.g., cell nuclei [interior]) and other objects, ( e.g., stroma and slide background [exterior])” ¶[0026].)
Chukka, Coudray, Couture, Faust, and Pan discloses all of the subject matter as described above except for specifically teaching “a cell border.” However, Jones in the same field of endeavor teaches a cell border (p. 535, Abstract, “finding the borders of cells in microscopy images” and p. 541 “algorithm is only responsible for computing cell-cell boundaries”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Chukka, Coudray, Pan, and Jones before the effective filing date of the claimed invention. The motivation for this combination of references would have been to more precisely and completely delineate individual cellular boundaries from the surrounding background tissue.
Claim 6.
The combination of Chukka, Coudray, Couture, Faust, Pan, and Jones teaches the computing system of claim 4, wherein the trained cell segmentation model is a pixel-resolution three-dimensional classification model (Chukka ¶0048 discloses a “RGB input image”) trained to classify a cell interior, a cell border, and a cell exterior (Chukka discloses an adaptive thresholding technique “to distinguish between darker objects ( e.g., cell nuclei [interior]) and other objects, ( e.g., stroma and slide background [exterior])” ¶[0026]; Jones p. 535, Abstract, “finding the borders of cells in microscopy images” and p. 541 “algorithm is only responsible for computing cell-cell boundaries).
The rationale provided for the rejection of claim(s) 5 is applicable to claim 6, mutatis mutandis.
Claims 11 is rejected under 35 U.S.C. 103 as being unpatentable over and Chukka, Coudray, Couture, and Faust as applied to claim 1 above, and further in view of Applicant Admitted Prior Art (“AAPA”).
Claim 11.
The combination of Chukka, Coudray, Couture, Faust teaches the computing system of claim 1, wherein the one or more biomarkers include at least one of a tumor-infiltrating lymphocyte (TIL) biomarker, a nucleus-to-cytoplasm (NC) ratio biomarker, a ploidy biomarker, a signet ring morphology biomarker, a programmed death-ligand 1 (PD-L 1) biomarker, a consensus molecular subtype (CMS) biomarker, a human epidermal growth factor receptor 2 (HER2) biomarker, or a homologous recombination deficiency (HRD) biomarker (the '249 specification's AAPA explicitly admits that utilizing predictive models to identify "CMS classifications" and "CMS category assignment" was known in the prior art (AAPA ¶¶ [0146], [0147]). Alternatively, Chukka teaches predicting ''gene status in breast carcinomas" (¶ [0023]), and a POSITA would readily recognize HER2 as the ubiquitous, standard gene status biomarker tested in breast carcinomas).
It would have been obvious to a person of ordinary skill in the art to configure the Chukka and Coudray deep learning system to specifically predict the Consensus Molecular Subtype (CMS) biomarkers admitted as prior art (AAPA). The AAPA establishes CMS classifications as known, clinically significant molecular profiles derived from RNA expression. An artisan would be strongly motivated to apply Coudray's technique of predicting genetic mutations directly from optical H&E tiles to these known CMS biomarkers, thereby expanding the system's prognostic capabilities without requiring costly or time-consuming physical sequencing assays.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
Prosecution Insights