DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1 and 11 are objected to because of the following informalities: claim terms “viable tumor” and “necrotic tumor” should recite “a viable tumor” and “necrotic tumor”, respectively, for proper antecedent basis. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7, 9-10, 12-17, and 19 are rejected are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without integration into a practical application or recitation of significantly more.
In the analysis below, the method of independent claim 1 is considered representative of independent claim 1 and 10 since all of the independent claims recite identical steps despite being directed to different statutory matter. Furthermore, independent claims 1 and 10 are directed to one of the four statutory categories of eligible subject matter (a process for independent claim 1; an apparatus for independent claim 10); thus, the claims pass Step 1 of the Subject Matter Eligibility Test (See flowchart in MPEP 2106).
Step 2A, prong 1 analysis:
The independent claims are directed to determining predicted outcomes of subjects with cancer from biomedical images by identifying a biomedical image of a tissue sample from a subject with cancer, the biomedical image having (i) a first region of interest (ROI) corresponding to viable tumor in the tissue sample and (ii) a second ROI corresponding to necrotic tumor in the tissue sample; determining (i) a first segment identifying the first ROI in the biomedical image and (ii) a second segment identifying the second ROI in the biomedical image; determining a ratio between a first size of the first segment associated with the viable tumor and a second size of the second segment associated with the necrotic tumor; generating, a value indicative of a predicted outcome of the cancer in the subject using the ratio; and storing an association between the subject and the value indicative of the predicted outcome.
Each of the above steps can be performed mentally. In particular, a skilled practitioner in medicine such as a medical doctor uses their own trained medical knowledge and human vision to observe biomedical images taken by machines to identify cancerous tissues and differentiating between a viable tumor and a necrotic tumor; a doctor uses their naked human vision to spot gross (large-scale) necrosis in a tumor during surgery or when slicing open an excised tissue sample or in images of the tissue sample; further the doctor notes different region of interest (ROIs) that are critical for evaluation on the images with pen such as drawing bounding boxes; the doctor takes drawn bounding box ratios of the x and y dimensions of the bounding boxes drawn for the identified viable and necrotic tumors, respectively; the doctor then does a diagnosis and indicates on a scale between 1 and 10 how bad the cancer is for the patient using their medical knowledge and recording the patients data in their file on a computer; therefore, this process can all be done mentally.
As such, the description in independent claims 1 and 10 is an abstract idea – namely, a mental process. Accordingly, the analysis under prong one of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Additional elements:
The additional element recited in independent claims 1 and 10 are a computing system, a machine learning model, and data structures.
Step 2A, prong 2 analysis:
The above-identified additional elements do not integrate the judicial exception into a practical application. A computing system, a machine learning model, and data structures are generic computers that simply automate what a human doctor already does in prognosis and diagnosis of tumors looking at medical imaging data.
Each of the other additional elements (a computing system, a machine learning model, and data structures) amounts to merely using different devices as tools to perform the claimed mental process. Implementing an abstract idea on a computer or using known generic devices does not integrate a judicial exception into a practical application (See MPEP 2106.05(f)).
Moreover, the additional elements of the claims do not recite an improvement in the functioning of a computer or other technology or technical field, the claimed steps are not performed using a particular machine, the claimed steps do not effect a transformation, and the claims do not apply the judicial exception in any meaningful way beyond generically linking the use of the judicial exception to a particular technological environment (See MPEP 2106.04(d)). Therefore, the analysis under prong two of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Step 2B:
Finally, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Each of the other additional elements (a computing system, a machine learning model, and data structures) are generic computer features which perform generic computer functions that are well-understood, routine, and conventional and do not amount to more than implementing the abstract idea with a computerized system. Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea).
Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation, and mere implementation on a generic computer does not add significantly more to the claims. Accordingly, the analysis under step 2B of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
For all of the foregoing reasons, independent claims 1 and 10 do not recite eligible subject matter under 35 USC 101.
Claims 2 and 12 recite classifying, by the computing system, the subject into a risk stratification category of a plurality of risk stratification categories based on a comparison of the value with a threshold, and maintaining, by the computing system, a measure of progression of the cancer in the subject using the risk stratification category of the subject over a plurality of time instances. This is all part of the diagnosis and evaluation for cancer a medical doctor does for a patient using their own knowledge and human vision to make the evaluations; therefore, this process can all be done mentally.
Claims 3 and 13 recite determining, by the computing system, the threshold to compare against, based on a plurality of values each indicative of predicted outcome determined for a respective subject of a plurality of subjects. This is all part of the diagnosis and evaluation for cancer a medical doctor does for a patient using their own knowledge and human vision to make the evaluations such as comparing healthy image data of other patients to the current patient; therefore, this process can all be done mentally.
Claims 4 and 14 recite wherein generating the value further comprises generating the value indicating at least one of an overall survival or a progression-free survival of the subject, based on the ratio of the first size of the first segment associated with the viable tumor and the second size of the second segment associated with the necrotic tumor. This is all part of the diagnosis and evaluation for cancer a medical doctor does for a patient using their own knowledge and human vision to make the evaluations whether a patient has cancer and how much the cancer has progressed; therefore, this process can all be done mentally.
Claims 5 and 15 recites wherein identifying the biomedical image further comprises receiving a corresponding plurality of biomedical images of a respective plurality of tissue samples obtained from an anatomical site for the cancer of the subject over a corresponding plurality of time instances, and wherein generating the value further comprises generating the value indicative of the predicted outcome of the cancer for the subject at a respective time instance of the plurality of time instances at which the tissue sample was obtained. This is all part of the diagnosis and evaluation for cancer a medical doctor does for a patient using their own knowledge and human vision to make the evaluations; the doctor keeps data over time of a patient developing cancer and evaluates how the progression of the cancer develops based on the images taken over time; therefore, this process can all be done mentally.
Claims 6 and 16 recite wherein identifying the biomedical image further comprises receiving, via an imaging acquirer, a plurality of biomedical images each corresponding to a whole slide image (WSI) of a respective tissue sample stained to differentiate the viable tumor and the necrotic tumor from a remainder of the tissue sample, and wherein determining the ratio further comprises determining the ratio between (i) a respective first size of the first segment associated with the viable tumor and (ii) a respective second size of the second segment associated with the necrotic tumor determined from each of the plurality of biomedical images. This is all part of the diagnosis and evaluation for cancer a medical doctor does for a patient using their own knowledge and human vision to make the evaluations such as comparing healthy image data of other patients to the current patient; the doctor views whole slide images and uses their knowledge to find the tumors in the images and determines sizes of the tumor segments in tissue; therefore, this process can all be done mentally.
Claims 7 and 17 recite wherein applying the machine learning model further comprises applying the machine learning model to the biomedical image to determine a plurality of segments, each of the plurality of segments corresponding to a respective morphological classification of a plurality of morphological classifications for the tissue sample. The doctor (specifically a pathologist or radiologist) uses their vision to determine a plurality of segments in a medical image, with each segment corresponding to a specific morphological classification of the tissue sample; this exact process is known as manual visual semantic segmentation; therefore, this process can all be done mentally.
Claims 9 and 19 recite wherein the machine learning model is established using a training dataset comprising a plurality of examples, each of the plurality of examples identifying (i) a respective second biomedical image of a second tissue sample having (a) a third ROI corresponding to viable tumor in the second tissue sample and (b) a fourth ROI corresponding to necrotic tumor in the second tissue and (ii) an annotation identifying the third ROI and the fourth ROI in the respective second biomedical image. The doctor trains and learns medicine using old imaging datasets to learn how to identify viable and necrotic tumors in medical school; a machine learning model is a generic computer automating the process a doctor already does using their own human vision; therefore, this process can all be done mentally.
Therefore, dependent claims 2-7, 9, 12-17, and 19 recite the same abstract idea of a mental process which can be performed in the mind with the aid of pen and paper, and are therefore also rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No.: 2024/0242835 (Giltnane et al.) (hereinafter Giltnane), in view of non-patent literature "Deep interactive learning: an efficient labeling approach for deep learning-based osteosarcoma treatment response assessment"; International Conference on Medical Image Computing and Computer-Assisted Intervention; Cham: Springer International Publishing, 2020 (Ho et al.) (hereinafter Ho).
Regarding claim 1, Giltnane teaches a method of determining predicted outcomes of subjects with cancer from biomedical images, comprising: (Giltnane, abstract: “In one embodiment, a digital pathology image processing system receives digital pathology images of histologic samples. Physical characteristics of a first histologic sample associated with a first digital pathology image are assessed. The first digital pathology image is segmented based on one or more regions of the first digital pathology image corresponding to tumor bed. The first digital pathology image is also segmented based on one or more regions corresponding to one or more predetermined histologic features. An assessment is generated regarding a specified condition in the first histologic sample based on the one or more regions corresponding to tumor bed and the one or more regions corresponding to the one or more predetermined histologic features. The condition may be associated with a level or degree of pathologic response. A user interface may display information related to the assessment.”)
identifying, by a computing system, a biomedical image of a tissue sample from a subject with cancer, the biomedical image having (i) a first region of interest (ROI) corresponding to viable tumor in the tissue sample and (ii) a second ROI corresponding to necrotic tumor in the tissue sample (Giltnane, para. [0052]; para. [0029]: “As an example, the digital pathology image processing system 110 may process images of tissue samples or tiles of the whole slide images of tissue samples generated by the digital pathology image processing system 110 to identify and process segments of the digital pathology images that correspond to tumor bed and/or that correspond to particular histologic features or evidence of the particular histologic features. As an example, the digital pathology image processing system 100 may identify histologic features in the digital pathology image that correspond to viable tumor cells, regions of viable tumor, necrotic tumor cells, regions of necrosis, tumor stroma cells, regions of tumor stroma, or other specified histologic features in the corresponding tissue sample.”; “The present embodiments may be used to assess MPR or pCR for certain types of cancers in the clinical setting and in real-world settings that can be used to enforce and further facilitate the goals of consistency of evaluations”; see steps 205 and 210 in FIG. 2 below;
PNG
media_image1.png
617
819
media_image1.png
Greyscale
)
applying, by the computing system, a machine learning model to the biomedical image to determine (i) a first segment identifying the first ROI in the biomedical image and (ii) a second segment identifying the second ROI in the biomedical image (Giltnane, para. [0070]-[0073]; FIG. 2; FIG. 3: “At step 215, the digital pathology image processing system 110 may segment the digital pathology image based on the area of the sample shown in the digital pathology image corresponding to the tumor bed. The segmentation may be performed, for example, by the image segmentation module 111. The image segmentation module 111 may use, for example, a first machine-learning model trained to characterize or predict regions of a digital pathology image as corresponding to the tumor bed of a particular sample. The first machine-learning model may be trained to recognize variations in color, structures within the image, and other signs of the tumor bed and classify regions of the image accordingly … At step 220, the digital pathology image processing system 110 may segment the digital pathology image based on the area of the sample shown in the digital pathology image corresponding to one or more predetermined types of histologic features (e.g., viable tumor cells, necrosis, and stroma). The segmentation may be performed, for example, by the image segmentation module 111. The image segmentation module 111 may use, for example, a second machine-learning model trained to characterize or predict regions of a digital pathology image as corresponding to each of the predetermined histologic features in a tumor bed. The second machine-learning model may be trained to characterize regions of the digital pathology image corresponding to predetermined histologic features associated with a specific type of assessment … As an example, when assessing pathologic response by cancerous lung tissue to certain treatments, the second machine-learning model may be configured to identify regions of the digital pathology image corresponding to viable tumor cells, tumor stroma, and necrosis … In particular embodiments, steps 215 and 220 may be collapsed into a single step by utilizing a single machine-learning model trained to characterize regions of the digital pathology image corresponding to tumor bed as well as the predetermined histologic features … At step 225, based on the segmented digital pathology image, the digital pathology image processing system 110 may compute the area of a sample shown in the digital pathology image corresponding to the tumor bed. The evaluation may be performed by a segmentation evaluation module 112. As described herein, the segmentation evaluation module 112 may determine the number of pixels or size of the region of the digital pathology image that may been segmented as corresponding to the tumor bed. The segmentation evaluation module 112 may determine an associated area of the sample based on the size of the digital pathology image … At step 230, based on the segmented digital pathology image, the digital pathology image processing system 110 may compute the area of a sample shown in the digital pathology image corresponding to the predetermined histologic features. The evaluation may also be performed by the segmentation evaluation module 112. As described herein, the segmentation evaluation module 112 may determine the number of pixels or size of the region of the digital pathology image that may been segmented as corresponding to each of the predetermined histologic features. Based on the size of the regions of the digital pathology image, the physical area of each of the regions may be determined.”; see steps 215, 220, 225, and 230 in FIG. 2 above;
PNG
media_image2.png
838
566
media_image2.png
Greyscale
determining, by the computing system, a ratio between a first size of the first segment associated with the viable tumor and a second size of the second segment (Giltnane, para. [0057]; FIG. 9; para. [0097]: “A segmentation evaluation module 112 of the digital pathology image processing system 100 may analyze the segmented digital pathology image after processing by the image segmentation module 111. The segmentation evaluation module 112 may analyze the segmented images to determine, for example, the relative area of each identified segment to the area of other identified segments, the image, and/or the sample as a whole. As an example, the segmentation evaluation module 112 may calculate the area of the segmented image produced by the first machine-learning module that is determined to relate to the tumor bed. To do so, the segmentation evaluation module 112 may determine the number of pixels in the digital pathology image that correspond to the region of the segmented image. The segmentation evaluation module 112 may further determine the remaining number of pixels in the digital pathology image that correspond to the sample. The segmentation evaluation module 112 may then take the ratio of the two numbers of pixels to determine a percentage of pixels in the digital pathology image that corresponds to the tumor bed. Through similar process, the segmentation evaluation module 112 may determine the number of pixels in the segmented digital pathology image that correspond to, e.g., viable tumor cells, necrosis, and tumor stroma. The segmentation evaluation module 112 may compare the number of pixels corresponding to each type of histologic feature to the number of pixels in the digital pathology image overall. As discussed herein, another useful denominator may be the number of pixels corresponding to just the tumor bed (e.g., not including other histologic features that are not directly relevant to the evaluation of the tumor). The result of the analysis performed by the segmentation evaluation module 112 may be a series of ratios of the relative area of the relevant image segments. For example, the output may include a percentage of the digital pathology image (or the sample or even the tumor bed in the sample) corresponding to viable tumor cells, a percentage corresponding to necrosis, and a percentage corresponding to tumor stroma.”; “The import of combinations or relative levels of different histologic features (e.g., the ratio of viable tumor to tumor stroma, or necrotic tumor to stroma) may be assessed by eliminating the bias and potential sampling error of subjective analysis by pathologists.”;
PNG
media_image3.png
615
781
media_image3.png
Greyscale
);
generating, by the computing system, a value indicative of a predicted outcome of the cancer in the subject using the ratio (Giltnane, para. [0079]; para. [0040]; para. [0097]; para. [0124]; para. [0080]; FIG. 11: “At step 270, the digital pathology image processing system 110 generates an assessment regarding a specified condition. In particular, a pathologic response assessment module 113 determines whether a specified condition is detected in the image, or in the case of a collection of digital pathology images being provided, is detected in the plurality of images. As an example, the determination may be based on whether one or more of the combined characteristics satisfy a certain threshold. The threshold may be set based on, for example, the amount or quality of the images, the physical characteristics of the samples depicted in the images, the type of tissue being assessed, or the type of condition being assessed. Various threshold may be used for the various characteristics. For example, an assessment may be rendered based on the relative percentage of the tumor bed comprises viable cells. If the percentage satisfies a first threshold (which may be based on the type of sample depicted in the digital pathology image), then a first type of assessment may be determined (e.g., MPR [major pathologic response] detected) if the percentage satisfies a second threshold, then a second type of assessment may be determined (e.g., pCR detected). Combinations of the characteristics may also be assessed together. As an example, if the percentage of the tumor bed includes a first type of histologic feature and the area of the sample bed satisfy certain thresholds or other requirements, a specified type of assessment may be determined.”; “For example, the tool may assist in determining whether weighted averages or non-weighted averages provide more evaluative feedback or whether the ratio of certain types of histologic features are indicative of certain patient outcomes.”; “For example, the biomarker thresholds for different types of cancers or treatment regimens may be quickly and easily evaluated when assessing MPR or pCR or other pathologic response. One could take data from clinical trials and explore a new cut-off based on digital reads for a specific disease and treatment setting as well as histology. This cut-off could be refined based on the addition of other biomarker data to develop a better surrogate for DFS, OS, RFS, EFS, or a particular type of pathologic response. The import of combinations or relative levels of different histologic features (e.g., the ratio of viable tumor to tumor stroma, or necrotic tumor to stroma) may be assessed by eliminating the bias and potential sampling error of subjective analysis by pathologists.”; “FIGS. 17A-17D illustrate four graphs showing differences between digitally assessed MPR versus manually assessed MPR with respect to DFS and OS. Disease-free survival (DFS) according to manually assessed MPR-yes showed a trend towards longer DFS vs. MPR-no that was not statistically significant (FIG. 17A).”; “At step 275, the digital pathology image processing system 110 prepares and provides an output corresponding to the determination of the assessment. The output generation module 114 may prepare a variety of types of outputs corresponding to the assessment. For example, the output may include a plain language statement of the assessment (e.g., “MPR Determined”) and/or may include a listing of relevant characteristics that led to the assessment (e.g., “MPR Determined; Viable Tumor 8%”). The plain language statement may be incorporated into a report or user interface providing additional details regarding the digital pathology image and/or sample. Additionally or alternatively, visualizations may be generated for the output, such as a visualization illustrating the various segmented portions of the digital pathology image that were used in making the assessment.”; see steps 270 and 275 in FIG. 2 above; a major pathologic response (MPR) is a strong predictor of improved long-term survival and disease-free outcomes in patients receiving treatment before surgery;
PNG
media_image4.png
604
753
media_image4.png
Greyscale
); and
storing, by the computing system, using one or more data structures, an association between the subject and the value indicative of the predicted outcome (Giltnane, para. [0102]: “As discussed herein, in addition to storing the results of individual samples and blocks, the data stored by the digital pathology image processing system 120 may be analyzed over time (and across clinical studies) to identify ongoing trends that may prove useful for providing clinical validation for the techniques discussed herein or to simply ensure that data is consistent. For example, by recording and collecting pathologist identification information for each sample assessment they make, trends may be identified in pathologist performance in assessing the true percentages or other values of interest.”).
Giltnane fails to explicitly teach
determining, by the computing system, a ratio between a first size of the first segment associated with the viable tumor and a second size of the second segment associated with the necrotic tumor.
Bo teaches
determining, by the computing system, a ratio between a first size of the first segment associated with the viable tumor and a second size of the second segment associated with the necrotic tumor (Bo, pages 5-6, Section 2.3 Treatment Response Assessment; FIG. 1; FIG. 3: “The final CNN model segments viable tumor and necrotic tumor on testing WSIs. Note necrotic tumor is a combination of necrosis with bone and necrosis without bone. The ratio of necrotic tumor to overall tumor in case-level estimated by a deep learning model, RDL, is defined as
PNG
media_image5.png
114
410
media_image5.png
Greyscale
(1)
where Pvt and Pnt are the number of pixels of viable tumor and necrotic tumor in a case, respectively.”;
PNG
media_image6.png
508
759
media_image6.png
Greyscale
;
PNG
media_image7.png
853
760
media_image7.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the step of determining, by the computing system, a ratio between a first size of the first segment associated with the viable tumor and a second size of the second segment, as taught by Giltnane, to include determining, by the computing system, a ratio between a first size of the first segment associated with the viable tumor and a second size of the second segment associated with the necrotic tumor, as taught by Bo.
The suggestion/motivation for doing so would have been that “the ratio of necrotic tumor to overall tumor post neoadjuvant chemotherapy is a well-known prognostic factor and correlates with patients’ survival; thus, for patients with localized disease who have undergone complete resection, if the ratio of tumor necrosis is greater than 90%, the 5-year survival is higher than 80%” (Bo, pages 1-2, para. 1, Introduction) and that “our experiments showed that the CNN model can successfully estimate the necrosis ratio known as a prognostic factor for patients’ survival for osteosarcoma in an objective and reproducible way” (Bo, page 9, Conclusion).
Therefore, it would have been obvious to combine Giltnane, with Bo, to obtain the invention as specified in claim 1.
Regarding claim 2, Giltnane, in view of Bo, teaches the method of claim 1, further comprising: classifying, by the computing system, the subject into a risk stratification category of a plurality of risk stratification categories based on a comparison of the value with a threshold, and maintaining, by the computing system, a measure of progression of the cancer in the subject using the risk stratification category of the subject over a plurality of time instances (Giltnane, para. [0033]: “Measurements or quantifications of the assessed regions may be compared. For example, the size of the region corresponding to each of the viable tumor, tumor stroma, and necrotic tumor is compared to the tumor bed. The resulting values, which may be used in the form of percentages are then compared to one or more predetermined threshold values. The predetermined threshold values may have been determined based on best practices or prior studies as being indicative of certain type”).s of response. For example, a tumor cell area percentage of less than 10% may be indicative of MPR. A tumor cell area percentage of less than 1% may be indicative of pCR. The resulting assessment is then output to a user. This pathologic response data may be used to identify a cut-off for digital response assessment that is prognostic or predictive with correlation to DFS, OS, RFS, or EFS, risk stratification, or patient selection, which could be specific to a histologic or molecular sub-type or a specific treatment regimen used.”).
Regarding claim 3, Giltnane, in view of Bo, teaches the method of claim 2, further comprising determining, by the computing system, the threshold to compare against, based on a plurality of values each indicative of predicted outcome determined for a respective subject of a plurality of subjects (Giltnane, para. [0033]; see rejection of claim 2 above discussing the thresholds found from best practices or prior studies with multiple subjects; para. [0036]: “In certain embodiments, MPR may be evaluated by comparing the computed average value of one or more of these histologic features to a predetermined threshold. The predetermined threshold may be based, for example, on the indication sought, the type of mass being evaluated, a category of the disease being evaluated, etc. For example, a block of resected sample of the tumor may be said to exhibit MPR where the average percentage of the block comprising viable tumor is below 10%. As this average is calculated across the number of slides, but necessarily relative to the surface area of each individual slide, this calculated number may be referred to as a non-weighted average.”).
Regarding claim 4, Giltnane, in view of Bo, teaches the method of claim 1, wherein generating the value further comprises generating the value indicating at least one of an overall survival or a progression-free survival of the subject, based on the ratio of the first size of the first segment associated with the viable tumor and the second size of the second segment associated with the necrotic tumor (Giltnane, para. [0079]; para. [0040]; para. [0097]; para. [0124]; para. [0080]; see rejection of claim 1 above discussing Overall survival (OS) rate, DFS (disease-free survival) in relation to the ratios calculated between the different values of stroma cells, viable tumor cells, and necrotic tumor cells in the tissue; Giltnane, para. [0003]: “Pathologic response, including pathologic complete response (pCR) and major pathologic response (MPR) is a histologic assessment providing an early measure of treatment efficacy MPR and pCR have been studied as a surrogate for disease-free survival (DFS), event-free survival (EFS), relapse-free survival (RFS) or overall survival (OS) and have been used as an efficacy endpoint in Phase II and Ill clinical trials studying neoadjuvant therapies in respectable non-small cell lung cancer (NSCLC) and breast cancer.”; para. [0023]; FIG. 17A: “FIGS. 17A-17D illustrate four graphs showing differences between digitally assessed MPR versus manually assessed MPR with respect to DFS and OS.”;
PNG
media_image8.png
617
746
media_image8.png
Greyscale
).
Regarding claim 5, Giltnane, in view of Bo, teaches the method of claim 1, wherein identifying the biomedical image further comprises receiving a corresponding plurality of biomedical images of a respective plurality of tissue samples obtained from an anatomical site for the cancer of the subject over a corresponding plurality of time instances, and wherein generating the value further comprises generating the value indicative of the predicted outcome of the cancer for the subject at a respective time instance of the plurality of time instances at which the tissue sample was obtained (Giltnane, para. [0102]-[0103]; para. [0114]: “As discussed herein, in addition to storing the results of individual samples and blocks, the data stored by the digital pathology image processing system 120 may be analyzed over time (and across clinical studies) to identify ongoing trends that may prove useful for providing clinical validation for the techniques discussed herein or to simply ensure that data is consistent. For example, by recording and collecting pathologist identification information for each sample assessment they make, trends may be identified in pathologist performance in assessing the true percentages or other values of interest … Because results are likely to be provide stronger evidence when the assessments are consistent, the digital pathology image processing system 120 provides the ability to track trends is assessments over time and to introduce repeatability to the analysis which was heretofore impracticable.”; “In addition, tracking information such as block identifying information and patient identifying information be used to track and compare clinical population results across a clinical study and potentially over time. For example, if a single patient submits multiple samples over time, tracking of dates and patient identifying information may be used to study effects of time and study the progress of a mass or tissue in an individual.”; “Additionally, the results reporting interface 1100 may include additional fields that may be customized for the particular study or operator. For example, the reporting interface 1100 may include fields to display the size of the sample area (e.g., tumor bed) at other points in time (e.g., pre-therapy, post-therapy), display an approximate percentage of the total mass examined, display the weighted and non-weighted percentages of other assessed or computed values (e.g., necrosis or stroma), etc.”).
Regarding claim 6, Giltnane, in view of Bo, teaches the method of claim 1, wherein identifying the biomedical image further comprises receiving, via an imaging acquirer, a plurality of biomedical images each corresponding to a whole slide image (WSI) of a respective tissue sample stained to differentiate the viable tumor and the necrotic tumor from a remainder of the tissue sample, (Giltnane, para. [0042]; para. [0051]; para. [0106]; FIG. 6A: “A digital pathology image generation system 120 may generate one or more whole slide images or other related digital pathology images, corresponding to a particular sample. For example, an image generated by digital pathology image generation system 120 may include a stained section of a biopsy sample. As another example, an image generated by digital pathology image generation system 120 may include a slide image (e.g., a blood film) of a liquid sample. As another example, an image generated by digital pathology image generation system 120 may include fluorescence microscopy such as a slide image depicting fluorescence in situ hybridization (FISH) after a fluorescent probe has been bound to a target DNA or RNA sequence.”; “As an example, the digital pathology image processing system 110 may process images of tissue samples or tiles of the whole slide images of tissue samples generated by the digital pathology image processing system 110 to identify and process segments of the digital pathology images that correspond to tumor bed and/or that correspond to particular histologic features or evidence of the particular histologic features. As an example, the digital pathology image processing system 100 may identify histologic features in the digital pathology image that correspond to viable tumor cells, regions of viable tumor, necrotic tumor cells, regions of necrosis, tumor stroma cells, regions of tumor stroma, or other specified histologic features in the corresponding tissue sample.; “FIGS. 6A-8B illustrate examples of digital pathology images and output annotations produced by the digital pathology image processing system 120. FIGS. 6A-6C show three examples of digital pathology images 600, 610, and 620. As shown in FIG. 6A, example original digital pathology image 600 is a scan of an H&E-stained image (shown in typical purple and pink colors). Other stains, used independently or in combination, may also be used based on the trained models and the assessment goals of the digital pathology image processing system 120. As shown in FIG. 6B, example annotated image 610 represents digital pathology image 600 after evaluation by the tumor bed model 320. Annotated image 610 includes areas marked by the tumor bed model 320 to indicate that they correspond to the tumor bed (areas 611 shown in red) the portions of image 610 without any annotation (area 612 shown in purple and pink colors) illustrate adjacent non-tumor tissue. As shown in FIG. 6C, example annotated image 620 represents digital pathology image 600 after evaluation by the tissue model 325. The annotated image 620 includes sections marked by the tissue model 325 to indicate whether they correspond to viable tumor cells (various small areas 621 shown in red), tumor stroma (areas 622 shown in orange), or necrosis (various areas 623 shown in dark gray). The portions of the image without any annotation (areas 624 shown in pink and purple) illustrate adjacent non-tumor tissue.”;
PNG
media_image9.png
601
888
media_image9.png
Greyscale
), and
wherein determining the ratio further comprises determining the ratio between (i) a respective first size of the first segment associated with the viable tumor and (ii) a respective second size of the second segment associated with the necrotic tumor determined from each of the plurality of biomedical images (Bo, pages 5-6, Section 2.3 Treatment Response Assessment; FIG. 1; FIG. 3; see rejection of claim 1 above showing numerous WSI images that are segmented to viable tumor cells and necrotic tumor cells and a ratio between both is found).
Regarding claim 7, Giltnane, in view of Bo, teaches the method of claim 1, wherein applying the machine learning model further comprises applying the machine learning model to the biomedical image to determine a plurality of segments, each of the plurality of segments corresponding to a respective morphological classification of a plurality of morphological classifications for the tissue sample (Giltnane, para. [0054]; para. [0070]; para. [0132]: “The image segmentation module 111 may comprise or use one or more trained machine-learning models to perform the image segmentation. As an example, the image segmentation module 111 may use a first machine-learning model to segment the image based on the regions of the digital pathology image that are determined, by the machine-learning module, to correspond to the tumor bed in the sample. The image segmentation module 111 may further use a second machine-learning model to segment the image based on the regions of the digital pathology image that correspond to one of the one or more histologic features of interest. As an example, the second machine-learning model may be configured to identify regions of the digital pathology image that correspond to viable tumor cells, necrosis, or tumor stroma. The second machine-learning model may further label the identified regions accordingly such that the output produced by the second machine-learning model includes coordinates or other designations of the digital pathology image and labels for whether the coordinates are associated with viable tumor cells, necrosis, or tumor stroma. To perform the image segmentations, the first machine-learning model and the second machine-learning model may perform a variety of analyses on the image or a subdivision (e.g., tile) of the image, including, but not limited to, edge detection, image heuristic analysis, image classification and comparison, object detection and classification, semantic segmentation, and instance segmentation.”; “The first machine-learning model may be trained to recognize variations in color, structures within the image, and other signs of the tumor bed and classify regions of the image accordingly. In particular embodiments, the image segmentation module 111 may produce a new instance of the digital pathology image including annotations corresponding to the segmented regions (e.g., to prevent the original digital pathology image from being permanently altered or destroyed; “A first model (“artifact model”) was applied to identify tissue present on each slide, and predicted the presence of artifacts (e.g., tissue folding or blurring) within the WSI. Only regions deemed to contain usable tissue (i.e., not artifact or background) were used for subsequent modeling. One subsequent model (“tumor bed model”) was developed to classify tissue pixels from the artifact model as tumor bed or non-tumor bed. Another model (“tissue model”) was developed separately to classify tissue pixels from the artifact model as cancer epithelium, cancer-associated stroma, necrosis, or normal tissue. These three models were deployed to classify each pixel corresponding to tissue within each WSI as tumor bed or non-tumor bed, and one of cancer epithelium, stroma, necrosis, or normal tissue. The tumor-bed model predictions were then transformed to reconcile with the tissue model by reassigning every non-tumor bed pixel categorized as cancer epithelium, stroma, or necrosis as tumor bed.”).
Regarding claim 8, Giltnane, in view of Bo, teaches the method of claim 1, wherein the first size identifies a first number of pixels of the first segment associated with the viable tumor, and wherein the second size identifies a second number of pixels of the second segment associated with the necrotic tumor (Giltnane, para. [0057]; para. [0072]: “As an example, the segmentation evaluation module 112 may calculate the area of the segmented image produced by the first machine-learning module that is determined to relate to the tumor bed. To do so, the segmentation evaluation module 112 may determine the number of pixels in the digital pathology image that correspond to the region of the segmented image. The segmentation evaluation module 112 may further determine the remaining number of pixels in the digital pathology image that correspond to the sample. The segmentation evaluation module 112 may then take the ratio of the two numbers of pixels to determine a percentage of pixels in the digital pathology image that corresponds to the tumor bed. Through similar process, the segmentation evaluation module 112 may determine the number of pixels in the segmented digital pathology image that correspond to, e.g., viable tumor cells, necrosis, and tumor stroma. The segmentation evaluation module 112 may compare the number of pixels corresponding to each type of histologic feature to the number of pixels in the digital pathology image overall. As discussed herein, another useful denominator may be the number of pixels corresponding to just the tumor bed (e.g., not including other histologic features that are not directly relevant to the evaluation of the tumor). The result of the analysis performed by the segmentation evaluation module 112 may be a series of ratios of the relative area of the relevant image segments. For example, the output may include a percentage of the digital pathology image (or the sample or even the tumor bed in the sample) corresponding to viable tumor cells, a percentage corresponding to necrosis, and a percentage corresponding to tumor stroma.”; “The evaluation may be performed by a segmentation evaluation module 112. As described herein, the segmentation evaluation module 112 may determine the number of pixels or size of the region of the digital pathology image that may been segmented as corresponding to the tumor bed. The segmentation evaluation module 112 may determine an associated area of the sample based on the size of the digital pathology image. As an example, the segmentation evaluation module 112 may use the physical characteristics of the sample and/or use metadata associated with the digital pathology image to compute the area of the sample.”).
Regarding claim 9, Giltnane, in view of Bo, teaches the method of claim 1, wherein the machine learning model is established using a training dataset comprising a plurality of examples, each of the plurality of examples identifying (i) a respective second biomedical image of a second tissue sample having (a) a third ROI corresponding to viable tumor in the second tissue sample and (b) a fourth ROI corresponding to necrotic tumor in the second tissue and (ii) an annotation identifying the third ROI and the fourth ROI in the respective second biomedical image (Giltnane, para. [0063]; para. [0038]; para. [0085]; FIG. 4: “As embodied herein, the training controller 115 may select, retrieve, and/or access training data that includes a set of digital pathology images. The training data may further include a corresponding set of labels and/or annotations for particular histologic features in shown in the digital pathology images. During training operations, e.g., for the first machine-learning model or the second machine-learning model, the training controller 115 may cause the image segmentation module 111 to segment a subset of the digital pathology images in the training data. The output for each of the digital pathology images may be compared to the annotations and/or labels for the training data. Based on the comparison, one or more scoring functions may be used to evaluate the levels of precision and accuracy of the machine-learning model under testing. The training process will be repeated many times and may be performed with one or more subsets or cuts of the training data. For example, during each training cycle, a randomly-sampled selection of the digital pathology images from the training data may be provided as input to image segmentation module 111.”; “A second model 325 may be trained to determine cell types- and/or regions-of-interest based on the condition being evaluated. As described herein, in the case of digital MPR assessment for lung cancers, the cell types- and/or regions-of-interest may include viable tumor cells, tumor stroma, and necrosis. The tissue model 325 may be trained to receive the same digital pathology image 310 and annotate or otherwise indicate which regions of the image include viable tumor cells, tumor stroma, and necrosis. The tissue model 325 may further determine the area of each type of cell within the tumor bed (e.g., the area of the tumor bed that comprises viable tumor cells, the area of the tumor bed that comprises tumor stroma, and the area of the tumor bed that comprises necrosis). As discussed, these cells contribute to the tumor bed, so the total area of each of the determined regions is expected to equal the area of the tumor bed. The relative area of each region is calculated as the area of the region divided by the total area of the tumor bed. The tissue model 325 for identifying different cell types- and/or regions-of-interest may also be trained to recognize predetermined histologic features and other visual features that are indicative of whether a given region of the digital pathology image correspond to, for example, viable tumor cells, tumor stroma, and necrosis.; “Training data may be obtained from existing clinical studies or assessments where initial pathologist work was reviewed and confirmed or corrected by a team of reviewing pathologists who agreed on the eventual assessment. As the machine-learning models are trained based on the annotations and assessments within the training data, the accuracy of the training data will directly influence the accuracy of the model.”;
PNG
media_image10.png
353
618
media_image10.png
Greyscale
).
Regarding claim 10, Giltnane, in view of Bo, teaches the method of claim 1, further comprising providing, by the computing system, information to define a treatment to administer to the cancer in the subject based on the association between the value and the subject, wherein the cancer includes one of bone cancer, lung cancer, breast cancer, or colon cancer (Giltnane, para. [0026]: “In neoadjuvant trials, the efficacy of the therapy cannot be determined until years later when EFS, DFS, RFS, and OS are available. These endpoints are therefore blunt tools with high potential for conflating factors and requiring considerable investment to track results over time. This may delay the time before definitive results can be determined as well as the overall value of the results. Neoadjuvant trials allow efficacy end points such as clinical and pathologic response to be determined in several months. Neoadjuvant treatments offer potential advantages, including the ability to treat micrometastatic disease and analyze the treatment-related effect on the primary tumor after or during either adjuvant or neoadjuvant therapy. Neoadjuvant treatment with surrogate measures of efficacy (e.g., surrogate endpoints) such as pathologic response or histologic treatment effect have the potential to accelerate curative therapies for this patient population. Determining response to neoadjuvant therapy may be useful not only as a surrogate endpoint, but also to differentiate the neoadjuvant treatment effect from the adjuvant treatment effect if patients receive both types of treatment during a clinical trial or in practice.”; “This reporting has led to the design of neoadjuvant therapy clinical trials for resectable lung cancer in which MPR is a primary or co-primary endpoint due to being a quantifiable measure for evaluating results by evaluating resected samples directly MPR or pCR assessment is currently performed manually by pathologists. In addition to being labor intensive and not scalable, the assessments often lack consistency between individual pathologists and among assessments by the same pathologist over time. In certain types of cancers in the clinical setting and elsewhere, that can be used to enforce and further facilitate the goals of consistency of evaluations and improve the overall collection of pathologic data. Moreover, there is a lack of automated tools to replace the subjective and labor intensive practices required to assess MPR or pCR at a scalable level.”; assessing MPR or pCR in patients using the machine learning assisted method for lung cancer tumors leads to using treatments such as neoadjuvant therapy more efficiently for doctors than doing manual evaluation).
Regarding claim 11, Fuchs teaches a system for determining predicted outcomes of subjects with cancer from biomedical images, comprising: a computing system having one or more processors coupled with memory, configured to: (Giltnane, para. [0138]: “In particular embodiments, computer system 1800 includes a processor 1802, memory 1804, storage 1806, an input/output (I/O) interface 1808, a communication interface 1810, and a bus 1812. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.”;
PNG
media_image11.png
471
482
media_image11.png
Greyscale
).
With regards to the remaining limitations of independent claims 11 and dependent claims 12-20, they recite the functions of the process of independent claim 1 and dependent claims 2-10, respectively, as apparatuses. Thus, the analyses in rejecting claims 1-10 are equally applicable to the remaining limitations of independent claim 11 and dependent claims 12-20 respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: U.S. Patent Application Publication No.: 2020/0272864 (Faust et al.) that teaches using machine learning for segmenting tumors into different classes including viable tumor and necrosis, displaying percentages of viable tumor and necrosis in the image as well as finding risk stratification categories, and predicting clinical outcomes.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL ADAM SHARIFF whose telephone number is 571-272-9741. The examiner can normally be reached M-F 8:30-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached on 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ADAM SHARIFF/
Examiner, Art Unit 2672
Prosecution Insights