DETAILED ACTION
Contents
Notice of Pre-AIA or AIA Status 2
Election/Restrictions 2
Claim Rejections - 35 USC § 101 2
Claim Rejections - 35 USC § 103 3
Conclusion 16
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group 1, claims 1-11, 17-24 in the reply filed on 3/27/26 is acknowledged.
Claims 1-24 are currently pending. Claims 12-16 are withdrawn from consideration.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-11, 17-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter as follows. In regards to claims 1 and 17, the claims recite an abstract idea that can be a mental process. The claims do not integrate the abstract idea into a practical application nor do the claims recite an inventive concept or provide significantly more within the limitations. Claims 2-11 do not integrate the exception into a practical application as well. Claims 18-24 also do not integrate the exception into a practical idea and are similar in scope to dependent claims 2-11.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimedinvention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 9, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (IEEE: “WSISA: Making Survival Prediction from Whole Slide Histopathological Images”) in view of Georgescu et al (US 2022/0076411 A1).
Regarding claim 1, Zhu teaches a method comprising: receiving a microscopy image associated with a test sample (see abstract, method; an effective Whole Slide Histopathological Images Survival Analysis framework); randomly selecting a set of sub-images from within the region of interest (see 3.1.1; The goal in this stage is to generate candidate patches from WSIs. Unlike extracting patches only from annotated ROIs, we argue that the heterogeneous patterns and their proportions in each WSI also count. We assume that those candidate patches randomly sampled from patients WSIs can catch the main patterns and their proportions. And candidate patches from different WSIs of one patient together reflect the patient’s survival risk. We set a fixed area sampling ratio to sample the candidate patches (patch size of 512 by 512, 0.5 microns per pixel), making sure a fixed proportion of pixels are sampled from each WSI. This step and the assumptions are the basis of following steps); generating a set of outcome predictions, each outcome prediction associated with a corresponding sub-image of the set of sub-images by providing the sub-image to a trained deep neural network (see 3.1.3; As pointed above, different candidate clusters contain distinguished pattern patches. Those patches may have various prediction power on patients’ survival. To distinguish their predicting power and select the candidate clusters, we train separate deep convolutional survival model (DeepCon vSurv) on each cluster. The DeepConvSurv models are trained patch-wise. That is, we assign each patch with the survival label from that patient. Then we train the model with all the patches in the cluster. We select the clusters with predicting accuracy a little bit better than random guess as the base learners in the aggregation step. The architecture of DeepConvSurvis showninTable1. The difference between DeepConvSurv and traditional deep models in classification or regression relies at the loss function. The loss function of DeepConvSurv is replaced with Cox model’s loss function (2). The corresponding models of the selected clusters become features generator in the aggregation stage.); aggregating the outcome predictions of the set of outcome predictions to generate an aggregate outcome prediction (see 3.1.4; After generating patient-wise weighted features, the last step is aggregating those features to make a final survival prediction. As pointed above, separate patches lack ability of representing patients’ holistic information. It is needed to integrate them to better predict patients’ survival. In this problem, from extensive experiments on three different cancer datasets, the simple Cox model with Lasso [23] can predict survivals very well based on the weighted features. The reasons are: 1) because the sample size is relatively small, simple model will not easily be overfitted; 2) if the features are highly related to the survival labels, simple model will work well. In WSISA, the prediction model can also be easily changed to other state of-the-arts models, such as random survival forests [9]. The algorithm for WSISA is shown in algorithm 1. It shows the general procedure of WSISA and will not include the details like splitting the training, validation and testing sets); and providing the aggregate outcome prediction associated with the microscopy image (see 3.1.4; Aggregation is the key stage of the framework that outputs patient-wise survival prediction values. This stage can be further divided into two main sub-steps: Generate weighted features and Aggregation. Generating Weighted Features This substep distinguishes our framework with traditional patch-based survival prediction methods to a large extent. We take the pro portions of various heterogeneous patterns into consideration. And we solve the problem of various numbers and sizes of WSIs among different patients by unifying the contributions from those WSIs. Here, we show how various patches’ contributions from one patient are unified. From the first stage, patches are extracted based on a fixed area sampling ratio. When keep the patch size fixed, the number of patches extracted are proportional to the WSI’s size for one WSI. The patient’s total number of patches is the sum of all the patches extracted from his/her WSIs. To estimate the weight of separate pattern, we need to count the number of patches in each cluster. Suppose there’re total ni patches extracted from patient i. For each selected cluster, the patient has nij patches in cluster j. Then the contribution of cluster j to the patients survival prediction can be calculated as: wij = nij ni , i∈{1,...,N},j∈{1,...,J}, (3) where N is the number of patients, J is the number of selected clusters and wij is the weight for cluster j in patient i. Since each patient may have different numbers of patches in a selected cluster, the features for that patient in the selected cluster are calculated as k=K xij = wij k=1 xijk/K, (4) where xij is the output features in cluster j for patient i. It can be either the prediction risks from each cluster or the output of FC layer in the DeepConvSurv. K is the number of patches for patient i in cluster j. By randomly sampling and setting a large enough sampling ratio, the weights for survival correlated patches can be estimated well for a patient.). Zhu does not teach expressly identifying a region of interest of the microscopy image for analysis.
Georgescu, in the same field of endeavor, teaches identifying a region of interest of the microscopy image for analysis (see abstract; A CNN is applied to a histological image to identify areas of interest. The CNN classifies pixels according to relevance classes including one or more classes indicating levels of interest and at least one class indicating lack of interest. The CNN is trained on a training data set including data which has recorded how pathologists have interacted with visualizations of histological images. In the trained CNN, the interest-based pixel classification is used to generate a segmentation mask that defines areas of interest).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu to utilize the cited limitations as suggested by Georgescu. The suggestion/motivation for doing so would have been to assist a user by better allocating for further examination (see 0025). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu, while the teaching of Georgescu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 9, Zhu teaches a convolutional neural network (see section 3.1.3).
Regarding claim 17, Zhu teaches a system comprising: receiving a microscopy image associated with a test sample (see abstract, method; an effective Whole Slide Histopathological Images Survival Analysis framework); randomly selecting a set of sub-images from within the region of interest (see 3.1.1; The goal in this stage is to generate candidate patches from WSIs. Unlike extracting patches only from annotated ROIs, we argue that the heterogeneous patterns and their proportions in each WSI also count. We assume that those candidate patches randomly sampled from patients WSIs can catch the main patterns and their proportions. And candidate patches from different WSIs of one patient together reflect the patient’s survival risk. We set a fixed area sampling ratio to sample the candidate patches (patch size of 512 by 512, 0.5 microns per pixel), making sure a fixed proportion of pixels are sampled from each WSI. This step and the assumptions are the basis of following steps); generating a set of outcome predictions, each outcome prediction associated with a corresponding sub-image of the set of sub-images by providing the sub-image to a trained deep neural network (see 3.1.3; As pointed above, different candidate clusters contain distinguished pattern patches. Those patches may have various prediction power on patients’ survival. To distinguish their predicting power and select the candidate clusters, we train separate deep convolutional survival model (DeepCon vSurv) on each cluster. The DeepConvSurv models are trained patch-wise. That is, we assign each patch with the survival label from that patient. Then we train the model with all the patches in the cluster. We select the clusters with predicting accuracy a little bit better than random guess as the base learners in the aggregation step. The architecture of DeepConvSurvis showninTable1. The difference between DeepConvSurv and traditional deep models in classification or regression relies at the loss function. The loss function of DeepConvSurv is replaced with Cox model’s loss function (2). The corresponding models of the selected clusters become features generator in the aggregation stage.); aggregating the outcome predictions of the set of outcome predictions to generate an aggregate outcome prediction (see 3.1.4; After generating patient-wise weighted features, the last step is aggregating those features to make a final survival prediction. As pointed above, separate patches lack ability of representing patients’ holistic information. It is needed to integrate them to better predict patients’ survival. In this problem, from extensive experiments on three different cancer datasets, the simple Cox model with Lasso [23] can predict survivals very well based on the weighted features. The reasons are: 1) because the sample size is relatively small, simple model will not easily be overfitted; 2) if the features are highly related to the survival labels, simple model will work well. In WSISA, the prediction model can also be easily changed to other state of-the-arts models, such as random survival forests [9]. The algorithm for WSISA is shown in algorithm 1. It shows the general procedure of WSISA and will not include the details like splitting the training, validation and testing sets); and providing the aggregate outcome prediction associated with the microscopy image (see 3.1.4; Aggregation is the key stage of the framework that outputs patient-wise survival prediction values. This stage can be further divided into two main sub-steps: Generate weighted features and Aggregation. Generating Weighted Features This substep distinguishes our framework with traditional patch-based survival prediction methods to a large extent. We take the pro portions of various heterogeneous patterns into consideration. And we solve the problem of various numbers and sizes of WSIs among different patients by unifying the contributions from those WSIs. Here, we show how various patches’ contributions from one patient are unified. From the first stage, patches are extracted based on a fixed area sampling ratio. When keep the patch size fixed, the number of patches extracted are proportional to the WSI’s size for one WSI. The patient’s total number of patches is the sum of all the patches extracted from his/her WSIs. To estimate the weight of separate pattern, we need to count the number of patches in each cluster. Suppose there’re total ni patches extracted from patient i. For each selected cluster, the patient has nij patches in cluster j. Then the contribution of cluster j to the patients survival prediction can be calculated as: wij = nij ni , i∈{1,...,N},j∈{1,...,J}, (3) where N is the number of patients, J is the number of selected clusters and wij is the weight for cluster j in patient i. Since each patient may have different numbers of patches in a selected cluster, the features for that patient in the selected cluster are calculated as k=K xij = wij k=1 xijk/K, (4) where xij is the output features in cluster j for patient i. It can be either the prediction risks from each cluster or the output of FC layer in the DeepConvSurv. K is the number of patches for patient i in cluster j. By randomly sampling and setting a large enough sampling ratio, the weights for survival correlated patches can be estimated well for a patient.). Zhu does not teach one or more processors; and one or more processor-readable media storing instructions which, when executed by one or more processors, cause performance of: identifying a region of interest of the microscopy image for analysis.
Georgescu, in the same field of endeavor, teaches one or more processors (see 0238; processor); and one or more processor-readable media storing instructions which, when executed by one or more processors (see 0282; Computing apparatus 500 preferably includes a main memory 515 and may also include a secondary memory 520. Main memory 515 provides storage of instructions and data for programs executing on processor 510, such as one or more of the functions and/or modules discussed above. It should be understood that computer readable program instructions stored in the memory and executed by processor 510 may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in and/or compiled from any combination of one or more programming languages, including without limitation Smalltalk, C/C++, Java, JavaScript, Perl, Visual Basic, .NET, and the like. Main memory 515 is typically semiconductor-based memory such as dynamic random access memory (DRAM) and/or static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), and the like, including read only memory (ROM)), cause performance of: identifying a region of interest of the microscopy image for analysis (see abstract; A CNN is applied to a histological image to identify areas of interest. The CNN classifies pixels according to relevance classes including one or more classes indicating levels of interest and at least one class indicating lack of interest. The CNN is trained on a training data set including data which has recorded how pathologists have interacted with visualizations of histological images. In the trained CNN, the interest-based pixel classification is used to generate a segmentation mask that defines areas of interest).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu to utilize the cited limitations as suggested by Georgescu. The suggestion/motivation for doing so would have been to assist a user by better allocating for further examination (see 0025). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu, while the teaching of Georgescu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claims 2-3, 18, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (IEEE: “WSISA: Making Survival Prediction from Whole Slide Histopathological Images”) with Georgescu et al (US 2022/0076411 A1), and further in view of Couetil et al (“Predicting melanoma survival and metastasis with interpretable histopathological features and machine learning models”).
Regarding claims 2-3, Zhu with Georgescu teaches all elements as mentioned above in claim 1. Zhu with Georgescu does not teach expressly outcome prediction of the set of outcome predictions corresponds to a prediction of disease progression within a future time period; metastasis of a tumor to a body region different from a body region associated with the test sample.
Couetil, in the same field of endeavor, teaches outcome prediction of the set of outcome predictions corresponds to a prediction of disease progression within a future time period (see introduction); metastasis of a tumor to a body region different from a body region associated with the test sample (see introduction).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu with Georgescu to utilize the cited limitations as suggested by Couetil. The suggestion/motivation for doing so would have been to identify rapid, accurate and cost-effective predictors (see pg. 1). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu with Georgescu, while the teaching of Couetil continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claims 18-19, the claim is analyzed as a system that implements the limitations of claims 2-3 (see rejection of claims 2-3).
Claims 4-6, 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (IEEE: “WSISA: Making Survival Prediction from Whole Slide Histopathological Images”) with Georgescu et al (US 2022/0076411 A1), and further in view of Wu et al (WO 2019/118544 A1).
Regarding claims 4-6, Zhu with Georgescu teaches all elements as mentioned above in claim 1. Zhu with Georgescu does not teach expressly a microscopy image that has not been physically stained; a virtually stained microscopy image; generated by a trained machine learning model different from the trained deep neural network.
Wu, in the same field of endeavor, teaches a microscopy image that has not been physically stained (see 0005); a virtually stained microscopy image (see 0005); generated by a trained machine learning model different from the trained deep neural network (see 0005).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu with Georgescu to utilize the cited limitations as suggested by Wu. The suggestion/motivation for doing so would have been to not damage the sample and provide real time evaluation (see 0039). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu with Georgescu, while the teaching of Wu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claims 20-22, the claim is analyzed as a system that implements the limitations of claims 4-6 (see rejection of claims 4-6).
Claims 7, 23 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (IEEE: “WSISA: Making Survival Prediction from Whole Slide Histopathological Images”) with Georgescu et al (US 2022/0076411 A1), and further in view of Li et al (FIW: “Fast Regions-of-Interest Detection in Whole Slide Histopathology Images”).
Regarding claim 7, Zhu with Georgescu teaches all elements as mentioned above in claim 1. Zhu with Georgescu does not teach expressly filtering a background region based on an annotation of the region of interest.
Li, in the same field of endeavor, teaches filtering a background region based on an annotation of the region of interest (see abstract and introduction).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu with Georgescu to utilize the cited limitations as suggested by Li. The suggestion/motivation for doing so would have been to reduce the complexity by faster localizing (see abstract). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu with Georgescu, while the teaching of Li continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 23, the claim is analyzed as a system that implements the limitations of claim 7 (see rejection of claim 7).
Claims 8, 24 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (IEEE: “WSISA: Making Survival Prediction from Whole Slide Histopathological Images”) with Georgescu et al (US 2022/0076411 A1), and further in view of Sadhwani et al (SR: “Comparative analysis of machine learning approaches to classify tumor mutation burden in lung adenocarcinoma using histopathology images”).
Regarding claim 8, Zhu with Georgescu teaches all elements as mentioned above in claim 1. Zhu with Georgescu does not teach expressly selected with uniform probability from within the region of interest.
Sadhwani, in the same field of endeavor, teaches selected with uniform probability from within the region of interest (see pg. 4-5).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu with Georgescu to utilize the cited limitations as suggested by Sadhwani. The suggestion/motivation for doing so would have been to enable a significant improvement in prediction (see abstract). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu with Georgescu, while the teaching of Sadhwani continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 24, the claim is analyzed as a system that implements the limitations of claim 8 (see rejection of claim 8).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (IEEE: “WSISA: Making Survival Prediction from Whole Slide Histopathological Images”) with Georgescu et al (US 2022/0076411 A1), and further in view of Duanmu et al (BI: “Aspatial attention guided deep learning system for prediction of pathological complete response using breast cancer histopathology images”).
Regarding claim 10, Zhu with Georgescu teaches all elements as mentioned above in claim 1. Zhu with Georgescu does not teach expressly an attention mechanism configured to utilize a region of interest within a sub-image to generate a corresponding outcome prediction.
Duanmu, in the same field of endeavor, teaches an attention mechanism configured to utilize a region of interest within a sub-image to generate a corresponding outcome prediction (see section 4.4).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu with Georgescu to utilize the cited limitations as suggested by Duanmu. The suggestion/motivation for doing so would have been to enable high accuracy for prediction (see conclusion). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu with Georgescu, while the teaching of Duanmu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (IEEE: “WSISA: Making Survival Prediction from Whole Slide Histopathological Images”) with Georgescu et al (US 2022/0076411 A1), and further in view of Meier et al (JP: “Hypothesis-free deep survival learning applied to the tumour microenvironment in gastric cancer”).
Regarding claim 11, Zhu with Georgescu teaches all elements as mentioned above in claim 1. Zhu with Georgescu does not teach expressly a median outcome prediction of the set of outcome predictions.
Meier, in the same field of endeavor, teaches an a median outcome prediction of the set of outcome predictions (see pg. 275).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Zhu with Georgescu to utilize the cited limitations as suggested by Meier. The suggestion/motivation for doing so would have been to enable new ways to predict risk (see abstract). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Zhu with Georgescu, while the teaching of Meier continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EDWARD PARK. The examiner’s contact information is as follows:
Telephone: (571)270-1576 | Fax: 571.270.2576 | Edward.Park@uspto.gov
For email communications, please notate MPEP 502.03, which outlines procedures pertaining to communications via the internet and authorization. A sample authorization form is cited within MPEP 502.03, section II.
The examiner can normally be reached on M-F 9-6 CST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer, can be reached on (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EDWARD PARK/
Primary Examiner, Art Unit 2666
Prosecution Insights