DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Application
Claims 1-10 are currently pending in this application.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim defines “computer-readable recoding medium” which is related to software per se. However, the claim does not define a non-transitory computer-readable recording medium and is thus non-statutory for that reason (i.e., "When functional descriptive material is recorded on some non- transitory computer-readable medium it becomes structurally and functionally interrelated to the non-transitory medium and will be statutory in most cases since use of technology permits the function of the descriptive material to be realized"- Guidelines Annex IV).
Allowable Subject Matter
Claims 2 and 6 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-5, 9, and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over AGHAEI et al. (Hereafter, “Aghaei”) [US 2025/0385005 A1].
In regards to claim 1, Aghaei discloses a tissue imaging processing method ([Abstract] A method for using a federated learning classifier in digital pathology includes distributing, by a centralized server, a global model to a plurality of client devices.), comprising: (a) obtaining a target image ([0072] The model architecture may comprise a pre-processing stage 305 comprising an image acquisition module 310 to generate or obtain input images including simplex image data (e.g., images where each has a single stain) and/or multiplex image data (e.g., an image having a plurality of stains).); (b) obtaining a dataset, enhancing the staining components of the dataset, and using an enhanced dataset for federated learning to construct a tumor segmentation model ([0010] Some embodiments of the present disclosure include a computer-implement method for using a federated learning classifier. The method includes distributing, by a centralized server, a global model configured to classify pathology images to a plurality of client devices; receiving, by the centralized server an updated model from at least one of the plurality of client devices, wherein the updated model has been further trained at the at least one of the plurality of client devices using a plurality of slide images and a plurality of corresponding annotations; aggregating, by the centralized server, the updated model with the global model to generate an updated global model; and distributing the updated global model to at least one of the plurality of client devices. [0059] The computing environment 200 can also include a second prediction model (e.g., a 2DCNN) for segmenting target regions (e.g., regions of tumor cells).), wherein the dataset includes a test image ([0060] In various embodiments, each prediction model 215a-n corresponding to the classifier subsystems 210a-n is separately additionally trained based on one or more sets of input image elements 220a-n. In some embodiments, each of the input image elements 220a-n include image data from one or more scanned slides. Each of the input image elements 220a-n may correspond to image data from a single specimen and/or a single day on which the underlying image data corresponding to the image was collected. [0064] Individually or an ensemble of trained prediction models can be deployed to process unlabeled input image elements 220h-n to segment non-target or target regions, provide image analysis, provide a diagnosis of disease for treatment or a prognosis for a subject such as a patient, or a combination thereof.), and the test image is an immunohistochemical image ([0076] In some embodiments, one of the images has been stained with at least one primary stain (hematoxylin or eosin (H&E)), while another one of the images has been stained in at least one of an IHC assay or an in-situ hybridization (ISH) assay for the identification of a specific biomarker. In some embodiments, one of the images has been stained with both hematoxylin and eosin, while another one of the images has been stained in at least one of an IHC assay or ISH assay for the identification of a specific biomarker.); (c) inputting the target image into the tumor segmentation model to determine a tumor area within the target image ([0073] The model architecture may further comprise a segmentation and masking module 350 to segment regions or biological structures such as lymphocyte aggregates or clusters of tumor cells in the input images, and generate a mask based on the segmented regions or biological structures, and an optional registration module 355 to map the identified regions or biological structures (e.g. tumor cells or immune cells) from a first image or first set of images within the input images to at least one additional image or a plurality of additional images.); (d) performing cell membrane staining detection on the tumor area, and further classifying the tumor area based on an integrity of the cell membrane and a staining level of the cell membrane ([0074] In some embodiments, the image acquisition module 310 generates or obtains images or image data of a biological sample having one or more stains (e.g. the images may be simplex images or multiplex images). [0079] Following image acquisition and/or unmixing, input images or unmixed image channel images are processed with an image analysis algorithm provided by the image analysis module 330 to identify and classify cells and/or nuclei. The procedures and algorithms described herein may be adapted to identify and classify various types of cells or cell nuclei based on features within the input images, including identifying and classifying tumor cells, non-tumor cells, stroma cells, lymphocytes, non-target stain, etc. One of ordinary skill in the art should appreciate that the nucleus, cytoplasm and membrane of a cell have different characteristics and that differently stained tissue samples may reveal different biological features. Specifically, one of ordinary skill in the art should appreciate that certain cell surface receptors can have staining patterns localized to the membrane, or localized to the cytoplasm. Thus, a “membrane” staining pattern is analytically distinct from a “cytoplasmic” staining pattern. Likewise, a “cytoplasmic” staining pattern and a “nuclear” staining pattern are analytically distinct. Each of these distinct staining patterns may be used as features for identifying cells and/or nuclei. For example, stromal cells may be strongly stained by FAP, whereas tumor epithelial cells may be strongly stained by EpCAM, while cytokeratins may be stained by panCK. Thus, by utilizing different stains different cell types may be differentiated and distinguished during image analysis to provide a classification solution. [0087] In some embodiments, the expression score is an H-score is used to assess the percentage of tumor cells with cell membrane staining graded as ‘weak,’ ‘moderate’ or ‘strong.’ The grades are summated to give an overall maximum score of 300 and a cut-off point of 100 to distinguish between a ‘positive’ and ‘negative. For example, a membrane staining intensity (0, 1+, 2+, or 3+) is determined for each cell in a fixed field of view (or here, each cell in a tumor or cell cluster). The H-score may simply be based on a predominant staining intensity, or more complexly, can include the sum of individual H-scores for each intensity level seen. In other embodiments, the expression score is an Allred score. The Allred score is a scoring system which looks at the percentage of cells that test positive for hormone receptors, along with how well the receptors show up after staining (this is called “intensity”).); and (e) grading based on the classification result ([0086] In some embodiments, a variety of marker expression scores are calculated for each stain or biomarker within each cell cluster within each image (simplex images or unmixed image channel images from a multiplex image) using the scoring module 340. The scoring module 340, in some embodiments, utilizes data acquired during the detection and classification of cells by the image analysis module 330.).
It would have been obvious to one of ordinary skill in the art at the time of the invention to incorporate the different embodiments of Aghaei to include the additional features for the system/method [Official Notice] in order to improve pathology machine-learning models using distributed data without compromising sensitive patient data through the movement to a central location [See Aghaei].
In regards to claim 3, the limitations of claim 1 have been addressed. Aghaei discloses wherein the step of using the enhanced dataset for federated learning to construct the tumor segmentation model further comprises: preprocessing the enhanced dataset, and inputting the preprocessing result into a federated learning server to construct the tumor segmentation model ([0010] Some embodiments of the present disclosure include a computer-implement method for using a federated learning classifier. The method includes distributing, by a centralized server, a global model configured to classify pathology images to a plurality of client devices; receiving, by the centralized server an updated model from at least one of the plurality of client devices, wherein the updated model has been further trained at the at least one of the plurality of client devices using a plurality of slide images and a plurality of corresponding annotations; aggregating, by the centralized server, the updated model with the global model to generate an updated global model; and distributing the updated global model to at least one of the plurality of client devices.).
In regards to claim 4, the limitations of claim 3 have been addressed. Aghaei discloses wherein the step of preprocessing the enhanced dataset and inputting the preprocessing results into the federated learning server to construct the tumor segmentation model further comprises: constructing a local model and a global model based on the learning rules of the federated learning server ([0056] In some embodiments, a Federated Learning (FL) system for Digital Pathology (DP) may be utilized to generate and distribute a global model (e.g., an aggregated global model) without exchanging sensitive or identifying data (e.g., patient data) between clients and/or a centralized system (e.g., a server). A server is configured to maintain and distribute the global model in an iterative process as updated models are received from clients. FIG. 1 depicts an example a FL DP system 100 that includes one or more servers 110 configured to maintain and distribute one or more global models 112, 114. The server 110 is in communication with one or more client systems 120, 130, 140 that may each include various DP equipment such as a workstation 122, 132, 142, a microscope 124, 134, 144, a digital slide scanner 126, 136, 146, and any other necessary equipment as would be understood by those skilled in the art. Each of the client systems may utilize one or more local models 128, 138, 148, 150 that are based on the global models 112, 114.); training the local model with the enhanced dataset to obtain a plurality of parameters of the local model ([0056] The client systems 120, 130, 140 may be utilized to further train the local models 128, 138, 148, 150. For example, the client systems 120, 130, 140 may receive patient data, classify the patient data using the local models 128, 138, 148, 150, receive user input regarding the classified patient data (e.g., from a pathologist or other medical professional utilizing a graphical user interface displaying the classified data), and update the local model 128, 138, 148, 150 based on the user input (e.g., each client retrains the global model by using a local training dataset).); inputting the plurality of parameters of the local model into the federated learning server, training the global model through a clustering pipeline ([0056] In various embodiments, the client devices are configured to periodically provide their local models 128, 138, 148, 150 to the centralized server 110. The centralized server 110 may then utilize the local models 128, 138, 148, 150 to update the global model 112, 114 (e.g., by updating weights in the global model) and distribute the updated global model 112, 114 to the client systems 120, 130, 140.); and training the global model with the enhanced dataset, inputting a parameter of the trained global model into the federated learning server to construct the tumor segmentation model ([0057] In some embodiments, after each iteration, the performance each of the updated local models 128, 138, 148, 150 may be ascertained using a validation dataset. When a local model 128, 138, 148, 150 has been determined to provide improved performance on the validation dataset, the local model may be incorporated into the global model 112, 114. The performance of the updated global model 112, 114 may also be validated with a validation dataset. If the global model 112, 114 has been improved, the updated global model 112, 114 may be distributed to all or some of the client devices 120, 130, 140. In some embodiments, a client may elect to not share their updated local model 128, 138, 148, 150, but still receive the updated global model 112, 114. In other embodiments, a client may elect to share their local model 128, 138, 148, 150, but not receive any updated global models 112, 114. In other embodiments, a client may elect to not share their updated local model 128, 138, 148, 150 and not receive the updated global model 112, 114. Thus, models that are generated at the client site are not controlled by the centralized server 110 and are shared with the centralized server 110 based on the client's discretion. Each client may have an independent validation dataset and may use the validation dataset to examine the performance of the model based on their quality standards. Based on this validation, the client may determine whether to deploy the global model 112, 114 or not. [0059] FIG. 2 shows a block diagram illustrates a computing environment 200 for non-tumor segmentation and image analysis using deep convolutional neural networks according to various embodiments. The computing environment 200 may employ the same type of prediction model or different types of prediction models trained to segment non-target regions, segment target regions, or provide image analysis of target regions.).
In regards to claim 5, the limitations of claim 1 have been addressed. Aghaei discloses wherein Step (d) further comprises: (d1) performing nuclear detection on the tumor area to determine a position of a nucleus within the tumor area ([0080] In some embodiments, tumor nuclei are automatically identified by first identifying candidate nuclei and then automatically distinguishing between tumor nuclei and non-tumor nuclei. [0081] For example, in some embodiments the images obtained as input are processed such as to detect nucleus centers (seeds) and/or to segment the nuclei.); (d2) performing cell membrane detection based on the position of the nucleus to determine a position of a cell membrane corresponding to the nucleus [0080-0081]; (d3) classifying the cell membrane based on a staining level of the cell membrane to obtain a pre-classification result ([0080] Methods of identifying, classifying, and/or scoring nuclei, cell membranes, and cell cytoplasm in images of biological samples having one or more stains are described in U.S. Pat. No. 7,760,927 (“the '927 patent”), the contents of which are incorporated herein in their entirety for all purposes. For example, the '927 patent describes an automated method for simultaneously identifying a plurality of pixels in an input image of a biological tissue stained with a biomarker, including considering a first color plane of a plurality of pixels in a foreground of the input image for simultaneous identification of cell cytoplasm and cell membrane pixels, wherein the input image has been processed to remove background portions of the input image and to remove counterstained components of the input image; determining a threshold level between cell cytoplasm and cell membrane pixels in the foreground of the digital image; and determining simultaneously with a selected pixel and its eight neighbors from the foreground if the selected pixel is cell cytoplasm pixel, a cell membrane pixel or a transitional pixel in the digital image using the determined threshold level. ); and (d4) classifying the tumor area based on the pre-classification result and the integrity of the cell membrane ([0087] In some embodiments, the expression score is an H-score is used to assess the percentage of tumor cells with cell membrane staining graded as ‘weak,’ ‘moderate’ or ‘strong.’ The grades are summated to give an overall maximum score of 300 and a cut-off point of 100 to distinguish between a ‘positive’ and ‘negative.).
Claim 9 lists all the same elements of claim 1, but in system form rather than method form. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claim 9. Furthermore, regarding claim 9, Aghaei discloses a tissue imaging processing system, comprising: a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the tissue imaging processing method ([0007] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.) according to claim 1 [See claim 1 above].
Claim 10 lists all the same elements of claim 1, but in computer-readable recording medium form rather than method form. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claim 10. Furthermore, regarding claim 10, Aghaei discloses a computer-readable recording medium for processing tissue imaging, comprising computer programs/instructions, wherein the computer programs/instructions, when executed by a processor ([0108] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and/or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods and/or part or all of one or more processes disclosed herein.), implement the method according to claim 1 [See claim 1 above].
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aghaei in view of Lemdani et al. (Hereafter, “Lemdani”) [US 2019/0392612 A1].
In regards to claim 7, the limitations of claim 1 have been addressed. Aghaei fails to explicitly disclose wherein Step (d3) further comprises: performing color deconvolution on each pixel of the cell membrane to convert it to a representation in hematoxylin-diaminobenzidine; classifying each pixel of the cell membrane based on its staining intensity in the diaminobenzidine channel according to a preset threshold to obtain the classification level of each pixel; and obtaining the pre-classification result based on the classification level of each pixel.
Lemdani discloses wherein Step (d3) further comprises: performing color deconvolution on each pixel of the cell membrane to convert it to a representation in hematoxylin-diaminobenzidine; classifying each pixel of the cell membrane based on its staining intensity in the diaminobenzidine channel according to a preset threshold to obtain the classification level of each pixel; and obtaining the pre-classification result based on the classification level of each pixel ([0021] the histopathological image comprises, in addition to the histological stain, a non-specific stain, for example hematoxylin or eosin; [0022] the step of separating the histological stain comprises a deconvolution of the colors of the histopathological image over the hue/saturation/lightness space; [0023] the histological stain is diaminobenzidine or hematoxylin-aminoethylcarbazole; [0024] the step of detecting biological cells of interest comprises a K-means classification using a centroid of the class of biological cells of interest and a centroid of the fibrosis class, the centroid of the class of biological cells of interest can then, preferentially, be taken equal to the minimum gray level of the intermediate image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Aghaei with the teachings of Lemdani in order to provide a global, objective, rapid measure of infiltration across an entire biological object and a method that works with virtual slides and different histological stains [See Lemdani].
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Aghaei in view of Descombes et al. (Hereafter, “Descombes”) [US 2023/0084952 A1].
In regards to claim 8, the limitations of claim 5 have been addressed. Aghaei fails to explicitly disclose wherein Step (d4) further comprises: obtaining the integrity of the cell membrane using a skeletonization algorithm; and refining the pre-classification result based on the integrity of the cell membrane to classify the tumor area.
Descombes discloses wherein Step (d4) further comprises: obtaining the integrity of the cell membrane using a skeletonization algorithm; and refining the pre-classification result based on the integrity of the cell membrane to classify the tumor area ([0079] The outputs of the methods are then locally normalized and binarized, and small elements are removed using connected-component analysis. The output is then skeletonized, and the obtained skeletons (representing cell membranes) are cleaned by pruning and removing spurious branches and loops using a sequential repeated dilation followed by skeletonization steps (preferably two).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Aghaei’s classification of tumor segments with the known use of skeletonization and pruning on cell membranes as taught by Descombes in order to improve the automated assessment of cellular structures [See Descombes].
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kaitlin A Retallick whose telephone number is (571)270-3841. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chris Kelley can be reached at (571) 272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KAITLIN A RETALLICK/Primary Examiner, Art Unit 2482
Prosecution Insights