DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/11/2025 is/are compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Office Action Summary
Claim(s) 1-3, 6, 8, 10-13, 16, 18, and 20-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chukka et al (US 2020/0342597 A1) in view of Liu et al (An End-to-End DL Histochemical Scoring System for Breast Cancer Tissue Microarray).
Claim(s) 7 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chukka et al (US 2020/0342597 A1) in view of Liu et al (An End-to-End DL Histochemical Scoring System for Breast Cancer Tissue Microarray), further in view of Smith (US 2003/0059851 A1).
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chukka et al (US 2020/0342597 A1) in view of Liu et al (An End-to-End DL Histochemical Scoring System for Breast Cancer Tissue Microarray), further in view of Morrison et al (US 2020/0231670 A1).
Claim(s) 4-5 and 14-15 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 6, 8, 10-13, 16, 18, and 20-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chukka et al (US 2020/0342597 A1) in view of Liu et al (An End-to-End DL Histochemical Scoring System for Breast Cancer Tissue Microarray).
Regarding claim(s) 1, 11, and 21, Chukka teaches a system comprising:
one or more data processors (Figure 2; and Paragraph [0050]); and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations (Figure 2; and Paragraph [0050]) comprising:
accessing a duplex immunohistochemistry (IHC) image of a slice of specimen, wherein the duplex IHC image comprises a depiction of cells associated with one or more of a first biomarker and a second biomarker corresponding to a disease (Paragraph [0059]: “the input images may be multiplex images, i.e. stained for multiple, different markers in a multiplex assay”; Paragraph [0064]: “By way of one non-limiting example, and in the context of detecting breast cancer, in some embodiments the tissue sample is stained in an IHC assay for the presence of one or biomarkers including an estrogen receptor marker, a progesterone receptor marker, a Ki-67 marker, or a HER2 marker”; and Paragraph [0128]: “the images received as input may be multiplex images, i.e. the image received is of a biological sample stained with more than one stain”);
generating, from the duplex IHC image, a first synthetic image depicting the first biomarker and a second synthetic image depicting the second biomarker (Paragraph [0128]: “the multiple image is first unmixed into its constituent channels, where each unmixed channel corresponds to a particular stain or signal […] the unmixed images (often referred to as “channel images” or “image channel images”) […] inter-marker heterogeneity may be determined with a first H&E image, a second multiplex image stained for a plurality of cluster of differentiation markers (CD3, CD8, etc.), and a plurality of simplex images each stained for a particular biomarker (e.g. ER, PR, Ki67, etc.). In this example, the multiplex image is first unmixed into its constituent channel images, and those channel images may be used along with the H&E image and the plurality of simplex images to determined inter-marker heterogeneity”; and Paragraph [0129]: “in a sample comprising one or more stains and hematoxylin, individual images may be produced for each channel of the one or more stains and hematoxylin”);
determining, for each of the first synthetic image and the second synthetic image (Paragraph [0128] – Paragraph [0129]), a set of features representing pixel intensities of the depiction of cells in the first synthetic image and the second synthetic image (Paragraph [0083]: “An “appearance feature” as used herein is, for example, a feature having been computed for a particular nucleus by comparing pixel intensity values of pixels contained in or surrounding a nuclear blob or seed used for identifying the nucleus, whereby the compared pixel intensities are derived from different image channels”; Paragraph [0084]: “A “background feature” is, for example, a feature being indicative of the appearance and/or stain presence in cytoplasm and cell membrane features of the cell comprising the nucleus for which the background feature was extracted from the image. A background feature and a corresponding metrics can be computed for a nucleus and a corresponding cell depicted in a digital image […]”; and Paragraph [0129]: “features extracted from these channels are useful in describing the different biological structures present within any image of a tissue”);
processing the set of features using a trained machine learning model, wherein an output of the processing corresponds to a (Paragraph [0093]: “[…] variability between the image data obtained from different biomarkers, a different classifier can be trained for each different biomarker so as to ensure better performance on unseen test data […]”; Paragraph [0096]: “a classifier is used to distinguish between tumor nuclei from other candidate nuclei which also may be further classified […] the random forest classifier may be trained by […] (iii) training the random forest classifier to distinguish between tumor nuclei and non-tumor nuclei based on the extracted features. The trained random forest classifier may then be applied to classify the nuclei in an unlabeled image into tumor nuclei and non-tumor nuclei”; and Paragraph [0054]: “a scoring module to provide an expression score based on the cell labels, an overlay generation module to provide identifying indicia as an overlay onto the unlabeled image”); and
outputting a result that corresponds to a (Paragraph [0019]: “the system further comprises a scoring module configured to count the cell labels in the unlabeled image and output an expression score based on ratios of the various counted cell labels. In some embodiments, scoring is performed within a pre-determined field-of-view within the unlabeled image”; Paragraph [0054]: “a scoring module to provide an expression score based on the cell labels, an overlay generation module to provide identifying indicia as an overlay onto the unlabeled image”; and Paragraph [0113]: “Following assignment of cell labels, the image may be further assessed (step 326) […] a scoring module may be employed to generate one or more expression scores […] based on the detected positive and negative tumor cells/nuclei, various slide level scores such as marker percent positivity may be computed using one or more methods”).
Chukka fails to teach processing the set of features using a trained machine learning model, wherein an output of the processing corresponds to a predicted expression level of the first biomarker and the second biomarker; and outputting a result that corresponds to a predicted characterization of the specimen with respect to the disease based on the output of the processing.
However, Liu teaches processing the set of features using a trained machine learning model, wherein an output of the processing corresponds to a predicted expression level of the first biomarker and the second biomarker (Figure 6; Equation 11; Table 3; Abstract: “we present an end-to-end deep learning system which directly predicts the H-Score automatically”; Page 5, 3.3 The H-Score Prediction Framework, 1st Paragraph: “Only the information useful for predicting the H-Score is presented to a deep CNN to estimate the H-Score of the input image”; and Page 6, 4.1 dataset, 1st Paragraph: “The tissues are cropped from a sample of one patient which are stained with three different nuclei activity biomarkers: ER, p53, and PgR”); and
outputting a result that corresponds to a predicted characterization of the specimen with respect to the disease based on the output of the processing (Figure 6; Equation 11; Table 3; Page 1, Right Col., 1st Paragraph: “The H-Scores of tissue samples stained with different biomarkers are combined together to determine the biological class of a case. Clinical decision making is to choose an appropriate treatment from a number of available treatment options according to the biological class of the tumour”; and Page 8, Right Col., 2nd Paragraph: “To evaluate the performance of our proposed RAMCNN and the two H-Score relevant images SINI and SITI […] The RGB-CNN takes the original RGB TMA image with the shape of 512 X 512 X 3 as input, and output the H-Score prediction”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the image analysis method of Chukka to incorporate the deep learning-based histochemical scoring technique taught by Liu, to predict biomarker expression levels from stain intensity information extracted from the respective biomarker-specific image channels. The motivation for this combination of references would have been to provide automated prediction of biomarker expression scores while reducing the time-consuming and subjective nature of manual histochemical scoring, as recognized by Liu (Abstract; and Section 1). Furthermore, because Chukka teaches processing multiple biomarker specific image channels (Paragraph [0092]: “the image analysis module 207 is run a first time to extract features and classify cells and/or nuclei in a first image; and then run a second time to extract features and classify cells and/or nucleic in a series of addition images, where the additional images may be other simplex images or unmixed image channel images”; and Paragraphs [0128] – [0129]), a person of ordinary skill in the art would have had reason to apply Liu's trained scoring approach to the respective channels to obtain predicted expression levels for the corresponding biomarkers. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim(s) 2 and 12, Chukka as modified by Liu teaches the computer-implemented method of claim 1, where Chukka teaches further comprising, prior to determining the set of features:
preprocessing the first synthetic image and the second synthetic image by applying color deconvolution to the first synthetic image and the second synthetic image (Paragraph [0128] – Paragraph [0129]; and Paragraph [0131]: “Unmixing the component signals of each input pixel enables retrieval and analysis of stain-specific channels, such as a hematoxylin channel and an eosin channel in H&E images, or a diaminobenzidine (DAB) channel and a counterstain (e.g., hematoxylin) channel in IHC images. The terms “unmixing” and “color deconvolution” (or “deconvolution”) or the like (e.g. “deconvolving,” “unmixed”) […]”).
Regarding claim(s) 3 and 13, Chukka as modified by Liu teaches the computer-implemented method of claim 1, further comprising, prior to determining the set of features:
processing the first synthetic image and the second synthetic image using another trained machine learning model, wherein another output of the processing identifies first depictions of cells of the first synthetic image predicted to depict the first biomarker and second depictions of cells of the second synthetic image predicted to depict the second biomarker (where Chukka teaches in Paragraph [0081]: “Following detection of the nuclei, features (or metrics) are derived from within the input image”; Paragraph [0092]; Paragraph [0111]: “unlabeled image is supplied to a trained multilayer neural network 220 (step 324). The skilled artisan will appreciate that the multilayer neural network 220 must have been trained to classify the types of cells present (or suspected to be present) in the unlabeled image”; and Paragraph [0128] – Paragraph [0129]; and where Liu teaches in Figure 6; Equation 11; Table 3; Abstract: “we present an end-to-end deep learning system which directly predicts the H-Score automatically”; Page 5, 3.2 Nuclei and Tumour Maps: “the important information pathologists use to come up with the H-Score is the number of nuclei and the number of tumour nuclei in the TMA image. We therefore need to extract these two pieces of information and we use two separate FCNs, one for segmenting all nucleus and the other for segmenting tumour nucleus only […]”; Page 5, 3.3 The H-Score Prediction Framework: “[…] Only the information useful for predicting the H-Score is presented to a deep CNN to estimate the H-Score of the input image […]”; and Page 6, 4.1 dataset, 1st Paragraph: “The tissues are cropped from a sample of one patient which are stained with three different nuclei activity biomarkers: ER, p53, and PgR”).
Regarding claim(s) 6 and 16, Chukka as modified by Liu teaches the computer-implemented method of claim 1, where Chukka teaches wherein the first biomarker comprises estrogen receptor proteins and the second biomarker comprises progesterone receptor proteins (Paragraph [0059]: “the input images may be multiplex images, i.e. stained for multiple, different markers in a multiplex assay”; Paragraph [0064]: “By way of one non-limiting example, and in the context of detecting breast cancer, in some embodiments the tissue sample is stained in an IHC assay for the presence of one or biomarkers including an estrogen receptor marker, a progesterone receptor marker, a Ki-67 marker, or a HER2 marker”; and Paragraph [0128]: “the images received as input may be multiplex images, i.e. the image received is of a biological sample stained with more than one stain”).
Regarding claim(s) 8 and 18, Chukka as modified by Liu teaches the computer-implemented method of claim 1, where Chukka teaches wherein a sample slice of the specimen comprises a first stain for the first biomarker and a second stain for the second biomarker (Paragraph [0059]: “the input images may be multiplex images, i.e. stained for multiple, different markers in a multiplex assay”; Paragraph [0064]: “By way of one non-limiting example, and in the context of detecting breast cancer, in some embodiments the tissue sample is stained in an IHC assay for the presence of one or biomarkers including an estrogen receptor marker, a progesterone receptor marker, a Ki-67 marker, or a HER2 marker”; and Paragraph [0128]: “the images received as input may be multiplex images, i.e. the image received is of a biological sample stained with more than one stain”).
Regarding claim(s) 10 and 20, Chukka as modified by Liu teaches the computer-implemented method of claim 1, where Chukka teaches further comprising performing subsequent processing to generate the result of the predicted characterization of the specimen, wherein performing the subsequent processing includes detecting depictions of a set of tumor cells, and wherein the result characterizes a presence of, quantity of and/or size of the set of tumor cells (Paragraph [0113]: “Following assignment of cell labels, the image may be further assessed (step 326) […] a scoring module may be employed to generate one or more expression scores […] based on the detected positive and negative tumor cells/nuclei, various slide level scores such as marker percent positivity may be computed using one or more methods”; Paragraph [0019]: “a cell detection module adapted to identify cells within an unlabeled image; a multilayer neural network trained to provide predictive labels for each pixel within the unlabeled image, the predictive labels corresponding to different cell types”; Paragraph [0112]: “he cell would be assigned the label of a tumor cell since the majority of the pixels were predicted to be tumor cell pixels”; and Paragraph [0080]: “additional spectral and/or shape features of the identified nuclear blob may be analyzed to determine if the nucleus or nuclear blob is a nucleus of a tumor cell or not […] nuclei of lymphocytes are identified by identifying all nuclear blobs of a minimum size or diameter which is significantly larger than the average size or diameter of a normal lung cell nucleus”).
Claim(s) 7 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chukka et al (US 2020/0342597 A1) in view of Liu et al (An End-to-End DL Histochemical Scoring System for Breast Cancer Tissue Microarray), further in view of Smith (US 2003/0059851 A1).
Regarding claim(s) 7 and 17, Chukka as modified by Liu teaches the method of claim 1, but do not specifically teach wherein the trained machine learning model comprises a linear regression model. However, Smith teaches wherein the trained machine learning model comprises a linear regression model (Table 11; Paragraph [0254]: “This proved very effective in producing a linear regression between the three paired data points in the table […] The "least squares" best estimate of the linear relation for ln(.SIGMA.OD) regressed upon ng mutant p53/mg protein is linear, and 100% of the linear shape of the regression line is accounted for by the co-variation between the quantity of mutant p53 in the calibration cells and their associated immunostaining”; and Paragraph [0257]: “The immunohistology regression line for the particular batch of slides was used to convert the cell imaging score into p53 concentration per mg protein, and to change this to a volumetric concentration, using the SW480 cell lysate concentration as a constant”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the trained machine learning model of Chukka as modified by Liu to comprise a linear regression model, as taught by Smith, because Smith demonstrates that a linear regression model fitted to calibration data provides a quantitative relationship between immunostaining intensity measurements and biomarker protein concentrations (Smith, Table 11, Figure 11, and Paragraph [0254] – Paragraph [0257]). The motivation for this combination of references would have been to make such a modification to improve the quantitative estimation of biomarker expression levels from image-derived staining intensity features and to account for variations in staining intensity between different staining batches, as taught by Smith (Paragraph [0255]). This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chukka et al (US 2020/0342597 A1) in view of Liu et al (An End-to-End DL Histochemical Scoring System for Breast Cancer Tissue Microarray), further in view of Morrison et al (US 2020/0231670 A1).
Regarding claim(s) 9 and 19, Chukka as modified by Liu teaches the computer-implemented method of claim 8, but do not specifically teach wherein the first stain comprises tetramethylrhodamine and the second stain comprises 4-Dimethylaminoazobenzene-4'- sulfonyl. However, Morrison teaches wherein the first stain comprises tetramethylrhodamine and the second stain comprises 4-Dimethylaminoazobenzene-4'- sulfonyl (Paragraph [0542]: “Exemplary fluorophores useful as labels include, but are not limited to, rhodamine derivatives, for example, tetramethylrhodamine, rhodamine B […] Exemplary chromophores include, but are not limited to, phenolphthalein, malachite green, nitroaromatics such as nitrophenyl, diazo dyes, dabsyl (4-dimethylaminoazobenzene-4′-sulfonyl), and the like”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the biomarker staining method of Chukka as modified by Liu to employ tetramethylrhodamine and 4-Dimethylaminoazobenzene-4'-sulfonyl as detectable labels for the respective biomarkers, as taught by Morrison, in order to provide alternative known labeling agents for detecting biomarker expression in immunoassays. This motivation is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Allowable Subject Matter
Claim(s) 4-5 and 14-15 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Relevant Prior Art Directed to State of Art
Barnes et al (US 2017/0309021 A1) are relevant prior art not applied in the rejection(s) above. Barnes discloses a digital image analysis method for co-expression analysis of multiple markers in a tissue sample comprising: accessing a plurality of single marker channel images, each of the plurality of single marker channel images being a digital image whose pixel intensity values correlate with the amount of a respective single marker in the tissue sample; computing a heat map of marker expression for each of the single marker channel images; identifying one or more candidate regions of interest (ROIs) in each heat map of marker expression; computing overlay masks comprising the identified one or more candidate ROIs from each heat map of marker expression, an overlay mask being an image mask masking out all pixels not belonging to one of the identified candidate ROIs of the heat map the overlay mask was derived from; identifying one or more co-localized ROIs from the overlay masks by identifying intersection areas and/or union areas of the candidate ROIS of two or more of the heat maps, mapping the identified intersection or union areas back to the two or more heat maps; and using the candidate ROIs or parts thereof which correspond to said mapped-back intersection or union areas as the one or more co-localized ROIs; and selectively analyzing pixel intensities of the co-localized ROIs of two or more of the heat maps for determining if the markers corresponding to the two or more heat maps are co-expressed in the tissue sample.
Nie et al (US 2021/0216746 A1) are relevant prior art not applied in the rejection(s) above. Nie discloses a system for identifying clusters of cells within a sample image of a biological specimen stained with a primary stain or stained for the presence of one or more biomarkers, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories to store computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: a. detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine comprises a convolutional neural network adapted to detect cellular features within the sample image; b. extracting cellular features from one or more layers of the convolutional neural network; and c. clustering the detected cells in the sample image based on the extracted cellular features to provide one or more homogeneous clusters of detected cells.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONGBONG NAH whose telephone number is (571) 272-1361. The examiner can normally be reached M - F: 9:00 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL MISTRY can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONGBONG NAH/Examiner, Art Unit 2674
