DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/29/2024, 02/19/2025, and 10/16/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 6, 9, 14, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Barnes (US 20170309021) in view of Taube (US 20240272161).
Regarding claim 1:
Barnes discloses: a method (¶ [0025] “FIG. 2B provides a flowchart outlining the steps in one method of the present disclosure for identifying co-localized regions of interest and, in some embodiments, counting cells in those identified co-localized regions of interest.” ¶ [0038] “The present disclosure provides systems and methods for automatic identification of co-localized regions of interest (ROI) in multiplex assays); comprising:
accessing a digital pathology image depicting at least part of a biological sample that is stained for a first type of biomarker and a second type of biomarker (¶ [0063] “As an initial step, the computer system receives a plurality of single marker channel images as input (step 220)”, where each of the plurality of single marker channels images provided comprise signals corresponding to a single marker (e.g. signals from a stain or a tag, including chromogens, fluorophores, quantum dots, etc.)”; ¶ [0064] “The plurality of single marker channel images may be derived from several sources, including (i) a multi-channel image of a tissue sample from a multiplex assay, where the tissue sample is stained with two or more markers (“multiplex image”)”);
unmixing the digital pathology image (¶ [0064] “…Unlike simplex images, multiplex images must be unmixed into the plurality of single marker channel images.”; ¶ [0065] “Methods of unmixing are well known to those of ordinary skill in the art and any method now known or later discovered may be used to “unmix” the multiplex images into the plurality of single marker channel images”); to generate:
a first synthetic singleplex image depicting the at least part of the biological sample for which the first type of biomarker is identified; and a second synthetic singleplex image depicting the at least part of the biological sample from which the second type of biomarker is identified (¶ [0064] “…Unlike simplex images, multiplex images must be unmixed into the plurality of single marker channel images. In some embodiments, however, each of the simplex images may also unmixed so as to separate signals corresponding to the marker in each image from signals corresponding to a counterstain”; ¶ [0065] “Methods of unmixing are well known to those of ordinary skill in the art and any method now known or later discovered may be used to “unmix” the multiplex images into the plurality of single marker channel images. In general, the unmixing process extracts stain-specific channels to determine local concentrations of individual stains using reference spectra that are well known for standard types of tissue and stain combinations. The pixel intensities of the respective single marker channel images correlate with the amount of stain specifically bound to said marker at corresponding locations in the tissue sample. The amount of bound stain, again, correlates with the amount of said marker and thus, with the expression level of said marker, at said tissue section location. The terms “unmixing” and “color deconvolution” (or “deconvolution”) or the like (e.g. “deconvolving,” “unmixed”) are used interchangeably.”
In view of applicant’s definition of “synthetic singleplex image” in the original specification in ¶ [0068]. Barnes’ unmixed single marker channel images, including separation from counterstain, teach or least render obvious the claimed first and second synthetic singleplex images);
applying a first machine-learning model to the first synthetic singleplex image; detect a first plurality of cells from the first synthetic singleplex image; and determine, for each cell of the first plurality of cells, a classification of a first set of classifications, the classification of the first set indicating whether the cell includes a biomarker having the first type of biomarker (¶ [0085] – ¶ [0090] teach marker-specific positive/negative classification “… “area of positive marker expression” is a digital image area indicating that a particular biomarker (that may be indicative of the presence of a particular cell type, e.g. an immune cell type) is expressed in a particular area of a tissue sample that corresponds to said image area…by comparing a digital image corresponding to a particular biomarker and respective image channel with an intensity threshold, areas of positive marker expression and areas of negative marker expression within said digital image can be determined…A candidate ROI for positive marker expression indicates regions of interest identified as having high values on a heat map, i.e. representing a tissue area with a high number of cells positive for a marker…By negative marker expression, it is meant that thresholded regions are negative to overexpression of the marker, i.e., the biomarker is not expressed or only weakly expressed in corresponding sections of the tissue slide”; ¶ [0135] – ¶ [0137] further teach “…machine learning techniques may be used for cell detection, such as statistical model matching learned from structured support vector machines (SVMs) to identify the cell-like regions … the counting of cells is performed using a convolutional neural network that has been trained for the task … The trained cell detector may be used to identify cellular structures, such as immune cells, in the channels of the image that correspond to multiple types of cell markers or other target structures such as a nucleus. The learning means may include generating a convolutional neural network (CNN) by analyzing a plurality of training images with ground truths labeled thereon. Subsequent to the training, a test image or image under analysis may be divided into a plurality of patches, each patch containing one or multiple channels that are classified according to a CNN, and a probability map may be generated representing a presence of the cell or other target structure within the image “) to:
applying a second machine-learning model to the second synthetic singleplex image to: detect a second plurality of cells from the second synthetic singleplex image; and determine, for each cell of the second plurality of cells, a classification of a second set of classifications, the classification of the second set indicating whether the cell includes a biomarker having the second type of biomarker; wherein the first set of classifications are different from the second set of classifications (¶ [0014] – ¶ [0015] “the area of intersection of two or more of the candidate ROIs is mapped to each of the single marker channel images, thereby identifying the co-localized ROIs in each of said images…one or more co-localized ROIs co-express three or more markers…compute a heat map of marker expression for each of a plurality of single marker channel images, wherein each of the plurality of single marker channel images comprise a single marker… ¶ [0137] “…The trained cell detector may be used to identify cellular structures, such as immune cells, in the channels of the image that correspond to multiple types of cell markers or other target structures such as a nucleus. The learning means may include generating a convolutional neural network (CNN) by analyzing a plurality of training images with ground truths labeled thereon. Subsequent to the training, a test image or image under analysis may be divided into a plurality of patches, each patch containing one or multiple channels that are classified according to a CNN, and a probability map may be generated representing a presence of the cell or other target structure within the image”.
Because Barnes teaches separate marker-specific channel images corresponding to different markers, and further teachers ML-based detection and classification in such channels, it would have been obvious to apply ML inference separately to the respective first and second marker-specific images, whether by separate models or by separate application of the same model, thereby yielding first and second marker-specific classification sets that are different because they correspond to different biomarkers.);
while Barnes discloses that the shared-coordinate, co-localized combination of marker-specific results in ¶ [0012] “…mapping the one or more co-localized ROIs to a same coordinate position in each of the plurality of single marker channel images; and estimating a number of cells in at least one of the determined one or more co-localized ROIs in each of the plurality of single marker channel images”; and ¶ [0014] “…two or more of the candidate ROIs is mapped to each of the single marker channel images, thereby identifying the co-localized ROIs in each of said images”.
Barnes does not expressly teach: merging the classifications of the first plurality of cells and the classifications of the second plurality of cells to generate merged classifications; and outputting the digital pathology image with merged classifications.
However, in the same field of endeavor, Taube teaches: merging the classifications of the first plurality of cells and the classifications of the second plurality of cells to generate merged classifications (¶ [0041] “…FIG. 15B shows a representative image of merged phenotype output following single-marker phenotyping (top left), and a corresponding output of each individual single-marker phenotype algorithm before merging (bottom and right).”; ¶ [0112] “…‘Single-marker’ phenotyping was also performed, whereby cells were assigned positive or negative status for each marker individually. Cell centers were then used to merge the six individual datasets into a single Cartesian coordinate system.”).
and outputting the digital pathology image with merged classifications (classifications (¶ [0041] “…FIG. 15B shows a representative image of merged phenotype output following single-marker phenotyping (top left), and a corresponding output of each individual single-marker phenotype algorithm before merging (bottom and right).”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Barnes to incorporate the teachings of Taube by including: merging the classifications of the first plurality of cells and the classifications of the second plurality of cells to generate merged classifications; and outputting the digital pathology image with merged classifications in order to derive a reliable cell-level multi-marker information from multiplex tissue images.
Regarding claim 6:
Barnes in view of Taube teaches the limitations of claim 1 as applied above.
Taube further teaches: wherein outputting the digital pathology image with merged classifications includes overlaying the merged classifications onto the digital pathology image (¶ [0042] “FIGS. 16A-16E show representative output from bespoke algorithms that facilitate visual inspection of segmentation and phenotyping performance. FIG. 16A shows a custom display that shows ˜1250 cells per view. A colored dot is placed on each cell in the mIF image indicating the lineage. Additionally, a dash is placed over the cell if PD-L1 (green dash) and/or PD-1 (cyan dash) is expressed. 20 views per specimen were visually inspected, except for the rare cases with less tissue availability. FIG. 16B shows the inspection of up to 25 randomly selected positive and negative cells/stamps for each marker across all HPFs in a given specimen.”)).
Regarding claims 9 and 17: the claims limitations are similar to those of claim 1; therefore, rejected in the same manner as applied above. Barnes teaches a system and a crm in ¶ [0015] “Another aspect of the present disclosure is a computer system for co-expression analysis of multiple markers in a tissue sample comprising one or more processors and at least one memory, the at least one memory storing non-transitory computer-readable instructions for execution by the one or more processors”.
Regarding claim 14: the claim limitations are similar to hose of claim 6; therefore, rejected in the same manner as applied above.
Claim(s) 2-3, 5, 10-11, 13, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Barnes (US 20170309021) in view of Taube (US 20240272161), Chefd’hotel (US 20190012787), herein after “Chefd”, and Chukka (US 20200342597).
Regarding claim 2:
Barnes in view of Taube teaches the limitations of claim 1 as applied above.
Barnes in view of Taube does not expressly teach: generating a first set of probability maps, wherein each probability map of the first set of probability maps includes a plurality of pixels and is associated with a classification of the first set of classifications, wherein the probability map identifies, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to the classification; and for each cell of the first plurality of cells: identifying a probability map of the first set of probability maps that includes the highest probability value for one or more pixels that represent the cell.
However, in a related field, Chefd teaches: wherein determining the classifications for the first plurality of cells includes:
generating a first set of probability maps (¶ [0061] “…Once the structures are mapped, the maps are fully connected, and the CNN operation (S407) outputs a map comprising a fully connected layer that is similar to the typical neural network to generate probabilistic labels for each class. The probability map generated represents a presence of each different type of immune cell or other target structure within the input patches”, ¶ [0063] “…A probability map is generated as the output of CNN. The probability map represents a presence of each different type of immune cell or other target structure within the input patches”);
wherein each probability map of the first set of probability maps includes a plurality of pixels (¶ [0063] “…for example cell, the centroid or center of the cell may be obtained by determining the centroid coordinates using a non-maximum suppression operation. By utilizing the non-maximum suppression operation, the local maximum in that region wherein that pixel has higher values than everything around it in that neighborhood is found, and therefore corresponds to the center or centroid of the identified structure, for example, the nucleus.”
and is associated with a classification of the first set of classifications (¶ [0061] “…Once the structures are mapped, the maps are fully connected, and the CNN operation (S407) outputs a map comprising a fully connected layer that is similar to the typical neural network to generate probabilistic labels for each class”; ¶ [0063] “…A probability map is generated as the output of CNN. The probability map represents a presence of each different type of immune cell or other target structure within the input patches”);
wherein the probability map identifies, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to the classification (¶ [0061] “…Once the structures are mapped, the maps are fully connected, and the CNN operation (S407) outputs a map comprising a fully connected layer that is similar to the typical neural network to generate probabilistic labels for each class”; ¶ [0063] “…A probability map is generated as the output of CNN. The probability map represents a presence of each different type of immune cell or other target structure within the input patches… By utilizing the non-maximum suppression operation, the local maximum in that region wherein that pixel has higher values than everything around it in that neighborhood is found, and therefore corresponds to the center or centroid of the identified structure, for example, the nucleus.”);
and for each cell of the first plurality of cells: identifying a probability map of the first set of probability maps that includes the highest probability value for one or more pixels that represent the cell (¶ [0061] “…the cell centroids may be obtained by determining immune cell coordinates using a non-maximum suppression operation (S408), which is a known edge thinning technique that can help to suppress all the gradient values to 0 except the local maxima, which indicates the location with the sharpest change of intensity value.”; ¶ [0063] “…the local maximum in that region wherein that pixel has higher values than everything around it in that neighborhood is found, and therefore corresponds to the center or centroid of the identified structure, for example, the nucleus. The final detection of the cells is shown in 432, with indicators 433 depicting locations of the centroids.”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Barnes and Taube to incorporate the teachings of Chefd , which includes a known and predictable way of implementing Barnes in view of Taube’s workflow though CNN-generated class-specific probability maps and identification of local maxima and centroids from those maps in order to represent marker-specific classification results in a spatially resolved form and to improve localization of detected cells from downstream cell-level assignment.
Barnes in view of Taube and in Chefd does not expressly teach: assigning the cell with a classification associated with the probability map.
However, in a related field, Chukka teaches: assigning the cell with a classification associated with the probability map (¶ [0112] “Using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325)… the assignment of a cell label to each identified individual cell comprises (i) quantifying a number of pixels bearing each predictive classification label within the identified individual cell; and (ii) assigning as the cell label the predictive label having the greatest quantity…the majority pixel label is used as the label of the cell”).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Barnes in view of Taube and Chefd to incorporate the teachings of Chukka, including converting outputs into a single cell-level label by evaluating the predictive classifications of the pixels associated with an identified cell and assigning the cell the label having the greatest support in order to obtain a robust final classification for each detected cell from the underlying pixel-level class information.
Regarding claim 3: the claim limitations are similar to those of claim 2; therefore, rejected in the same manner as applied above.
Regarding claim 5:
Chukka further teaches: wherein the first type of biomarker is an estrogen receptor protein and the second type of biomarker is a progesterone receptor protein (¶ [0064] “…the tissue sample is stained in an IHC assay for the presence of one or biomarkers including an estrogen receptor marker, a progesterone receptor marker”)).
Regarding claims 10-11, 13: the claims limitations are similar to those of claims 2-3, and 15; therefore, rejected in the same manner as applied above.
Regarding claims 18-19: the claims limitations are similar to those of claims 2-3; therefore, rejected in the same manner as applied above.
Claim(s) 4 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Barnes (US 20170309021) in view of Taube (US 20240272161) and Nie (WO 2021076605).
Regarding claim 4:
Barnes in view of Taube teaches the limitations of claim 1 as applied above.
Barnes in view of Taube does not expressly teach: wherein the first machine-learning model and/or the second machine-learning model includes a U-Net model.
However, in a related field, Nie teaches: wherein the first machine-learning model and/or the second machine-learning model includes a U-Net model (¶ [0028] “FIG. 3 shows an exemplary U-Net in accordance with various embodiments.”)
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Barnes in view of Taube to incorporate the teachings of Nie by including wherein the first machine-learning model and/or the second machine-learning model includes a U-Net model in order to obtain a well-known, robust deep-learning architecture for cell detection and classification in the marker-specific images, which would have been a predicable substitution of one known CNN architecture for another.
Regarding claim 12: the claim limitations are similar to those of claim 4; therefore, rejected in the same manner as applied above.
Claim(s) 7, 15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Barnes (US 20170309021) in view of Taube (US 20240272161) and Yip (US 20200258223).
Regarding claim 7:
Barnes in view of Taube teaches the limitations of claim 1 as applied above.
Barnes in view of Taube does not expressly teach: wherein the digital pathology image with merged classifications is used as a training image for training a third machine-learning model
However, in a related field, Yip teaches: wherein the digital pathology image with merged classifications is used as a training image for training a third machine-learning model (¶ [0379] “Annotated images 1818 may be provided to the deep learning framework 1802 for training of the various classification modules, using techniques described above… the entire histopathology image is provided to the framework 1802 for training” and Taube provides the merged image as applied in claim 1.)
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Barnes in view of Taube to incorporate the teachings of Yip by including wherein the digital pathology image with merged classifications is used as a training image for training a third machine-learning model in order to use labeled pathology-image outputs as supervised training data for a further classification model.
Regarding claims 15 and 20: the claims limitations are similar to those of claim 7; therefore, rejected in the same manner as applied above.
Allowable Subject Matter
Claims 8 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665
Prosecution Insights