DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over CHEN (WO 2018115055 A1, claims priority of US Provisional Parent Application 62/438,354, December 22, 2016), and in view of ASHOK (WO 2016138041 A2), and further in view of SETHI (US 20180232883 A1, claims priority of FOREIGN APPLICATION PRIORITY DATA: 2017-02-13).
Re Claim 1, CHEN discloses a method comprising:
a) training an untrained machine learnable device to predict status of a diagnostic, prognostic, or theragnostic feature in stained tissue samples, the untrained machine learnable device being trained with a characterized set of digital images of stained tissue samples, each digital image of the characterized set having a known status for the diagnostic, prognostic, or theragnostic feature and an associated 2-dimensional grid of spatial locations, training of the untrained machine learnable device (see CHEN: e.g., Fig. 1, and, --Immunohistochemical (IHC) slide staining can be utilized to identify proteins in cells of a tissue section and hence is widely used in the study of different types of cells, such as cancerous cells and immune cells in biological tissue. In the context of staining for immune cells, the immunological data indicates the type, density, and location of the immune cells within tumor samples and this data is of particular interest to pathologists in determining a patient survival prediction.--, in [0001]-[0002]; and, --a computer-implemented method for analysis of a tissue sample comprising: receiving first and second input images; performing an analysis of the first image, including deriving features from the first image; registering the first and second images by mapping at least a portion of the first image to the second image to form a registered image; performing an analysis of the registered image, including deriving features from the registered image; merging the features derived from the first image and features derived from the second image, wherein the features derived from one of the first image or the registered image include probability features; and classifying nuclei in one of the first or second images based on the merged features set.--, in [0004]-[0005], and [0019]; and, -- After H&E image features, biomarker image features, and probability image features are derived, they are merged together and used to classifying nuclei within at least one of the input images. In some embodiments, the classifier is trained and then used to distinguish different cell nuclei or staining responses. … ensembles of classifiers generally outperform monolithic solutions..Learning ensembles or multiple classifier systems (Support Vector Machine or Adaboost, described below) are methods for improving classification accuracy through aggregation of several similar classifiers' predictions and thereby reducing either the bias or variance of the individual classifiers. [00166] In some embodiments, the classification module is a Support Vector Machine ("SVM"). In general, a SVM is a classification technique, which is based on statistical learning theory where a nonlinear input data set is converted into a high dimensional linear feature space via kernels for the non-linear case…. classification is performed using a Bootstrap aggregating technique. Bootstrap aggregating (bagging) is a machine learning ensemble meta- algorithm, designed to improve the stability and accuracy of machine learning used in statistical classification and regression. It also reduces variance and helps to avoid overfitting.--, in [0160]-[0167]) including steps of:
identifying a plurality of extracted features in each digital image of the characterized set of digital images (see CHEN: e.g., --A voting response matrix is created by processing each pixel that accumulates a vote through a voting kernel. The kernel is based on the gradient direction computed at that particular pixel and an expected range of minimum and maximum nucleus size and a voting kernel angle (typically in the range [π/4, π/8]). In the resulting voting space, local maxima locations that have a vote value higher than a predefined threshold value are saved out as seed points. Extraneous seeds may be discarded later during subsequent segmentation or classification processes--, in [0077]-[0079], and, --[0082] Feature Extraction [0083] Following detection of the nuclei, features (or metrics) are derived, such as with a feature extraction module 215. In general, the feature extraction module 215 receives image data, derives certain metrics based on the received image data, and outputs those derived metrics for combination with the separately computed probability features (step 305). In some embodiments, separate metrics are computed from the biomarker image and from the H&E image. [0084] More specifically, metrics may be derived from features of the identified nuclei or from within a patch surrounding an identified nucleus in both the H&E and biomarker images. For example, a feature metric can be a numerical data value being indicative of quantitative properties of a particular feature, a histogram, a distribution, or the like. In some embodiments, feature metrics are computed for each nucleus based on their visual properties and descriptors, e.g. morphology features, appearance features, background features, etc. In other embodiments, features are computed from within an image patch surrounding an identified nucleus. In some embodiments, the various feature metrics derived from the detected nuclei of the H&E and biomarker images are supplied as vectors of metrics and, together with metrics derived from the generated probability map (step 304 or 317), are supplied to the classification module 216 for classification (step 307 or 314).--, in [0082]-[0101]; and, -- [00106] (C) Metrics Derived from Background Features [00107] A "background feature" is, for example, a feature being indicative of the appearance and/or stain presence in cytoplasm and cell membrane features of the cell comprising the nucleus for which the background feature was extracted from the image. A background feature and a corresponding metrics can be computed for a nucleus and a corresponding cell depicted in a digital image e.g. by identifying a nuclear blob or seed representing the nucleus; analyzing a pixel area (e.g. a ribbon of 20 pixels - about 9 microns - thickness around the nuclear blob boundary) directly adjacent to the identified set of cells are computed in, therefore capturing appearance and stain presence in cytoplasm and membrane of the cell with this nucleus together with areas directly adjacent to the cell. These metrics are similar to the nuclear appearance features, but are computed in a ribbon of about 20 pixels (about 9 microns) thickness around each nucleus boundary, therefore capturing the appearance and stain presence in the cytoplasm and membrane of the cell having the identified nucleus together with areas directly adjacent to the cell. Without wishing to be bound by any particular theory, the ribbon size is selected because it is believed that it captures a sufficient amount of background tissue area around the nuclei that can be used to provide useful information for nuclei discrimination.--, in [0106]-[0107]; and, --The cooccurrence is computed of the pixel intensity at location (x;y) in Ci and the pixel intensity at location (x+dx; y+dy) in Cj. It is believed that the CCM offers that advantage of capturing the spatial relationship between different tissue structures (highlighted in different channels), without the need of explicitly segmenting them.--, in [0120]);
associating a value for each extracted feature with each spatial location to form a set of feature maps, each extracted feature map providing values for a extracted feature over the associated 2-dimensional grid of spatial locations (see CHEN: e.g., --A voting response matrix is created by processing each pixel that accumulates a vote through a voting kernel. The kernel is based on the gradient direction computed at that particular pixel and an expected range of minimum and maximum nucleus size and a voting kernel angle (typically in the range [π/4, π/8]). In the resulting voting space, local maxima locations that have a vote value higher than a predefined threshold value are saved out as seed points. Extraneous seeds may be discarded later during subsequent segmentation or classification processes--, in [0077]-[0079], and, --[0082] Feature Extraction [0083] Following detection of the nuclei, features (or metrics) are derived, such as with a feature extraction module 215. In general, the feature extraction module 215 receives image data, derives certain metrics based on the received image data, and outputs those derived metrics for combination with the separately computed probability features (step 305). In some embodiments, separate metrics are computed from the biomarker image and from the H&E image. [0084] More specifically, metrics may be derived from features of the identified nuclei or from within a patch surrounding an identified nucleus in both the H&E and biomarker images. For example, a feature metric can be a numerical data value being indicative of quantitative properties of a particular feature, a histogram, a distribution, or the like. In some embodiments, feature metrics are computed for each nucleus based on their visual properties and descriptors, e.g. morphology features, appearance features, background features, etc. In other embodiments, features are computed from within an image patch surrounding an identified nucleus. In some embodiments, the various feature metrics derived from the detected nuclei of the H&E and biomarker images are supplied as vectors of metrics and, together with metrics derived from the generated probability map (step 304 or 317), are supplied to the classification module 216 for classification (step 307 or 314).--, in [0082]-[0101]; and, -- [00106] (C) Metrics Derived from Background Features [00107] A "background feature" is, for example, a feature being indicative of the appearance and/or stain presence in cytoplasm and cell membrane features of the cell comprising the nucleus for which the background feature was extracted from the image. A background feature and a corresponding metrics can be computed for a nucleus and a corresponding cell depicted in a digital image e.g. by identifying a nuclear blob or seed representing the nucleus; analyzing a pixel area (e.g. a ribbon of 20 pixels - about 9 microns - thickness around the nuclear blob boundary) directly adjacent to the identified set of cells are computed in, therefore capturing appearance and stain presence in cytoplasm and membrane of the cell with this nucleus together with areas directly adjacent to the cell. These metrics are similar to the nuclear appearance features, but are computed in a ribbon of about 20 pixels (about 9 microns) thickness around each nucleus boundary, therefore capturing the appearance and stain presence in the cytoplasm and membrane of the cell having the identified nucleus together with areas directly adjacent to the cell. Without wishing to be bound by any particular theory, the ribbon size is selected because it is believed that it captures a sufficient amount of background tissue area around the nuclei that can be used to provide useful information for nuclei discrimination.--, in [0106]-[0107]; and, --The cooccurrence is computed of the pixel intensity at location (x;y) in Ci and the pixel intensity at location (x+dx; y+dy) in Cj. It is believed that the CCM offers that advantage of capturing the spatial relationship between different tissue structures (highlighted in different channels), without the need of explicitly segmenting them.--, in [0120]); and
inputting the set of extracted feature maps to the untrained machine learnable device to form associations therein between the set of extracted feature maps and the known status for the diagnostic, prognostic, or theragnostic feature thereby creating a trained machine learnable device (see Chen: e.g., -- After H&E image features, biomarker image features, and probability image features are derived, they are merged together and used to classifying nuclei within at least one of the input images. In some embodiments, the classifier is trained and then used to distinguish different cell nuclei or staining responses. … ensembles of classifiers generally outperform monolithic solutions..Learning ensembles or multiple classifier systems (Support Vector Machine or Adaboost, described below) are methods for improving classification accuracy through aggregation of several similar classifiers' predictions and thereby reducing either the bias or variance of the individual classifiers. [00166] In some embodiments, the classification module is a Support Vector Machine ("SVM"). In general, a SVM is a classification technique, which is based on statistical learning theory where a nonlinear input data set is converted into a high dimensional linear feature space via kernels for the non-linear case…. classification is performed using a Bootstrap aggregating technique. Bootstrap aggregating (bagging) is a machine learning ensemble meta- algorithm, designed to improve the stability and accuracy of machine learning used in statistical classification and regression. It also reduces variance and helps to avoid overfitting.--, in [0160]-[0167]); and
b) predicting feature of a stained tissue sample of unknown status for the diagnostic feature (see Chen: e, g., --, an H&E tissue slide is used for the initial primary diagnosis to detect, grade and stage cancer type for a particular tissue indication (breast, prostate, lung cancer etc.). IHC tissue slides, on the other hand, are typically used for cancer subtyping for prognostic and predictive purposes. The tissue morphology, i.e. tumorous glandular regions, cells and lymphocytes and lymphatic regions and stromal regions and cells are easily distinguishable in H&E tissue slide. The IHC tissue slides, stained with either a simplex or multiplex IHC chromogenic assay (DAB, Fast Red, Dual stained), are used to detect and quantify antigen/protein overexpression in the tumor, immune or vascular regions in the tissue. In the manual process to review and interpret IHC slides either under a microscope or on a digital read of a whole slide capture on a computer monitor, pathologists typically also review the corresponding regions in H&E tissue slide images for a better visual understanding of the tissue morphology and disambiguate tissue structures, that may be similar looking in the chromogenic IHC tissue slide.--, in [0015]-[0017]);
Chen however does explicitly disclose predicting a status for the diagnostic, prognostic, or theragnostic feature of a stained tissue sample of unknown status for the diagnostic feature;
ASHOK discloses predicting a status for the diagnostic, prognostic, or theragnostic feature of a stained tissue sample of unknown status for the diagnostic feature;
ASHOK discloses predicting a status for the diagnostic, prognostic, or theragnostic feature of a stained tissue sample of unknown status for the diagnostic feature (see ASHOK: e.g., -- The cell also in frequent embodiments is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer. Often, the sample is evaluated for the presence of HER 2 expression. In frequent embodiments, the cell is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) HER 2 expression, grade, lympho-vascular invasion, lymph node invasion, ductal carcinoma in situ, lobular carcinoma in situ, extra-nodal extension, positive surgical margins, LAPP, and/or MAPP. [0039] Also often, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) bladder cancer. In frequent embodiments, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) grade, lymph node invasion, squamous differentiation, glandular differentiation, and/or lymph invasion, LAPP, and/or MAPP. [0040] In certain embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer. In frequent embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer grade, LAPP, and/or MAPP.--, in [0038]-[0040]),
CHEN and ASHOK are combinable as they are in the same field of endeavor: to analysis of pathological stained tissue sample images based on measurements and information of biomarkers. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify CHEN’s method using ASHOK’s teachings by including predicting a status for the diagnostic, prognostic, or theragnostic feature of a stained tissue sample of unknown status for the diagnostic feature to CHEN’s analysis of the pathological image in order to include the analysis cells/tissues obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer (e.g., immunohistochemistry, sequencing, etc.). (see ASHOK: e.g., in [0038]-[0040]);
CHEN as modified by ASHOK further disclose obtaining a sample digital image for the stained tissue sample, the digital image having an associated 2-dimensional grid of spatial locations (see CHEN: e.g., Fig. 1, and, --Immunohistochemical (IHC) slide staining can be utilized to identify proteins in cells of a tissue section and hence is widely used in the study of different types of cells, such as cancerous cells and immune cells in biological tissue. In the context of staining for immune cells, the immunological data indicates the type, density, and location of the immune cells within tumor samples and this data is of particular interest to pathologists in determining a patient survival prediction.--, in [0001]-[0002]; --a computer-implemented method for analysis of a tissue sample comprising: receiving first and second input images; performing an analysis of the first image, including deriving features from the first image,--, in [0004]-[0005], [0014]-[0015], and [0019]);
associating a value for each extracted feature with each spatial location of the digital image (see CHEN: e.g., --A voting response matrix is created by processing each pixel that accumulates a vote through a voting kernel. The kernel is based on the gradient direction computed at that particular pixel and an expected range of minimum and maximum nucleus size and a voting kernel angle (typically in the range [π/4, π/8]). In the resulting voting space, local maxima locations that have a vote value higher than a predefined threshold value are saved out as seed points. Extraneous seeds may be discarded later during subsequent segmentation or classification processes--, in [0077]-[0079], and, --[0082] Feature Extraction [0083] Following detection of the nuclei, features (or metrics) are derived, such as with a feature extraction module 215. In general, the feature extraction module 215 receives image data, derives certain metrics based on the received image data, and outputs those derived metrics for combination with the separately computed probability features (step 305). In some embodiments, separate metrics are computed from the biomarker image and from the H&E image. [0084] More specifically, metrics may be derived from features of the identified nuclei or from within a patch surrounding an identified nucleus in both the H&E and biomarker images. For example, a feature metric can be a numerical data value being indicative of quantitative properties of a particular feature, a histogram, a distribution, or the like. In some embodiments, feature metrics are computed for each nucleus based on their visual properties and descriptors, e.g. morphology features, appearance features, background features, etc. In other embodiments, features are computed from within an image patch surrounding an identified nucleus. In some embodiments, the various feature metrics derived from the detected nuclei of the H&E and biomarker images are supplied as vectors of metrics and, together with metrics derived from the generated probability map (step 304 or 317), are supplied to the classification module 216 for classification (step 307 or 314).--, in [0082]-[0101]; and, -- [00106] (C) Metrics Derived from Background Features [00107] A "background feature" is, for example, a feature being indicative of the appearance and/or stain presence in cytoplasm and cell membrane features of the cell comprising the nucleus for which the background feature was extracted from the image. A background feature and a corresponding metrics can be computed for a nucleus and a corresponding cell depicted in a digital image e.g. by identifying a nuclear blob or seed representing the nucleus; analyzing a pixel area (e.g. a ribbon of 20 pixels - about 9 microns - thickness around the nuclear blob boundary) directly adjacent to the identified set of cells are computed in, therefore capturing appearance and stain presence in cytoplasm and membrane of the cell with this nucleus together with areas directly adjacent to the cell. These metrics are similar to the nuclear appearance features, but are computed in a ribbon of about 20 pixels (about 9 microns) thickness around each nucleus boundary, therefore capturing the appearance and stain presence in the cytoplasm and membrane of the cell having the identified nucleus together with areas directly adjacent to the cell. Without wishing to be bound by any particular theory, the ribbon size is selected because it is believed that it captures a sufficient amount of background tissue area around the nuclei that can be used to provide useful information for nuclei discrimination.--, in [0106]-[0107]; and, --The cooccurrence is computed of the pixel intensity at location (x;y) in Ci and the pixel intensity at location (x+dx; y+dy) in Cj. It is believed that the CCM offers that advantage of capturing the spatial relationship between different tissue structures (highlighted in different channels), without the need of explicitly segmenting them.--, in [0120]);
Chen as modified by ASHOK however still do not explicitly disclose to form a test set of extracted feature maps for the stained tissue sample of unknown status;
SETHI discloses associating a value for each extracted feature with each spatial location of the digital image form a test set of extracted feature maps for the stained tissue sample of unknown status (see SETHI: e.g., -- The tissue component classification map is then used to extract features such as area, circumference, length of major and minor axis, parameters of best fit circle, ellipse or polyhedron, graph based features encoding the concordance of labels among neighboring pixels, etc. for each POI in a patient tissue image. These features are then inputted to a pre-trained penalized logistic regression based disease classifier that assigns probabilities to each POI belonging to one of the four sub-types viz. normal, dysplasia, hyperplasia, and carcinoma. This local disease classification at each POI is aggregated into a report that computes the percent of POIs for each of the four sub-types viz. normal, dysplasia, hyperplasia, and carcinoma. Based on the report generated by this embodiment of the present disclosure and pathologist's report based on visual inspection of the adjacent H&E stained tissue sample appropriate treatment selections can be made by an oncologist.--, in [0089]);
CHEN (as modified by ASHOK) and SETHI are combinable as they are in the same field of endeavor: to analysis of pathological stained tissue sample images based on measurements and information of biomarkers. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify CHEN (as modified by ASHOK)’s method using SETHI’s teachings by including associating a value for each extracted feature with each spatial location of the digital image form a test set of extracted feature maps for the stained tissue sample of unknown status to CHEN (as modified by ASHOK)’s analysis of cells {such as tumor cells, tissues in a pathological stained tissue sample images} in order to improve the accuracy in determining tumoral classification and prognosis of tumoral tissue (see SETHI: e.g., in abstract, and in [001]-[0012], [0071]-[0078], [0085], and [0089]);
CHEN as modified by ASHOK and SETHI further disclose inputting the test set of extracted feature maps to the trained machine learnable device to obtain a predicted status for the status of the diagnostic, prognostic, or theragnostic feature for the stained tissue sample (see Chen: e.g., -- After H&E image features, biomarker image features, and probability image features are derived, they are merged together and used to classifying nuclei within at least one of the input images. In some embodiments, the classifier is trained and then used to distinguish different cell nuclei or staining responses. … ensembles of classifiers generally outperform monolithic solutions..Learning ensembles or multiple classifier systems (Support Vector Machine or Adaboost, described below) are methods for improving classification accuracy through aggregation of several similar classifiers' predictions and thereby reducing either the bias or variance of the individual classifiers. [00166] In some embodiments, the classification module is a Support Vector Machine ("SVM"). In general, a SVM is a classification technique, which is based on statistical learning theory where a nonlinear input data set is converted into a high dimensional linear feature space via kernels for the non-linear case…. classification is performed using a Bootstrap aggregating technique. Bootstrap aggregating (bagging) is a machine learning ensemble meta- algorithm, designed to improve the stability and accuracy of machine learning used in statistical classification and regression. It also reduces variance and helps to avoid overfitting.--, in [0160]-[0167]; also see ASHOK: e.g., -- The cell also in frequent embodiments is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer. Often, the sample is evaluated for the presence of HER 2 expression. In frequent embodiments, the cell is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) HER 2 expression, grade, lympho-vascular invasion, lymph node invasion, ductal carcinoma in situ, lobular carcinoma in situ, extra-nodal extension, positive surgical margins, LAPP, and/or MAPP. [0039] Also often, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) bladder cancer. In frequent embodiments, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) grade, lymph node invasion, squamous differentiation, glandular differentiation, and/or lymph invasion, LAPP, and/or MAPP. [0040] In certain embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer. In frequent embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer grade, LAPP, and/or MAPP.--, in [0038]-[0040]; and, -- [00219] In order to make clinical predictions, the machine learning algorithm has been trained. For training, biomarker data from 70% cells of a particular sample (with known Gleason score and adverse pathology) is fed into the algorithm. Subsequently the algorithm analyzes data from the remaining cells (30%) to make predictions about the LAPP and MAPP of the population. To determine the accuracy of our assay, the predictions made by the algorithm were compared to known Gleason scores and adverse pathology data. FIG. 38D demonstrates high sensitivity and specificity for the present methods to predict Gleason score, and distinguishes between samples with different Gleason scores. Remarkably, Gleason 3+4 (marked 7-) from Gleason 4+3 (marked 7+) were discerned in samples with high confidence, as seen by the ROC curve and associated statistics (FIG. 38 A). Moreover, these data demonstrate wide distribution of LAPP scores within the same Gleason group (FIG. 38B), indicating that the present diagnostic methods provide an evaluation of the tumorigenic potential of a sample that is more quantitative than, or is complementary to, the current Gleason scoring system. [00220] These data demonstrate, for example, that: (i) it is feasible to isolate and maintain tumor- derived cells; (ii) a panel of phenotypic biomarkers may be accurately measured; (iii) it is possible to train the machine learning algorithm to achieve increased accuracy to predict LAPP and MAPP; and (iv) the methods are capable of risk stratifying samples with the same Gleason score with high accuracy. Additionally, the machine learning algorithm is demonstrated to predict seminal vesicle invasion (FIG. 38C & D).--, in [00219]-[00220]; also see SETHI: e.g., -- (b) generating a pre-processed image from the images of the patient tissue; (c) locating one or more points of interest in the pre-processed image using a first classifier; (d) generating a disease spatial map with the probability of a disease state at the points-of-interest in the tissue image by using a second classifier; and (e) aggregating the probability of the disease condition at each point-of-interest located in step (c) to obtain the disease class scores of the disease state for the patient tissue. In further embodiments, the first classifier is a nucleus detector comprising one or more pre-trained neural networks, convolutional neural networks, or recurrent neural networks etc. In further embodiments, the probability of the disease state is aggregated from two or more images of the patient tissue from the same patient. In further embodiments, the method undergoes periodic or continuous validation. [0011] …cause the processor to: (a) input one or more images of the patient tissue, wherein the tissue has been treated with a stain comprising one or more stain components; (b) generate disease spatial maps with the probability of a disease state at one or more points-of-interest by using a combined classifier into one out of two or more disease classes--, in [0010]-[0012]; -- The objective of using a pre-processing module 203 is to take a magnified tissue image 160 and produce a pre-processed image 204 such that it is advantageous for the other modules such as points-of-interest detection 205 or local disease classification 207 to work with the pre-processed image 204 as compared to the magnified tissue image 160.--, in [0061], and, -- POI classifier training module 455 is a machine learning system or a classifier such as a convolutional neural network with associated parameters such as weights and hyper-parameters such as depth, filter sizes, etc. The sub-images of fixed window size from the training set 454 may be inputted into the convolutional neural network and their respective labels are the desired output. The convolutional neural network may be trained or optimized to minimize a cost function, such as cross-entropy between the correct labels and the neural network predictions by using an optimization method such as gradient descent. The objective of the training or optimization process is to reduce the difference between the classifier output and the known output (POI vs. non-POI status). With sufficient training using example input-output pairs, a machine learning based POI classifier training module learns to predict labels for the central pixel of input sub-image windows of pre-processed tissue image. For POI detection in new patient tissue images, a set of parameters and hyper-parameters 499 of the POI classifier training module 455 are saved in the memory of a computing unit. A validation data set similar to the training data set with pre-processed images along with their known POI locations may be used to ascertain the performance of the POI classifier. If the performance is unsatisfactory on the validation dataset, more training data may be collected, or the architecture (determined by hyper-parameters) of the classifier may be changed.--, and, -- one convolutional neural network. In some embodiments of the disease class scoring 105, the steps of pre-processing, detecting POIs, and computing a local disease classification may be done in a single module trained using multi-class label prediction techniques. For example, as shown in FIG. 9, the three color channels 912, 914, 916 of a color normalized (pre-processed) image are input into a single convolutional neural network or CNN (CNN 1), which can be used to examine all possible sub-images of a fixed size, and make a simultaneous decision about whether the point at the center of that sub-image is a POI, and if so, what the probability of each disease class is at that location in form of POI detection and disease class maps 942, 944. These maps can be used by a disease class aggregator module to produce disease class scores 952.--, in [0071]-[0078], and [0085]).
Re Claim 2, CHEN as modified by ASHOK and SETHI further disclose wherein the extracted features include morphological features (see CHEN: e.g., --A voting response matrix is created by processing each pixel that accumulates a vote through a voting kernel. The kernel is based on the gradient direction computed at that particular pixel and an expected range of minimum and maximum nucleus size and a voting kernel angle (typically in the range [π/4, π/8]). In the resulting voting space, local maxima locations that have a vote value higher than a predefined threshold value are saved out as seed points. Extraneous seeds may be discarded later during subsequent segmentation or classification processes--, in [0077]-[0079], and, --[0082] Feature Extraction [0083] Following detection of the nuclei, features (or metrics) are derived, such as with a feature extraction module 215. In general, the feature extraction module 215 receives image data, derives certain metrics based on the received image data, and outputs those derived metrics for combination with the separately computed probability features (step 305). In some embodiments, separate metrics are computed from the biomarker image and from the H&E image. [0084] More specifically, metrics may be derived from features of the identified nuclei or from within a patch surrounding an identified nucleus in both the H&E and biomarker images. For example, a feature metric can be a numerical data value being indicative of quantitative properties of a particular feature, a histogram, a distribution, or the like. In some embodiments, feature metrics are computed for each nucleus based on their visual properties and descriptors, e.g. morphology features, appearance features, background features, etc. In other embodiments, features are computed from within an image patch surrounding an identified nucleus. In some embodiments, the various feature metrics derived from the detected nuclei of the H&E and biomarker images are supplied as vectors of metrics and, together with metrics derived from the generated probability map (step 304 or 317), are supplied to the classification module 216 for classification (step 307 or 314).--, in [0082]-[0101]; and, -- [00106] (C) Metrics Derived from Background Features [00107] A "background feature" is, for example, a feature being indicative of the appearance and/or stain presence in cytoplasm and cell membrane features of the cell comprising the nucleus for which the background feature was extracted from the image. A background feature and a corresponding metrics can be computed for a nucleus and a corresponding cell depicted in a digital image e.g. by identifying a nuclear blob or seed representing the nucleus; analyzing a pixel area (e.g. a ribbon of 20 pixels - about 9 microns - thickness around the nuclear blob boundary) directly adjacent to the identified set of cells are computed in, therefore capturing appearance and stain presence in cytoplasm and membrane of the cell with this nucleus together with areas directly adjacent to the cell. These metrics are similar to the nuclear appearance features, but are computed in a ribbon of about 20 pixels (about 9 microns) thickness around each nucleus boundary, therefore capturing the appearance and stain presence in the cytoplasm and membrane of the cell having the identified nucleus together with areas directly adjacent to the cell. Without wishing to be bound by any particular theory, the ribbon size is selected because it is believed that it captures a sufficient amount of background tissue area around the nuclei that can be used to provide useful information for nuclei discrimination.--, in [0106]-[0107]; and, --The cooccurrence is computed of the pixel intensity at location (x;y) in Ci and the pixel intensity at location (x+dx; y+dy) in Cj. It is believed that the CCM offers that advantage of capturing the spatial relationship between different tissue structures (highlighted in different channels), without the need of explicitly segmenting them.--, in [0120]).
Re Claim 3, CHEN as modified by ASHOK and SETHI further disclose wherein the morphological features describe shape, texture, and color of cellular and/or sub-cellular components (see Chen: e.g., -- Nuclei detection based on radial symmetry voting is executed on color image intensity data and makes explicit use of the a priori domain knowledge that the nuclei are elliptical shaped blobs with varying sizes and eccentricities. To accomplish this, along with color intensities in the input image, image gradient information is also used in radial symmetry voting and combined with an adaptive segmentation process to precisely detect and localize the cell nuclei. A "gradient" as used herein is, for example, the intensity gradient of pixels calculated for a particular pixel by taking into consideration an intensity value gradient of a set of pixels surrounding said particular pixel. Each gradient may have a particular "orientation" relative to a coordinate system whose x- and y- axis are defined by two orthogonal edges of the digital image. For instance, nuclei seed detection involves defining a seed as a point which is assumed to lie inside a cell nucleus and serve as the starting point for localizing the cell nuclei. The first step is to detect seed points associated with each cell nuclei using a highly robust approach based on the radial symmetry to detect elliptical-shaped blobs, structures resembling cell nuclei. The radial symmetry approach operates on the gradient image using a kernel based voting procedure.--, in [0078]).
Re Claim 4, CHEN as modified by ASHOK and SETHI further disclose wherein the cellular and/or sub-cellular components include individual cells, mitotic figures, cell nucleus, vacuoles in the cytoplasm, extra cellular space, and nucleolus (see CHEN: e.g., --A voting response matrix is created by processing each pixel that accumulates a vote through a voting kernel. The kernel is based on the gradient direction computed at that particular pixel and an expected range of minimum and maximum nucleus size and a voting kernel angle (typically in the range [π/4, π/8]). In the resulting voting space, local maxima locations that have a vote value higher than a predefined threshold value are saved out as seed points. Extraneous seeds may be discarded later during subsequent segmentation or classification processes--, in [0077]-[0079], and, --[0082] Feature Extraction [0083] Following detection of the nuclei, features (or metrics) are derived, such as with a feature extraction module 215. In general, the feature extraction module 215 receives image data, derives certain metrics based on the received image data, and outputs those derived metrics for combination with the separately computed probability features (step 305). In some embodiments, separate metrics are computed from the biomarker image and from the H&E image. [0084] More specifically, metrics may be derived from features of the identified nuclei or from within a patch surrounding an identified nucleus in both the H&E and biomarker images. For example, a feature metric can be a numerical data value being indicative of quantitative properties of a particular feature, a histogram, a distribution, or the like. In some embodiments, feature metrics are computed for each nucleus based on their visual properties and descriptors, e.g. morphology features, appearance features, background features, etc. In other embodiments, features are computed from within an image patch surrounding an identified nucleus. In some embodiments, the various feature metrics derived from the detected nuclei of the H&E and biomarker images are supplied as vectors of metrics and, together with metrics derived from the generated probability map (step 304 or 317), are supplied to the classification module 216 for classification (step 307 or 314).--, in [0082]-[0101]; and, -- [00106] (C) Metrics Derived from Background Features [00107] A "background feature" is, for example, a feature being indicative of the appearance and/or stain presence in cytoplasm and cell membrane features of the cell comprising the nucleus for which the background feature was extracted from the image. A background feature and a corresponding metrics can be computed for a nucleus and a corresponding cell depicted in a digital image e.g. by identifying a nuclear blob or seed representing the nucleus; analyzing a pixel area (e.g. a ribbon of 20 pixels - about 9 microns - thickness around the nuclear blob boundary) directly adjacent to the identified set of cells are computed in, therefore capturing appearance and stain presence in cytoplasm and membrane of the cell with this nucleus together with areas directly adjacent to the cell. These metrics are similar to the nuclear appearance features, but are computed in a ribbon of about 20 pixels (about 9 microns) thickness around each nucleus boundary, therefore capturing the appearance and stain presence in the cytoplasm and membrane of the cell having the identified nucleus together with areas directly adjacent to the cell. Without wishing to be bound by any particular theory, the ribbon size is selected because it is believed that it captures a sufficient amount of background tissue area around the nuclei that can be used to provide useful information for nuclei discrimination.--, in [0106]-[0107]; also see SETHI: e.g., --pre-processing module reduces the memory requirements for further processing and improves the accuracy of tumor sub-type detection by selecting clinically relevant bands from the available spectral information at each pixel of an HSI. Each 20-band HSI pixel is given as input to a pre-trained hierarchical clustering based classifier to determine if the pixel belonged to colon epithelium, stroma, non-epithelial Globlet cells or background. The output of the hierarchical clustering based classifier is a tissue component classification map that assigns one of the four labels mentioned above to each pixel location. Noticeably, each tissue component in a patient tissue image serves as a POI in this example. The tissue component classification map is then used to extract features such as area, circumference, length of major and minor axis, parameters of best fit circle, ellipse or polyhedron, graph based features encoding the concordance of labels among neighboring pixels, etc. for each POI in a patient tissue image. These features are then inputted to a pre-trained penalized logistic regression based disease classifier that assigns probabilities to each POI belonging to one of the four sub-types viz. normal, dysplasia, hyperplasia, and carcinoma. This local disease classification at each POI is aggregated into a report that computes the percent of POIs for each of the four sub-types viz. normal, dysplasia, hyperplasia, and carcinoma. Based on the report generated by this embodiment of the present disclosure and pathologist's report based on visual inspection of the adjacent H&E stained tissue sample appropriate treatment selections can be made by an oncologist.--, in [0089]).
Re Claim 5, CHEN as modified by ASHOK and SETHI further disclose wherein the extracted features include colorimetric features and using RGB pixels is that these are "features" that describe the colors within a structured biologic element (see Chen: e.g., -- Nuclei detection based on radial symmetry voting is executed on color image intensity data and makes explicit use of the a priori domain knowledge that the nuclei are elliptical shaped blobs with varying sizes and eccentricities. To accomplish this, along with color intensities in the input image, image gradient information is also used in radial symmetry voting and combined with an adaptive segmentation process to precisely detect and localize the cell nuclei--, in [0078], --different image channels present in any of the input images (e.g. multiplex images comprising multiple signals corresponding to different biomarkers) must first be separated such as by color deconvolution (also referred to as "unmixing") to decompose the original RGB image into separate image channels…. in some types of image analysis, cell detection and classification can be done directly on the input RGB image or some other derived images (like HSV, CIELab) from the RGB image--, in [0148]).
Re Claim 6, CHEN as modified by ASHOK and SETHI further disclose wherein the untrained machine learnable device is a computer executing instructions for a neural network (see SETHI: e.g., -- (b) generating a pre-processed image from the images of the patient tissue; (c) locating one or more points of interest in the pre-processed image using a first classifier; (d) generating a disease spatial map with the probability of a disease state at the points-of-interest in the tissue image by using a second classifier; and (e) aggregating the probability of the disease condition at each point-of-interest located in step (c) to obtain the disease class scores of the disease state for the patient tissue. In further embodiments, the first classifier is a nucleus detector comprising one or more pre-trained neural networks, convolutional neural networks, or recurrent neural networks etc. In further embodiments, the probability of the disease state is aggregated from two or more images of the patient tissue from the same patient. In further embodiments, the method undergoes periodic or continuous validation. [0011] …cause the processor to: (a) input one or more images of the patient tissue, wherein the tissue has been treated with a stain comprising one or more stain components; (b) generate disease spatial maps with the probability of a disease state at one or more points-of-interest by using a combined classifier into one out of two or more disease classes--, in [0010]-[0012]; -- The objective of using a pre-processing module 203 is to take a magnified tissue image 160 and produce a pre-processed image 204 such that it is advantageous for the other modules such as points-of-interest detection 205 or local disease classification 207 to work with the pre-processed image 204 as compared to the magnified tissue image 160.--, in [0061], and, -- POI classifier training module 455 is a machine learning system or a classifier such as a convolutional neural network with associated parameters such as weights and hyper-parameters such as depth, filter sizes, etc. The sub-images of fixed window size from the training set 454 may be inputted into the convolutional neural network and their respective labels are the desired output. The convolutional neural network may be trained or optimized to minimize a cost function, such as cross-entropy between the correct labels and the neural network predictions by using an optimization method such as gradient descent. The objective of the training or optimization process is to reduce the difference between the classifier output and the known output (POI vs. non-POI status). With sufficient training using example input-output pairs, a machine learning based POI classifier training module learns to predict labels for the central pixel of input sub-image windows of pre-processed tissue image. For POI detection in new patient tissue images, a set of parameters and hyper-parameters 499 of the POI classifier training module 455 are saved in the memory of a computing unit. A validation data set similar to the training data set with pre-processed images along with their known POI locations may be used to ascertain the performance of the POI classifier. If the performance is unsatisfactory on the validation dataset, more training data may be collected, or the architecture (determined by hyper-parameters) of the classifier may be changed.--, and, -- one convolutional neural network. In some embodiments of the disease class scoring 105, the steps of pre-processing, detecting POIs, and computing a local disease classification may be done in a single module trained using multi-class label prediction techniques. For example, as shown in FIG. 9, the three color channels 912, 914, 916 of a color normalized (pre-processed) image are input into a single convolutional neural network or CNN (CNN 1), which can be used to examine all possible sub-images of a fixed size, and make a simultaneous decision about whether the point at the center of that sub-image is a POI, and if so, what the probability of each disease class is at that location in form of POI detection and disease class maps 942, 944. These maps can be used by a disease class aggregator module to produce disease class scores 952.--, in [0071]-[0078], and [0085]).
Re Claim 7, CHEN as modified by ASHOK and SETHI further disclose wherein the untrained machine learnable device is a computer executing instructions for a convolutional neural network (see SETHI: e.g., -- (b) generating a pre-processed image from the images of the patient tissue; (c) locating one or more points of interest in the pre-processed image using a first classifier; (d) generating a disease spatial map with the probability of a disease state at the points-of-interest in the tissue image by using a second classifier; and (e) aggregating the probability of the disease condition at each point-of-interest located in step (c) to obtain the disease class scores of the disease state for the patient tissue. In further embodiments, the first classifier is a nucleus detector comprising one or more pre-trained neural networks, convolutional neural networks, or recurrent neural networks etc. In further embodiments, the probability of the disease state is aggregated from two or more images of the patient tissue from the same patient. In further embodiments, the method undergoes periodic or continuous validation. [0011] …cause the processor to: (a) input one or more images of the patient tissue, wherein the tissue has been treated with a stain comprising one or more stain components; (b) generate disease spatial maps with the probability of a disease state at one or more points-of-interest by using a combined classifier into one out of two or more disease classes--, in [0010]-[0012]; -- The objective of using a pre-processing module 203 is to take a magnified tissue image 160 and produce a pre-processed image 204 such that it is advantageous for the other modules such as points-of-interest detection 205 or local disease classification 207 to work with the pre-processed image 204 as compared to the magnified tissue image 160.--, in [0061], and, -- POI classifier training module 455 is a machine learning system or a classifier such as a convolutional neural network with associated parameters such as weights and hyper-parameters such as depth, filter sizes, etc. The sub-images of fixed window size from the training set 454 may be inputted into the convolutional neural network and their respective labels are the desired output. The convolutional neural network may be trained or optimized to minimize a cost function, such as cross-entropy between the correct labels and the neural network predictions by using an optimization method such as gradient descent. The objective of the training or optimization process is to reduce the difference between the classifier output and the known output (POI vs. non-POI status). With sufficient training using example input-output pairs, a machine learning based POI classifier training module learns to predict labels for the central pixel of input sub-image windows of pre-processed tissue image. For POI detection in new patient tissue images, a set of parameters and hyper-parameters 499 of the POI classifier training module 455 are saved in the memory of a computing unit. A validation data set similar to the training data set with pre-processed images along with their known POI locations may be used to ascertain the performance of the POI classifier. If the performance is unsatisfactory on the validation dataset, more training data may be collected, or the architecture (determined by hyper-parameters) of the classifier may be changed.--, and, -- one convolutional neural network. In some embodiments of the disease class scoring 105, the steps of pre-processing, detecting POIs, and computing a local disease classification may be done in a single module trained using multi-class label prediction techniques. For example, as shown in FIG. 9, the three color channels 912, 914, 916 of a color normalized (pre-processed) image are input into a single convolutional neural network or CNN (CNN 1), which can be used to examine all possible sub-images of a fixed size, and make a simultaneous decision about whether the point at the center of that sub-image is a POI, and if so, what the probability of each disease class is at that location in form of POI detection and disease class maps 942, 944. These maps can be used by a disease class aggregator module to produce disease class scores 952.--, in [0071]-[0078], and [0085]).
Re Claim 8, CHEN as modified by ASHOK and SETHI further disclose wherein the convolutional neural network includes a plurality of convolutional layers and a plurality of pooling layers (see SETHI: e.g., -- (b) generating a pre-processed image from the images of the patient tissue; (c) locating one or more points of interest in the pre-processed image using a first classifier; (d) generating a disease spatial map with the probability of a disease state at the points-of-interest in the tissue image by using a second classifier; and (e) aggregating the probability of the disease condition at each point-of-interest located in step (c) to obtain the disease class scores of the disease state for the patient tissue. In further embodiments, the first classifier is a nucleus detector comprising one or more pre-trained neural networks, convolutional neural networks, or recurrent neural networks etc. In further embodiments, the probability of the disease state is aggregated from two or more images of the patient tissue from the same patient. In further embodiments, the method undergoes periodic or continuous validation. [0011] …cause the processor to: (a) input one or more images of the patient tissue, wherein the tissue has been treated with a stain comprising one or more stain components; (b) generate disease spatial maps with the probability of a disease state at one or more points-of-interest by using a combined classifier into one out of two or more disease classes--, in [0010]-[0012]; -- The objective of using a pre-processing module 203 is to take a magnified tissue image 160 and produce a pre-processed image 204 such that it is advantageous for the other modules such as points-of-interest detection 205 or local disease classification 207 to work with the pre-processed image 204 as compared to the magnified tissue image 160.--, in [0061], and, -- POI classifier training module 455 is a machine learning system or a classifier such as a convolutional neural network with associated parameters such as weights and hyper-parameters such as depth, filter sizes, etc. The sub-images of fixed window size from the training set 454 may be inputted into the convolutional neural network and their respective labels are the desired output. The convolutional neural network may be trained or optimized to minimize a cost function, such as cross-entropy between the correct labels and the neural network predictions by using an optimization method such as gradient descent. The objective of the training or optimization process is to reduce the difference between the classifier output and the known output (POI vs. non-POI status). With sufficient training using example input-output pairs, a machine learning based POI classifier training module learns to predict labels for the central pixel of input sub-image windows of pre-processed tissue image. For POI detection in new patient tissue images, a set of parameters and hyper-parameters 499 of the POI classifier training module 455 are saved in the memory of a computing unit. A validation data set similar to the training data set with pre-processed images along with their known POI locations may be used to ascertain the performance of the POI classifier. If the performance is unsatisfactory on the validation dataset, more training data may be collected, or the architecture (determined by hyper-parameters) of the classifier may be changed.--, and, -- one convolutional neural network. In some embodiments of the disease class scoring 105, the steps of pre-processing, detecting POIs, and computing a local disease classification may be done in a single module trained using multi-class label prediction techniques. For example, as shown in FIG. 9, the three color channels 912, 914, 916 of a color normalized (pre-processed) image are input into a single convolutional neural network or CNN (CNN 1), which can be used to examine all possible sub-images of a fixed size, and make a simultaneous decision about whether the point at the center of that sub-image is a POI, and if so, what the probability of each disease class is at that location in form of POI detection and disease class maps 942, 944. These maps can be used by a disease class aggregator module to produce disease class scores 952.--, in [0071]-[0078], and [0085]).
Re Claim 9, CHEN as modified by ASHOK and SETHI further disclose wherein the convolutional neural network further includes a global mean layer and a batch-normalization layer (see SETHI: e.g., -- (b) generating a pre-processed image from the images of the patient tissue; (c) locating one or more points of interest in the pre-processed image using a first classifier; (d) generating a disease spatial map with the probability of a disease state at the points-of-interest in the tissue image by using a second classifier; and (e) aggregating the probability of the disease condition at each point-of-interest located in step (c) to obtain the disease class scores of the disease state for the patient tissue. In further embodiments, the first classifier is a nucleus detector comprising one or more pre-trained neural networks, convolutional neural networks, or recurrent neural networks etc. In further embodiments, the probability of the disease state is aggregated from two or more images of the patient tissue from the same patient. In further embodiments, the method undergoes periodic or continuous validation. [0011] …cause the processor to: (a) input one or more images of the patient tissue, wherein the tissue has been treated with a stain comprising one or more stain components; (b) generate disease spatial maps with the probability of a disease state at one or more points-of-interest by using a combined classifier into one out of two or more disease classes--, in [0010]-[0012]; -- The objective of using a pre-processing module 203 is to take a magnified tissue image 160 and produce a pre-processed image 204 such that it is advantageous for the other modules such as points-of-interest detection 205 or local disease classification 207 to work with the pre-processed image 204 as compared to the magnified tissue image 160.--, in [0061], and, -- POI classifier training module 455 is a machine learning system or a classifier such as a convolutional neural network with associated parameters such as weights and hyper-parameters such as depth, filter sizes, etc. The sub-images of fixed window size from the training set 454 may be inputted into the convolutional neural network and their respective labels are the desired output. The convolutional neural network may be trained or optimized to minimize a cost function, such as cross-entropy between the correct labels and the neural network predictions by using an optimization method such as gradient descent. The objective of the training or optimization process is to reduce the difference between the classifier output and the known output (POI vs. non-POI status). With sufficient training using example input-output pairs, a machine learning based POI classifier training module learns to predict labels for the central pixel of input sub-image windows of pre-processed tissue image. For POI detection in new patient tissue images, a set of parameters and hyper-parameters 499 of the POI classifier training module 455 are saved in the memory of a computing unit. A validation data set similar to the training data set with pre-processed images along with their known POI locations may be used to ascertain the performance of the POI classifier. If the performance is unsatisfactory on the validation dataset, more training data may be collected, or the architecture (determined by hyper-parameters) of the classifier may be changed.--, and, -- one convolutional neural network. In some embodiments of the disease class scoring 105, the steps of pre-processing, detecting POIs, and computing a local disease classification may be done in a single module trained using multi-class label prediction techniques. For example, as shown in FIG. 9, the three color channels 912, 914, 916 of a color normalized (pre-processed) image are input into a single convolutional neural network or CNN (CNN 1), which can be used to examine all possible sub-images of a fixed size, and make a simultaneous decision about whether the point at the center of that sub-image is a POI, and if so, what the probability of each disease class is at that location in form of POI detection and disease class maps 942, 944. These maps can be used by a disease class aggregator module to produce disease class scores 952.--, in [0071]-[0078], and [0085]).
Re Claim 10, CHEN as modified by ASHOK and SETHI further disclose determining treatment for a subject from a subjects' predicted status for the status of the diagnostic feature and then treating the subject (see SETHI: e.g., -- The tissue component classification map is then used to extract features such as area, circumference, length of major and minor axis, parameters of best fit circle, ellipse or polyhedron, graph based features encoding the concordance of labels among neighboring pixels, etc. for each POI in a patient tissue image. These features are then inputted to a pre-trained penalized logistic regression based disease classifier that assigns probabilities to each POI belonging to one of the four sub-types viz. normal, dysplasia, hyperplasia, and carcinoma. This local disease classification at each POI is aggregated into a report that computes the percent of POIs for each of the four sub-types viz. normal, dysplasia, hyperplasia, and carcinoma. Based on the report generated by this embodiment of the present disclosure and pathologist's report based on visual inspection of the adjacent H&E stained tissue sample appropriate treatment selections can be made by an oncologist.--, in [0089]).
Re Claim 11, CHEN as modified by ASHOK and SETHI further disclose wherein the subject is treated with a chemotherapeutic agent See SETHI: e.g., --[0076] Many disease classes and POI types may be used consistently with the spirit of the disclosure. For example, molecular sub-types of cancer such as luminal, basal, and absence thereof can be used to plan specific treatments. Similarly, treatment outcomes determined using years of follow up after specific treatments can be used as classes such as “likely to metastasize after chemotherapy” vs. “unlikely to metastasize after chemotherapy” so that prognostic models that predict treatment effectiveness and disease course can be built. Additionally, patient survival endpoints determined by years of follow up after specific treatments can be used as classes such as “alive” vs. “dead” to develop survival outcome prediction models.--, in [0076]; and, --[0089] In this example, we describe how an embodiment of the present disclosure can be used to process microspectroscopic images of colon tissue for early stage tumor detection and sub-type identification. Most colorectal tumors can be classified into one of the four sub-types viz. normal, hyperplasia, dysplasia and carcinoma in their increasing order of aggressiveness. Numerous studies have reported that early detection, sub-type identification and treatment of an individual colon tumor can improve the survival outcome and patient's quality of life through appropriate treatment recommendations selected on the basis of tumor sub-type. Microspectroscopic images of tissue samples captured using Fourier Transform Infra-Red (FTIR) or Quantum Cascade Laser (QCL) scanners offer molecular specificity of vibration spectroscopy and spatial resolution of optical microscopy without altering the chemical composition of the sample as done by the conventional staining methods such as H&E or IHC staining. As a result, these methods are becoming increasingly popular for detection and assessment of various diseases including colon cancer.--, in [0088]-[0089]).
Re Claim 12, CHEN as modified by ASHOK and SETHI further disclose wherein the diagnostic feature, prognostic, or theragnostic is presence or absence of a biomarker (see CHEN: e.g., --performing an analysis of the registered image, including deriving features from the registered image; merging the features derived from the first image and features derived from the second image, wherein the features derived from one of the first image or the registered image include probability features; and classifying nuclei in one of the first or second images based on the merged features set.--, in [0004]-[0005], and, --(c) running a nucleus detection module and/or a feature extraction module to derive features from the first image; (d) running the nucleus detection module and/or the feature extraction module to derive features from the registered image; (e) running a segmentation module to identify tissues types and/or cell types from one of the first image or the registered image; (f) running a region map generation module on the identified tissue types and/or cell types such that a probability features map be generated; (g) running a feature extraction module to derive probability features from the probability features map; (h) running a classification module to classify nuclei in one of the first image or the registered image based on the features derived from the first image, the features derived from the registered image, and the derived probability features. In some embodiments, the method further comprises (i) running a scoring module to score the classified nuclei. In some embodiments, the image in which the nuclei are classified is the image comprising signals corresponding to biomarkers (e.g. a biomarker image).--, in [0008]-[0009], and, --[0082] Feature Extraction [0083] Following detection of the nuclei, features (or metrics) are derived, such as with a feature extraction module 215. In general, the feature extraction module 215 receives image data, derives certain metrics based on the received image data, and outputs those derived metrics for combination with the separately computed probability features (step 305). In some embodiments, separate metrics are computed from the biomarker image and from the H&E image. [0084] More specifically, metrics may be derived from features of the identified nuclei or from within a patch surrounding an identified nucleus in both the H&E and biomarker images. For example, a feature metric can be a numerical data value being indicative of quantitative properties of a particular feature, a histogram, a distribution, or the like. In some embodiments, feature metrics are computed for each nucleus based on their visual properties and descriptors, e.g. morphology features, appearance features, background features, etc. In other embodiments, features are computed from within an image patch surrounding an identified nucleus. In some embodiments, the various feature metrics derived from the detected nuclei of the H&E and biomarker images are supplied as vectors of metrics and, together with metrics derived from the generated probability map (step 304 or 317), are supplied to the classification module 216 for classification (step 307 or 314).--, in [0082]-[0101]).
Re Claim 13, CHEN as modified by ASHOK and SETHI further disclose wherein the stained tissue sample is a putative breast cancer sample (see Chen: e.g., ----, an H&E tissue slide is used for the initial primary diagnosis to detect, grade and stage cancer type for a particular tissue indication (breast, prostate, lung cancer etc.). IHC tissue slides, on the other hand, are typically used for cancer subtyping for prognostic and predictive purposes. The tissue morphology, i.e. tumorous glandular regions, cells and lymphocytes and lymphatic regions and stromal regions and cells are easily distinguishable in H&E tissue slide. The IHC tissue slides, stained with either a simplex or multiplex IHC chromogenic assay (DAB, Fast Red, Dual stained), are used to detect and quantify antigen/protein overexpression in the tumor, immune or vascular regions in the tissue. In the manual process to review and interpret IHC slides either under a microscope or on a digital read of a whole slide capture on a computer monitor, pathologists typically also review the corresponding regions in H&E tissue slide images for a better visual understanding of the tissue morphology and disambiguate tissue structures, that may be similar looking in the chromogenic IHC tissue slide.--, in [0015]-[0017]; also see ASHOK: e.g., -- The cell also in frequent embodiments is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer. Often, the sample is evaluated for the presence of HER 2 expression. In frequent embodiments, the cell is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) HER 2 expression, grade, lympho-vascular invasion, lymph node invasion, ductal carcinoma in situ, lobular carcinoma in situ, extra-nodal extension, positive surgical margins, LAPP, and/or MAPP. [0039] Also often, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) bladder cancer. In frequent embodiments, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) grade, lymph node invasion, squamous differentiation, glandular differentiation, and/or lymph invasion, LAPP, and/or MAPP. [0040] In certain embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer. In frequent embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer grade, LAPP, and/or MAPP.--, in [0038]-[0040]).
Re Claim 14, CHEN as modified by ASHOK and SETHI further disclose wherein the biomarker is selected from ER, HER2, PR, Ki67, and cytokeratin markers (see CHEN: e.g., --classifying nuclei in the input image corresponding to the biomarker image based on the merged features set. In some embodiments, the biomarker image comprises signals corresponding to a presence of at least one of an estrogen receptor (ER) marker, a progesterone receptor (PR) marker, a Ki-67 marker, or a HER2 marker. In some embodiments, the biomarker image comprises signals corresponding to a presence of a PD-L1 marker, CD3 marker or CD8 marker. In some embodiments, the computer-implemented method further comprises the step of scoring the classified nuclei.--, in [0004]-[0005], and [0014]-[0015]).
Re Claim 15, CHEN as modified by ASHOK and SETHI further disclose wherein the biomarker is ER with the predicted status being used to determine specific treatments (see CHEN: e.g., --classifying nuclei in the input image corresponding to the biomarker image based on the merged features set. In some embodiments, the biomarker image comprises signals corresponding to a presence of at least one of an estrogen receptor (ER) marker, a progesterone receptor (PR) marker, a Ki-67 marker, or a HER2 marker. In some embodiments, the biomarker image comprises signals corresponding to a presence of a PD-L1 marker, CD3 marker or CD8 marker. In some embodiments, the computer-implemented method further comprises the step of scoring the classified nuclei.--, in [0004]-[0005], and [0014]-[0015]; also see ASHOK: e.g., -- The cell also in frequent embodiments is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer. Often, the sample is evaluated for the presence of HER 2 expression. In frequent embodiments, the cell is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) HER 2 expression, grade, lympho-vascular invasion, lymph node invasion, ductal carcinoma in situ, lobular carcinoma in situ, extra-nodal extension, positive surgical margins, LAPP, and/or MAPP. [0039] Also often, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) bladder cancer. In frequent embodiments, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) grade, lymph node invasion, squamous differentiation, glandular differentiation, and/or lymph invasion, LAPP, and/or MAPP. [0040] In certain embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer. In frequent embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer grade, LAPP, and/or MAPP.--, in [0038]-[0040]; also --[0076] Many disease classes and POI types may be used consistently with the spirit of the disclosure. For example, molecular sub-types of cancer such as luminal, basal, and absence thereof can be used to plan specific treatments. Similarly, treatment outcomes determined using years of follow up after specific treatments can be used as classes such as “likely to metastasize after chemotherapy” vs. “unlikely to metastasize after chemotherapy” so that prognostic models that predict treatment effectiveness and disease course can be built. Additionally, patient survival endpoints determined by years of follow up after specific treatments can be used as classes such as “alive” vs. “dead” to develop survival outcome prediction models.--, in [0076]; and, --[0089] In this example, we describe how an embodiment of the present disclosure can be used to process microspectroscopic images of colon tissue for early stage tumor detection and sub-type identification. Most colorectal tumors can be classified into one of the four sub-types viz. normal, hyperplasia, dysplasia and carcinoma in their increasing order of aggressiveness. Numerous studies have reported that early detection, sub-type identification and treatment of an individual colon tumor can improve the survival outcome and patient's quality of life through appropriate treatment recommendations selected on the basis of tumor sub-type. Microspectroscopic images of tissue samples captured using Fourier Transform Infra-Red (FTIR) or Quantum Cascade Laser (QCL) scanners offer molecular specificity of vibration spectroscopy and spatial resolution of optical microscopy without altering the chemical composition of the sample as done by the conventional staining methods such as H&E or IHC staining. As a result, these methods are becoming increasingly popular for detection and assessment of various diseases including colon cancer.--, in [0088]-[0089]).
Re Claim 16, CHEN as modified by ASHOK and SETHI further disclose wherein the biomarker is ER, PR, and HER2 with the predicted status indicating prognosis (see Chen: e.g., ----, an H&E tissue slide is used for the initial primary diagnosis to detect, grade and stage cancer type for a particular tissue indication (breast, prostate, lung cancer etc.). IHC tissue slides, on the other hand, are typically used for cancer subtyping for prognostic and predictive purposes. The tissue morphology, i.e. tumorous glandular regions, cells and lymphocytes and lymphatic regions and stromal regions and cells are easily distinguishable in H&E tissue slide. The IHC tissue slides, stained with either a simplex or multiplex IHC chromogenic assay (DAB, Fast Red, Dual stained), are used to detect and quantify antigen/protein overexpression in the tumor, immune or vascular regions in the tissue. In the manual process to review and interpret IHC slides either under a microscope or on a digital read of a whole slide capture on a computer monitor, pathologists typically also review the corresponding regions in H&E tissue slide images for a better visual understanding of the tissue morphology and disambiguate tissue structures, that may be similar looking in the chromogenic IHC tissue slide.--, in [0015]-[0017]; also see ASHOK: e.g., -- The cell also in frequent embodiments is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer. Often, the sample is evaluated for the presence of HER 2 expression. In frequent embodiments, the cell is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) HER 2 expression, grade, lympho-vascular invasion, lymph node invasion, ductal carcinoma in situ, lobular carcinoma in situ, extra-nodal extension, positive surgical margins, LAPP, and/or MAPP. [0039] Also often, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) bladder cancer. In frequent embodiments, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) grade, lymph node invasion, squamous differentiation, glandular differentiation, and/or lymph invasion, LAPP, and/or MAPP. [0040] In certain embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer. In frequent embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer grade, LAPP, and/or MAPP.--, in [0004]-[0005], [0019], [0038]-[0040]).
Re Claim 17, CHEN as modified by ASHOK and SETHI further disclose wherein the biomarker is E-cadherin and PIK3CA with the predicted status being used to differentiate between subtypes of breast cancer (see Chen: e.g., ----, an H&E tissue slide is used for the initial primary diagnosis to detect, grade and stage cancer type for a particular tissue indication (breast, prostate, lung cancer etc.). IHC tissue slides, on the other hand, are typically used for cancer subtyping for prognostic and predictive purposes. The tissue morphology, i.e. tumorous glandular regions, cells and lymphocytes and lymphatic regions and stromal regions and cells are easily distinguishable in H&E tissue slide. The IHC tissue slides, stained with either a simplex or multiplex IHC chromogenic assay (DAB, Fast Red, Dual stained), are used to detect and quantify antigen/protein overexpression in the tumor, immune or vascular regions in the tissue. In the manual process to review and interpret IHC slides either under a microscope or on a digital read of a whole slide capture on a computer monitor, pathologists typically also review the corresponding regions in H&E tissue slide images for a better visual understanding of the tissue morphology and disambiguate tissue structures, that may be similar looking in the chromogenic IHC tissue slide.--, in [0015]-[0017]; also see ASHOK: e.g., -- The cell also in frequent embodiments is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer. Often, the sample is evaluated for the presence of HER 2 expression. In frequent embodiments, the cell is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) HER 2 expression, grade, lympho-vascular invasion, lymph node invasion, ductal carcinoma in situ, lobular carcinoma in situ, extra-nodal extension, positive surgical margins, LAPP, and/or MAPP. [0039] Also often, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) bladder cancer. In frequent embodiments, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) grade, lymph node invasion, squamous differentiation, glandular differentiation, and/or lymph invasion, LAPP, and/or MAPP. [0040] In certain embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer. In frequent embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer grade, LAPP, and/or MAPP.--, in [0004]-[0005], [0019], [0038]-[0040]).
Re Claim 18, CHEN as modified by ASHOK and SETHI further disclose wherein the stained tissue sample is a putative cancer sample (see Chen: e.g., -- an H&E tissue slide is used for the initial primary diagnosis to detect, grade and stage cancer type for a particular tissue indication (breast, prostate, lung cancer etc.). IHC tissue slides, on the other hand, are typically used for cancer subtyping for prognostic and predictive purposes. The tissue morphology, i.e. tumorous glandular regions, cells and lymphocytes and lymphatic regions and stromal regions and cells are easily distinguishable in H&E tissue slide. The IHC tissue slides, stained with either a simplex or multiplex IHC chromogenic assay (DAB, Fast Red, Dual stained), are used to detect and quantify antigen/protein overexpression in the tumor, immune or vascular regions in the tissue. In the manual process to review and interpret IHC slides either under a microscope or on a digital read of a whole slide capture on a computer monitor, pathologists typically also review the corresponding regions in H&E tissue slide images for a better visual understanding of the tissue morphology and disambiguate tissue structures, that may be similar looking in the chromogenic IHC tissue slide.--, in [0015]).
Re Claim 19, CHEN as modified by ASHOK and SETHI further disclose wherein the stained tissue sample is a putative lung cancer sample (see CHEN: e.g., --performing an analysis of the registered image, including deriving features from the registered image; merging the features derived from the first image and features derived from the second image, wherein the features derived from one of the first image or the registered image include probability features; and classifying nuclei in one of the first or second images based on the merged features set.--, in [0004]-[0005], and, --(c) running a nucleus detection module and/or a feature extraction module to derive features from the first image; (d) running the nucleus detection module and/or the feature extraction module to derive features from the registered image; (e) running a segmentation module to identify tissues types and/or cell types from one of the first image or the registered image; (f) running a region map generation module on the identified tissue types and/or cell types such that a probability features map be generated; (g) running a feature extraction module to derive probability features from the probability features map; (h) running a classification module to classify nuclei in one of the first image or the registered image based on the features derived from the first image, the features derived from the registered image, and the derived probability features. In some embodiments, the method further comprises (i) running a scoring module to score the classified nuclei. In some embodiments, the image in which the nuclei are classified is the image comprising signals corresponding to biomarkers (e.g. a biomarker image).--, in [0008]-[0009], and, --, an H&E tissue slide is used for the initial primary diagnosis to detect, grade and stage cancer type for a particular tissue indication (breast, prostate, lung cancer etc.). IHC tissue slides, on the other hand, are typically used for cancer subtyping for prognostic and predictive purposes. The tissue morphology, i.e. tumorous glandular regions, cells and lymphocytes and lymphatic regions and stromal regions and cells are easily distinguishable in H&E tissue slide. The IHC tissue slides, stained with either a simplex or multiplex IHC chromogenic assay (DAB, Fast Red, Dual stained), are used to detect and quantify antigen/protein overexpression in the tumor, immune or vascular regions in the tissue. In the manual process to review and interpret IHC slides either under a microscope or on a digital read of a whole slide capture on a computer monitor, pathologists typically also review the corresponding regions in H&E tissue slide images for a better visual understanding of the tissue morphology and disambiguate tissue structures, that may be similar looking in the chromogenic IHC tissue slide.--, in [0015]-[0017]).
Re Claim 20, CHEN as modified by ASHOK and SETHI further disclose wherein the biomarker is EGFR, KRAS, c-Met, MET, and ALK (see CHEN: e.g., -- the features derived from one of the first image or the registered image include probability features; and classifying nuclei in the input image corresponding to the biomarker image based on the merged features set. In some embodiments, the biomarker image comprises signals corresponding to a presence of at least one of an estrogen receptor (ER) marker, a progesterone receptor (PR) marker, a Ki-67 marker, or a HER2 marker. In some embodiments, the biomarker image comprises signals corresponding to a presence of a PD-L1 marker, CD3 marker or CD8 marker.--, in [0002]-[0005]; see SETHI: e.g., in [0088]; also see ASHOK: e.g., -- The cell also in frequent embodiments is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) breast cancer. Often, the sample is evaluated for the presence of HER 2 expression. In frequent embodiments, the cell is obtained from a breast sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) HER 2 expression, grade, lympho-vascular invasion, lymph node invasion, ductal carcinoma in situ, lobular carcinoma in situ, extra-nodal extension, positive surgical margins, LAPP, and/or MAPP. [0039] Also often, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) bladder cancer. In frequent embodiments, the cell is obtained from a bladder sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) grade, lymph node invasion, squamous differentiation, glandular differentiation, and/or lymph invasion, LAPP, and/or MAPP. [0040] In certain embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer. In frequent embodiments, the cell is obtained from a kidney sample and the prognostic indicator, expression profile, or pathology potential determination comprises (predicting) kidney cancer grade, LAPP, and/or MAPP.--, in [0004]-[0005], [0019], [0038]-[0040]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights