Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-16 AND 18-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Yip et al (Pub. No.: U.S. 2021/0256690 A1) in view of Chefd’hotel et al (U.S. Pub No: 2017/0098310 A1).
Regarding claim 1, Yip discloses a computer-implemented method for analyzing an image of a slide corresponding to a pathology specimen, the method comprising (see page 1, paragraph, [0002] the present disclosure relates to examining “digital images” to detect, quantify, and/or characterize cancer-related biomarker(s) and, more particularly, to detect, quantify, and/or characterize such biomarkers from analysis of one or more “histopathology slide” images. Also page 3, paragraph, [0019] a computer-implemented method of identifying biomarkers in “digital image” of a hematoxylin and eosin (H&E) stained slide of target tissue (pathology specimen), the method comprises: receiving the digital image to an image-based biomarker prediction system having one or more processors; performing an image tiling process, using the one or more processors, on the digital image by separating the digital image into a plurality of tile images, where each of the plurality of tile images contains a different portion of the digital image; applying, using the one or more processors, the plurality of tile images to a multiscale deep learning framework comprising one or more trained deep learning multiscale classifier models, each trained deep learning multiscale classifier models being trained to classify a different tissue classification for each tile image and determining a tissue classification for each of the plurality of tile images, using the multiscale deep learning framework; identifying, using the one or more processors, cells within the digital image using a trained cell segmentation model; and from the tissue classification determined for each tile image and from the identified cells within the digital image, identifying a predicted presence of one or more biomarkers associated with the digital image);
Determining using an artificial intelligence (AI) system, a salient region overlay for at least one digitized image of the pathology specimen (see page 3, paragraph, [0021] in accordance with another example, a computer-implemented method of identifying biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of target tissue, the method comprises: receiving the digital image to an image-based biomarker prediction system having one or more processors; separating, using the one or more processors, the digital image into a plurality of tile images, where each of the plurality of tile images contains a different portion of the digital image (salient image); applying, using the one or more processors, the plurality of tile images to a deep learning (AI), framework comprising one or more trained biomarker classification models, each trained biomarker classification model being trained to classify a different biomarker; predicting, using the one or more processors, a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models; from the predicted biomarker classifications of each of the tile images, determining a predicted presence of one or more biomarkers in the target tissue; and generating a report containing the digital image and a “digital overlay” visualizing the predicted presence of the one or more biomarkers.
Also, page 7, paragraph, [0094] biomarkers may be identified through any of the following models. Any models referenced herein may be implemented as artificial intelligence engines and may include gradient boosting models, random forest models, neural networks (NN), regression models, Naive Bayes models, or machine learning algorithms (MLA). A MLA or a NN may be trained from a training data set. In an exemplary prediction profile, a training data set may include imaging, pathology, clinical, and/or molecular reports and details of a patient, such as those curated from an EHR or genetic sequencing reports.
Also, page 9, paragraph, [0123] the imaging-based biomarker prediction system 102 is part of a comprehensive biomarker prediction, patient diagnosis, and patient treatment system. For example, the imaging-based biomarker prediction system 102 may be coupled to communicate predicted biomarker information, tumor prediction, and tumor state information to external systems, including a computer-based pathology lab/oncology system 118 that may receive a generated biomarker report including image “overlay mapping” and use the same for further diagnosing cancer state of the patient and for identifying matching therapies for use in treating the patient. The imaging-based biomarker prediction system 102 may further send generated reports to a computer system 120 of the patient's primary care provider and to a physician clinical records system 122 for data basing the patients report with previously generated reports on the patient and/or with databases of generated reports on other patients for use in future patient analyses, including deep learning analyses, such as those described herein.
Finally, page 17, paragraph, [0195] the controller 302 generates image tiles 407, accessing one or more tiling masks 409, and tile metadata 411, which the controller 302 feeds as inputs to the deep learning framework 402, for determining predicted biomarker and/or tumor status and metrics, which are then provided to an overlay report generator 404 for generating a biomarker and tumor report 406. Optionally, the report 406 may include an overlay of the histopathology image and further include biomarker scoring data, such as percentage TILs, in an example);
representing the salient region overlay by a by a binary segmentation of the at least one digitized image indicating if each pixel has a salient feature present (see page 12, paragraph, [0152] the cell segmentation model of the module may be configured as a three-class “semantic segmentation” FCN model developed by modifying a UNet classifier replacing a loss function with a cross-entropy function, focal loss function, or mean square error function to form a three-class segmentation model. Three-class nature of the FCN model means that the cell segmentation model 316 may be configured as a first pixel-level FCN model, that identifies and assigns each pixel of image data into a cell-subunit class: (i) cell interior, (ii) a cell border, or (iii) a cell exterior. This is provided by way of example. The segmentation size of the module model 316 may be determined based on the type of cell to be segmented. For both TILs biomarkers, for example, the model 316 may be configured to perform lymphocyte identification and segmentation using a three-class FCN model. For example, the cell segmentation model 316 may be configured to “classify pixels” in an image as corresponding to the (i) interior, (ii) border, or (iii) exterior of lymphocyte cell.
Also, page 23, paragraph, [0249] the overlay map generator 324 may display the digital overlays as transparent or opaque layers that cover the histopathology image, aligned such that the image location shown in the overlay and the histopathology image are in the same location on the display. The “overlay map” may have varying degrees of transparency. The degree of transparency may be adjustable by the user, in a dynamic reporting mode of the overlay map generator 324. The overlay map generator 326 may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class. The overlay map may be displayed as a “heatmap” showing different tissue classifications and having different pixel intensity levels that correspond to different biomarker status levels, e.g., in the TILs example, showing higher intensity pixels for tissue regions having higher predicted TILs status (higher %) and lower intensity pixels for tissue regions having lower predicted TILs status (lower %).
Finally, page 33, paragraph, [0371] FIGS. 16A-16F illustrate input histopathology images received by the imaging-based biomarker prediction system 1400, corresponding overlay maps generated by the system 1400 to predict the location of IHC PD-L1 biomarker, and corresponding IHC stained tissue images used as references to determine the accuracy of overlay maps. The IHC stained tissue images were obtained from a test cohort but were not applied to the system 1200 during model training. FIGS. 16A-16C illustrate a representative PD-L1 positive biomarker classification example. FIG. 16A displays an input H&E image; FIG. 16B displays a probability map overlaid on the H&E image: FIG. 16C displays a PD-L1 IHC stain for reference. FIGS. 16D-16F illustrate a representative PD-L1 negative biomarker classification example. FIG. 16D displays an input H&E image; FIG. 16E displays a probability map overlaid on the H&E image; and FIG. 16F displays a PD-L1 IHC stain for reference. The color bar indicates the predicted probability of the tumor PD-L1+ class);
suppressing, based on a diagnostic value for each pixel, one or more non-salient regions of the at least one digitized image (see page 3, paragraph, [0029] the method further comprises: for each H&E slide training image, performing a tile selection process that infers a class status for each tile image in the H&E slide training image; and based on inferred class status, “discarding tile image ”s not corresponding to a desired class, before performing the tile-based tissue classification analysis on each of the H&E slide training images, such that the tile-based tissue classification analysis is performed on only selected tile images of the H&E slide training image.
Also, paragraphs, [0281] in an example, also at process, the controller “removes these pixels” by converting the image to a grayscale image, passing the grayscale image through a Gaussian blur filter that mathematically adjusts the original grayscale value of each pixel to a blurred grayscale value to create a blurred image. Other filters may be used to blur the image. Then, for each pixel, the controller 302 subtracts the blurred grayscale value from the original grayscale value to create a difference grayscale value. In one example, if a difference grayscale value of a pixel is less than a user-defined threshold, it may indicate that the blur filter did not significantly alter the original grayscale value and the pixel in the original image was located in a blurred region. The difference grayscale values may be compared to a threshold to create a binary mask that indicates where the blurred regions are that may be designated as non-tissue regions. A mask may be a copy of an image, where the colors, RGB values, or other values in the pixels are adjusted to show the presence or absence of an object of a certain type to show the location of all objects of that type. For example, the binary mask may be generated by setting the binary value of each pixel to 0 if the pixel has a difference grayscale value less than a user-defined blur threshold and setting the binary value of each pixel to 1 if the pixel has a difference grayscale value higher than or equal to a user-defined blur threshold. The regions of the binary mask that have pixel binary values of 0 indicate blurred areas in the original image that may be designated as non-tissue.
and normalizing the salient region overlay to obtain a variable (see page 9, paragraph, [0117] FIG. 1, the imaging-based biomarker prediction system 102 includes an image pre-processing sub-system 114 that performs initial image processing to enhance image data for faster processing in training a machine learning framework and for performing biomarker prediction using a trained deep learning framework. In the illustrated example, the image pre-processing sub-system 114 performs a normalization process on received image data, including one or more of color normalization 114a, intensity normalization 114b, and imaging source normalization 114c, to compensate for and correct for differences in the received image data. While in some examples the imaging-based biomarker prediction system 102 receives medical images, in other examples the sub-system 114 is able to generate medical images, either from received histopathology slides or from other received images, such as generating composite histopathology images by aligning shifted histopathology images to compensate from vertical/horizontal shift. This image pre-processing allows a deep learning framework to more efficiently analyze images across large data sets (e.g., over 1000s, 10000s, to 100000s, to 1000000s of medical images), thereby resulting in faster training and faster analysis processing. Also, page 12, paragraph, [0144] in the illustrated example, images having tile-level labeling the pipeline 315 includes tissue detection processes and image tiling processes. These processes may be performed on all received imaged data, only on training image data, only on received image data for analysis, or some combination thereof. In some examples, the tissue detection process, for example, may be excluded to expedite deep learning training, image analysis, and/or biomarker prediction. Indeed, any of the processes of the controller 302 may be performed in a dedicated biomarker prediction system or distributed for performance by externally-connected systems. For example, a histopathology imaging system may be configured to perform normalization processes before sending image data to the biomarker prediction system. In some examples, the biomarker prediction system may communicate an executable normalization software package to the connected external systems that configures those systems to perform normalization or other pre-processing.
However, regarding claim 1, Yip discloses (page 25, paragraph, [0279] and [0281-0282], referring to “removes or filtering” different portion pixels value.
But does not explicitly state “suppressing, based on the value for each pixel”.
On the other hand, Chefd’hotel, in the same field of “generalizable and interpretable deep learning model for predicting microsatellite instability from histopathology slide images”, teaches page 2, paragraph, [0016] a mask can be, for example, an image consisting of one bit per pixel. Some areas of the mask having a particular bit, e.g. "0", are considered as masking bits and may be used for hiding, deleting or “suppressing” intensity values of pixels of other images onto which the mask is mapped. For example, using a thresholder saliency edge image as an edge mask may imply generating a mask in which all pixels whose saliency value is below a minimum value are represented as mask pixels which can be overlaid on the initial image 702 to filter out background noise signals.
Therefore, it would have been obvious to one having ordinary skill in the art at the time the invention was made to modify Yip invention according to the teaching of Chefd’hotel because to combine, “removes these pixels” by converting the image to a grayscale image, passing the grayscale image through a Gaussian blur filter that mathematically adjusts the original grayscale value of each pixel to a blurred grayscale value to create a blurred image”, that is taught by Yip with the further procedure, “an image consisting of one bit per pixel. Some areas of the mask having a particular bit, e.g. "0", are considered as masking bits for hiding, deleting or suppressing intensity values of pixels of other images onto which the mask is mapped”, taught by Chefd’hotel, which provides an improved method of analysis and visualization of salient features of histopathology slide images, which can easily be implemented in analyzing images.
Regarding claim 2, Yip discloses the computer-implemented method of claim 1, further comprising converting the salient region overlay into a tissue map (see page 23, paragraph, [0249] the overlay map generator 324 may display the digital overlays as transparent or opaque layers that cover the histopathology image, aligned such that the image location shown in the overlay and the histopathology image are in the same location on the display. The overlay map may have varying degrees of transparency. The degree of transparency may be adjustable by the user, in a dynamic reporting mode of the overlay map generator 324. The overlay map generator 326 may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class. The overlay map may be displayed as a heatmap showing different tissue classifications and having different pixel intensity levels that correspond to different biomarker status levels, e.g., in the TILs example, showing higher intensity pixels for tissue regions having higher predicted TILs status (higher %) and lower intensity pixels for tissue regions having lower predicted TILs status (lower %). Also, page 27, paragraph, [0305] the architecture 1200, e.g., in the tissue classifier module 306, saves this 3-dimensional probability data array, and the overlay map generator 324 “converts” the tissue class label probabilities for each small square tile into a tissue class overlay map. In an example, the overlay map generator 324 may compare the probabilities stored in each vector to determine the largest probability value associated with each small square tile. The tissue class label associated with that largest value may be assigned to that small square tile and only the assigned labels will be displayed in the tissue class overlay map).
Regarding claim 3, Yip discloses the computer-implemented method of claim 1, further comprising: detecting the salient feature using the AI system on the at least one digitized image to produce a tissue visualization (page 3, paragraph, [0021], the plurality of tile images to a deep learning framework comprising one or more trained biomarker classification models, each trained biomarker classification model being trained to classify a different biomarker; predicting, using the one or more processors, a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models; from the predicted biomarker classifications of each of the tile images, determining a predicted presence of one or more biomarkers in the target tissue; and generating a report containing the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers. Also, page 7, paragraph, [0094] biomarkers may be identified through any of the following models. Any models referenced herein may be implemented as artificial intelligence engines and may include gradient boosting models, random forest models, neural networks (NN), regression models, Naive Bayes models, or machine learning algorithms (MLA). A MLA or a NN may be trained from a training data set. In an exemplary prediction profile, a training data set may include imaging, pathology, clinical, and/or molecular reports and details of a patient, such as those curated from an EHR or genetic sequencing reports).
Regarding claim 4, Yip discloses the computer-implemented method of claim 1, wherein the at least one digitized image comprises related case information, patient information and information from a clinical system (page 9, paragraphs, [0123-0124] in some examples, the imaging-based biomarker prediction system 102 is part of a comprehensive biomarker prediction, patient diagnosis, and patient treatment system. For example, the imaging-based biomarker prediction system 102 may be coupled to communicate predicted biomarker information, tumor prediction, and tumor state information to external systems, including a computer-based pathology lab/oncology system 118 that may receive a generated biomarker report including image overlay mapping and use the same for further diagnosing cancer state of the patient and for identifying matching therapies for use in treating the patient. The imaging-based biomarker prediction system 102 may further send generated reports to a computer system 120 of the patient's primary care provider and to a physician clinical records system 122 for data basing the patients report with previously generated reports on the patient and/or with databases of generated reports on other patients for use in future patient analyses, including deep learning analyses, such as those described herein. To analyze the received histopathology image data and other data, the imaging-based biomarker prediction system 102 includes a deep learning framework 150 that implements various machine learning techniques to generate trained classifier models for image-based biomarker analysis from received training sets of image data or sets of image data and other patient information. With trained classifier models, the deep learning framework 150 is further used to analyze and diagnose the presence of image-based biomarkers in subsequent images collected from patients. In this manner, images and other data of previously treated and analyzed patients is utilized, through the trained models, to provide analysis and diagnosis capabilities for future patients).
Regarding claim 5, Yip discloses the computer-implemented method of claim 1, further comprising alerting a user when the salient region overlay is available (page 9, paragraph, [0123], the imaging-based biomarker prediction system is part of a comprehensive biomarker prediction, patient diagnosis, and patient treatment system. For example, the imaging-based biomarker prediction system may be coupled to communicate predicted biomarker information, tumor prediction, and tumor state information to external systems, including a computer-based pathology lab/oncology system that may receive a “generated biomarker report” including image overlay mapping and use the same for further diagnosing cancer state of the patient and for identifying matching therapies for use in treating the patient. The imaging-based biomarker prediction system 102 may further send generated “reports to a computer system of the patient's primary care provider and to a physician” clinical records system for data basing the patients report with previously generated reports on the patient and/or with databases of generated reports on other patients for use in future patient analyses, including deep learning analyses, such as those described herein. Also, page 23, paragraphs, [0248-0249] FIGS. 10A and 10B illustrate examples of a digital overlay maps created by the overlay map generator of system 300, for example. These overlay maps may be generated as static digital reports displayed to clinicians or as dynamic reports allowing user interaction through a graphical user interface (GUI). FIG. 10A illustrates a tissue class overlay map generated by the overlay map generator 324. FIG. 1013 illustrates a cell outer edge overlay map generated by the overlay map generator 324. In an example, the overlay map generator 324 may display the digital overlays as transparent or opaque layers that cover the histopathology image, aligned such that the image location shown in the overlay and the histopathology image are in the same location on the display. The overlay map may have varying degrees of transparency. The degree of transparency may be adjustable by the user, in a dynamic reporting mode of the overlay map generator. The overlay map generator may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class).
Regarding claim 7, Yip discloses the computer-implemented method of claim 1, further comprising: indicating, by the AI system, a diagnostic value for each pixel (see claim 1, also page 15, paragraph, [0177] when training a tile based deep learning network to predict a biomarker classification label for each tile utilizes a strongly supervised approach to generate biomarker labels to identify the HRD status (Positive or Negative) of individual cells. Single cell RNA sequencing may be used alone, or in combination with laser guided micro-dissection to extract one cell at a time, to achieve labels for each cell. In one example a cell segmentation model may be incorporated to first get the outline of the cells, then an artificial intelligence engine may classify the pixel values inside each of the cell contours according to biomarker status. In another example masks of the image may be generated where HRD positive cells are assigned a first value and HRD negative cells are assigned a second value. A single scale deep learning framework may then be trained using slides with masks to identify cells that express HRD).
Regarding claim 9, Yip discloses the he computer-implement method of claim 1, further comprising developing a pipeline to archive a plurality of processed images and/or prospective patient data (see page 12, paragraphs, [0143-0146] the image discriminator determines which images are to be provided to a slide-level label pipeline 313 for feeding into the deep learning framework single-scale classifier module and which images are to be provided to a tile-level label pipeline 315 for feeding into the deep learning framework multiscale classifier. In examples in which the image discriminator 314 sends unlabeled images to the pipeline 315, the pipeline 315 includes a multiple instance learning (MIL) controller, discussed further herein, configured to convert all or portions of these histopathology images to tile-labeled images. to expedite tissue detection of the trained tissue classifier, the tissue detection process of the pipeline 315 may perform initial tissue identification, to locate and segment the tissue regions of interest for biomarker analysis. Such issue tissue identification may include, for example, identifying tissue boundaries and segmenting an image into tissue and non-tissue regions, so that metadata identifying the tissue regions is stored with the image data to expedite processing and prevent biomarker analysis attempts on non-tissue regions or on regions not corresponding to the tissue to be examined).
Regarding claim 11, Yip discloses the system of claim 10, further comprising converting the salient region overlay into a tissue map (see page 23, paragraph, [0249] the overlay map generator 324 may display the digital overlays as transparent or opaque layers that cover the histopathology image, aligned such that the image location shown in the overlay and the histopathology image are in the same location on the display. The overlay map may have varying degrees of transparency. The degree of transparency may be adjustable by the user, in a dynamic reporting mode of the overlay map generator 324. The overlay map generator 326 may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class. The overlay map may be displayed as a heatmap showing different tissue classifications and having different pixel intensity levels that correspond to different biomarker status levels, e.g., in the TILs example, showing higher intensity pixels for tissue regions having higher predicted TILs status (higher %) and lower intensity pixels for tissue regions having lower predicted TILs status (lower %). Also, page 27, paragraph, [0305] the architecture 1200, e.g., in the tissue classifier module 306, saves this 3-dimensional probability data array, and the overlay map generator 324 “converts” the tissue class label probabilities for each small square tile into a tissue class overlay map. In an example, the overlay map generator 324 may compare the probabilities stored in each vector to determine the largest probability value associated with each small square tile. The tissue class label associated with that largest value may be assigned to that small square tile and only the assigned labels will be displayed in the tissue class overlay map).
Regarding claim 12, Yip discloses the system of claim 10, further comprising: detecting the salient feature using the AI system on the at least one digitized image to produce a tissue visualization (page 3, paragraph, [0021], the plurality of tile images to a deep learning framework comprising one or more trained biomarker classification models, each trained biomarker classification model being trained to classify a different biomarker; predicting, using the one or more processors, a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models; from the predicted biomarker classifications of each of the tile images, determining a predicted presence of one or more biomarkers in the target tissue; and generating a report containing the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers. Also, page 7, paragraph, [0094] biomarkers may be identified through any of the following models. Any models referenced herein may be implemented as artificial intelligence engines and may include gradient boosting models, random forest models, neural networks (NN), regression models, Naive Bayes models, or machine learning algorithms (MLA). A MLA or a NN may be trained from a training data set. In an exemplary prediction profile, a training data set may include imaging, pathology, clinical, and/or molecular reports and details of a patient, such as those curated from an EHR or genetic sequencing reports).
Regarding claim 13, Yip discloses the system of claim 10, wherein the at least one digitized image comprises related case information, patient information and information from a clinical system (page 9, paragraphs, [0123-0124] in some examples, the imaging-based biomarker prediction system 102 is part of a comprehensive biomarker prediction, patient diagnosis, and patient treatment system. For example, the imaging-based biomarker prediction system 102 may be coupled to communicate predicted biomarker information, tumor prediction, and tumor state information to external systems, including a computer-based pathology lab/oncology system 118 that may receive a generated biomarker report including image overlay mapping and use the same for further diagnosing cancer state of the patient and for identifying matching therapies for use in treating the patient. The imaging-based biomarker prediction system 102 may further send generated reports to a computer system 120 of the patient's primary care provider and to a physician clinical records system 122 for data basing the patients report with previously generated reports on the patient and/or with databases of generated reports on other patients for use in future patient analyses, including deep learning analyses, such as those described herein. To analyze the received histopathology image data and other data, the imaging-based biomarker prediction system 102 includes a deep learning framework 150 that implements various machine learning techniques to generate trained classifier models for image-based biomarker analysis from received training sets of image data or sets of image data and other patient information. With trained classifier models, the deep learning framework 150 is further used to analyze and diagnose the presence of image-based biomarkers in subsequent images collected from patients. In this manner, images and other data of previously treated and analyzed patients is utilized, through the trained models, to provide analysis and diagnosis capabilities for future patients).
Regarding claim 14, Yip discloses the system of claim 10, further comprising alerting a user when the salient region overlay is available (page 9, paragraph, [0123], the imaging-based biomarker prediction system is part of a comprehensive biomarker prediction, patient diagnosis, and patient treatment system. For example, the imaging-based biomarker prediction system may be coupled to communicate predicted biomarker information, tumor prediction, and tumor state information to external systems, including a computer-based pathology lab/oncology system that may receive a “generated biomarker report” including image overlay mapping and use the same for further diagnosing cancer state of the patient and for identifying matching therapies for use in treating the patient. The imaging-based biomarker prediction system 102 may further send generated “reports to a computer system of the patient's primary care provider and to a physician” clinical records system for data basing the patients report with previously generated reports on the patient and/or with databases of generated reports on other patients for use in future patient analyses, including deep learning analyses, such as those described herein. Also, page 23, paragraphs, [0248-0249] FIGS. 10A and 10B illustrate examples of a digital overlay maps created by the overlay map generator of system 300, for example. These overlay maps may be generated as static digital reports displayed to clinicians or as dynamic reports allowing user interaction through a graphical user interface (GUI). FIG. 10A illustrates a tissue class overlay map generated by the overlay map generator 324. FIG. 1013 illustrates a cell outer edge overlay map generated by the overlay map generator 324. In an example, the overlay map generator 324 may display the digital overlays as transparent or opaque layers that cover the histopathology image, aligned such that the image location shown in the overlay and the histopathology image are in the same location on the display. The overlay map may have varying degrees of transparency. The degree of transparency may be adjustable by the user, in a dynamic reporting mode of the overlay map generator. The overlay map generator may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class).
Regarding claim 16, Yip discloses the system of claim 10, further comprising: indicating, by the AI system, a diagnostic value for each pixel (see claim 1, also page 15, paragraph, [0177] when training a tile based deep learning network to predict a biomarker classification label for each tile utilizes a strongly supervised approach to generate biomarker labels to identify the HRD status (Positive or Negative) of individual cells. Single cell RNA sequencing may be used alone, or in combination with laser guided micro-dissection to extract one cell at a time, to achieve labels for each cell. In one example a cell segmentation model may be incorporated to first get the outline of the cells, then an artificial intelligence engine may classify the pixel values inside each of the cell contours according to biomarker status. In another example masks of the image may be generated where HRD positive cells are assigned a first value and HRD negative cells are assigned a second value. A single scale deep learning framework may then be trained using slides with masks to identify cells that express HRD).
Regarding claim 18, Yip discloses the system of claim 10, further comprising developing a pipeline to archive a plurality of processed images and/or prospective patient data (see page 12, paragraphs, [0143-0146] the image discriminator determines which images are to be provided to a slide-level label pipeline 313 for feeding into the deep learning framework single-scale classifier module and which images are to be provided to a tile-level label pipeline 315 for feeding into the deep learning framework multiscale classifier. In examples in which the image discriminator 314 sends unlabeled images to the pipeline 315, the pipeline 315 includes a multiple instance learning (MIL) controller, discussed further herein, configured to convert all or portions of these histopathology images to tile-labeled images. to expedite tissue detection of the trained tissue classifier, the tissue detection process of the pipeline 315 may perform initial tissue identification, to locate and segment the tissue regions of interest for biomarker analysis. Such issue tissue identification may include, for example, identifying tissue boundaries and segmenting an image into tissue and non-tissue regions, so that metadata identifying the tissue regions is stored with the image data to expedite processing and prevent biomarker analysis attempts on non-tissue regions or on regions not corresponding to the tissue to be examined).
Regarding claim 20, Yip discloses the non-transitory computer readable medium of claim 19, further comprising converting the salient region overlay into a tissue map (see page 23, paragraph, [0249] the overlay map generator 324 may display the digital overlays as transparent or opaque layers that cover the histopathology image, aligned such that the image location shown in the overlay and the histopathology image are in the same location on the display. The overlay map may have varying degrees of transparency. The degree of transparency may be adjustable by the user, in a dynamic reporting mode of the overlay map generator 324. The overlay map generator 326 may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class. The overlay map may be displayed as a heatmap showing different tissue classifications and having different pixel intensity levels that correspond to different biomarker status levels, e.g., in the TILs example, showing higher intensity pixels for tissue regions having higher predicted TILs status (higher %) and lower intensity pixels for tissue regions having lower predicted TILs status (lower %). Also, page 27, paragraph, [0305] the architecture 1200, e.g., in the tissue classifier module 306, saves this 3-dimensional probability data array, and the overlay map generator 324 “converts” the tissue class label probabilities for each small square tile into a tissue class overlay map. In an example, the overlay map generator 324 may compare the probabilities stored in each vector to determine the largest probability value associated with each small square tile. The tissue class label associated with that largest value may be assigned to that small square tile and only the assigned labels will be displayed in the tissue class overlay map).
With regard to claims 6, 10, 15 and 19 the arguments analogous to those presented above for claims 1, 2, 3, 4, 5, 7, 9, 11, 12, 13, 14, 16, 18 and 20 are respectively applicable to claims 6, 10, 15 and 19.
Allowable Subject Matter
Claims 8 and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Seyed Azarian whose telephone number is (571) 272-7443. The examiner can normally be reached on Monday through Thursday from 6:00 a.m. to 7:30 p.m.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Matthew Bella, can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information about the PAIR system, see http:// pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/SEYED H AZARIAN/Primary Examiner, Art Unit 2667
June 8, 2026
Prosecution Insights