DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/28/2026 has been entered.
Response to Arguments
Applicant’s arguments (see remarks), filed 05/28/2026, with respect to the claims 1-4, 6-12, 15-17, 29, and 43 have been fully considered but respectfully, are not persuasive.
The applicant argues on page 10, “The Office Action contends that Weisenfeld teaches receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of a physically expanded sample of a tissue or a cell specimen. This reference mentions the concept of "expansion microscopy" because it discusses "a preparative step of expansion microscopy." (Weisenfeld at [0152]). However, this reference fails to disclose receiving and processing a physically expanded sample as recited in claim 1.”
In response, the Office respectfully disagrees. Based on the breadth of the claim language, the prior art by WEISENFELD et al. (US 20210150707 A1) explicitly teaches receiving, with at least one processor (Fig. 11. Paragraph [0532]-WEISENFELD discloses FIG. 11 is a block diagram illustrating a system for tissue classification. The system 1100 includes one or more processing units CPU(s) 1102. Please also read paragraph [0535-0539]), image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen (Fig. 11. Paragraph [0504]-WEISENFELD discloses a biological sample can have regions that show morphological feature(s) that may indicate the presence of disease or the development of a disease phenotype. In paragraph [0506]-WEISENFELD discloses [0506] A region of interest can be identified in a biological sample using a variety of different techniques, e.g., expansion microscopy, bright field microscopy, dark field microscopy, phase contrast microscopy, electron microscopy, fluorescence microscopy, reflection microscopy, interference microscopy, and confocal microscopy, and combinations thereof. For example, the staining and imaging of a biological sample can be performed to identify a region of interest. Please also read paragraph [0154-0158, 0473-0475, 0529, 0554, 0591-0592, and 0600-0617]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-8, 10, 12, 15-17, 29, and 43 are rejected under 35 U.S.C. 103 as being unpatentable over YIP et al. (US 20200258223 A1), hereinafter referenced as YIP in view of WEISENFELD et al. (US 20210150707 A1), hereinafter referenced as WEISENFELD, and in further view of YANG et al. (US 20160335747 A1), hereinafter referenced as YANG and in further view of SULLIVAN et al. (US 20220180964 A1), hereinafter referenced as SULLIVAN.
Regarding claim 1, YIP explicitly teaches a computer implemented method (Fig. 1. Paragraph [0111]-YIP discloses FIG. 1 illustrates a prediction system 100 capable of analyzing digital images of histopathology slides of a tissue sample and determining the likelihood of biomarker presence in that tissue. In paragraph [0138]-YIP discloses FIG. 3 illustrates an implementation of the imaging-based biomarker prediction system 102, and more particularly, of the deep learning framework 150 (wherein the framework 300 includes a pre-processing controller 302, a deep learning framework cell segmentation module 304, a deep learning framework multiscale classifier module 306, a deep learning framework single-scale classifier module 307, and a deep learning post-processing controller 308). In paragraph [0411]-YIP discloses FIG. 38 illustrates an example computing device 3800 for implementing the imaging-based biomarker prediction system 100 of FIG. 1) comprising:
segmenting (Fig. 1. Paragraph [0130]-YIP discloses a histopathology image may be segmented. In paragraph [0131]-YIP discloses in system 100, the deep learning framework 150 further includes a trained image classifier module 170. In paragraph [0133]-YIP discloses the module 170 may further include a cell segmenter 176 that identifies cells within a histopathology image, including cell borders, interiors, and exteriors. Please also see Fig. 3 and read paragraph [0151, 0155-0156 and 0176]), with the at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111 and 0113]), the at least one image to define a plurality of single-cell images (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images. The biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. In paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect single-cell analysis data or detection of cellular products. Please also see Fig. 3 and read paragraph [0176, 0207 and 0292-0293]);
YIP fails to explicitly teach receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
However, WEISENFELD explicitly teaches receiving, with at least one processor (Fig. 11. Paragraph [0532]-WEISENFELD discloses FIG. 11 is a block diagram illustrating a system for tissue classification. The system 1100 includes one or more processing units CPU(s) 1102. Please also read paragraph [0535-0539]), image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen (Fig. 11. Paragraph [0504]-WEISENFELD discloses a biological sample can have regions that show morphological feature(s) that may indicate the presence of disease or the development of a disease phenotype. In paragraph [0506]-WEISENFELD discloses a region of interest can be identified in a biological sample using a variety of different techniques, e.g., expansion microscopy. Please also read paragraph [0154-0158, 0473-0475, 0529, 0554, 0591-0592, and 0600-0617]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP of having a computer implemented method, with the teachings of WEISENFELD of having receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
Wherein YIP’s method having receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
The motivation behind the modification would have been to obtain a method that improves machine learning model training, accuracy and classifications as well as the resolution for spatial analysis, since both YIP and WEISENFELD concern cellular image analysis. Wherein YIP’s systems and methods provides improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while WEISENFELD’s systems and methods that improves the capture of analytes and resolution for spatial analysis. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and WEISENFELD et al. (US 20210150707 A1), Abstract and Paragraph [0496, 0515, and 0564].
Although YIP explicitly teaches applying, with the at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0113]), a filter to at least one single-cell image of the plurality of single-cell images, resulting in a plurality of images filtered having a plurality of different second resolutions than the first resolution (Fig. 3. Paragraph [0118]-YIP explicitly teaches in multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, at a first image resolution, downsampling that image to a second image resolution, and then performing a normalization on the downsampled histopathology image, such as color and/or intensity normalization, and removing non-tissue objects from the image (wherein multiple versions of images are created by applying a 10-layer format with 2x downsampling at each layer and applying various types of filters to remove artifacts and improve the quality of each image). Please also read paragraph [0264-0272 and 0309-0310] (wherein the process of 2x upsampling is discussed but not implemented)),
YIP fails to explicitly teach applying, with the at least one processor, a filter to at least one image of the plurality of images by iteratively decreasing a kernel size of the filter iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image; and
However, YANG explicitly teaches applying, with the at least one processor, a filter to at least one image (Fig. 3D, #102 called input data. Paragraph [0032]) of the plurality of images (Fig. 3D. Paragraph [0032]-YANG discloses FIG. 3D is a flow diagram of an example methodology 200 for removing blur from an image. The method 200 may be performed, for example, in whole or in part by the computing device 1000 of FIG. 6. The method 200 begins by receiving 102 input data representing a blurry digital image and applying 104 a variable scale filter to the blurry digital image at an original resolution to obtain a set of filtered observations at different scale levels. The variable scale filter may include a set of Gaussian noise filters with decreasing radius, or a set of directional noise filters) by iteratively decreasing a kernel size of the filter (Fig. 3D. Paragraph [0032]-YANG discloses the variable scale filter may include a set of Gaussian noise filters with decreasing radius, or a set of directional noise filters), resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image (Fig. 3. Paragraph [0032]-YANG discloses the variable scale filter includes a first filter and a second filter that is different from the first filter. The method 200 includes applying 106a the first filter to the blurry digital image at the original resolution to obtain a first observation at a first scale level, and applying 106b the second filter to the blurry digital image at the original resolution to obtain a second observation at a second scale level that is smaller than the first scale level. The applying step 106b may be repeated at iteratively smaller scale levels to obtain additional filtered observations of the blurry image. The estimated blur kernel may become accurate enough for non-blind deblurring before reaching the finest resolution level).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD of having a computer implemented method comprising: receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of a physically expanded sample of a tissue or a cell specimen, with the teachings of YANG of having applying, with the at least one processor, a filter to at least one image of the plurality of images by iteratively decreasing a kernel size of the filter iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image.
Wherein YIP’s method having applying, with the at least one processor, a filter to at least one single-cell image of the plurality of single-cell images by iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one single-cell image; and
The motivation behind the modification would have been to obtain a method that improves machine learning model training, accuracy and classifications as well as the blur removal, since both YIP and YANG concern image analysis and gaussian filters. Wherein YIP’s systems and methods improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while YANG’s systems and methods that improves the ability to remove blur from a single image by accumulating a blur kernel. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and YANG et al. (US 20160335747 A1), Abstract and Paragraph [0017].
Although YIP explicitly teaches training, with the at least one processor, a machine learning model to predict a classification of single-cell images (Fig. 3. Paragraph [0123]-YIP discloses to analyze the received histopathology image data and other data, the imaging-based biomarker prediction system 102 includes a deep learning framework 150 that implements various machine learning techniques to generate trained classifier models for image-based biomarker analysis from received training sets of image data or sets of image data and other patient information. In paragraph [0124]-YIP discloses by labeling the image data 162a according to associations with the other data types, the imaging-based biomarker prediction system may train an image classifier module to predict the one or more different data types from image data 162a. Please also read paragraph [0115, 0138, 0176, 0190-0199 and 0293-0294]) based on multiple fine-tuning iterations (Fig. 3. Paragraph [0377]-YIP discloses training may be performed with weakly supervised learning. This process may be repeated many times, given enough collections and tiles as input to the neural network, it will learn to differentiate tiles with different classes with higher accuracy as more iterations are performed. Please also read paragraph [0264-0265, 0273]).
YIP is silent on wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
However, SULLIVAN explicitly teaches wherein each iteration of the multiple fine-tuning iterations (Fig. 2. Paragraph [0132]-SULLIVAN discloses the neural network model can be iteratively trained based on the input and by comparing the output to variants and variants with significance, to generate a trained neural network model. At a verification stage and/or execution stage, the trained neural network model can then be executed to generate an estimate output that closely anticipates the variants and/or variants with significance of samples and/or contact matrices) is based on at least one image of the plurality of images (Fig. 2. Paragraph [0127]-SULLIVAN discloses the data preparation module 310 can normalize the sequencing reads or contact matrix from the sample or set of samples to a common format and/or a common scale. The preparation module 310 can normalize a set of images representing the information from the sample or set of samples to a common image size of 256 pixels by 256 pixels and to a common image file format of Tagged Image File Format (TIFF). In paragraph [0129]-SULLIVAN discloses the karyotyping by sequencing variant detector 315 recursively uses the first machine learning model 316 (e.g., a CNN model), creating increasing resolution contact matrixes between classification steps, to precisely identify a set of structural variants of the desired resolution. Each structural variant from the set of structural variants are then classified using the second machine learning model 321 (e.g., a KNN model) of the karyotyping by sequencing variant analyzer 320 to predict a set of clinical significance and/or biological significance of the set of structural variants. In paragraph [0133]-SULLIVAN discloses the CNN captures relationships in a contact matrix by the application of a series of convolutional filters of various dimensions. Please also read paragraph [0088, 0115 and 0131]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD and in further view of YANG of having a computer implemented method comprising: receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of a physically expanded sample of a tissue or a cell specimen, with the teachings of SULLIVAN of having wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
Wherein YIP’s method having training, with the at least one processor, a machine learning model to predict a classification of single-cell images based on multiple fine-tuning iterations, wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
The motivation behind the modification would have been to obtain a method that improves machine learning model training, accuracy and classifications, since both YIP and SULLIVAN concern image analysis and cellular images. Wherein YIP’s systems and methods improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while SULLIVAN’s systems and methods improve model training and performance. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and SULLIVAN et al. (US 20220180964 A1), Abstract and Paragraph [0591-0600].
Regarding claim 2, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches wherein the at least one image comprises an image of a plurality of cells (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images (wherein medical images may include histopathology slides such as digital H&E stained slide images, IHC stained slide images, or digital images of any other staining protocols)), and wherein segmenting the at least one image to define the plurality of single-cell images (Fig. 1. Paragraph [0130]-YIP discloses a histopathology image may be segmented. In paragraph [0131]-YIP discloses in system 100, the deep learning framework 150 further includes a trained image classifier module 170. In paragraph [0133]-YIP discloses the module 170 may further include a cell segmenter 176 that identifies cells within a histopathology image, including cell borders, interiors, and exteriors. Please also see Fig. 3 and read paragraph [0151, 0155-0156 and 0176]) comprises:
identifying a location of each cell of the plurality of cells of the at least one image based on a pixel coordinate of each cell of the plurality of cells of the at least one image (Fig. 3. Paragraph [0212]-YIP discloses the process 712 may access a stored image processing library and use that library to find contours around the cell interior class. The process 712 may perform a cell registration process. The cell border class (denoted by locations with a 1 value in each mask) ensures separation between neighboring cell interiors. This generates a list of every contour on each mask. The process 712 determines the coordinates of the contour's centroid (center of mass), from which the process 712 produces a centroid list. Next, to generate outputs that are in the coordinate space defined by the entire received image instead of the coordinate space that is specific to a single tile in the image, each coordinate in the contour lists and the centroid lists is shifted. The process 714 performs the same processes as the process 712, but on the lymphocyte classes. In paragraph [0217]-YIP discloses the process 716 has the coordinates for each cell centroid, the coordinates for the top-left corner of each tissue classification tile, and the size of each tissue classification tile, and is configured to determine the parent tile for each cell based on its centroid location. Please also read paragraph [0112 and 0152]); and
defining for each of the plurality of cells, a single-cell image based on the location of a cell of the plurality of cells (Fig. 3. Paragraph [0152]-YIP discloses the module 304 receives tiled, sub-images from the pipeline 315, and the cell segmentation model 316 determines the list of locations of all lymphocytes, and those locations are compared to the other three class model's list of all cells determined from the model 316. In paragraph [0155]-YIP discloses a UNet model can recognize the outer edges of many types of cells and may classify each cell according to cell shape or its location within a tissue class region assigned by the tissue classification module 320. Further in paragraph [0217]-YIP discloses the process 716, each cell is binned into one of the tissue classification tiles (from process 706) based on location. Please also read paragraph [0166 and 0188]).
wherein the single-cell image comprises the cell of the plurality of cells and a microenvironment of the cell of the plurality of cells (Fig. 3. Paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect various types of data, such as single-cell analysis data or detection of cellular products (including proteins, lipids, and other molecules) indicating the presence of specific cell populations, including effector data, stimulatory data, regulatory data, inflammatory data, chemoattractive data (wherein imaging metrics and analysis may include cell shape, cell area, cell perimeter, cell convex area ratio, cell circularity, cell convex perimeter area, cell length, lymphocyte %, cellular characteristics, cell textures, the clustering of cell types of tissue classes based on spacing and density of classified cells, the spacing and distance of tissue class classified tiles, probability of neighboring cell types, biomarkers such as HRD status, DNA ploidy scores, karyotypes, CMS scores, chromosomal instability (CIN) status, signet ring morphology scores, NC ratios, cellular pathway activation status, and any visually detectable features. Please also see Fig. 1 and read paragraph [0090, 0107, 0176, 0188, 0207 and 0292-0293]).
Regarding claim 3, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 2, YIP further teaches wherein the image of the plurality of cells (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images. The biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. In paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect single-cell analysis data or detection of cellular products. Please also see Fig. 3 and read paragraph [0176, 0207 and 0292-0293]) comprises a plurality of nuclei (Fig. 3. Paragraph [0151-YIP discloses the cell segmentation model 316 may be configured as a first pixel-level FCN model, that identifies and assigns each pixel of image data into a cell-subunit class: (i) cell interior, (ii) a cell border, or (iii) a cell exterior. In paragraph [0166]-YIP discloses the cell segmentation model 316 may be trained to analyze an input image and assign one of the three classes to each pixel, define cells as a group of adjacent nucleus pixels and all cytoplasm pixels between the nucleus pixels and the next nearest border pixels, and then for each cell, the biomarker classification model 322 may be configured to calculate the nucleus (wherein image sets containing nuclei may also be annotated and used for training a machine learning model). Please also read paragraph [0163, 0187-0188 and 0209]), and wherein prior to identifying the location of each cell of the plurality of cells of the at least one image (Fig. 3. Paragraph [0116]-YIP discloses in FIG. 1, the imaging-based biomarker prediction system 102 includes an image pre-processing sub-system 114 that performs initial image processing to enhance image data for faster processing in training a machine learning framework and for performing biomarker prediction using a trained deep learning framework. Further in paragraph [0152]-YIP discloses the module 304 receives tiled, sub-images from the pipeline 315, and the cell segmentation model 316 determines the list of locations of all lymphocytes, and those locations are compared to the other three class model's list of all cells determined from the model 316. Please also read paragraph [0210, 0212 and 0216]), the method further comprises:
blurring the at least one image by decreasing the resolution of the at least one image to facilitate identification of the plurality of nuclei (Fig. 3. Paragraph [0118]-YIP discloses in multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, at a first image resolution, downsampling that image to a second image resolution, and then performing a normalization on the downsampled histopathology image, such as color and/or intensity normalization, and removing non-tissue objects from the image. In paragraph [0271]-YIP discloses the pre-processing controller 302 can receive an image from the file having a resolution that is higher than the optimal resolution and downsample the image at a ratio that achieves the optimal resolution, at process 1106. In paragraph [0279]-YIP discloses the controller 302 removes these pixels (wherein the pixels represent artifacts, markings or blurred areas) by converting the image to a grayscale image, passing the grayscale image through a Gaussian blur filter. Other filters may be used to blur the image. Please also read paragraph [0146]).
Regarding claim 4, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches further comprising:
assigning, with at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111 and 0113]), a label (Fig. 1. Paragraph [0130]-YIP discloses a histopathology image may be segmented and each segment of the image may be labeled according to one or more data types that may be classified to that segment. The histopathology image may be labeled as a whole according to the one or more data types that may be classified to the image or at least one segment of the image. Further in paragraph [0132]-YIP discloses the trained image classifier module 170 includes trained tissue classifiers 172, trained by the module 160 using one or more training image sets, to identify and classify tissue type in regions/areas of received image data. These trained tissue classifiers are trained to identify biomarkers via the tissue classification, where these include single-scale configured classifiers 172a and multiscale classifiers 172b. Please also see Fig. 3) to the at least one single-cell images (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images. The biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. In paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect single-cell analysis data or detection of cellular products. Please also see Fig. 3 and read paragraph [0176, 0207 and 0292-0293]).
Regarding claim 6, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches wherein the resolution of the at least one single-cell image increases until the resolution of the at least one single-cell image of the plurality of single-cell images is at the first resolution of the plurality of single-cell images (Fig. 31. Paragraph [0265]-YIP discloses each digital image file received by the pre-processing controller 302, at 1102, contains multiple versions of the same image content, and each version has a different resolution. In paragraph [0273]-YIP discloses the pre-processing controller 302 retrieves the 40× magnification layer, then downsamples the image in that layer at a 2× downsampling ratio to create an image with the optimal resolution of 20× magnification. Furthermore, in paragraph [0310]-YIP discloses these added and replacement layers convert a CNN to a tile-resolution FCN without requiring the upsampling performed in the later layers of traditional pixel-resolution FCNs. Upsampling is a method by which a new version of an original image can be created with a higher resolution value than the original image. However, upsampling is a time-consuming, computation-intense process. Therefore, it would have been obvious to a person of ordinary skill in the art to increase the resolution of an image. YIP teaches multi-scale and single scale processing of single-cell images, iteratively downsampling images to reach an optimal resolution, and improving the quality of images by applying various filters. In addition, YIP discusses the possibility of using 2x upsampling layers, which iteratively increase image resolution. Although downsampling offers advantages in terms of data size, speed and computational resources, upsampling improves the detail of images and allows images to be matched to an optimal resolution. Thus, it would have been obvious to use techniques to increase the resolution of a single-cell image images when the optimal resolution is lower than necessary and/or computational resources are available).
Regarding claim 7, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 6, YIP fails to explicitly teach wherein the fine tuning is repeated until the resolution of the at least one single-cell image of the plurality of single-cell images is a highest resolution.
However, SULLIVAN explicitly teaches wherein the fine tuning is repeated (Fig. 2. Paragraph [0132]-SULLIVAN discloses the neural network model can be iteratively trained based on the input and by comparing the output to variants and variants with significance, to generate a trained neural network model. At a verification stage and/or execution stage, the trained neural network model can then be executed to generate an estimate output that closely anticipates the variants and/or variants with significance of samples and/or contact matrices) until the resolution of the at least one single-cell image of the plurality of single-cell images is a highest resolution (Fig. 2. Paragraph [0127]-SULLIVAN discloses the data preparation module 310 can normalize the sequencing reads or contact matrix from the sample or set of samples to a common format and/or a common scale. The preparation module 310 can normalize a set of images representing the information from the sample or set of samples to a common image size of 256 pixels by 256 pixels and to a common image file format of Tagged Image File Format (TIFF). In paragraph [0129]-SULLIVAN discloses the karyotyping by sequencing variant detector 315 recursively uses the first machine learning model 316 (e.g., a CNN model), creating increasing resolution contact matrixes between classification steps, to precisely identify a set of structural variants of the desired resolution. Each structural variant from the set of structural variants are then classified using the second machine learning model 321 (e.g., a KNN model) of the karyotyping by sequencing variant analyzer 320 to predict a set of clinical significance and/or biological significance of the set of structural variants. In paragraph [0133]-SULLIVAN discloses the CNN captures relationships in a contact matrix by the application of a series of convolutional filters of various dimensions. Please also read paragraph [0088, 0115 and 0131]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN of having a computer implemented method comprising: receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of a physically expanded sample of a tissue or a cell specimen, with the teachings of SULLIVAN of having wherein the fine tuning is repeated until the resolution of the at least one single-cell image of the plurality of single-cell images is a highest resolution.
Wherein YIP’s method having wherein the fine tuning is repeated until the resolution of the at least one single-cell image of the plurality of single-cell images is a highest resolution.
The motivation behind the modification would have been to obtain a method that improves machine learning model training, accuracy and classifications, since both YIP and SULLIVAN concern image analysis and cellular images. Wherein YIP’s systems and methods improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while SULLIVAN’s systems and methods improve model training and performance. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and SULLIVAN et al. (US 20220180964 A1), Abstract and Paragraph [0591-0600].
Regarding claim 8, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches wherein the filter is a Gaussian filter (Fig. 3. Paragraph [0273]-YIP discloses at process 1106, after the pre-processing controller 302 obtains an image with an optimal resolution, it locates all parts of the image that depict tumor sample tissue and digitally eliminates debris, pen marks, and other non-tissue objects. In paragraph [0274]-YIP discloses at process 1106, the pre-processing controller 302 differentiates between tissue and non-tissue regions of the image and uses Gaussian blur removal to edit pixels with non-tissue objects. In an example, any control tissue on a slide that is not part of the tumor sample tissue can be detected and labeled as control tissue by the tissue detector or manually labeled by a human analyst as control tissue that should be excluded from the downstream tile grid projections. In paragraph [0278]-YIP discloses at process 1106, the controller 302 eliminates pixels in the image that have low local variability (wherein these pixels represent artifacts, markings, or blurred areas). In paragraph [0279]-YIP discloses at process 1106, the controller 302 removes these pixels by converting the image to a grayscale image, passing the grayscale image through a Gaussian blur filter that mathematically adjusts the original grayscale value of each pixel to a blurred grayscale value to create a blurred image. Other filters may be used to blur the image).
Regarding claim 10, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches further comprising:
manipulating an image perspective of the at least one single-cell image of the plurality of single-cell images to provide a plurality of augmented single-cell images, wherein manipulating the image perspective comprises randomly flipping, randomly rotating, randomly shearing along an axis, and/or randomly translating at least one single-cell image of the plurality of single-cell images, and/or adding random noise to the at least one single-cell image of the plurality of single-cell images (Fig. 3. Paragraph [0336]-YIP each histopathology image can exhibit large degrees of variation in visual features, including tumor appearance, so a training set may include digital slide images that are highly dissimilar to better train the model for the variety of slides that it may analyze. Images in training data may also be subjected to data augmentation (including rotating, scaling, color jitter, etc.), before being used to train the model).
Regarding claim 12, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches wherein the machine learning model comprises a convolutional neural network (CNN) (Fig. 1. Paragraph [0135]-YIP discloses the trained image classifier module 170 and associated classifiers may be configured with an image-analysis adapted machine learning techniques, including, for example, deep learning techniques, including, by way of example, a CNN model and, more particular, a tile-resolution CNN, that in some examples is implemented as a FCN model, and, more particularly still, implemented as a tile-resolution FCN model, etc. Please also read paragraph [0042, 0149-0150]).
Regarding claim 15, YIP explicitly teaches a system (Fig. 1, #100 and #3800 called a prediction system and a computing device, respectively. Paragraph [0111 and 0411]-YIP discloses FIG. 1 illustrates a prediction system 100 capable of analyzing digital images of histopathology slides of a tissue sample and determining the likelihood of biomarker presence in that tissue. In paragraph [0138]-YIP discloses FIG. 3 illustrates an implementation of the imaging-based biomarker prediction system 102, and more particularly, of the deep learning framework 150 (wherein the framework 300 includes a pre-processing controller 302, a deep learning framework cell segmentation module 304, a deep learning framework multiscale classifier module 306, a deep learning framework single-scale classifier module 307, and a deep learning post-processing controller 308). In paragraph [0411]-YIP discloses FIG. 38 illustrates an example computing device 3800 for implementing the imaging-based biomarker prediction system 100 of FIG. 1. Please also see Fig. 3) comprising at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]) programmed or configured to:
segment (Fig. 1. Paragraph [0130]-YIP discloses a histopathology image may be segmented. In paragraph [0131]-YIP discloses in system 100, the deep learning framework 150 further includes a trained image classifier module 170. In paragraph [0133]-YIP discloses the module 170 may further include a cell segmenter 176 that identifies cells within a histopathology image, including cell borders, interiors, and exteriors. Please also see Fig. 3 and read paragraph [0151, 0155-0156 and 0207-0216]) the at least one image to define a plurality of single-cell images (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images. The biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. In paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect single-cell analysis data or detection of cellular products. Please also see Fig. 3 and read paragraph [0176, 0207 and 0292-0293]);
Although YIP explicitly teaches receive image data (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images. The biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. In paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect single-cell analysis data or detection of cellular products. Please also see Fig. 3 and read paragraph [0176, 0207 and 0292-0293]) associated with at least one image at a first resolution (Fig. 1. Paragraph [0116]-YIP discloses in FIG. 1, the imaging-based biomarker prediction system 102 includes an image pre-processing sub-system 114 that performs initial image processing to enhance image data for faster processing in training a machine learning framework and for performing biomarker prediction using a trained deep learning framework. Further in paragraph [0118]-YIP discloses in a multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, at a first image resolution. Please also read paragraph [0139, 0313 and 0329-0330]);
YIP fails to explicitly teach receive image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
However, WEISENFELD explicitly teaches receive image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen (Fig. 1. Paragraph [0506]-WEISENFELD discloses a region of interest can be identified in a biological sample using a variety of different techniques, e.g., expansion microscopy, bright field microscopy, dark field microscopy, phase contrast microscopy, electron microscopy, fluorescence microscopy, reflection microscopy, interference microscopy, and confocal microscopy, and combinations thereof. Further in paragraph [0152]-WEISENFELD discloses a biological sample embedded in a hydrogel can be isometrically expanded. In paragraph [0154]- WEISENFELD discloses Isometric expansion can be performed by anchoring one or more components of a biological sample to a gel, followed by gel formation, proteolysis, and swelling. In paragraph [0156]-WEISENFELD discloses isometric expansion of the sample can increase the spatial resolution of the subsequent analysis of the sample. Isometric expansion of the biological sample can result in increased resolution in spatial profiling (e.g., single-cell profiling). In paragraph [0157]-WEISENFELD discloses Isometric expansion can enable three-dimensional spatial resolution of the subsequent analysis of the sample. Further in paragraph [0158]-WEISENFELD discloses a biological sample is isometrically expanded to a volume at least 2×, 2.1×, 2.2×, 2.3×, 2.4×, 2.5×, 2.6×, 2.7×, 2.8×, 2.9×, 3×, 3.1×, 3.2×, 3.3×, 3.4×, 3.5×, 3.6×, 3.7×, 3.8×, 3.9×, 4×, 4.1×, 4.2×, 4.3×, 4.4×, 4.5×, 4.6×, 4.7×, 4.8×, or 4.9× its non-expanded volume).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP of having a system comprising at least one processor programmed or configured to: segment the at least one image to define a plurality of single-cell images; and train a machine learning model to predict a classification of single-cell images based on multiple fine-tuning iterations, wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images, with the teachings of WEISENFELD of having receive image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
Wherein YIP’s system having receive image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
The motivation behind the modification would have been to obtain a system that improves machine learning model training, accuracy and classifications as well as the resolution for spatial analysis, since both YIP and WEISENFELD concern cellular image analysis. Wherein YIP’s systems and methods provides improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while WEISENFELD’s systems and methods that improves the capture of analytes and resolution for spatial analysis. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and WEISENFELD et al. (US 20210150707 A1), Abstract and Paragraph [0496, 0515, and 0564].
Although YIP explicitly teaches apply a filter to at least one single-cell image of the plurality of single-cell images, resulting in a plurality of images having a plurality of different second resolutions than the first resolution (Fig. 3. Paragraph [0118]-YIP explicitly teaches in multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, at a first image resolution, downsampling that image to a second image resolution, and then performing a normalization on the downsampled histopathology image, such as color and/or intensity normalization, and removing non-tissue objects from the image (wherein multiple versions of images are created by applying a 10-layer format with 2x downsampling at each layer and applying various types of filters to remove artifacts and improve the quality of each image). Please also read paragraph [0264-0272 and 0309-0310] (wherein the process of 2x upsampling where an image’s resolution is repeatedly increased is discussed but not implemented));
YIP fails to explicitly teach apply a filter to at least one image of the plurality of images by iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image.
However, YANG explicitly teaches apply a filter to at least one image (Fig. 3D, #102 called input data. Paragraph [0032]) of the plurality of images (Fig. 3D. Paragraph [0032]-YANG discloses FIG. 3D is a flow diagram of an example methodology 200 for removing blur from an image. The method 200 begins by receiving 102 input data representing a blurry digital image and applying 104 a variable scale filter to the blurry digital image at an original resolution to obtain a set of filtered observations at different scale levels. The variable scale filter may include a set of Gaussian noise filters with decreasing radius, or a set of directional noise filters) by iteratively decreasing a kernel size of the filter (Fig. 3D. Paragraph [0032]-YANG discloses the variable scale filter may include a set of Gaussian noise filters with decreasing radius, or a set of directional noise filters), resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image (Fig. 3. Paragraph [0032]-YANG discloses the variable scale filter includes a first filter and a second filter that is different from the first filter. The method 200 includes applying 106a the first filter to the blurry digital image at the original resolution to obtain a first observation at a first scale level, and applying 106b the second filter to the blurry digital image at the original resolution to obtain a second observation at a second scale level that is smaller than the first scale level. The applying step 106b may be repeated at iteratively smaller scale levels to obtain additional filtered observations of the blurry image. The estimated blur kernel may become accurate enough for non-blind deblurring before reaching the finest resolution level);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD of having a system comprising at least one processor programmed or configured to: receive image data associated with at least one image at a first resolution, the at least one image comprising an image of a physically expanded sample of a tissue or a cell specimen, with the teachings of YANG of having apply a filter to at least one image of the plurality of images by iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image.
Wherein YIP’s system having apply a filter to at least one single-cell image of the plurality of single-cell images by iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one single-cell image.
The motivation behind the modification would have been to obtain a system that improves machine learning model training, accuracy and classifications as well as the blur removal, since both YIP and YANG concern image analysis and gaussian filters. Wherein YIP’s systems and methods improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while YANG’s systems and methods that improves the ability to remove blur from a single image by accumulating a blur kernel. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and YANG et al. (US 20160335747 A1), Abstract and Paragraph [0017].
Although YIP explicitly teaches train a machine learning model to predict a classification of single-cell images (Fig. 3. Paragraph [0123]-YIP discloses to analyze the received histopathology image data and other data, the imaging-based biomarker prediction system 102 includes a deep learning framework 150 that implements various machine learning techniques to generate trained classifier models for image-based biomarker analysis from received training sets of image data or sets of image data and other patient information. In paragraph [0124]-YIP discloses by labeling the image data 162a according to associations with the other data types, the imaging-based biomarker prediction system may train an image classifier module to predict the one or more different data types from image data 162a. Please also read paragraph [0115, 0138, 0176, 0190-0199 and 0293-0294]) based on multiple fine-tuning iterations, wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images (Fig. 3. Paragraph [0377]-YIP discloses training may be performed with weakly supervised learning. This process may be repeated many times, given enough collections and tiles as input to the neural network, it will learn to differentiate tiles with different classes with higher accuracy as more iterations are performed. Please also read paragraph [0264-0273, 0376, and 0378-0386]).
YIP is silent on wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
However, SULLIVAN explicitly teaches wherein each iteration of the multiple fine-tuning iterations (Fig. 2. Paragraph [0132]-SULLIVAN discloses the neural network model can be iteratively trained based on the input and by comparing the output to variants and variants with significance, to generate a trained neural network model. At a verification stage and/or execution stage, the trained neural network model can then be executed to generate an estimate output that closely anticipates the variants and/or variants with significance of samples and/or contact matrices) is based on at least one image of the plurality of images (Fig. 2. Paragraph [0127]-SULLIVAN discloses the data preparation module 310 can normalize the sequencing reads or contact matrix from the sample or set of samples to a common format and/or a common scale. The preparation module 310 can normalize a set of images representing the information from the sample or set of samples to a common image size of 256 pixels by 256 pixels and to a common image file format of Tagged Image File Format (TIFF). In paragraph [0129]-SULLIVAN discloses the karyotyping by sequencing variant detector 315 recursively uses the first machine learning model 316 (e.g., a CNN model), creating increasing resolution contact matrixes between classification steps, to precisely identify a set of structural variants of the desired resolution. Each structural variant from the set of structural variants are then classified using the second machine learning model 321 (e.g., a KNN model) of the karyotyping by sequencing variant analyzer 320 to predict a set of clinical significance and/or biological significance of the set of structural variants. In paragraph [0133]-SULLIVAN discloses the CNN captures relationships in a contact matrix by the application of a series of convolutional filters of various dimensions. Please also read paragraph [0088, 0115 and 0131]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD and in further view of YANG of having a system comprising at least one processor programmed or configured to: receive image data associated with at least one image at a first resolution, the at least one image comprising an image of a physically expanded sample of a tissue or a cell specimen, with the teachings of SULLIVAN of having wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
Wherein YIP’s system having train a machine learning model to predict a classification of single-cell images based on multiple fine-tuning iterations, wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
The motivation behind the modification would have been to obtain a system that improves machine learning model training, accuracy and classifications, since both YIP and SULLIVAN concern image analysis and cellular images. Wherein YIP’s systems and methods improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while SULLIVAN’s systems and methods improve model training and performance. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and SULLIVAN et al. (US 20220180964 A1), Abstract and Paragraph [0591-0600].
Regarding claim 16, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the system of claim 15, YIP further teaches wherein the at least one image comprises an image of a plurality of cells (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images. The biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. In paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect single-cell analysis data or detection of cellular products. Please also see Fig. 3 and read paragraph [0176, 0207 and 0292-0293]), and wherein when segmenting the at least one image to define the plurality of single-cell images (Fig. 3. Paragraph [0130]-YIP discloses a histopathology image may be segmented. In paragraph [0131]-YIP discloses in system 100, the deep learning framework 150 further includes a trained image classifier module 170. In paragraph [0133]-YIP discloses the module 170 may further include a cell segmenter 176 that identifies cells within a histopathology image, including cell borders, interiors, and exteriors. Please also read paragraph [0151, 0155-0156 and 0176]), the at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111 and 0113]) is further programmed or configured to:
identify a location of each cell of the plurality of cells of the at least one image based on a pixel coordinate of each cell of the plurality of cells of the at least one image (Fig. 3. Paragraph [0212]-YIP discloses the process 712 may access a stored image processing library and use that library to find contours around the cell interior class. The process 712 may perform a cell registration process. The cell border class (denoted by locations with a 1 value in each mask) ensures separation between neighboring cell interiors. This generates a list of every contour on each mask. The process 712 determines the coordinates of the contour's centroid (center of mass), from which the process 712 produces a centroid list. Next, to generate outputs that are in the coordinate space defined by the entire received image instead of the coordinate space that is specific to a single tile in the image, each coordinate in the contour lists and the centroid lists is shifted. The process 714 performs the same processes as the process 712, but on the lymphocyte classes. In paragraph [0217]-YIP discloses the process 716 has the coordinates for each cell centroid, the coordinates for the top-left corner of each tissue classification tile, and the size of each tissue classification tile, and is configured to determine the parent tile for each cell based on its centroid location. Please also read paragraph [0112 and 0152]); and
define for each of the plurality of cells, a single-cell image based on the location of a cell of the plurality of cells (Fig. 3. Paragraph [0152]-YIP discloses the module 304 receives tiled, sub-images from the pipeline 315, and the cell segmentation model 316 determines the list of locations of all lymphocytes, and those locations are compared to the other three class model's list of all cells determined from the model 316. In paragraph [0155]-YIP discloses a UNet model can recognize the outer edges of many types of cells and may classify each cell according to cell shape or its location within a tissue class region assigned by the tissue classification module 320. Further in paragraph [0217]-YIP discloses the process 716, each cell is binned into one of the tissue classification tiles (from process 706) based on location. Please also read paragraph [0166 and 0188]).
wherein the single-cell image comprises the cell of the plurality of cells and a microenvironment of the cell of the plurality of cells (Fig. 3. Paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect various types of data, such as single-cell analysis data or detection of cellular products (including proteins, lipids, and other molecules) indicating the presence of specific cell populations, including effector data, stimulatory data, regulatory data, inflammatory data, chemoattractive data (wherein imaging metrics and analysis may include cell shape, cell area, cell perimeter, cell convex area ratio, cell circularity, cell convex perimeter area, cell length, lymphocyte %, cellular characteristics, cell textures, the clustering of cell types of tissue classes based on spacing and density of classified cells, the spacing and distance of tissue class classified tiles, probability of neighboring cell types, biomarkers such as HRD status, DNA ploidy scores, karyotypes, CMS scores, chromosomal instability (CIN) status, signet ring morphology scores, NC ratios, cellular pathway activation status, and any visually detectable features. Please also see Fig. 1 and read paragraph [0090, 0107, 0176, 0188, 0207 and 0292-0293]).
Regarding claim 17, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the system of claim 16, YIP further teaches wherein the image of the plurality of cells comprises a plurality of nuclei (Fig. 3. Paragraph [0151-YIP discloses the cell segmentation model 316 may be configured as a first pixel-level FCN model, that identifies and assigns each pixel of image data into a cell-subunit class: (i) cell interior, (ii) a cell border, or (iii) a cell exterior. In paragraph [0166]-YIP discloses the cell segmentation model 316 may be trained to analyze an input image and assign one of the three classes to each pixel, define cells as a group of adjacent nucleus pixels and all cytoplasm pixels between the nucleus pixels and the next nearest border pixels, and then for each cell, the biomarker classification model 322 may be configured to calculate the nucleus (wherein image sets containing nuclei may also be annotated and used for training a machine learning model). Please also read paragraph [0163, 0187-0188 and 0209]), and wherein prior to identifying the location of each cell of the plurality of cells of the at least one image (Fig. 3. Paragraph [0149]-YIP discloses the deep learning multiscale classifier module 304 is configured to perform cell segmentation through a cell segmentation model 316, where cell segmentation may be a pixel-level process of the histopathology image from normalization process 310. Further in paragraph [0152]-YIP discloses the module 304 receives tiled, sub-images from the pipeline 315, and the cell segmentation model 316 determines the list of locations of all lymphocytes, and those locations are compared to the other three class model's list of all cells determined from the model 316. Please also read paragraph [0210, 0212 and 0216]), the at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111-0113])) is further programmed or configured to:
blur the at least one image by decreasing the resolution of the at least one image to facilitate identification of the plurality of nuclei (Fig. 3. Paragraph [0118]-YIP discloses in multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, at a first image resolution, downsampling that image to a second image resolution, and then performing a normalization on the downsampled histopathology image, such as color and/or intensity normalization, and removing non-tissue objects from the image. In paragraph [0271]-YIP discloses the pre-processing controller 302 can receive an image from the file having a resolution that is higher than the optimal resolution and downsample the image at a ratio that achieves the optimal resolution, at process 1106. In paragraph [0279]-YIP discloses the controller 302 removes these pixels (wherein the pixels represent artifacts, markings or blurred areas) by converting the image to a grayscale image, passing the grayscale image through a Gaussian blur filter that mathematically adjusts the original grayscale value of each pixel to a blurred grayscale value to create a blurred image. Other filters may be used to blur the image. Please also read paragraph [0146]).
Regarding claim 29, YIP explicitly teaches a computer program product comprising at least one non-transitory computer readable medium including one or more instructions (Fig. 38. Paragraph [0411]-FIG. 38 illustrates an example computing device 3800 for implementing the imaging-based biomarker prediction system 100 of FIG. 1. System 100 may be implemented on the computing device 3800 and in particular on one or more processing units 3810, which may represent Central Processing Units (CPUs), and/or on one or more or Graphical Processing Units (GPUs) 3811, including clusters of CPUs and/or GPUs. The system 100 may be stored on and implemented from one or more non-transitory computer-readable media 3812 of the computing device 3800. The computer-readable media 3812 may include an operating system 3814 and the deep learning framework 3816 having elements corresponding to that of deep learning framework 300, including the pre-processing controller 302, classifier modules 304 and 306, and the post-processing controller 308. The computer-readable media 3812 may store trained deep learning models, executable code, etc. used for implementing the techniques herein) that, when executed by at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111-0113]), cause the at least one processor to:
segment (Fig. 1. Paragraph [0130]-YIP discloses a histopathology image may be segmented. In paragraph [0131]-YIP discloses in system 100, the deep learning framework 150 further includes a trained image classifier module 170. In paragraph [0133]-YIP discloses the module 170 may further include a cell segmenter 176 that identifies cells within a histopathology image, including cell borders, interiors, and exteriors. Please also see Fig. 3 and read paragraph [0151, 0155-0156 and 0176]) the at least one image to define a plurality of single-cell images (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images. The biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. In paragraph [0138]-YIP discloses the organoid modeling lab 116 may collect single-cell analysis data or detection of cellular products. Please also see Fig. 3 and read paragraph [0176, 0207 and 0292-0293]);
Although YIP explicitly teaches receive image data associated with at least one image at a first resolution (Fig. 1. Paragraph [0115]-YIP discloses the imaging-based biomarker prediction system 102 is communicatively coupled to receive medical images, for example of histopathology slides such as digital H&E stained slide images, IHC stained slide images, or digital images of any other staining protocols (wherein images are received from any number of medical image data sources such as physician clinical records systems 106 or histopathology image repositories 110. In paragraph [0116]-YIP discloses in FIG. 1, the imaging-based biomarker prediction system 102 includes an image pre-processing sub-system 114 that performs initial image processing to enhance image data for faster processing in training a machine learning framework and for performing biomarker prediction using a trained deep learning framework. Further in paragraph [0118]-YIP discloses in a multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, at a first image resolution. Please also read paragraph [0139, 0313 and 329-0330]);
YIP fails to explicitly teach receive image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
However, WEISENFELD explicitly teaches receive image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen (Fig. 1. Paragraph [0506]-WEISENFELD discloses a region of interest can be identified in a biological sample using a variety of different techniques, e.g., expansion microscopy, bright field microscopy, dark field microscopy, phase contrast microscopy, electron microscopy, fluorescence microscopy, reflection microscopy, interference microscopy, and confocal microscopy, and combinations thereof. Further in paragraph [0152]-WEISENFELD discloses a biological sample embedded in a hydrogel can be isometrically expanded. In paragraph [0154]- WEISENFELD discloses Isometric expansion can be performed by anchoring one or more components of a biological sample to a gel, followed by gel formation, proteolysis, and swelling. In paragraph [0156]-WEISENFELD discloses isometric expansion of the sample can increase the spatial resolution of the subsequent analysis of the sample. Isometric expansion of the biological sample can result in increased resolution in spatial profiling (e.g., single-cell profiling). In paragraph [0157]-WEISENFELD discloses Isometric expansion can enable three-dimensional spatial resolution of the subsequent analysis of the sample. Further in paragraph [0158]-WEISENFELD discloses a biological sample is isometrically expanded to a volume at least 2×, 2.1×, 2.2×, 2.3×, 2.4×, 2.5×, 2.6×, 2.7×, 2.8×, 2.9×, 3×, 3.1×, 3.2×, 3.3×, 3.4×, 3.5×, 3.6×, 3.7×, 3.8×, 3.9×, 4×, 4.1×, 4.2×, 4.3×, 4.4×, 4.5×, 4.6×, 4.7×, 4.8×, or 4.9× its non-expanded volume).
. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP of having a computer program product, with the teachings of WEISENFELD of having receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
Wherein YIP’s computer program product having receiving, with at least one processor, image data associated with at least one image at a first resolution, the at least one image comprising an image of physically expanded sample of a tissue or a cell specimen.
The motivation behind the modification would have been to obtain a computer program product that improves machine learning model training, accuracy and classifications as well as the resolution for spatial analysis, since both YIP and WEISENFELD concern cellular image analysis. Wherein YIP’s systems and methods provides improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while WEISENFELD’s systems and methods that improves the capture of analytes and resolution for spatial analysis. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and WEISENFELD et al. (US 20210150707 A1), Abstract and Paragraph [0496, 0515, and 0564].
Although YIP explicitly teaches apply a filter to at least one single-cell image of the plurality of single-cell images, resulting in a plurality of images filtered having a plurality of different second resolutions than the first resolution (Fig. 3. Paragraph [0118]-YIP explicitly teaches in multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, at a first image resolution, downsampling that image to a second image resolution, and then performing a normalization on the downsampled histopathology image, such as color and/or intensity normalization, and removing non-tissue objects from the image (wherein multiple versions of images are created by applying a 10-layer format with 2x downsampling at each layer and applying various types of filters to remove artifacts and improve the quality of each image). Please also read paragraph [0264-0272 and 0309-0310] (wherein the process of 2x upsampling is discussed but not implemented)).
YIP fails to explicitly teach apply a filter to at least one image of the plurality of images by iteratively decreasing a kernel size of the filter iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image; and
However, YANG explicitly teaches apply a filter to at least one image (Fig. 3D, #102 called input data. Paragraph [0032]) of the plurality of images (Fig. 3D. Paragraph [0032]-YANG discloses FIG. 3D is a flow diagram of an example methodology 200 for removing blur from an image. The method 200 begins by receiving 102 input data representing a blurry digital image and applying 104 a variable scale filter to the blurry digital image at an original resolution to obtain a set of filtered observations at different scale levels. The variable scale filter may include a set of Gaussian noise filters with decreasing radius, or a set of directional noise filters) by iteratively decreasing a kernel size of the filter (Fig. 3D. Paragraph [0032]-YANG discloses the variable scale filter may include a set of Gaussian noise filters with decreasing radius, or a set of directional noise filters), resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image (Fig. 3. Paragraph [0032]-YANG discloses the variable scale filter includes a first filter and a second filter that is different from the first filter. The method 200 includes applying 106a the first filter to the blurry digital image at the original resolution to obtain a first observation at a first scale level, and applying 106b the second filter to the blurry digital image at the original resolution to obtain a second observation at a second scale level that is smaller than the first scale level. The applying step 106b may be repeated at iteratively smaller scale levels to obtain additional filtered observations of the blurry image. The estimated blur kernel may become accurate enough for non-blind deblurring before reaching the finest resolution level).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD of having a computer program product, with the teachings of MIKHNO of having apply a filter to at least one image of the plurality of images by iteratively decreasing a kernel size of the filter iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one image; and
Wherein YIP’s computer program product having apply a filter to at least one single-cell image of the plurality of single-cell images by iteratively decreasing a kernel size of the filter, resulting in a plurality of images filtered with different kernel sizes having a plurality of different second resolutions than the first resolution, wherein each iteration increases a resolution of the at least one single-cell image.
The motivation behind the modification would have been to obtain a computer program product that improves machine learning model training, accuracy and classifications as well as the blur removal, since both YIP and YANG concern image analysis and gaussian filters. Wherein YIP’s systems and methods improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while YANG’s systems and methods that improves the ability to remove blur from a single image by accumulating a blur kernel. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and YANG et al. (US 20160335747 A1), Abstract and Paragraph [0017].
Although YIP explicitly teaches train a machine learning model to predict a classification of single-cell images (Fig. 3. Paragraph [0123]-YIP discloses to analyze the received histopathology image data and other data, the imaging-based biomarker prediction system 102 includes a deep learning framework 150 that implements various machine learning techniques to generate trained classifier models for image-based biomarker analysis from received training sets of image data or sets of image data and other patient information. In paragraph [0124]-YIP discloses by labeling the image data 162a according to associations with the other data types, the imaging-based biomarker prediction system may train an image classifier module to predict the one or more different data types from image data 162a. Please also read paragraph [0115, 0138, 0176, 0190-0199 and 0293-0294]) based on multiple fine-tuning iterations (Fig. 3. Paragraph [0377]-YIP discloses training may be performed with weakly supervised learning. This process may be repeated many times, given enough collections and tiles as input to the neural network, it will learn to differentiate tiles with different classes with higher accuracy as more iterations are performed. Please also read paragraph [0264-0273, 0376, and 0378-0386])
YIP is silent on wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
However, SULLIVAN explicitly teaches wherein each iteration of the multiple fine-tuning iterations (Fig. 2. Paragraph [0132]-SULLIVAN discloses the neural network model can be iteratively trained based on the input and by comparing the output to variants and variants with significance, to generate a trained neural network model. At a verification stage and/or execution stage, the trained neural network model can then be executed to generate an estimate output that closely anticipates the variants and/or variants with significance of samples and/or contact matrices) is based on at least one image of the plurality of images (Fig. 2. Paragraph [0127]-SULLIVAN discloses the data preparation module 310 can normalize the sequencing reads or contact matrix from the sample or set of samples to a common format and/or a common scale. The preparation module 310 can normalize a set of images representing the information from the sample or set of samples to a common image size of 256 pixels by 256 pixels and to a common image file format of Tagged Image File Format (TIFF). In paragraph [0129]-SULLIVAN discloses the karyotyping by sequencing variant detector 315 recursively uses the first machine learning model 316 (e.g., a CNN model), creating increasing resolution contact matrixes between classification steps, to precisely identify a set of structural variants of the desired resolution. Each structural variant from the set of structural variants are then classified using the second machine learning model 321 (e.g., a KNN model) of the karyotyping by sequencing variant analyzer 320 to predict a set of clinical significance and/or biological significance of the set of structural variants. In paragraph [0133]-SULLIVAN discloses the CNN captures relationships in a contact matrix by the application of a series of convolutional filters of various dimensions. Please also read paragraph [0088, 0115 and 0131]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN of having a computer program product comprising at least one non-transitory computer readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to: receive image data associated with at least one image at a first resolution, the at least one image comprising an image of a physically expanded sample of a tissue or a cell specimen, with the teachings of SULLIVAN of having wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
Wherein YIP’s computer program product having train a machine learning model to predict a classification of single-cell images based on multiple fine-tuning iterations, wherein each iteration of the multiple fine-tuning iterations is based on at least one image of the plurality of images.
The motivation behind the modification would have been to obtain a computer program product that improves machine learning model training, accuracy and classifications, since both YIP and SULLIVAN concern image analysis and cellular images. Wherein YIP’s systems and methods improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while SULLIVAN’s systems and methods improve model training and performance. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and SULLIVAN et al. (US 20220180964 A1), Abstract and Paragraph [0591-0600].
Regarding claim 43, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches comprising:
removing, with the at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111-0113]), a background from the at least one image (Fig. 1. Paragraph [0117]-YIP discloses the image pre-processing sub-system 114 may perform further image processing that removes artifacts and other noise from received images by doing preliminary tissue detection 114d, for example, to identify regions of the images corresponding to histopathology stained tissue for subsequent analysis, classification, and segmentation. In paragraph [0118]-YIP discloses in multiscale configuration where image data is to be analyzed on a tile-basis, image pre-processing includes receiving an initial histopathology image, and removing non-tissue objects from the image (wherein the background is non-tissue objects). Further in paragraph [0360]-YIP discloses a process 1508 displays images associated with a tissue masking step of process 1502. An assembled probability map generated by the process 1504 is passed through this tissue mask to remove background. Both background and marker area are removed by the masking algorithm of the process 1508. Please also read paragraph [0206 and 0283]);
sorting, with the at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111-0113]), the at least one single-cell image of the plurality of single-cell images into one category of a plurality of categories (Fig. 28. Paragraph [0404]-YIP discloses in FIG. 28, a process 2800 is provided for determining a proposed immunotherapy treatment for a patient using the imaging-based biomarker predictor system 102 of FIG. 1, and in particular the biomarker prediction of the deep learning framework 300 of FIG. 3. At a process 2806, the trained deep learning framework applies the images to a trained tissue classifier model and a trained biomarker segmentation model to determine biomarker status of the tissue regions of the image. A trained cell segmentation classifier model is further used by the process 2806. The process 2806 generates biomarker status and biomarker metrics for the image. As shown in FIG. 29, the output from the process 2806 may be provided to a process 2808 and implemented on a tumor therapy decision system 2900. Please also see Fig. 1 and 3, and read paragraph [0123-0125, 0157-0166, 0241-0250, and 0405-0411]) based on a value output by the machine learning model after it is trained (Fig. 1. Paragraph [0208]-YIP discloses the trained tissue classification model is configured to classify each tile into different tissue classes (e.g., tumor, stroma, normal epithelium, etc.). The trained tissue classification model calculates the class probability for each class stored in the model. The process 706 then determines the most likely class and assigns that class to the tile. The process 706 may output, as a result, a list of lists. Each nested interior list serves as nested classification that describes a single tile and contains the position of the tile, the probabilities that the tile is each of the classes contained in the model, and the identity of the most probable class (wherein the prediction system, including the post-processing controller 308, may determine various prediction values/scores during and after training/implementation including biomarker prediction metrics such as tumor purity, number of tiles classified as a particular tissue class, number of cells, number of tumor infiltrating lymphocytes, clustering of cell types or tissue classes, densities of cell types or tissue classes, tumor cell characteristics—roundness, length, nuclei density, stroma thickness around tumor tissue, image pixel data stats, predicted patient survival, PD-L1 status, MSI, TMB, origin of a tumor, and immunotherapy/therapy response. Please also see Fig. 3 and 7, and read paragraph [0123-0125, 0187-0188, 0205-0211]);
generating, with the at least one processor (Fig. 38, #3810 called processing units. Paragraph [0411]. Please also read paragraph [0111-0113]), a communication comprising a predicted treatment outcome based on which category of the plurality of categories the at least one single-cell image of the plurality of single-cell images is sorted into (Fig. 28. Paragraph [0404]-YIP discloses the output from the process 2806 may be provided to a process 2808 and implemented on a tumor therapy decision system 2900 (such as may be part of a genomic sequencing system, oncology system, chemotherapy decision system, immunotherapy decision system, or other therapy decision system) that determines a tumor type based on the received data, including based on the biomarker metrics, genomic sequencing data, etc. The system 2900 analyzes the biomarker status and/or biomarker metrics and other received molecular data against available immunotherapies 2902, at a process 2810, and the system 2900 recommends a matched listing of possible tumor-type specific immunotherapies 2904, filtered from the list of available immunotherapies 2902, in the form of a matched therapy report. Please also read paragraph [0183-0187, 0241-0246 and 0405-0411]); and
displaying, with the at least one processor, data associated with the communication via a graphical user interface (GUI) on a user device (Fig. 31. Paragraph [0410]-YIP discloses referring to FIG. 31, a GUI generated display 3100 having a panel 3102 showing an entire histopathology image 3104 and an enlarged portion (1.3× zoom factor) of that image 3104 displayed as window 3106. The panel 3102 further includes a magnification factor corresponding to the window 3106 and a tumor content report. FIG. 34 illustrates a drop down menu 3108 listing a series of classifications that a user can select for generating an classification overlay map that will be displayed on the display 3100. FIG. 35 illustrates the resulting display 3100 with an overlay map showing tumor classified tissue, demonstrating, in this example, that the tissue had been divided into tiles, and the tiles having classifications are shown. In the example of FIG. 35, the classification illustrated is a tumor classification. FIG. 36 illustrates another example classification overlay mapping, this one of a cell classification, epithelium, immune, stroma, tumor, or other. FIG. 37 illustrates a magnified cell classification overlay mapping that showing classifications may indeed may be displayed at a magnification sufficient to differential different cells with an histopathology image. Please also read paragraph [0245-0260]).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over YIP et al. (US 20200258223 A1), hereinafter referenced as YIP in view of WEISENFELD et al. (US 20210150707 A1), hereinafter referenced as WEISENFELD, and in further view of YANG et al. (US 20160335747 A1), hereinafter referenced as YANG and in further view of SULLIVAN et al. (US 20220180964 A1), hereinafter referenced as SULLIVAN and in further view of DITTAMORE et al. (US 20220260574 A1), hereinafter referenced as DITTAMORE.
Regarding claim 9, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP further teaches wherein the machine learning model (Fig. 3. Paragraph [0086]-YIP discloses the deep learning frameworks include a multiscale configuration that uses a tiling strategy to accurately capture structural and local histology of various diseases (e.g., cancer tumor prediction). These multiscale configurations perform classification on (labeled or unlabeled) histopathology images using classifiers trained to classify tiles of received histopathology images. The multiscale configurations contain tile-level tissue classifiers, i.e., classifiers trained using tile-based deep learning training. In some examples, the multiscale configurations contain pixel-level cell classifiers and cell segmentation models. Please also read paragraph [0093 and 0123-0125]) outputs a classification value for the at least one single-cell image of the plurality of single-cell images (Fig. 3. Paragraph [0132] The trained image classifier module 170 includes trained tissue classifiers 172, trained by the module 160 using one or more training image sets, to identify and classify tissue type in regions/areas of received image data. In some examples, these trained tissue classifiers are trained to identify biomarkers via the tissue classification, where these include single-scale configured classifiers 172a and multiscale classifiers 172b. In paragraph [0133]-YIP discloses the module 170 may further include other trained classifiers, including, trained cell classifiers 174 that identify biomarkers via cell classification. The module 170 may further include a cell segmenter 176 that identifies cells within a histopathology image, including cell borders, interiors, and exteriors), and wherein training the machine learning model to predict a classification of the at least one single-cell image of the plurality of single-cell images (Fig. 3. Paragraph [0190]-YIP discloses in training mode, where the deep learning frameworks in the system 300 are trained, various training data may be obtained (wherein training image data 401 is provided to the pre-processing controller and may be high resolution and low resolution histopathology images, digitally and/or manually annotated, annotated tissue image data from various tissue types, computer generated, synthetic image data, segmented cells, image data of labeled biomarkers (e.g., the biomarkers discussed herein), either slide-level label or tile-level labels). Please also read [0174, 0176 and 0198]) further comprises:
based on determining that the classification predicted for the at least one single-cell image does not match a classification of the at least one single-cell image, automatically correcting the classification of the at least one single-cell image to match the classification predicted for the at least one single-cell image (Fig. 3. Paragraph [0333]-YIP discloses the training set images are converted to input training image matrices and processed by the tissue classifier module 306 to assign a tissue class label to each tile image of the training image. If the tissue classifier module 306 does not accurately label the validation set of training images to match the corresponding annotations added by a human analyst, the weights of each layer of the deep learning network may be adjusted automatically by stochastic gradient descent through backpropagation until the tissue classifier module 306 accurately labels most of the validation set of training images. Please also read paragraph [0386]).
YIP fails to explicitly teach determining whether an accuracy value of the machine learning model is above a threshold value; based on determining that the accuracy value of the machine learning model is above the threshold value, determining whether the classification predicted for the at least one single-cell image matches a classification of the at least one single-cell image.
However, DITTAMORE explicitly teaches determining whether an accuracy value of the machine learning model (Paragraph [0184]-DITTAMORE discloses the disclosed methods encompass the use of a predictive model. The disclosed methods encompass comparing a measurable feature with a reference feature. Analyzing a measurable feature encompasses one or more of a support vector machine classification algorithm, a machine learning algorithm, or a combination thereof. In paragraph [0185]-DITTAMORE discloses an analytic classification process can use any one of a variety of statistical analytic methods to manipulate the quantitative data and provide for classification of the sample. Examples of useful methods include machine learning algorithms and other methods known to those skilled in the art) is above a threshold value (Paragraph [0187]-DITTAMORE discloses the predictive ability of a model can be evaluated according to its ability to provide a quality metric, e.g. AUROC (area under the ROC curve) or accuracy, of a particular value, or range of values. A desired quality threshold is a predictive model that will classify a sample with an accuracy of at least about 0.7, at least about 0.75, at least about 0.8, at least about 0.85, at least about 0.9, at least about 0.95, or higher. As an alternative measure, a desired quality threshold can refer to a predictive model that will classify a sample with an AUC of at least about 0.7, at least about 0.75, at least about 0.8, at least about 0.85, at least about 0.9, or higher);
based on determining that the accuracy value of the machine learning model is above the threshold value (Paragraph [0187]-DITTAMORE discloses ROC analysis can be used to select the optimal threshold under a variety of clinical circumstances, balancing the inherent tradeoffs that exist between specificity and sensitivity. In paragraph [0188]-DITTAMORE discloses the relative sensitivity and specificity of a predictive model can be adjusted to favor either the specificity metric or the sensitivity metric, where the two metrics have an inverse relationship), determining whether the classification predicted for the at least one single-cell image (Paragraph [0089]-DITTAMORE discloses the non-enrichment CTC analysis platform described herein enables the methods of the invention by allowing for single cell resolution and accurate genomic profiling of heterogeneous CTC populations (wherein the term CTC is a “circulating tumor cell” related to cancer that is present in a biological sample and can be present as single cells or clusters, the term biological sample can be any sample that contains CTCs and CTC data can be generated with any microscopic method known in the art). In paragraph [0120]-DITTAMORE discloses phenotypic parameters are analyzed by a classifier that utilizes the models and/or algorithms to predict 15 cell types. Based on the classifications of the cell types, a determination is made as to whether a sample contains or does not contain cell type K. In paragraph [0121]-DITTAMORE discloses after computation, each cell will have 15 probabilities of being one of the 15 cell types. Then each cell is ranked by its 15 probabilities, and the cell is determined as one cell type with the highest probability. Please also read paragraph [0143, 0153 and 0178]) matches a classification of the at least one single-cell image (Paragraph [0186]-DITTAMORE discloses classification can be made according to predictive modeling methods that set a threshold for determining the probability that a sample belongs to a given class. The probability preferably is at least 50%, or at least 60%, or at least 70%, or at least 80%, or at least 90% or higher. Classifications also can be made by determining whether a comparison between an obtained dataset and a reference dataset yields a statistically significant difference. If so, then the sample from which the dataset was obtained is classified as not belonging to the reference dataset class. Conversely, if such a comparison is not statistically significantly different from the reference dataset, then the sample from which the dataset was obtained is classified as belonging to the reference dataset class); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches of having a computer implemented method, with the teachings of DITTAMORE of having wherein the steps are repeated until the resolution of the at least one single-cell image of the plurality of single-cell images is a highest resolution.
Wherein YIP’s method having wherein the steps are repeated until the resolution of the at least one single-cell image of the plurality of single-cell images is a highest resolution.
The motivation behind the modification would have been to obtain a method that improves machine learning model training, accuracy and classifications as well enables accurate genomic profiling of heterogeneous CTC populations, since both YIP and DITTAMORE concern cellular image analysis. Wherein YIP’s systems and methods provides improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while DITTAMORE’s systems and methods enables accurate genomic profiling of heterogeneous CTC populations by allowing for single cell resolution. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and DITTAMORE et al. (US 20220260574 A1), Abstract and Paragraph [0089].
Claims 11 is rejected under 35 U.S.C. 103 as being unpatentable over YIP et al. (US 20200258223 A1), hereinafter referenced as YIP in view of WEISENFELD et al. (US 20210150707 A1), hereinafter referenced as WEISENFELD, and in further view of YANG et al. (US 20160335747 A1), hereinafter referenced as YANG and in further view of SULLIVAN et al. (US 20220180964 A1), hereinafter referenced as SULLIVAN and in further view of NATAN et al. (US 20190120767 A1), hereinafter referenced as NATAN.
Regarding claim 11, YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches the computer implemented method of claim 1, YIP fails to explicitly teach wherein the at least one image has a resolution of 5 nm to 250 nm per pixel.
However, NATAN explicitly teaches wherein the at least one image (Fig. 1A-B. Paragraph [0031]-NATAN discloses super-resolution fluorescence microscopy is a type of light microscopy that provides images at a higher resolution than permitted by the limit of diffraction. Using visible light and high numerical aperture objectives, conventional microscopy images are limited to a resolution of about 250 nm. Super-resolution images can be taken at much higher resolution, currently as high as 5 nm. Super-resolution imaging includes any microscopy techniques that result in a resolution of at least about 250 nm, 200 nm, 150 nm, 100 nm, 50 nm, 25 nm, 20 nm, 15 nm, 10 nm, or 5 nm. In some embodiments, the resolution is from about 200 nm to 5 nm, 150 to 10 nm, 100 to 5 nm).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of YIP in view of WEISENFELD and in further view of YANG and in further view of SULLIVAN explicitly teaches of having a computer implemented method comprising: receiving, with at least one processor, image data associated with at least one image at a first resolution, with the teachings of NATAN of having wherein the at least one image has a resolution of 5 nm to 250 nm per pixel.
Wherein YIP’s method having wherein the at least one image has a resolution of 5 nm to 250 nm per pixel.
The motivation behind the modification would have been to obtain a method that improves machine learning model training, accuracy and classifications as well as improves the resolution of the super-resolution microscopy, since both YIP and NATAN concern cellular image analysis. Wherein YIP’s systems and methods provides improves the accuracy, classification and training of a machine learning model by disrupting shift invariance and the convergence speed and stability during training, while NATAN’s systems and methods that improves the resolution of the super-resolution microscopy. Please see YIP et al. (US 20200258223 A1), Paragraph [0094, 0365, 0370 and 0379] and NATAN et al. (US 20190120767 A1), Abstract and Paragraph [0046 and 0104].
Allowable Subject Matter
Claim 14 is therefrom objected to as being dependent upon rejected base claim 1 respectively but would be allowable if rewritten in independent form including all of the limitations of the base claims and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 14, the prior arts fail to explicitly teach, wherein the sample is expanded by permeating the sample with a polymer monomer composition comprising an a,p- unsaturated carbonyl monomer, comprising an acrylate, methacrylate, acrylamide, or methacrylamide monomer for producing a water-swellable (co)polymer, and an enal able to polymerize with the acrylate, methacrylate, acrylamide, or methacrylamide monomer; and polymerizing the polymer monomer composition with the enal to form a swellable material containing the cell or tissue sample, resulting in covalent linking of the enal to both the swellable material and a biomaterial in the sample, as claimed in claim 14.
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
BHARTI et al. (US 20210117729 A1)-a method is provided for non-invasively predicting characteristics of one or more cells and cell derivatives. The method includes training a machine learning model using at least one of a plurality of training cell images representing a plurality of cells and data identifying characteristics for the plurality of cells. The method further includes receiving at least one test cell image representing at least one test cell being evaluated, the at least one test cell image being acquired non-invasively and based on absorbance as an absolute measure of light, and providing the at least one test cell image to the trained machine learning model. Using machine learning based on the trained machine learning model, characteristics of the at least one test cell are predicted. The method further includes generating, by the trained machine learning model, release criteria for clinical preparations of cells based on the predicted characteristics of the at least one test cell......................Please see Fig. 1-2 and 6-7. Abstract.
STATE et al. (US 20210201571 A1)- A computer-implemented method for 3D reconstruction including obtaining 2D images and, for each 2D image, camera parameters which define a perspective projection. The 2D images all represent a same real object. The real object is fixed. The method also includes obtaining, for each 2D image, a smooth map. The smooth map has pixel values, and each pixel value represents a measurement of contour presence. The method also includes determining a 3D modeled object that represents the real object. The determining iteratively optimizes energy. The energy rewards, for each smooth map, projections of silhouette vertices of the 3D modeled object having pixel values representing a high measurement of contour presence. This forms an improved solution for 3D reconstruction.....................Please see Para. [0115]. Abstract.
TANG et al. (US 20180300855 A1)- A method and system for processing an image operates by: filtering a first real image to obtain a first feature map therefor with performances of image features improved; upscaling the obtained first feature map to improve a resolution thereof, the feature map with improved resolution forming a second feature map; and constructing, from the second feature map, a second real image having enhanced performances and a higher resolution than that of the first real image.......................Please see Fig. 1-3. Abstract.
Varadarajan et al. (US 20210018503 A1)- Presented herein are methods of evaluating cellular activity by: placing a cell population on an area; assaying for a dynamic behavior of the cell population as a function of time; identifying cell(s) of interest based on the dynamic behavior; characterizing a molecular profile of the cell(s); and correlating the obtained information. The assayed dynamic behavior can include cellular activation, cellular inhibition, cellular interaction, protein expression, protein secretion, cellular proliferation, changes in cellular morphology, motility, cell death, cell cytotoxicity, cell lysis, and combinations thereof. Sensors associated with the area may be utilized to facilitate assaying. Molecular profiles of the cell(s) can then be characterized by various methods, such as DNA analysis, RNA analysis, and protein analysis. The dynamic behavior and molecular profile can then be correlated for various purposes, such as predicting clinical outcome of a treatment, screening cells, facilitating a treatment, diagnosing a disease, and monitoring cellular activity......................Please see Fig. 6-7. Abstract.
OZCAN et al. (US 20190333199 A1)- A microscopy method includes a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of images comprising co-registered pairs of high-resolution microscopy images or image patches of a sample and their corresponding low-resolution microscopy images or image patches of the same sample. A microscopy input image of a sample to be imaged is input to the trained deep neural network which rapidly outputs an output image of the sample, the output image having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast.......................Please see Fig. 3-6. Abstract.
PRICE et al. (US 20020186874 A1)- In an image segmentation system that processes image objects by digital filtration, a digital filter is defined. The digital filter includes a neighborhood operator for processing intensity values of neighborhoods of pixels in a pixel array. A first pixel array is received defining a pixelated image including one or more objects and a background and a second pixel array is received that defines a reference image. The reference image includes at least one object included in the pixelated image in a background. In the reference image, pixels included in the at least one object are distinguished from pixels included in the background by a predetermined amount of contrast. Pixels of the first and second images are compared to determine a merit value; the merit value is used to compute neighborhood operator values; and, the neighborhood operator is applied to images in order to create or enhance contrast between objects and background in the images........................Please see Fig. 3-7. Abstract.
SALTZ et al. (US 20200388029 A1)- A system associated with quantifying a density level of tumor-infiltrating lymphocytes, based on prediction of reconstructed TIL information associated with tumoral tissue image data during pathology analysis of the tissue image data is disclosed. The system receives digitized diagnostic and stained whole-slide image data related to tissue of a particular type of tumoral data. Defined are regions of interest that represents a portion of, or a full image of the whole-slide image data. The image data is encoded into segmented data portions based on convolutional autoencoding of objects associated with the collection of image data. The density of tumor-infiltrating lymphocytes is determined of bounded segmented data portions for respective classification of the regions of interest. A classification label is assigned to the regions of interest. It is determined whether an assigned classification label is above a pre-determined threshold probability value of lymphocyte infiltrated. The threshold probability value is adjusted in order to re-assign the classification label to the regions of interest based on a varied sensitivity level of density of lymphocyte infiltrated. A trained classification model is generated based on the re-assigned classification labels to the regions of interest associated with segmented data portions using the adjusted threshold probability value. An unlabeled image data set is received to iteratively classify the segmented data portions based on a lymphocyte density level associated with portions of the unlabeled image data set, using the trained classification model. Tumor-infiltrating lymphocyte representations are generated based on prediction of TIL information associated with classified segmented data portions. A refined TIL representation based on prediction of the TIL representations is generated using the adjusted threshold probability value associated with the classified segmented data portions. A corresponding method and computer-readable device are also disclosed......................Please see Fig. 1-2. Abstract.
AKILESH et al. (US 20210310058 A1)- Disclosed herein are methods of detecting a target viral nucleic acid sequence, determining the localization of the target viral nucleic acid sequence, and/or quantifying the number of target viral nucleic acid sequences in a cell. This method may be used on small target nucleic acid sequences, and may be referred to as Nano-FISH or viral Nano-FISH.......................Please see Fig. 1, 3, 41 and 45. Abstract.
Hosseini et al. (US 20200349707 A1)- Various image diagnostic systems, and methods of operating thereof, are disclosed herein. Example embodiments relate to operating the image diagnostic system to identify one or more tissue types within an image patch according to a hierarchical histological taxonomy, identifying an image patch associated with normal tissue, generating a pixel-level segmented image patch for an image patch, generating an encoded image patch for an image patch of at least one tissue, searching for one or more histopathological images, and assigning an image patch to one or more pathological cases.....................Please see Fig. 2-3. Abstract.
BOYDEN et al. (US 20190064037 A1)- The invention provides a method for preparing an expanded biological specimen suitable for microscopic analysis. Expanding the biological sample can be achieved by binding, e.g., anchoring, key biomolecules to a polymer network and swelling, or expanding, the polymer network, thereby moving the biomolecules apart as further described below. As the biomolecules are anchored to the polymer network isotropic expansion of the polymer network retains the spatial orientation of the biomolecules resulting in an expanded, or enlarged, biological specimen.....................Please see Fig. 1. Abstract.
MOEN et al. (US 20200364857 A1)- Disclosed herein include systems and methods for biological object tracking and lineage construction. Also disclosed herein include cloud-based systems and methods for allocating computational resources for deep learning-enabled image analysis of biological objects. Also disclosed herein include systems and methods for annotating and curating biological object tracking-specific training datasets....................Please see Fig. 1-2 and 5-6. Abstract.
KARAM et al. (US 20190303720 A1)- Embodiments of a deep learning enabled generative sensing and feature regeneration framework which integrates low-end sensors/low quality data with computational intelligence to attain a high recognition accuracy on par with that attained with high-end sensors/high quality data or to optimize a performance measure for a desired task are disclosed......................Please see para. [0096-0100]. Abstract.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Aaron Bonansinga whose telephone number is (703) 756-5380 The examiner can normally be reached on Monday-Friday, 9:00 a.m. - 6:00 p.m. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Chineyere Wills-Burns can be reached by phone at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON TIMOTHY BONANSINGA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights