DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on February 4th, 2025 has been considered and the listed references were noted.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 316B.
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
The disclosure is objected to because of the following informalities:
In Paragraph [0016], “...utilizing a machine model train to…” should be replaced with “…utilizing a machine model trained to…”.
Appropriate correction is required.
Status of Claims
Claims 1-8, 11-16, 19-21, 26-27, and 29 are pending. Claims 9-10, 17-18, 22-25, and 28 are canceled.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 12 and 13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 12 recites the limitation "the at least one second bag of patches", but not the “at least one bag of patches” introduced in its parent claim 1. The “at least one second bag of patches” is first introduced in Claim 13, from which Claim 12 does not depend. There is insufficient antecedent basis for this limitation in the claim. Suggested correction: Amend Claim 12 to read “the at least one bag of patches”
Claim 13 recites the limitation “the one or more second feature maps”, but not the “one or more feature maps” introduced in its parent claim 1. The “one or more second feature maps” is not introduced in any other independent or dependent claim. There is insufficient antecedent basis for this limitation in the claim. Suggested correction: Amend Claim 13 to read “the one or more feature maps”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 5, 8, 12, 21, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Yip (US 2022/0101519) in view of Koller (US 20210366577) and Yang (CN 112101451).
Regarding Claim 1, Yip discloses A method, comprising:(Yip, Paragraph [0399], discloses “...the architecture 1800 may be used to trained neural networks with convolution layers creating a feature map that is input to fully connected classification layers, such as AlexNet or VGG….”);
; and “a third layer trained to generate the prediction of the image class label based at least in part on the one or more normalized feature maps” (Yip, Paragraph [0310] and Figure 12A, discloses: “FIG. 12A illustrates the layers of an example of the layer structure of the architecture 1200. FIG. 12B illustrates example output sizes for different layers and resulting sub-layers of the architecture 1200, showing the tile-resolution FCN configuration. As shown, the tile-resolution FCN configuration included in the tissue classifier module 306 has additional layers of 1×1 convolution in a skip connection, downsampling by a factor of 8 in a skip connection, and a confidence map layer, and replaces an average pooling layer with a concatenation layer, and a fully connected FCN layer with a 1×1 convolution and Softmax layer. The added layers convert a classification task into a classification-segmentation task. This means that instead of receiving and classifying a whole image as one tissue class label, the added layers allow the tile-resolution FCN to classify each small tile in the user-defined grid as a tissue class.”
PNG
media_image1.png
634
513
media_image1.png
Greyscale
Yip, Paragraph [0364], discloses “The feature maps of the small FOV branches are downsampled by 8 to match the dimensions of the ResNet-18 feature map. These feature maps are concatenated before passing through a softmax output to produce a PD-L1 biomarker prediction (confidence) map.”; From these two paragraphs and Figure 12A, we can see that there is clearly a fully connected and pooling layer used to classify the image class labels based on the feature maps seen in Paragraph [0364]); and “outputting, by the one or more processors, the prediction of the image class label” (Yip, Paragraph [0175], discloses “Training a tile based deep learning network to predict a biomarker classification label for each tile of the whole slide image may be performed using any of the methods described herein.”; It is important to note that a tile is analogous to the patch described in the claims in specification of this application. In the Koller reference later on in this analysis, the image segmentation process (through sectioning an image into tiles (aka “patches”) will be explained in further detail). Yip does not explicitly disclose “A method, comprising: segmenting, by one or more processors, an image into a plurality of patches”, “grouping, by the one or more processors, the plurality of patches into at least one bag of patches”, “grouping, by the one or more processors, the plurality of patches into at least one bag of patches; inputting, by the one or more processors, the at least one bag of patches into a machine- learning model trained to generate a prediction of an image class label based on the at least one bag of patches, the machine-learning model including”, or “a second layer trained to normalize the one or more feature maps utilizing a set of batch normalization parameters determined from the at least one bag of patches to generate one or more normalized feature maps”.
However, in an analogous field of endeavor, Koller discloses segmenting an image into a plurality of patches that “the phenotypic assay is an image and can be prepared for the machine learning model. For example, the image can be sectioned into tiles and/or elements in the images can be labeled (e.g., labeled cell types, labeled boundaries of cells, etc.) prior to inputting into the machine learning model” (Koller, Paragraph [0263]) whereas “the non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to perform steps comprising: applying a ML-enabled cellular disease model using at least a prediction generated from the machine learning model developed using embodiments of the method for developing the machine learning model described above” (Koller, Paragraph [0035]). Koller also discloses an entire methodology for grouping the plurality of patches into at least one bag of patches and inputting the bag of patches into a machine-learning model trained to generate a prediction of an image class label based on the at least one bag of patches, seen in Paragraphs [0580]-[0582]:
PNG
media_image2.png
149
458
media_image2.png
Greyscale
PNG
media_image2.png
149
458
media_image2.png
Greyscale
PNG
media_image3.png
470
458
media_image3.png
Greyscale
In Paragraphs [0580]-[0582], we can definitively see the segmenting of the image through sectioning the image into tiles as described previously, as the invention groups them based on their cellular phenotypes by inputting them into the machine learning model. Once these tiles (or patches) are inputted into the model, the model can then predict the histology scores of each tile to assign class labels to identify them accurately. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the first and third layers as well as the technique for outputting an image class label seen in Yip with the method of segmenting the image into patches, grouping the patches, and inputting the patches into a machine learning model to predict the image class label based on the bag of patches seen in Koller to achieve a more complete method for generating image class labels for different sections of the whole-slide image. By combining the elements from both the Yip and Koller methods, one of ordinary skill in the art can effectively obtain image class labels based on the state of the tissue cells on each section of the whole slide image. Thus, it would have been obvious for one of ordinary skill in the art to combine the Yip and Koller references to achieve the same limitations described in the Claim 1.
The combination of Yip and Koller does not explicitly disclose “a second layer trained to normalize the one or more feature maps utilizing a set of batch normalization parameters determined from the at least one bag of patches to generate one or more normalized feature maps”. However, in an analogous field of endeavor, Yang discloses the following in Paragraphs [0040] - [0042]:
PNG
media_image4.png
184
649
media_image4.png
Greyscale
It is clear from the aforementioned paragraphs that the feature maps are normalized through batch normalization, allowing for consistency throughout the analysis of breast cancer tissue that the Yang method describes. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the image class label prediction method seen in the combination of Yip and Koller with the technique of using batch normalization to normalize one or more feature maps seen in Yang to achieve an improved method. By using the method seen in the combination of Yip and Koller with the normalization technique seen in Yang, one of ordinary skill in the art ensures that each whole-slide image that is analyzed has a balanced patch set before being analyzed to determine multiple image class labels. Thus, it would have been obvious for one of ordinary skill in the art to combine the Yip, Koller, and Yang references to achieve the same method described in Claim 1
Regarding Claim 2, the combination of Yip, Koller and Yang discloses “The method of claim 1, wherein the image comprises only one whole-slide image (WSI)” (Yip, Paragraph [0399], discloses “The processes 712 and 714 are performed for each received tile and are used to express the information in the mask arrays in terms of coordinates that are in the coordinate space of the original whole-slide image.”; From this, we can see that although there are tiles used throughout the use of this invention, there still exists only one whole-slide image.) From this, we can see that although there are tiles used throughout the use of this invention, there still exists only one whole-slide image).
Regarding Claim 3, the combination of Yip, Koller, and Yang discloses “The method of claim 1, further comprising receiving, by the one or more processors, the image, wherein the image comprises an image of a tissue sample” (Yip, Paragraph [0020], discloses “...receiving a plurality of digital images of H&E stained training slides of training tissue samples corresponding to the respective biomarker to an image-based biomarker prediction system having one or more processors…”).
Regarding Claim 5, the combination of Yip, Koller, and Yang discloses “The method of claim 1, wherein the image comprises a histological stain image” (Koller, Paragraph [0579], discloses “Liver biopsies were obtained from patients, liver tissues were sliced, and tissue slices underwent immunohistochemistry staining. Histological slides were individually imaged and used to train a machine learning model.”); “a fluorescence in situ hybridization (FISH) image” (Koller, Paragraph [0233] and Figure 2C, discloses “As an example, high-dimensional phenotypic assay data may include image data e.g., high-resolution microscopy data or immunohistochemistry image data captured of the cell or population of cells. Additional examples of phenotypic assay data include cell sequencing data, protein expression data, gene expression data, cell metabolic data, cell morphology data, or cell interaction data. Further examples of phenotypic assay data include functional data, such as electrophysiological functional data for cardiac cells and electroencephalogram (EEG) or electrocorticography (ECoG) for brain cells. As shown in FIG. 2C, examples of phenotypic assays include high content imaging (e.g., cellular microscopy) as well as single cell RNA-sequencing. Additional phenotypic assays include ATACseq, assays for measuring protein expression levels, RNA-FISH, and other disease-specific assays.”
PNG
media_image5.png
546
874
media_image5.png
Greyscale
Here, RNA-FISH means RNA Fluorescence In-Situ Hybridization, which is the same as a FISH image), “an immunofluorescence (IF) image” (Koller, Paragraph [0416], discloses “In some scenarios, in vitro cells are plated in wells and then stained e.g., using primary/secondary antibodies that are fluorescently tagged. In some embodiments, the in vitro cells are fixed prior to imaging. In some embodiments, the in vitro cells can undergo live cell imaging to observe changes in the cellular phenotypes over time.”; As seen from Paragraph [0416], antibodies are fluorescently tagged to be seen clearly within the slide prior to imaging, and antibodies are a vital part of the immune system. Therefore, this can be considered an example of immunofluorescence imaging), or a hematoxylin and eosin (H&E) image” (Koller, Paragraph [0246], discloses “As another example, for immunohistochemistry imaging, cells can be stained using hematoxylin/eosin stains. Images can be captured using any suitable microscopy including bright field microscopy and phase contrast microscopy.”).
Regarding Claim 8, the combination of Yip, Koller, and Yang discloses “The method of claim 1, wherein the machine-learning model comprises one or more convolutional neural networks (CNNs), a multiple-instance learning (MIL) machine-learning model, or a multiple-instance learning convolutional neural network (MILCNN) machine-learning model” (Koller, Paragraph [0580], discloses “In a preferred embodiment, a convolutional neural network (CNN) is deployed to analyze histological image data. Specifically, the CNN is deployed using a multiple instance learning (MIL) approach, where features from multiple tiles (instances) within a biopsy are combined to predict the pathologist scores.”).
Regarding Claim 12, the combination of Yip, Koller, and Yang discloses “The method of claim 1, wherein the set of batch normalization parameters corresponds to only the at least one second bag of patches” (Yip, Paragraph [0026], discloses “...performing a pixel-based cell segmentation analysis on each of the H&E slide training images; optionally performing a tile-based biomarker classification analysis on each of the H&E slide training images…”; Paragraph [0150] discloses “the deep learning multiscale classifier module 304 is configured to perform cell segmentation through a cell segmentation model 316, where cell segmentation may be a pixel-level process of the histopathology image from normalization process 310. In other examples, this pixel-level process may be performed on image tiles received from the pipeline 315. In some examples, the cell segmentation process of the framework 304 results in classifications that biomarker classifications, because some of the biomarkers identified herein are determined from cell level analysis, in contrast to tissue level analysis.”; Yip, Paragraph [0378], discloses: “training may be performed with weakly supervised learning that involves only image level labeling, and no local labeling of tissue, cell, tumor, etc. The architectures may be configured with a label-less training front end having an algorithm with customized cost function that chooses which tile(s) should be used as the input with specific label. The process may be iterative, first, treating each histopathology image as a collection of tiles, where a single label of the image is applied to all the tiles in the collection. The tiles may be applied to an inference pipeline, such as through a network like ResNet 34, Inception-v3, or FCN, and predefined tile selection criteria such as probabilities of the neural network output may be used to select which output image tiles will be provided as an input to the same neural network for next round. This process may be repeated many times, given enough collections and tiles as input to the neural network, it will learn to differentiate tiles with different classes with higher accuracy as more iterations are performed.”; Paragraph [0026] and [0150] describes the pixel-level analysis and batch normalization on the various tiles, while [0378] describes the iteration of this process. Taking this into consideration, by using the model to iterate over different images multiple times, this presents a second bag of tiles (aka patches) to be used with the model for further training and diagnostic measures).
Regarding Claim 21, the combination of Yip, Koller, and Yang discloses “The method of claim 1, wherein the image class label comprises an indication of a genetic biomarker of a tissue sample captured in the image” (Koller, Paragraph [0150], discloses: “the disease factor analysis system 205 receives or performs a genetic analysis on tissue samples obtained from individuals, such as individuals 210 that have the particular disease.”; Koller, Paragraph [0193], discloses “inputting labels to the different candidate patients can involve distinguishing the candidate patients based on their subject data, an example of which includes distinguishing patients based on their expression of biomarkers that are associated with one of the labels. In various embodiments, the inputting of labels to the candidate patients involves applying one or more trained predictive models that have been previously trained to distinguish between the two labels based on biomarker data. For example, a predictive model may be a classifier that analyzes, as input, biomarker data of a patient, and then outputs a prediction as to the label. The predictive model may analyze one or more biomarkers, such as a panel of biomarkers, for determining a prediction of the label.”).
Claim 27 recites a system with elements corresponding to the steps recited in Claim 1. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yip, Koller, and Yang references, presented in rejection of 1, apply to this claim. Finally, the combination of Yip, Koller, and Yang references discloses a processor, a memory, and a non-transitory computer-readable storage media (for example, see Koller, Paragraph [0045]).
Claims 4, 6, and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Koller and Yang, and further in view of Georgescu (US 2022/0076411).
Regarding Claim 4, the combination of Yip, Koller, and Yang discloses “The method of claim 1” (Please refer to the above-described analysis for Claim 1) (Georgescu, Paragraph [0116]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Yip, Koller, and Yang with the technique of having each patch represent a plurality of pixels corresponding to the region of an image seen in Georgescu to result in an improved method for generating a plurality of patches. By combining the method seen in the combination of Yip, Koller, and Yang with the Georgescu technique, wherein a patch comprises pixels corresponding to an image region, one of ordinary skill in the art allows not only for different sections of the image to be analyzed, but allows for analysis to be performed on a pixel-level to understand the finer details of the whole-slide image as a whole. Therefore, it would have been obvious for one of ordinary skill in the art to combine the Yip, Koller, Yang, and Georgescu references to achieve the same method described in Claim 4.
Regarding Claim 6, the combination of Yip, Koller, and Yang discloses “The method of claim 1 wherein the first layer comprises one or more convolutional layers” (Yip, Paragraph [0366], discloses “The architecture 1550 includes a stack of convolutional layers interleaved with “shortcut connections,” which skip intermediate layers.”); the second layer comprises one or more batch normalization layers (Yang, Paragraphs [0040]-[0042], disclose the following:
PNG
media_image4.png
184
649
media_image4.png
Greyscale
); and “the third layer comprises an output layer”. The combination of Yip, Koller, and Yang does not explicitly disclose “the third layer comprises an output layer”. However, in an analogous field of endeavor, Georgescu discloses “In our current implementation, in each successive convolution stage, as the dimensions decrease, the depth increases, so that the convolution layers are of ever increasing depth as well as ever decreasing dimensions, and in each successive transpose convolution stage, as the dimensions increase, the depth decreases, so that the deconvolution layers are of ever decreasing depth as well as ever increasing dimensions. The final convolution layer then has a maximum depth as well as minimum dimensions. Instead of the approach of successive depth increases and decreases through respectively the convolution and deconvolution stages, an alternative would be to design a neural network in which every layer except the input layer and the output layer has the same depth.” (Georgescu, Paragraph [0033]). As seen in Paragraph [0033], the output layer is clearly described as the final convolution layer after going through multiple convolution stages. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Yip, Koller, and Yang with the technique of having an output layer seen in Georgescu to improve the method of Claim 6 in the same way.
Regarding Claim 7, the combination of Yip, Koller, Yang, and Georgescu discloses “The method of claim 6, wherein the machine-learning model further comprises a pooling layer and a fully connected layer” (Yip, Paragraph [0399], discloses “…the architecture 1800 may be used to trained neural networks with convolution layers creating a feature map that is input to fully connected classification layers, such as AlexNet or VGG…”; Yip, Paragraph [0317], discloses “...a convolution layer title that includes “/n”, where n is a number, indicates that there is a downsampling (also known as pooling) of the result matrix produced by that layer”).
Claims 11 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Koller and Yang, and further in view of Zhu (US 2020/0226440).
Regarding Claim 11, the combination of Yip, Koller, and Yang discloses “The method of claim 1” (Please refer to the above-described analysis for Claim 1) image outputted by a convolutional layer. Batch normalization is to perform standardization on each feature image according to the feature image's own mean and variance, for example” (Zhu, Paragraph [0074]). Here, although the excerpt mentions the mean and variance of the feature image, we have to take note of the fact that the image is sectioned off into different patches, which in turn count as a bag of patches for that specific feature within that feature image. This paragraph is further discussed in Claims 13 and 29, where we further see that patches from the feature image are involved in the mini-batch normalization process. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in Yip, Koller, and Yang with the technique of having a mean and variance from the bag of patches seen in Zhu to improve the method of Claim 11 in the same way.
Regarding Claim 13, the combination of Yip, Koller, and Yang, discloses “The method of claim 1, wherein the machine-learning model was trained by: receiving, by the one or more processors, a training image; segmenting, by the one or more processors, the training image into a second plurality of patches” (Yip, Paragraph [0200], discloses “As part of a training process, at a block 602, tile-labeled histopathology images are received at the deep learning framework 300. The histopathology images may be of any type herein, but are illustrated as digital H&E slide images in this example. These images may be training images of a previously-determined and labeled (and thus known) cancer type (e.g., for supervised learning configurations).”; Paragraph [0019] discloses “...receiving the digital image to an image-based biomarker prediction system having one or more processors; performing an image tiling process...”; Paragraph [0026] discloses “… performing tile-based tissue classification analysis on each of the H&E slide training images ….”; “grouping, by the one or more processors, the second plurality of patches into at least one second bag of patches” (Yip, Paragraph [0026] discloses: “...performing a pixel-based cell segmentation analysis on each of the H&E slide training images; optionally performing a tile-based biomarker classification analysis on each of the H&E slide training images…”); “inputting, by the one or more processors, the at least one second bag of patches into the machine-learning model to generate a prediction of a second image class label based on the at least one second bag of patches” (Yip, Paragraph [0037], discloses “In some examples, the method further comprises: providing each tile image to a tile selection process that infers a class status for each tile image in the H&E slide training image; and based on inferred class status, selectively discarding tile images based on a tile selection criteria before applying the remaining plurality of tile images to the deep learning framework.”); wherein: “the first layer is trained to generate one or more feature maps based on the at least one second bag of patches” (Yip, Paragraph [0399], discloses “...the architecture 1800 may be used to trained neural networks with convolution layers creating a feature map that is input to fully connected classification layers, such as AlexNet or VGG….”); (Yip, Paragraph [0175], discloses “Training a tile based deep learning network to predict a biomarker classification label for each tile of the whole slide image may be performed using any of the methods described herein.”). The combination of Yip, Koller, and Yang does not explicitly disclose “the second layer is trained to normalize the one or more second feature maps utilizing a set of mini-batch normalization parameters determined from the at least one second bag of patches to generate one or more second normalized feature maps”. However, in an analogous field of endeavor, Zhu discloses the following in Paragraphs [0074]-[0075]:
PNG
media_image6.png
428
341
media_image6.png
Greyscale
As we can see from the aforementioned paragraphs, this specific layer has specific mini-batch normalization parameters as a mini batch is incorporated into the batch normalization process. Also note that the elements from the patches are incorporated into the standardization formula, allowing for this normalization layer to be analogous to the second bag of patches within the limitation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Yip, Koller, and Yang with the mini-batch normalization parameters seen in Zhu to result in an improved method for predicting image class labels for the training image. By combining the mini batch normalization parameters seen in Zhu with the method seen in Yip, Koller, and Yang, one of ordinary skill in the art can perform the same method described in Claim 1 on a training image, but with a different set of parameters to make it distinct from the current patches from the whole-slide image to train the model adequately to mitigate the amount of errors present in the model when using the method. Therefore, it would have been obvious for one of ordinary skill in the art to combine the Yip, Koller, Yang, and Zhu references to achieve the same method described in Claim 13.
Claims 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Koller and Yang, and further in view of Zhu and Georgescu.
Regarding Claim 14, the combination of Yip, Koller, Yang, and Zhu discloses “The method of claim 13” (Please refer to the above-described analysis for Claim 13) the output patches of the CNN can be stitched together taking account of any discrepancies. Our approach can however, if desired, also be applied to a random sample of patches over the WSI which are of the same or different magnification, as in the prior art, or as might be carried out by a pathologist.” (Georgescu, Paragraph [0116]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Yip, Koller, Yang, and Zhu with the technique of having each patch represent a plurality of pixels corresponding to the region of an image seen in Georgescu to result in an improved method for generating a second plurality of patches. By combining the method seen in the combination of Yip, Koller, and Yang with the Georgescu technique, wherein a patch comprises pixels corresponding to an image region, one of ordinary skill in the art allows not only for different sections of the image to be analyzed, but allows for analysis to be performed on a pixel-level to understand the finer details of the training whole-slide image. Therefore, it would have been obvious for one of ordinary skill in the art to combine the Yip, Koller, Yang, Zhu and Georgescu references to achieve the same method described in Claim 14.
Regarding Claim 15, the combination of Yip, Koller, Yang, and Zhu discloses “The method of claim 1 wherein the first layer comprises one or more convolutional layers” (Yip, Paragraph [0366], discloses “The architecture 1550 includes a stack of convolutional layers interleaved with “shortcut connections,” which skip intermediate layers.”); the second layer comprises one or more batch normalization layers (Yang, Paragraphs [0040]-[0042], disclose the following:
PNG
media_image4.png
184
649
media_image4.png
Greyscale
); and “the third layer comprises an output layer”. The combination of Yip, Koller, Yang and Zhu does not explicitly disclose “the third layer comprises an output layer”. However, in an analogous field of endeavor, Georgescu discloses “In our current implementation, in each successive convolution stage, as the dimensions decrease, the depth increases, so that the convolution layers are of ever increasing depth as well as ever decreasing dimensions, and in each successive transpose convolution stage, as the dimensions increase, the depth decreases, so that the deconvolution layers are of ever decreasing depth as well as ever increasing dimensions. The final convolution layer then has a maximum depth as well as minimum dimensions. Instead of the approach of successive depth increases and decreases through respectively the convolution and deconvolution stages, an alternative would be to design a neural network in which every layer except the input layer and the output layer has the same depth.” (Georgescu, Paragraph [0033]). As seen in Paragraph [0033], the output layer is clearly described as the final convolution layer after going through multiple convolution stages. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Yip, Koller, Yang and Zhu with the technique of having an output layer seen in Georgescu to improve the method of Claim 15 in the same way.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Koller and Yang, and further in view of Zhu, Georgescu, and El Yaniv (US 2017/0286830).
Regarding Claim 16, the combination of Yip, Koller, Yang, Zhu, and Georgescu (which, for the sake of brevity, will be abbreviated as YKYZG) discloses “The method of claim 15” (Please refer to the above-described analysis for Claim 15) Paragraphs [0067-[0069]:
PNG
media_image7.png
553
457
media_image7.png
Greyscale
It is important to note that the running variance is computed for the batch that is being normalized, and after that running variance is computed, that then gets further incorporated into the shift-based batch normalization technique, allowing for the training set to have statistical information while normalizing the data. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of YKYZG with the El Yaniv technique of obtaining a running variance from the batch normalization process to apply to mini-batch normalization parameters to have a more complete image prediction method for the training image. By using the El Yaniv technique with the method seen in YKYZG, one of ordinary skill in the art can have these statistical parameters in advance to further train the model when using it to represent the entire training dataset acquired from the training image. Therefore, it would have been obvious for one of ordinary skill in the art to combine the Yip, Koller, Yang, Zhu, Georgescu, and El Yaniv references to achieve the same method described in Claim 16.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Koller and Yang, and further in view of Zhu and Labatie (US 2023/0098994, w/ EFD of September 29th, 2021).
Regarding Claim 19, the combination of Yip, Koller, Yang, and Zhu discloses “The method of claim 13” (Please refer to the above-described analysis for Claim 13) . The combination of Yip, Koller, Yang, and Zhu is not relied on to disclose “wherein the set of mini-batch normalization parameters comprises a mini-batch mean and a mini-batch variance”. However, in an analogous field of endeavor, Labatie discloses “Many deep neural networks employ a technique known as ‘Batch Normalisation’ to improve training. Batch Normalisation works by normalising intermediate tensors of the network, i.e. the outputs of the various layers of the network, to zero mean and unit variance for each mini-batch of training data, by computing a mean and variance of the elements of all training examples of the mini-batch, a single mean being computed for each channel in the intermediate tensors of the entire mini-batch. In other words, a mean and variance are computed for each channel, across all spatial dimensions and training examples of the mini-batch. For convolutional neural networks, Batch Normalisation is applied over a given mini-batch for all channels and spatial dimensions.”. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Yip, Koller, Yang, and Zhu with the technique of using a mini-batch mean and mini-batch variance seen in Labatie to achieve a more complete method for predicting image class labels of the training image. By using the Labatie technique of using a mini batch mean and variance, one of ordinary skill in the art can use both mini-batch normalization parameters to standardize the data before continuing to use the machine learning model to predict image class labels. Therefore, it would have been obvious for one of ordinary skill in the art to combine the Yip, Koller, Yang, Zhu, and Labatie references to achieve the same method described in Claim 19.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Koller and Yang, and further in view of Zhu and Shaul (US 20220237788).
Regarding Claim 20, the combination of Yip, Koller, Yang, and Zhu discloses “The method of claim 13” (Please refer to the above-described analysis for Claim 14) (Shaul, Paragraph [0347]) whereas “The training image 900 is split into a plurality of tiles.” (Shaul, Paragraph [0355]). Shaul also discloses that “A second set of tile pairs is created. Each tile pair of the second set comprises the start tile and a “distant” tile in respect to the start tile. For example, this step can comprise creating as many tile pairs as distant tiles are contained in the image 800 outside of the second circle. Alternatively, this step can comprise randomly selecting a subset of the available distant tiles and creating a tile pair for each of the selected distant tiles by adding the start tile to the selected distant tile” (Shaul, Paragraph [0358]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Yip, Koller, Yang, and Zhu with the technique of randomly sampling patches of pixels from the second bag of patches seen in Shaul to achieve a more realistic approach when using the image class label prediction method. By combining the Shaul technique of randomly sampling the second bag of patches with the method seen in the combination of Yip, Koller, Yang and Zhu, one of ordinary skill in the art can ensure the machine learning model does not train from the same patches constantly to ensure the model can identify specific sets of information without being biased towards one specific patch within the whole-slide image. Therefore, it would have been obvious for one of ordinary skill in the art to combine the Yip, Koller, Yang, Zhu, and Shaul references to achieve the same method described in Claim 20.
Claim 26 rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Koller, and in further view of Yang.
Regarding Claim 26, Yip discloses “A method of treating subject with cancer, comprising: characterizing a tissue sample comprising the cancer from the subject” (Yip, Paragraph [0014], discloses “The present application presents an imaging-based biomarker prediction system formed of a deep learning framework configured and trained to directly learn from histopathology slide images and predict the presence of biomarkers in medical images. In examples, deep learning frameworks are configured and trained to analyze histopathology images and identify a plurality of different biomarkers. In various examples, these deep learning frameworks are configured to include different trained biomarker classifiers each configured to receive unlabeled histopathology images and provide different biomarker predictions for those images. These biomarker predictions may then be used to reduce a large set of available immunotherapies to a reduced, small subset of targeted immunotherapies that medical professionals may use to treat patients. As such, in various examples, deep learning frameworks are provided that identify biomarkers indicating the presence of a tumor, a tumor state/condition, or information about a tumor of the tissue sample, from which a set of target immunotherapies can be determined.”); (Yip, Paragraph [0015], discloses “In some examples, the predicted biomarker status reports may be input to network-accessible next generation sequencing systems for driving subsequent genomic sequencing, or input to computerized cancer therapy decision systems for filtering therapy listings down to biomarker-determined matched therapies.”). Yip does not explicitly disclose “as having a genetic biomarker according to the method of claim 21”. However, in an analogous field of endeavor, we can recall that Koller discloses from claim 21 that “the disease factor analysis system 205 receives or performs a genetic analysis on tissue samples obtained from individuals, such as individuals 210 that have the particular disease” (Koller, Paragraph [0150]), where he further describes that “inputting labels to the different candidate patients can involve distinguishing the candidate patients based on their subject data, an example of which includes distinguishing patients based on their expression of biomarkers that are associated with one of the labels. In various embodiments, the inputting of labels to the candidate patients involves applying one or more trained predictive models that have been previously trained to distinguish between the two labels based on biomarker data. For example, a predictive model may be a classifier that analyzes, as input, biomarker data of a patient, and then outputs a prediction as to the label. The predictive model may analyze one or more biomarkers, such as a panel of biomarkers, for determining a prediction of the label.” (Koller, Paragraph [0151]). Both references clearly describe predicting and showing genetic biomarkers in great detail, but Koller demonstrates the fact that the patients are shown expressing specific biomarkers that contribute to the condition of the tissue associated with a specific organ in their body that could potentially relate with cancer. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the tissue sample characterization and cancer treatment administration techniques seen in Yip with the technique of identifying a genetic biomarker from the method described in Claim 21 seen in Koller to achieve a complete method of treating subjects with cancer.
Claim 29 is rejected under 35 U.S.C. 103 as being unpatentable over Yip in view of Zhu.
Regarding Claim 29, Yip discloses “A method, comprising: receiving, by one or more processors, a training image; segmenting, by the one or more processors, the training image into a plurality of patches” (Yip, Paragraph [0200], discloses “As part of a training process, at a block 602, tile-labeled histopathology images are received at the deep learning framework 300. The histopathology images may be of any type herein, but are illustrated as digital H&E slide images in this example. These images may be training images of a previously-determined and labeled (and thus known) cancer type (e.g., for supervised learning configurations).”; Paragraph [0019] discloses “...receiving the digital image to an image-based biomarker prediction system having one or more processors; performing an image tiling process...”; Paragraph [0026] discloses “… performing tile-based tissue classification analysis on each of the H&E slide training images ….”); “grouping, by the one or more processors, the plurality of patches into at least one bag of patches” (Yip, Paragraph [0026], discloses “...performing a pixel-based cell segmentation analysis on each of the H&E slide training images; optionally performing a tile-based biomarker classification analysis on each of the H&E slide training images…”); “training a first layer to generate one or more feature maps based on the at least one bag of patches” (Yip, Paragraph [0399], discloses “...the architecture 1800 may be used to trained neural networks with convolution layers creating a feature map that is input to fully connected classification layers, such as AlexNet or VGG….”); “training a third layer to generate the prediction of an image class label for the training image based at least in part on the one or more normalized feature maps.” (Yip, Paragraph [0175], discloses “Training a tile based deep learning network to predict a biomarker classification label for each tile of the whole slide image may be performed using any of the methods described herein.”). Yip does not explicitly disclose “training a second layer to normalize the one or more feature maps utilizing a set of mini- batch normalization parameters from the one or more normalized feature maps”. However, in an analogous field of endeavor, Zhu discloses the following in Paragraphs [0074]-[0075]:
PNG
media_image8.png
669
533
media_image8.png
Greyscale
As mentioned before in Claim 13, this specific layer has specific mini-batch normalization parameters as a mini batch is incorporated into the batch normalization process. Also note that the elements from the patches are incorporated into the standardization formula, allowing for this normalization layer to be analogous to the second bag of patches within the limitation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the segmenting and grouping of patches as well as the first and third layers seen in Yip with the mini-batch normalization parameters seen in Zhu to result in an improved method for predicting image class labels for the training image. By combining the mini batch normalization parameters seen in Zhu with the patch techniques and first and third layers seen in Yip, one of ordinary skill in the art can perform an adequate image class label prediction analysis on training image with parameters that associate with the mini batch normalization for the patches seen from the whole-slide training image. Therefore, it would have been obvious for one of ordinary skill in the art to combine the Yip and Zhu references to achieve the same method described in Claim 29.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kamath (US 20190286945) teaches a device that forms a neural network envelope cell comprising a plurality of convolution-based filters in series or parallel. The device constructs a convolutional neural network by stacking copies of the envelope cell in series.
Kapur (US 20250342588, w/ an EFD of September 9th, 2019) teaches systems and methods for receiving a target electronic image corresponding target specimen, applying a machine learning system to the target electronic image to identify a ROI, an determine an expression level, category, and/or prescence of a biomarker.
Wirch (US 20210407076) teaches a method where whole-slide images (WSIs) are aligned with a multi-resolution registration algorithm, normalized for improved processing, annotated by an expert user, and divided into image patches.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SORIE I KOROMA JR whose telephone number is (571)272-9259. The examiner can normally be reached Monday - Friday 8AM-6:00PM; Alternate Fridays Off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SORIE I KOROMA JR/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights