DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Claims 16-25 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected species, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 2 October, 2026.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 3, and 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) in view of Arnold et al. (US 20220207730 A1).
Regarding claim 1, Li et al. disclose a computer-implemented method for classifying an input image, the method comprising: generating a feature map for the input image using a trained neural network that includes at least one convolutional layer (“The SegMitos model trained with concentric label for mitosis detection. In the label map, the small white circle denotes the estimated mitotic region and the green ring denotes the ignored region when computing loss. Segmentation network yields a pixel-wise segmentation map, Fig. 2, We propose a novel concentric label as well as a concentric loss, making it possible to train a dense prediction model with weak annotation”, p167, Mitosis segmentation based on fully convolutional network, p168) [CNN necessarily involves producing feature maps]; generating a plurality of concentric crops of the feature map (To tackle this problem, we expand the single-pixel label to a novel label with concentric circles, where the inside circle is a mitotic region and the ring around the inside circle is a “middle ground”, abstract, “Therefore, we propose a novel concentric label based on the weak centroid annotation. Firstly, we use a circle which centered at the annotated pixel as an estimated mitotic region. Then considering that there may be some mitotic pixels outside the circle, we encompass the mitotic circle using a larger circle” p166); generating an output vector that represents a characteristic of a structure depicted in a center region of the input image using information from each of the plurality of concentric crops (To tackle this problem, we expand the single-pixel label to a novel label with concentric circles, where the inside circle is a mitotic region and the ring around the inside circle is a “middle ground”, abstract, “Therefore, we propose a novel concentric label based on the weak centroid annotation. Firstly, we use a circle which centered at the annotated pixel as an estimated mitotic region. Then considering that there may be some mitotic pixels outside the circle, we encompass the mitotic circle using a larger circle” p166, “The first one is a circle centered at the original centroid pixel, as shown in Fig. 4 (a). We regard the circular region as a mitotic cell,” p169); and determining a classification result by processing the output vector (“The output of SegMitos is a segmentation map in which the score of a pixel denotes its confidence of belonging to the mitosis class. We stitch the segmentation maps of these patches to get the response map of the full HPF. The prediction map is noisy due to the existence of ambiguous cells and noise”, “For a blob, if its mean score ¯S is higher than a threshold s 1 and meanwhile its area A is larger than a threshold a 1, this blob will be kept as a detection, otherwise, it will be filtered out” p169).
While CNNs typically involve generating an output vector that represents a characteristic of a structure, and determining a classification result by processing the output vector is also the standard way CNNs operate, another reference is added to make this explicit.
Arnold et al. teach generating a feature map for the input image using a trained neural network that includes at least one convolutional layer (The flow 300 can include individually providing each of the tiles 308 to the first trained model 312. In some configurations, the first trained model 312 can include a convolutional neural network (CNN). In some configurations, the first trained model 312 can be trained to generate a number of feature maps based on an input tile. Thus, the first trained model can function as a feature extractor, [0049], “The flow 300 can include providing the feature maps to a first attention module 316. In some configurations, the first attention module 316 can include a multilayer perceptron (MLP). The first attention module 316 can generate a first number of attention values 320 based on the feature maps generated by the first trained model 312. In some configurations, the first attention module 316 can generate an attention value for a tile based on the feature maps associated with the tile”, [0052], CNN feature extractor: In some configurations, a Vggl1 model with batch normalization (Vggl1bn) is used as the backbone for the feature extractor in both 5× and 10× models. The Vggl1bn may be initialized with weights pretrained on ImageNet. The feature extractor was first trained on the tile-level dataset for tile classification. After that, the fully connected layers were replaced by a 1×1 convolutional layer to reduce the feature map dimension, [0086]) generating an output vector that represents a characteristic of a structure depicted in the input image using information from each of the plurality of concentric crops and determining a classification result by processing the output vector (In some configurations, the clustering 332 can include clustering the first number of tiles 308 based on the feature maps and the first number of attention values 320. In some configurations, the flow 300 can include reducing each feature map associated with each tile to a one-dimensional vector. In some configurations, the flow 300 can include reducing feature maps of size 512×4×4 reduced to a 64×4×4 map after a final 1×1 convolution layer, and flattening the 64×4×4 map to form a 1024×1 vector, [0055], After training the first extractor, the fully connected layers can be replaced by a 1×1 convolutional layer to reduce the feature map dimension, outputs of which can be flattened and used as instance feature vectors V in the MIL model for slide classification, [0062], In some configurations, process 400 can reduce feature maps of size 512×4×4 to 64×4×4 after the 1×1 convolution, and then flattened to form a 1024×1 vector using a fully connected layer embedded it into a 1024×1 instance feature vector., [0064], The feature extractor was first trained on the tile-level dataset for tile classification. After that, the fully connected layers were replaced by a 1×1 convolutional layer to reduce the feature map dimension, outputs of which were flattened and used as instance feature vectors V in the MIL model for slide classification., [0086], The flow 700 can include flattening the feature maps and feeding the feature maps into a fully connected layer with 256 nodes, followed by ReLU and dropout layers (in training only), which can output the first number of feature vectors 716, [0098], At 912, the process 900 can receive feature vectors associated with the first number of tiles from the first trained model. In some configurations, the feature vectors can be the feature vectors 716 in FIG. 7. At 916, the process 900 can generate a first number of attention values based on the feature vectors associated with the first number of tiles. In some configurations, the process 900 can provide each of the feature vectors to a first attention model, [0122]-[0123]) [tiles of Arnold et al. interpreted as the concentric crops, in combination with the actual concentric crops of Li et al. this teaches the entirety of the limitations].
Li et al. and Arnold et al. are in the same art of using CNNs for processing slide images (Li et al., p168, p170; Arnold et al., abstract, [0064]). The combination of Arnold et al. with Li et al. will enable processing an output vector. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the output vector of Arnold et al. with the invention of Li et al. as this was known at the time of filing, the combination would have predictable results, and as Arnold et al. indicate “In some configurations of the present disclosure, an attention-based multiple instance learning (MIL) model is provided that can predict slide-level labels, but also provide visualization of relevant regions using inherent attention maps. Unlike previous work that relied on labor intensive labels, such as manually drawn regions of interest (ROIs) around glands, our model is trained using labels, such as slide-level labels, also known as weak labels, which can be easily retrieved from pathology reports” “The model achieved state-of-the-art performance, with a classification accuracy of 85.11% on a hold-out test set consisting of 860 slides from 227 patients. ROI-level classification” ([0025]) therefore reducing the need for manual intervention and increasing classification accuracy.
Regarding claim 2, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. and Arnold et al. further indicate the structure depicted in the center region of the input image is a structure to be classified (Our method, called SegMitos, is illustrated in Fig. 2. We train a segmentation network based on fully convolutional network (FCN) (Long et al., 2015) with the mitosis data. The SegMitos model produces a segmentation map where each pixel represents its confidence of belonging to the mitosis class, p166, Our SegMitos model is based on a fully convolutional network designed for semantic segmentation on natural images. Fully convolutional network (Long et al., 2015) adapts all fully connected layers in conventional classification network to convolutional layers, p168,
PNG
media_image1.png
710
948
media_image1.png
Greyscale
Arnold et al., In some configurations, the first extractor can be trained based on a tile-level dataset. In some configurations, the tile-level dataset can include a number of slides annotated at the pixel-level (i.e., each pixel is labeled as benign, low-grade, and high grade). The low-grade and high-grade classifications can be normalized to “cancer” for the first extractor., [0062], CNN feature extractor: In some configurations, a Vggl1 model with batch normalization (Vggl1bn) is used as the backbone for the feature extractor in both 5× and 10× models. The Vggl1bn may be initialized with weights pretrained on ImageNet. The feature extractor was first trained on the tile-level dataset for tile classification, [0086]).
Regarding claim 3, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. further indicate the structure depicted in the center region of the input image is a biological cell (Our method, called SegMitos, is illustrated in Fig. 2. We train a segmentation network based on fully convolutional network (FCN) (Long et al., 2015) with the mitosis data. The SegMitos model produces a segmentation map where each pixel represents its confidence of belonging to the mitosis class, p166, Our SegMitos model is based on a fully convolutional network designed for semantic segmentation on natural images. Fully convolutional network (Long et al., 2015) adapts all fully connected layers in conventional classification network to convolutional layers, p168,
PNG
media_image1.png
710
948
media_image1.png
Greyscale
Regarding claim 5, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. further indicate the classification result predicts that the input image depicts a mitotic figure (
PNG
media_image1.png
710
948
media_image1.png
Greyscale
A deep learning based automatic and accurate mitosis detection method is proposed in this paper. We utilize a semantic segmentation model to segment breast histopathology images. Based on the probability map, we develop a filter mechanism to seek out mitotic cells from candidates, p177).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) and Arnold et al. (US 20220207730 A1) as applied to claim 1 above, further in view of Xia et al. (US 20210319560 A1).
Regarding claim 4, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. partly indicate for each of the plurality of concentric crops, a center of the feature map is coincident with a center of the crop (“The first one is a circle centered at the original centroid pixel, as shown in Fig. 4 (a). We regard the circular region as a mitotic cell. Based on a statistic analysis in 2012 MITOSIS training set, the average area of mitotic cells is about 590 pixels. Considering that the shape of a mitotic cell is irregular and unknown, we make the circle small so that the pixels within the circle incline to be positive. Specifically, we try two different configurations in circle label: the circle label with 12-pixels radius and the circle label with 15- pixels radius”, p169) however another reference is added to make this explicit.
Xia et al. teach for each of the plurality of concentric crops, a center of the feature map is coincident with a center of the crop (“After the training for the first segmentation sub-network is completed, the plurality of input images having unified resolution passes through the trained first segmentation sub-network to obtain four feature maps having the resolution of 576×576×96, 288×288×48, 144×144×24, and 72×72×12, and two probability outputs having resolution of 72×72×12. According to the probability outputs of the low resolution, a rough segmentation result for the heart cavity is obtained by using maximum value comparison, where the resolution is 72×72×12. Based on the rough segmentation result, the coordinates of the center of gravity of the heart cavity are calculated, and areas which have fixed sizes and are capable of fully covering the target cavity are cropped from the four feature maps having the resolution of 576×576×96, 288×288×48, 144×144×24, and 72×72×12 by taking the coordinates of the center of gravity as the center. In one example, an area having a size of 30×20×12 is cropped from the feature map of 72×72×12, an area having a size of 60×40×24 is cropped from the feature map of 144×144×24, an area having a size of 120×80×48 is cropped from the feature map of 288×288×48, and an area having a size of 240×160×96 is cropped from the feature map of 576×576×96”, [0118]).
Li et al. and Arnold et al. and Xia et al. are in the same art of using CNNs for processing biological images (Li et al., p168, p170; Arnold et al., abstract, [0064]; Xia et al., [0015], [0061]). The combination of Xia et al. with Li et al. and Arnold et al. will enable having a center of the feature map is coincident with a center of the crop. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the center of Xia et al. with the invention of Li et al. and Arnold et al. as this was known at the time of filing, the combination would have predictable results, and as Xia et al. indicate “Embodiments of the present disclosure implement target object positioning and segmentation at the same time in a process of image processing, and the image processing precision is improved while the speed of image processing is guaranteed” (abstract) providing a speed improvement when inventions are combined.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) and Arnold et al. (US 20220207730 A1) as applied to claim 1 above, further in view of Fuchs et al. (US 20200285890 A1).
Regarding claim 6, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. and Arnold et al. do not explicitly disclose generating a latent embedding of at least a portion of the input image using a trained encoder that includes at least one convolutional layer, wherein generating the feature map also uses the latent embedding.
Fuchs et al. teach generating a latent embedding of at least a portion of the input image using a trained encoder that includes at least one convolutional layer, wherein generating the feature map also uses the latent embedding (Cancer histopathology images exhibit high intra- and inter-heterogeneity because of their size (as large as tens of billions of pixels) While other studies have performed unsupervised clustering of whole slide tiles based on image features, they have been used to address the problem of image segmentation and relied on clustering a developed latent space. This study adjusts the latent space with each iteration of clustering, [0084], “Referring now to FIG. 9D, depicted is a block diagram of a convolution stack 950 in the encoder block 942 of the image reconstruction model 908 in the system 900. Each convolution stack 950 in the convolution stack 950 can have a set of transform layers 952A-N (hereinafter generally referred to as transform layers 952). The set of transform layers 952 can include one or more weights to modify or otherwise process the input to produce or generate an output feature map 956. The input may include one of the tiles 940 when the convolution stack 950 is the first in the encoder block 942. The input may include a resultant feature map 956 when the convolution stack 950 is not the first in the encoder block 942. The output feature map 956 may correspond to one of the embedding representations 940′. The set of transform layers 952 can be arranged in series, with an output of one transform layer 952 fed as an input to a succeeding transform layer 952. Each transform layer 952 may have a non-linear input-to-output characteristic. In some embodiments, the set of transform layers 952 may be a convolutional neural network (CNN), including a convolutional layer, a normalization layer, and an activation layer (e.g., a rectified linear unit (ReLU)), among others”, [0126]).
Li et al. and Arnold et al. and Fuchs et al. are in the same art of using CNNs for processing slide images (Li et al., p168, p170; Arnold et al., abstract, [0064]; Fuchs et al., [0015], [0126]). The combination of Fuchs et al. with Li et al. and Arnold et al. will enable using a latent embedding. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the latent embedding of Fuchs et al. with the invention of Li et al. and Arnold et al. as this was known at the time of filing, the combination would have predictable results, and as Fuchs et al. indicate “There is an urgent need for efficient identification of prognostically relevant cellular and structural morphologies from limited histology datasets of rare cancers, such as ICC, to build risk stratification systems which are currently lacking across many cancer types. Computational pathology offers a new set of tools, and more importantly, a new way of approaching the historical challenges of subtyping cancers using computer vision-based deep learning, leveraging the digitization of pathology slides, and taking advantage of the latest advances in computational processing power. Presented herein is a new deep learning-based model which uses a unique neural network-based clustering approach to group together histology based on visual similarity. With this visual dictionary, clusters may be interpreted as subtypes and train a survival model, showing significant results for the first time in ICC.” ([0082]) and “In order to effectively cluster this extremely large amount of high intra-variance data into subsets which are based on similar morphologies, a neural network-based clustering cost-function, shown to outperform clustering techniques on images of hand-written digits, may be combined with a novel deep convolutional architecture” ([0084]) demonstrating a benefit in the combination of inventions to improve health applications and a benefit in being able to handle large amounts of data.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) and Arnold et al. (US 20220207730 A1) as applied to claim 1 above, further in view of Fuchs et al. (US 20210295528 A1).
Regarding claim 7, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Arnold et al. further partly indicate generating each of a plurality of feature maps using a respective one of a plurality of final layers of the trained neural network, wherein generating the feature map uses a concatenation of the plurality of feature maps (Furthermore, exposing the last layer allows features to be easily added from the heuristic feature extractor 122. This way the CNN 124, instead of providing a classification, outputs a feature vector, which can be easily concatenated with other features and then classified by the classifier 128. Hence the CNN 124 becomes just a feature extractor, but the features it extracts are "learned" from the data examples, while the heuristic features are hand crafted by a designer/user, [0021]) however another reference is added to make this more explicit.
Fuchs et al. teach generating each of a plurality of feature maps using a respective one of a plurality of final layers of the trained neural network, wherein generating the feature map uses a concatenation of the plurality of feature maps (FIG. 2A depicts example images. A convolutional neural network can predict 7 classes: (a) viable tumor, (b) necrosis with bone, (c) necrosis without bone, (d) normal bone, (e) normal tissue, (f) cartilage, and (g) blank. The goal is to accurately segment viable tumor and necrotic tumor on osteosarcoma whole slide images for automated treatment response assessment, [0024], FIG. 3A depicts example images. An example of Deep Interactive Learning (DIaL). (a) An original training whole slide image, (b) an exhaustive annotation, (c) an initial annotation, (d) the first prediction from a CNN trained by the initial annotation, [0026], A Deep Multi-Magnification Network (DMMN) may be used for multi-class tissue segmentation where the model looks at patches in multiple magnifications for accurate predictions. Specifically, DMMN is composed of three half-channeled U-Nets, U-Net-20×, U-Net-10×, and U-Net-5×, where in-put patches of these U-Nets are in 20×, 10×, and 5× magnifications, respectively, with size of 256×256 pixels centered at the same location. Intermediate feature maps in decoders of U-Net-10× and U-Net-5× are center-cropped and concatenated to a decoder of U-Net-20× to enrich feature maps. The final prediction patch of DMMN is generated in size of 256×256 pixels in 20× magnification, [0052]).
Li et al. and Arnold et al. and Fuchs et al. are in the same art of using CNNs for processing slide images (Li et al., p168, p170; Arnold et al., abstract, [0064]; Fuchs et al., [0023]- [0026]). The combination of Fuchs et al. with Li et al. and Arnold et al. will enable concatenation of the plurality of feature maps. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the concatenation of the plurality of feature maps of Fuchs et al. with the invention of Li et al. and Arnold et al. as this was known at the time of filing, the combination would have predictable results, and as Fuchs et al. indicate “Deep Interactive Learning (DIaL) may be used to minimize pathologists' annotation time by iteratively annotating mislabeled regions to improve a model. DIaL may be used with a pretrained model from a different cancer type to reduce manual training annotation on pancreatic pathology images” ([0048]) providing a cost savings motivation to combining inventions.
Claim(s) 8 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) and Arnold et al. (US 20220207730 A1) as applied to claim 1 above, further in view of Yeh et al. (US 20230128432 A1).
Regarding claim 8, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. further partly indicate generating the output vector using information from each of the plurality of concentric crops includes: for each of the plurality of concentric crops, generating a corresponding one of a plurality of feature vectors using at least one pooling operation; and generating the output vector using information from each of the plurality of feature vectors (
PNG
media_image2.png
346
486
media_image2.png
Greyscale
) however another reference is added to make this more explicit.
Yeh et al. teach generating the output vector using information from each of the plurality of concentric crops includes: for each of the plurality of concentric crops, generating a corresponding one of a plurality of feature vectors using at least one pooling operation; and generating the output vector using information from each of the plurality of feature vectors (Disclosed are an object detection method and a convolution neural network. The method is performed through hierarchical architecture of the CNN and includes extracting groups of augmented feature maps from an input image through a backbone and two other groups of feature maps, identifying positive and negative samples with an IOU-based sampling scheme to be proposals for foreground and background through a proposal-sampling classifier, mapping the proposals to regions on the groups of augmented feature maps through the region proposal module, pooling the regions to fixed scale feature maps based on ROI aligning, fusing the fixed scale feature maps, and flattening the fused feature maps to generate an ROI feature vector through an ROI aligner for object classification and box regression, abstract, The described embodiments concern one or more methods, systems, apparatuses, and computer readable mediums storing processor-executable process steps for training a convolution neural network to perform object detection and instance segmentation. In the embodiments, during the course of forward propagation in training a convolutional neural network (CNN), the CNN receives an image, generates a group of augmented feature maps, pools a region on a corresponding feature map of the group of augmented feature maps corresponding to a proposal or an ROI (region of interest) on the image to a fixed size feature map, flattens the fixed size feature map to an ROI feature vector, fuses the ROI feature vectors from the group of augmented feature maps to generate a fused ROI vector for object classification and anchor box regression, [0027], classifies at least one ground-truth object bounded by the multiple anchor boxes, mitotic cell, [0034], “Step 400: Pool a region on each feature map of the group of augmented feature maps mapped from each of multiple proposals on the input image to a fixed size feature map, flatten the fixed size feature map to generate an ROI feature vector, and fuse the ROI feature vectors generated from the group of augmented feature maps to generate a fused ROI vector through the ROI aligner for object classification and anchor box regression. A highlighted point in the current step resides in the use of an ROI-aligning scheme that pools the mapped regions on the group of augmented feature maps to the corresponding fixed size feature maps in an accurate fashion”, [0056]).
Li et al. and Yeh et al. are in the same art of imaging mitotic cells and pooling (Li et al., abstract, p168; Yeh et al., [0049], [0055]). The combination of Yeh et al. with Li et al. and Arnold et al. will enable generating a corresponding one of a plurality of feature vectors using at least one pooling operation. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the generating of Yeh et al. with the invention of Li et al. and Arnold et al. as this was known at the time of filing, the combination would have predictable results, and as Yeh et al. indicate “An objective of the present invention is to provide an object detection method and a convolution neural network (CNN) capable of extracting features from input images broadly ranging from spatial features to semantic features for enhancing accuracy in object classification and box regression of a proposal regardless of its size” ([0004]) and “As a result of the processes that deliver diversified features to the group of augmented feature maps and accurately transform the features in the group of augmented feature maps to the flattened ROI feature vector for the final stage of object classification and box regression, the method significantly improves the performance in object classification and box regression” ([0009]) suggesting a probable increase in accuracy that will result from combining inventions.
Regarding claim 12, Li et al. and Arnold et al. and Yeh et al. disclose the computer-implemented method of claim 8. Arnold et al. further indicate generating the output vector using the plurality of feature vectors comprises: generating a second plurality of feature vectors using a trained model, comprising applying the trained model separately to each of the plurality of feature vectors (At 912, the process 900 can receive feature vectors associated with the first number of tiles from the first trained model. In some configurations, the feature vectors can be the feature vectors 716 in FIG. 7. At 916, the process 900 can generate a first number of attention values based on the feature vectors associated with the first number of tiles. In some configurations, the process 900 can provide each of the feature vectors to a first attention model, [0122]-[0123]); and generating the output vector using information from each of the second plurality of feature vectors (At 932, the process 900 can receive feature vectors associated with the subset of tiles from the second trained model. In some configurations, the feature vectors can be the feature vectors 746 in FIG. 7. At 936, the process 900 can generate a second number of attention values based on the feature vectors associated with the subset of tiles. In some configurations, the process 900 can provide each of the feature vectors to a second attention model, [0127]-[0128]).
Claim(s) 9 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) and Arnold et al. (US 20220207730 A1) and Yeh et al. (US 20230128432 A1) as applied to claim 8 above, further in view of Quan et al. (US 20240046460 A1).
Regarding claim 9, Li et al. and Arnold et al. and Yeh et al. disclose the computer-implemented method of claim 8. Li et al. and Arnold et al. and Yeh et al. do not explicitly disclose generating the output vector using information from each of the plurality of feature vectors comprises generating the output vector using a weighted sum of the plurality of feature vectors.
Quan et al. teach generating the output vector using information from each of the plurality of feature vectors comprises generating the output vector using a weighted sum of the plurality of feature vectors (S51: transforming the M-dimensional feature vector corresponding to the tumor patch into a low-dimensional feature vector; S52: performing attention pooling on the low-dimensional feature vector corresponding to the tumor patch to obtain a weight of the tumor patch; S53: seeking a weighted sum for low-dimensional feature vectors corresponding to all of the tumor patches to obtain a low-dimensional feature vector of the whole target whole slide image; and S54: mapping the low-dimensional feature vector of the whole target whole slide image to a classification space, and performing softmax activation to obtain the MSI score, [0021]-[0024], S53: Seek a weighted sum for low-dimensional feature vectors corresponding to all of the tumor patches to obtain a low-dimensional feature vector of the whole target whole slide image, [0081], A weighted sum for all tumor patches is sought to obtain a 512-dimensional feature vector of the whole target whole slide image, [0085]).
Li et al. and Arnold et al. and Quan et al. are in the same art of using CNNs for processing slide images (Li et al., p168, p170; Arnold et al., abstract, [0064]; Quan et al., [0048], [0085]).The combination of Quan et al. with Li et al. and Arnold et al. and Yeh et al. will enable generating the output vector using a weighted sum of the plurality of feature vectors. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the generating of Quan et al. with the invention of Li et al. and Arnold et al. and Yeh et al. as this was known at the time of filing, the combination would have predictable results, and as Quan et al. indicate “An objective of the present disclosure is to provide a method for predicting an MSI from a whole slide image based on a self-attention mechanism, an electronic device and a computer-readable storage medium each implementing the method. By screening out tumor patches for feature encoding, and performing classification and prediction with a self-attention algorithm, the present disclosure improves an accuracy of MSI prediction” ([0013]) demonstrating an improved accuracy that will result from the combination.
Regarding claim 13, Li et al. and Arnold et al. and Yeh et al. disclose the computer-implemented method of claim 12. Li et al. and Arnold et al. and Yeh et al. do not explicitly disclose generating the output vector using information from each of the second plurality of feature vectors comprises generating the output vector using a weighted sum of the second plurality of feature vectors.
Quan et al. teach generating the output vector using information from each of the second plurality of feature vectors comprises generating the output vector using a weighted sum of the second plurality of feature vectors (S51: transforming the M-dimensional feature vector corresponding to the tumor patch into a low-dimensional feature vector; S52: performing attention pooling on the low-dimensional feature vector corresponding to the tumor patch to obtain a weight of the tumor patch; S53: seeking a weighted sum for low-dimensional feature vectors corresponding to all of the tumor patches to obtain a low-dimensional feature vector of the whole target whole slide image; and S54: mapping the low-dimensional feature vector of the whole target whole slide image to a classification space, and performing softmax activation to obtain the MSI score, [0021]-[0024], S53: Seek a weighted sum for low-dimensional feature vectors corresponding to all of the tumor patches to obtain a low-dimensional feature vector of the whole target whole slide image, [0081], A weighted sum for all tumor patches is sought to obtain a 512-dimensional feature vector of the whole target whole slide image, [0085]).
Li et al. and Arnold et al. and Quan et al. are in the same art of using CNNs for processing slide images (Li et al., p168, p170; Arnold et al., abstract, [0064]; Quan et al., [0048], [0085]).The combination of Quan et al. with Li et al. and Arnold et al. and Yeh et al. will enable generating the output vector using a weighted sum of the plurality of feature vectors. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the generating of Quan et al. with the invention of Li et al. and Arnold et al. and Yeh et al. as this was known at the time of filing, the combination would have predictable results, and as Quan et al. indicate “An objective of the present disclosure is to provide a method for predicting an MSI from a whole slide image based on a self-attention mechanism, an electronic device and a computer-readable storage medium each implementing the method. By screening out tumor patches for feature encoding, and performing classification and prediction with a self-attention algorithm, the present disclosure improves an accuracy of MSI prediction” ([0013]) demonstrating an improved accuracy that will result from the combination.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) and Arnold et al. (US 20220207730 A1) as applied to claim 1 above, further in view of Hegde et al. (US 20230207134 A1).
Regarding claim 14, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. further indicate a center region of the input image depicts a biological cell (Our method, called SegMitos, is illustrated in Fig. 2. We train a segmentation network based on fully convolutional network (FCN) (Long et al., 2015) with the mitosis data. The SegMitos model produces a segmentation map where each pixel represents its confidence of belonging to the mitosis class, p166, Our SegMitos model is based on a fully convolutional network designed for semantic segmentation on natural images. Fully convolutional network (Long et al., 2015) adapts all fully connected layers in conventional classification network to convolutional layers, p168,
PNG
media_image1.png
710
948
media_image1.png
Greyscale
Li et al. and Arnold et al. do not explicitly disclose selecting the input image as a patch of a larger image using a second trained neural network.
Hegde et al. teach selecting the input image as a patch of a larger image using a second trained neural network (A core element of our example deep learning system (DLS) included multiple convolutional neural network (CNN) modules with shared weights, and an average pooling layer that merges image features computed by these modules (see FIG. 1). For each case, cropped image patches were uniformly and randomly sampled from tissue-containing areas across all slides available for a given case. Next, image features were extracted for each patch by a convolutional neural network (CNN). These patch-level features were averaged (on a per-channel basis) and fed to a fully connected layer. Our custom loss function divided the follow-up period into four discrete bins depending on right-censorship time and outcome. As such, the model was designed to output a probability distribution over discretized survival times, [0040], A two stage model approach is contemplated in which the ML models 120a-n of the DLS are used in a first stage to generate features from the whole-slide images (mitosis, granular, tubule formation etc.) and those features along with case metadata are supplied to a second neural network model which generates an output of molecular prediction. Additionally, the architecture of FIG. 1 can be further trained to predict cancer spread beyond the local tumor environment, for example distant metastasis to lymph nodes or other tissue in breast cancer patients, [0080]).
Li et al. and Hegde et al. are in the same art of imaging mitotic cells and pooling (Li et al., abstract, p168; Hegde et al., [0056], [0080]). The combination of Hegde et al. with Li et al. and Arnold et al. will enable selecting the input image as a patch of a larger image using a second trained neural network. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the selecting of Hegde et al. with the invention of Li et al. and Arnold et al. as this was known at the time of filing, the combination would have predictable results, and as Hegde et al. indicate “Predicting patient prognosis in oncology underlies important clinical decisions regarding treatment and monitoring. In this work, we assessed the potential to improve predictions of disease-specific survival using a deep learning system trained without human annotations for known morphological features or regions of interest” ([0050]) suggesting a clinical benefit that may result from combining inventions.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (“Weakly supervised mitosis detection in breast histopathology images using concentric loss”) and Arnold et al. (US 20220207730 A1) as applied to claim 1 above, further in view of Zheng et al. (US 12100209 B2).
Regarding claim 15, Li et al. and Arnold et al. disclose the computer-implemented method of claim 1. Li et al. and Arnold et al. do not disclose processing the output vector comprises applying a sigmoid function to the output vector.
Zheng et al. teach processing the output vector comprises applying a sigmoid function to the output vector w.sub.a,b=sigmoid(w(v.sub.a,b, v.sub.a,a)), sigmoid( ) is an S-type function, v.sub.a,b is the feature vector corresponding to the target subject a and the target subject b, v.sub.a,a is the feature vector corresponding to the target subject a, and w( ) is an inner product function, col. 2, lines 55-60
PNG
media_image3.png
166
260
media_image3.png
Greyscale
col. 4, lines 1-20
PNG
media_image4.png
176
268
media_image4.png
Greyscale
col. 13, lines 1-15
extracting relational vector features using a sigmoid function of the t feature vectors to obtain the relational vector features; wherein the location features, the attribute features, the posture features, and the relational vector features are extracted by a same convolutional neural network, claim 9).
Li et al. and Arnold et al. and Quan et al. are in the same art of using CNNs for processing images (Li et al., p168, p170; Arnold et al., abstract, [0064]; Zheng et al., claim 9). The combination of Zheng et al. with Li et al. and Arnold et al. will enable applying a sigmoid function to the output vector. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the applying a sigmoid function to the output vector of Hegde et al. with the invention of Li et al. and Arnold et al. as this was known at the time of filing, the combination would have predictable results, and as Zheng et al. indicate “Currently, image description can only perform a low-level semantic description on the image. Only a single-subject single action (for example, the man is surfing on the sea in FIG. 1A) or a multi-subject single action (for example, a group of students are doing morning exercises in FIG. 1B) can be described. However, image description cannot perform a panoramic semantic description on the image. Relationships between a plurality of subjects, relationships between the subjects and actions, and relationships between the actions (for example, a man sees that a woman is knocked down by a vehicle in FIG. 1C) cannot be described” (col. 1, lines 25-40) and “In the solution, the higher-level panoramic semantic description can be obtained based on location features, attribute features, and posture features of a plurality of target subjects in a plurality of frames of images and relational vector features between the plurality of target subjects in the plurality of frames of images, to better reflect relationships between the plurality of subjects, relationships between the subjects and actions, and relationships between the actions in the images” (col. 1, line 65 – col. 2, line 10) suggesting an improvement to accuracy when inventions are combined.
Allowable Subject Matter
Claims 10 and 11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE ENTEZARI whose telephone number is (571)270-5084. The examiner can normally be reached 10-7 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent M Rudolph can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE M ENTEZARI HAUSMANN/Primary Examiner, Art Unit 2671
