DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 7, 9-10, 23-27 and 29-30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Abousamra et al. (IDS: “WEAKLY-SUPERVISED DEEP STAIN DECOMPOSITION FOR MULTIPLEX IHC IMAGES”, 2020) in view of Shrivastava et al. (US 11335093 B2).
Regarding claims 1 and 30, Abousamra et al. disclose a method of performing stain deconvolution on a colour target image (In this paper, we propose a novel deep autoencoder for color decomposition of multiplex images. The autoencoder learns to predict concentrations of different stains across the image so their combination recovers the original image, p482), the method comprising: providing a stain deconvolution network (autoencoder learns to predict concentrations of different stains across the image so their combination recovers the original image, p482) comprising and system for performing stain deconvolution on a colour target image, the system comprising: control and processing circuitry comprising at least one processor and memory, said memory comprising instructions executable by said at least one processor for performing operations (autoencoder/neural networks inherently require use of a computer system) comprising: a plurality of convolutional auto-encoder neural networks, each convolutional auto-encoder neural network being configured to process a respective input dataset to generate a respective output dataset, each input dataset comprising a respective two-dimensional array having dimensions equal to the pixel dimensions of the colour target image (The number of channels decreases through the encoder and increases back through the decoder. The final layer is a 1 x 1 convolution followed by a squaring operation to ensure the concentration map is all positive, We show results of the four methods in four rows: ColorAE, ColorUNet, PseudoInv and LassoReg
PNG
media_image1.png
182
384
media_image1.png
Greyscale
, p483); an absorbance calculation module configured to employ an absorbance model to generate a colour absorbance image, the colour absorbance image being generated by processing a plurality of stain vectors and a plurality of stain concentration maps, wherein each stain vector and each stain concentration map is associated with a respective stain (
PNG
media_image2.png
354
400
media_image2.png
Greyscale
, p482, To get proper concentration maps for pixels belonging to the background tissue it is important to have in S a background stain vector that represents the plain tissue color, Fig. 2. Using this superpixel-based stain labeling, we now assign a single stain to each pixel, called its label stain, p483,
PNG
media_image3.png
178
386
media_image3.png
Greyscale
); the absorbance calculation module being operatively coupled to the plurality of convolutional auto-encoder neural networks, such that each stain concentration map is obtained from the output dataset of a respective convolutional auto-encoder neural network (We propose an autoencoder which predicts the concentration of each stain at each pixel,
PNG
media_image4.png
150
396
media_image4.png
Greyscale
p482); and training the stain deconvolution network according to a stain deconvolution loss function comprising: a first loss component configured to minimize generation loss associated with the colour target image; and a second loss component configured to facilitate separation between the stain concentration maps; such that after the training (The auto-encoder is trained on two loss terms, reconstruction loss and label consistency loss. The reconstruction loss ensures the predicted stain concentrations correctly restore the original image. The label consistency loss enforces the
concentration prediction is consistent with human annotation, p482) [generation loss interpreted as reconstruction loss, second loss interpreted as label consistency loss], the stain concentration maps respectively represent deconvoluted stain concentration maps of the stains within the colour target image (
PNG
media_image5.png
558
404
media_image5.png
Greyscale
, Fig. 4 shows sample predictions on a raw image. In the top block, we show the predicted concentration map for different stains. The darker a pixel is, the higher the concentration is. A pixel is white if its concentration is near zero. We show results of the four methods in four
rows: ColorAE, ColorUNet, PseudoInv and LassoReg. The five columns correspond to the five different stains. Recall for ColorUNet we use the softmax layer output as the concentration
map. In the bottom block, we show the derived stain segmentation, i.e., each pixel takes the stain with the maximal concentration. A pixel is white if it does not take the stain. This stain segmentation can be delivered to domain expert for downstream analysis, p483).
Abousamra et al. imply but do not explicitly disclose each input dataset comprising a respective two-dimensional array having dimensions equal to the pixel dimensions of the colour target image.
Shrivastava et al. teach a system for performing stain deconvolution on a colour target image, the system comprising: control and processing circuitry comprising at least one processor and memory, said memory comprising instructions executable by said at least one processor for performing operations (col. 13, lines 4-31) comprising a plurality of convolutional neural networks, each convolutional neural network being configured to process a respective input dataset to generate a respective output dataset, each input dataset comprising a respective two-dimensional array having dimensions equal to the pixel dimensions of the colour target image (colorization machine learning model, col. 7, lines 25-26, The embedding neural network 206 is a convolutional neural network (CNN), that is, a neural network that includes one or more convolutional neural network layers, The output of the embedding neural network 206 after processing an input video frame (e.g., the reference frame 202 or the target frame 204) can be represented as a three-dimensional (3D) matrix of numerical values, with two “spatial” dimensions and one “channel” dimension. The embedding corresponding to a pixel of the input video frame at a particular spatial position (e.g., defined by (x,y) coordinates in the input video frame) is determined by extracting a portion of the embedding neural network output at the corresponding spatial position (i.e., along the channel dimension). In some cases, the spatial dimensionality of the embedding neural network output may be the same as the input video frame, e.g., the input video frame may have a spatial dimensionality of 256×256, and the embedding neural network output may have a spatial dimensionality of 256×256 with 200 channels, The embedding neural network 206 can have any appropriate neural network architecture. In one example, the embedding neural network architecture may include a ResNet-18 neural network architecture followed by a five layer 3D convolutional neural network. The system 100 trains the embedding network 206 to generate embeddings 210 that can be used to colorize the target frame 204 by generating predicted colors 212 of the target frame 204 based on the reference colors 214 of the reference frame 202., col. 7, line 35 – col. 8, line 19).
Abousamra et al. and Shrivastava et al. are in the same art of neural networks and processing color images (Abousamra et al., abstract, p482; Shrivastava et al., col. 1, lines 55-60). The combination of Shrivastava et al. with Abousamra et al. will enable having the output size the same as the input. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the size of Shrivastava et al. with the invention of Abousamra et al. as this was known at the time of filing, the combination would have predictable results, and as one of a limited number of possibilities (either maintaining size or increasing or decreasing size) would have been obvious to try and a design choice.
Regarding claim 2, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose the absorbance calculation module is further configured such that the colour absorbance image is generated by calculating a sum, over each stain, of the product the stain vector, the stain concentration map, and a stain spectral correction factor; wherein each stain has an associated stain spectral correction factor; and wherein each stain spectral correction factor is updated during the training, according to minimization of the stain deconvolution loss function (
PNG
media_image6.png
610
404
media_image6.png
Greyscale
, p482) [stacking interpreted as sum].
Regarding claim 7, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose the absorbance calculation module is further configured such that the calculation of the colour absorbance image includes a colour background vector; wherein the colour background vector is updated during the training, according to minimization of the stain deconvolution loss function (To get proper concentration maps for pixels belonging to the background tissue it is important to have in S a background stain vector that represents the plain tissue color, Fig. 2, p483).
Regarding claim 9, Abousamra et al. and Shrivastava et al. disclose the method according to claim 7. Abousamra et al. further disclose the stain deconvolution network comprises a background neural network, the background neural network being configured to determine the colour background vector, and wherein the background neural network is trained according to the stain deconvolution loss function (
PNG
media_image7.png
472
394
media_image7.png
Greyscale
, p483).
Regarding claim 10, Abousamra et al. and Shrivastava et al. disclose the method according to claim 7. Abousamra et al. further disclose the background neural network is an encoder-decoder network (The autoencoder starts with a 1 x 1 convolution that transforms
the input into 128 channels number of channels decreases through the encoder and increases back through the decoder p483
PNG
media_image8.png
362
742
media_image8.png
Greyscale
).
Regarding claim 23, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose at least one input dataset is randomly generated (We train our method and baselines on randomly extracted patches from the tumor area in 6 whole slide images of pancreatic cancer tissue, p483).
Regarding claim 24, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose at least two of the input datasets are a common input dataset (in-house dataset of multiplex images with 6 stains labeling 5 immune cell types, p482, For training, we sample 300 patches of size 400 400. For testing, we also randomly sample patches, but patch size varies depending on the evaluation strategy. The reference stain color vectors were obtained by averaging random samples from multiple locations (Fig. 2), p483).
Regarding claim 25, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose at least one of convolutional auto-encoder neural network includes a skip connection (We show results of the four methods in four rows: ColorAE, ColorUNet, PseudoInv and LassoReg, p483) [U-net architecture inherently has skip connections].
Regarding claim 26, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose during at least an initial portion of the training, computation of the loss function is augmented using at least one transformation of the input datasets (Our training loss is a weighted sum of the reconstruction loss and the label consistency loss. The weights are tuned empirically. The architecture of the network is shown in Fig. 5. The autoencoder starts with a 1 x 1 convolution that transforms the input into 128 channels, p483).
Regarding claim 27, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose the colour target image is a first colour tile of a main color image, and wherein the stain vectors obtained after training are final stain vectors, the method further comprising employing the final stain vectors when performing stain deconvolution of another colour tile of the main colour image (
PNG
media_image9.png
272
450
media_image9.png
Greyscale
We train our method and baselines on randomly extracted patches from the tumor area in 6 whole slide images of pancreatic cancer tissue. We test all models on patches from 4 different whole slide images. All patches have a resolution of 0.174 microns per pixel. For training, we sample 300 patches of size 400 400. For testing, we also randomly sample patches, but patch size varies depending on the evaluation strategy. The reference stain color vectors were obtained by averaging random samples from multiple locations (Fig. 2), p483) [patch = tile] [loss indicates at some point when loss hits a condition it will be the “final” stain vector].
Regarding claim 29, Abousamra et al. and Shrivastava et al. disclose the method according to claim 1. Abousamra et al. further disclose employing the stain concentration maps to perform stain quantification (
PNG
media_image5.png
558
404
media_image5.png
Greyscale
).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Abousamra et al. (“WEAKLY-SUPERVISED DEEP STAIN DECOMPOSITION FOR MULTIPLEX IHC IMAGES”, 2020) and Shrivastava et al. (US 11335093 B2) as applied to claim 2 above, further in view of Pengo et al. (“Efficient Blind Spectral Unmixing of Fluorescently Labeled Samples Using Multi-Layer Non-Negative Matrix Factorization”, 2013).
Regarding claim 3, Abousamra et al. and Shrivastava et al. disclose the method according to claim 2. Shrivastava et al. further indicate the stain spectral correction factors are defined according to stain spectral correction factor parameters of the stain deconvolution network, and wherein the stain spectral correction factor parameters are initialized prior to the training (To enable the system 100 to effectively generate target labels 108 for target video frames 106, the system 100 includes a colorization training subsystem 114 which is configured to train the embedding neural network 116. The training subsystem 114 trains the embedding neural network 116 over multiple training iterations to determine trained values of the embedding neural network parameters from initial values of the embedding neural network parameters. The training subsystem 114 can train the embedding neural network 116 on large amounts of readily available unlabeled color video data without requiring manual human supervision (e.g., without requiring a human to manually annotated pixel labels on the video data), col. 6, lines 5-20, After generating the estimated target colors 126, the training subsystem 114 adjusts the current values of the embedding neural network parameters to cause the system 100 to colorize the target video frame 106 more accurately. More specifically, the training subsystem 114 adjusts the current values of the embedding neural network parameters based on a difference between: (i) the (actual) target colors 130 of the target pixels in the target video frame 106, and (ii) the estimated target colors 126 of the target pixels in the target video frame 106. The training subsystem 114 adjusts the current values of the embedding neural network parameters using a gradient 132 of a loss function 134 with respect to the current values of the embedding neural network parameters. The loss function 134 depends on an error between the actual target colors 130 of the target pixels and the estimated target colors 126 of the target pixels, col. 6, line 60 – col. 7, line 10) however another reference is added to further teach this limitation.
Pengo et al. teach the stain spectral correction factors are defined according to stain spectral correction factor parameters of the stain deconvolution network, and wherein the stain spectral correction factor parameters are initialized prior to the training (
PNG
media_image10.png
456
348
media_image10.png
Greyscale
, p3,
PNG
media_image11.png
336
336
media_image11.png
Greyscale
,
PNG
media_image12.png
418
346
media_image12.png
Greyscale
, p5-6).
Abousamra et al. and Pengo et al. are in the same art of unmixing (Abousamra et al., abstract; Pengo et al., abstract). The combination of Pengo et al. with Abousamra et al. and Shrivastava et al. will enable stain spectral correction factor parameters are initialized prior to the training. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the initialization of Pengo et al. with the invention of Abousamra et al. and Shrivastava et al. as this was known at the time of filing, the combination would have predictable results, and as Pengo et al. indicate, “When two or more fluorescent dyes are used in the same preparation, or one dye is used in the presence of autofluorescence, the separation of the fluorescent emissions can become problematic. Various approaches have been recently proposed to solve this problem. Among them, blind non-negative matrix factorization is gaining interest since it requires little assumptions about the spectra and concentration of the fluorochromes. In this paper, we propose a novel algorithm for blind spectral separation that addresses some of the shortcomings of existing solutions: namely, their dependency on the initialization and their slow convergence. We apply this new algorithm to two relevant problems in fluorescence microscopy: autofluorescence elimination and spectral unmixing of multi-labeled samples. Our results show that our new algorithm performs well when compared with the state-of-the-art approaches for a much faster implementation.” (abstract) suggesting the combination of inventions will lead to faster convergence and computational efficiency.
Allowable Subject Matter
Claims 4-6, 8, 11-22, and 28 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following art is cited as relevant to the allowed claims but not sufficient alone or in combination to disclose, teach, or fairly suggest the above claims in their entirety and/or do not predate the application:
“Deep learning-based image analysis methods for brightfield-acquired multiplex
immunohistochemistry images”:
PNG
media_image13.png
884
766
media_image13.png
Greyscale
“BEER-LAMBERT AUTOENCODER FOR UNSUPERVISED STAIN REPRESENTATION
LEARNING AND DECONVOLUTION IN MULTI-IMMUNOHISTOCHEMICAL
BRIGHTFIELD HISTOLOGY IMAGES”:
PNG
media_image14.png
1028
818
media_image14.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE ENTEZARI whose telephone number is (571)270-5084. The examiner can normally be reached 10-7 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent M Rudolph can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE M ENTEZARI HAUSMANN/Primary Examiner, Art Unit 2671
Prosecution Insights