DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 8, and 15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process of determining image normalization parameters) without significantly more.
Claim 1 recite(s):
“…determining one or more normalization parameters for a first channel of the one or more imaging channels, the one or more normalization parameters for the first channel based on a first relationship between the first imaging device type and a second imaging device type, the second imaging device type being different from the first imaging device type”; Which can be reasonability interpreted as a human observer mentally determining a normalization parameter for an image based on a relationship between the imaging device used to capture the image and a desired imaging device type.
This judicial exception is not integrated into a practical application because of additional elements:
“…receiving, from an imaging device of a first imaging device type, a first autofluorescence image of a first tissue sample, the first autofluorescence image comprising one or more imaging channels”; is/are generically recited extra-solution activity of data gathering.
“…applying the one or more normalization parameters for the first channel to the first channel of the first autofluorescence image”; is/are generically recited computer method(s) that does/do not add a meaningful limitation to the abstract idea because it/they amount to simply implementing the abstract idea on a computer and pertain to a generically recited image normalization.
“…outputting the normalized first channel of the first autofluorescence image”; is/are generically recited extra-solution activity of data outputting.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because of additional elements:
“…receiving, from an imaging device of a first imaging device type, a first autofluorescence image of a first tissue sample, the first autofluorescence image comprising one or more imaging channels”; is/are well-understood, routine, and conventional extra-solution activity of data gathering.
“…applying the one or more normalization parameters for the first channel to the first channel of the first autofluorescence image”; is/are well-understood, routine, and conventional computer method(s) that does/do not add a meaningful limitation to the abstract idea because it/they amount to simply implementing the abstract idea on a computer and pertain to a well-understood, routine, and conventional image normalization.
“…outputting the normalized first channel of the first autofluorescence image”; is/are well-understood, routine, and conventional extra-solution activity of data outputting.
As per claim(s) 8 & 15, arguments made in rejecting claim(s) 1 are analogous.
Note that claim 8 recites additional elements: “a non-transitory computer-readable medium” and “one or more processors in communication with the non-transitory computer-readable medium”, which are generically recited and well-understood, routine, and conventional.
In addition, claim 15 recites additional elements: “A non-transitory computer-readable medium” and “processor-executable instructions configured to cause one or more processors to…”, which are generically recited and well-understood, routine, and conventional.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 5-7, 12-14, and 19-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 5, the limitation “The method of claim 4, wherein the plurality of training tissue sample images comprise: autofluorescence images of one or more tissue samples generated by the second imaging device type, wherein the autofluorescence images are obtained when the one or more tissue samples generated by the second imaging device type are unstained;”, is indefinite since the plurality of training tissue sample images comprise unstained images, while parent claim 4 recites the limitation “wherein the plurality of training tissue sample images are stained tissue samples”, which requires the plurality of training tissue sample images to be stained. For the purposes of examination, the limitation is interpreted as “The method of claim 3 [[4]], wherein the plurality of training tissue sample images comprise: autofluorescence images of one or more tissue samples generated by the second imaging device type, wherein the autofluorescence images are obtained when the one or more tissue samples generated by the second imaging device type are unstained;”.
As per claim(s) 12, arguments made in rejecting claim(s) 5 are analogous.
Claim 6 recites the limitation "determining, using the machine learning model, a first prediction of a first stained tissue sample image based on the one or more normalized imaging channels of the second autofluorescence image". There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the limitation is interpreted as "determining, using the machine learning model, a first prediction of a first stained tissue sample image based on .
As per claim(s) 13 and 19, arguments made in rejecting claim(s) 6 are analogous.
Regarding claims 7, 14, and 20, it/they is/are rejected under 112b for inheriting and failing to cure the deficiencies of the parent claim 6, 13, and 19, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 8-11, and 15-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ozcan et al. (US-20250046069-A1) hereinafter referenced as Ozcan, in view of Gurcan et al. (Histopathological Image Analysis: A Review) hereinafter referenced as Gurcan.
Regarding claim 1, Ozcan discloses: A method, comprising: receiving, from an imaging device of a first imaging device type, being different from a second imaging device type, a first autofluorescence image of a first tissue sample, the first autofluorescence image comprising one or more imaging channels (Ozcan: Abstract: “A deep learning-based virtual HER2 IHC staining method uses a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images,”;
0032: “With reference to FIG. 1a, in one embodiment, the trained, deep neural network 10 receives a single autofluorescence image 20 of a label-free tissue sample 22. In other embodiments, for example, where multiple excitation channels are used, there may be multiple autofluorescence images 20 of the label-free tissue sample 22 that are input to the trained, deep neural network 10 (e.g., one image per channel). These channels may include, for example, DAPI, FITC, TxRed, and Cy5 which are obtained using different filters/filter cubes in the imaging device 110. For example, multiple autofluorescence images 20 obtained with different filters/filter cubes in the imaging device 110 can be input to the trained, deep neural network 10 (e.g., two or more different channels).”; Wherein each image captured using a filter, constitutes a single channel image.);
determining one or more normalization parameters for a first channel of the one or more imaging channels, applying the one or more normalization parameters for the first channel to the first channel of the first autofluorescence image (Ozcan: 0051: “The autofluorescence images 20 of the unlabeled tissue sections were captured using a standard fluorescence microscope 110 (IX-83, Olympus) with a ×40/0.95 NA (UPLSAPO, Olympus) objective lens. Four fluorescent filter cubes, including DAPI (Semrock DAPI-5060C-OFX, EX 377/50 nm, EM 447/60 nm), FITC (Semrock FITC-2024B-OFX, EX 485/20 nm, EM 522/24 nm), TxRed (Semrock TXRED-4040C-OFX, EX 562/40 nm, EM 624/40 nm), and Cy5 (Semrock CY5-4040C-OFX, EX 628/40 nm, EM 692/40 nm) were used to capture the autofluorescence images 20 at different excitation-emission wavelengths. Each autofluorescence image 20 was captured with a scientific complementary metal-oxide-semiconductor (sCMOS) image sensor (ORCA-flash4.0 V2, Hamamatsu Photonics) with an exposure time of 150 ms, 500 ms, 500 ms, and 1000 ms for DAPI, FITC, TxRed, and Cy5 filters, respectively. Images were normalized for the four (4) channels by their respective exposure times. Thus, DAPI images (for training and test) were divided by their exposure time of 150 ms. The other channels were normalized to their respective exposure times. The image acquisition process was controlled by pManager (version 1.4) microscope automation software. After the standard IHC HER2 staining is complete, the bright-field WSIs were acquired using a slide scanner microscope (AxioScan Z1, Zeiss) with a ×20/0.8 NA objective lens (Plan-Apo).”); and
outputting the normalized first channel of the first autofluorescence image (Ozcan: 0038: “The virtual HER2 staining of breast tissue sample 22 was demonstrated by training deep neural network (DNN) models 10 with a dataset of twenty-five (25) breast tissue sections collected from nineteen (19) unique patients, constituting in total 20,910 image patches, each with 1024×1024 pixels. Once a DNN model 10 was trained, it virtually stained the unlabeled tissue sections using their autofluorescence microscopic images 20 captured with DAPI, FITC, TxRed, and Cy5 filter cubes (see Methods section), matching the corresponding bright-field images of the same field-of-views, captured after standard IHC HER2 staining.”; Wherein the normalized autofluorescence images were output and then processed by a virtual staining model.).
Ozcan does not disclose expressly: determining one or more normalization parameters for a first channel of the one or more imaging channels, the one or more normalization parameters for the first channel based on a first relationship between the first imaging device type and a second imaging device type, the second imaging device type being different from the first imaging device type.
Thus, Ozcan does not disclose expressly: one or more normalization parameters for the autofluorescence microscopic channel images based on a first relationship between the source fluorescence microscope imaging device type and a target slide scanner microscope device type.
Gurcan discloses: determining one or more normalization parameters for one or more imaging channels, the one or more normalization parameters for the one or more imaging channels based on a first relationship between a source image and a target image (Gurcan: Section: 3.1. Color Normalization: “One of the first steps essential for both fluorescent and bright field microscopy image analysis is color and illumination normalization. This process reduces the differences in tissue samples due to variation in staining and scanning conditions. The illumination can be corrected either using calibration targets or estimating the illumination pattern from a series of images by fitting polynomial surfaces [29]. Another approach is to match the histograms of the images. Software that corrects for spectral and spatial illumination variations is becoming a standard package provided by most bright field manufacturers. This is an essential step for algorithms that heavily depend on color space computations.”; Wherein the histograms of a source image are matched to a target image.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the histogram color and illumination normalization taught by Gurcan for the autofluorescence image normalization disclosed by Ozcan by normalizing the source autofluorescence images based on a target slide scanner image. The suggestion/motivation for doing so would have been “This process reduces the differences in tissue samples due to variation in staining and scanning conditions…This is an essential step for algorithms that heavily depend on color space computations.” (Gurcan: Section: 3.1. Color Normalization; Wherein a reduction of sample differences between the target images and a source image allows for improvements ). Further, one skilled in the art could have substituted one known element for another, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ozcan with Gurcan to obtain the invention as specified in claim 1.
Regarding claim 2, Ozcan in view of Gurcan discloses: The method of claim 1, further comprising: determining, using a machine learning model, a prediction of a stained tissue sample image based on one or more normalized imaging channels of the first autofluorescence image; and outputting the prediction of the stained tissue sample image (Ozcan: 0009: “The method includes providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched immunohistochemical (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample; obtaining one or more autofluorescence images of the label-free tissue sample with a fluorescence imaging device; inputting the one or more autofluorescence images of the label-free tissue sample to the trained, deep neural network; and the trained, deep neural network outputting the digitally stained IHC microscopic image of the label-free tissue sample that reveal the features specific to the at least one target biomarker that appears substantially equivalent to a corresponding image of the same label-free tissue sample had it been IHC stained chemically.”;
0038: “The virtual HER2 staining of breast tissue sample 22 was demonstrated by training deep neural network (DNN) models 10 with a dataset of twenty-five (25) breast tissue sections collected from nineteen (19) unique patients, constituting in total 20,910 image patches, each with 1024×1024 pixels. Once a DNN model 10 was trained, it virtually stained the unlabeled tissue sections using their autofluorescence microscopic images 20 captured with DAPI, FITC, TxRed, and Cy5 filter cubes (see Methods section), matching the corresponding bright-field images of the same field-of-views, captured after standard IHC HER2 staining.”; Wherein the normalized autofluorescence images, as taught by Gurcan, were processed by a virtual staining model in order to output a predicted digitally stained IHC microscopic image.).
Regarding claim 3, Ozcan in view of Gurcan discloses: The method of claim 2, wherein the machine learning model is trained using a plurality of training tissue sample images, wherein each tissue sample image of the plurality of training tissue sample images is generated by the second imaging device type (Ozcan: 0009: “The method includes providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched immunohistochemical (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample; obtaining one or more autofluorescence images of the label-free tissue sample with a fluorescence imaging device; inputting the one or more autofluorescence images of the label-free tissue sample to the trained, deep neural network; and the trained, deep neural network outputting the digitally stained IHC microscopic image of the label-free tissue sample that reveal the features specific to the at least one target biomarker that appears substantially equivalent to a corresponding image of the same label-free tissue sample had it been IHC stained chemically.”).
Regarding claim 4, Ozcan in view of Gurcan discloses: The method of claim 3, wherein the plurality of training tissue sample images are stained tissue samples (Ozcan: 0009: “The method includes providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched immunohistochemical (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample;”).
As per claim(s) 8, arguments made in rejecting claim(s) 1 are analogous. In addition, paragraph 0029 of Ozcan discloses a system comprising a computing device for implementing virtual staining methods, thus disclosing “A system comprising: a non-transitory computer-readable medium; one or more processors in communication with the non-transitory computer-readable medium”.
As per claim(s) 9, arguments made in rejecting claim(s) 2 are analogous.
As per claim(s) 10, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 11, arguments made in rejecting claim(s) 4 are analogous.
As per claim(s) 15, arguments made in rejecting claim(s) 1 are analogous. In addition, paragraph 0029 of Ozcan discloses a system comprising a computing device for implementing virtual staining methods, thus disclosing “A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors”.
As per claim(s) 16, arguments made in rejecting claim(s) 2 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 18, arguments made in rejecting claim(s) 4 are analogous.
Claim(s) 5-7, 12-14, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ozcan in view of Gurcan, and further in view of Klaiman (US-20210005308-A1).
Regarding claim 5, Ozcan in view of Gurcan discloses: The method of claim 4 (Claim limitation is interpreted according to the rejection of claim 5 under 35 U.S.C. 112(b) disclosed above.), wherein the plurality of training tissue sample images comprise: images of the one or more tissue samples generated by the second imaging device type, wherein the images are obtained when the one or more tissue samples generated by the second imaging device type are stained (Ozcan: 0009: “The method includes providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched immunohistochemical (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample;”).
Ozcan in view of Gurcan does not disclose expressly: wherein the plurality of training tissue sample images comprise: autofluorescence images of one or more tissue samples generated by the second imaging device type, wherein the autofluorescence images are obtained when the one or more tissue samples generated by the second imaging device type are unstained.
Klaiman discloses: A method for processing acquired stained, or unstained, images captured by a microscope and generating a predicted virtual stain output image (Klaiman: 0140-0141: “ the received acquired image is input to the MLL. Although many different types of acquired images can be used in various embodiments of the invention, it is important that the type of acquired image used is identical or very similar to the type of images used during the training phase of the MLL. For example, if the acquired image is an autofluorescence image, the MLL is required to have been trained also on autofluorescence images of tissue samples rather than X-ray images. And if the received acquired image depicts a tissue sample having been stained with three marker specific first stains A, B and C, the MLL is required to have been trained also on images of tissue samples having been stained with marker specific first stains A, B C rather than D or E. Next in step 108 , the MLL automatically transforms the acquired image into an output image.”) wherein, a plurality of training tissue sample images comprise: autofluorescence images of one or more tissue samples generated by a second imaging device type, wherein the autofluorescence images are obtained when one or more tissue samples generated by the second imaging device type are unstained (Klaiman: 0045: “the method further comprises acquiring, by an image acquisition system, the acquired image. The image acquisition system can be, for example, a bright field microscope, a fluorescence microscope or an X-ray microscope…A fluorescence microscope can be used in particular for acquiring images of tissue samples having been stained with one or more biomarker-specific stains consisting of e.g. an antibody that is directly or indirectly coupled to a fluorophore, or for generating autofluorescence images, or for generating images of non-biomarker specific fluorescent stains.”); and images of the one or more tissue samples generated by the second imaging device type, wherein the images are obtained when the one or more tissue samples generated by the second imaging device type are stained (Klaiman: 0056: “The generation of the MLL further comprises staining the training tissue samples with a second biomarker-specific stain. The second biomarker-specific stain is adapted to selectively stain the second biomarker in the training tissue samples. The generation of the MLL further comprises acquiring, by the image acquisition system, a plurality of second training images. Each second training image depicts a respective one of the training tissue samples having been stained with the second biomarker-specific stain. The generation of the MLL further comprises inputting the first and second training images pair wise into an untrained version of the MLL. Each pair of training images depicts the same training tissue sample and is pixel-wise aligned to each other. The generation of the MLL further comprises training the MLL such that the MLL learns to explicitly or implicitly identify regions in the second training images depicting tissue regions in the training tissue samples which are predicted to comprise the second biomarker, whereby the MLL uses intensity information contained in the first training image which depicts the same training tissue sample for the prediction.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the deep neural network model trained to transform autofluorescence images into virtually stained bright-field equivalent images disclosed by Ozcan in view of Gurcan with the machine learning model trained to transform autofluorescence images into virtually stained autofluorescence images taught by Klaiman. The suggestion/motivation for doing so would have been “Irrespective of whether or not the tissue sample depicted in the acquired image is stained or not, the image was acquired under conditions under which the tissue's autofluorescence signal was the most prominent signal captured by the image acquisition system. Thus, it is possible that the acquired image shows a mixture of autofluorescence signal and some staining signals, but is hereby assumed that the most prominent signal is derived from autofluorescence. Autofluorescence images of tissue samples are examples of images captured by a fluorescent microscope.” (Klaiman: 0135; Wherein the mixture of stained images within the acquired images allows for greater flexibility for image acquisition for virtual staining processing.). Further, one skilled in the art could have substituted one known element for another, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ozcan in view of Gurcan with Klaiman to obtain the invention as specified in claim 5.
Regarding claim 6, Ozcan in view of Gurcan discloses: The method of claim 1, wherein a machine learning model is trained using a plurality of normalized autofluorescence imaging channels of tissue sample images including the normalized first channel of the first autofluorescence image, each channel normalized using first normalization parameters based on a second relationship between the first imaging device type and the second imaging device type (Ozcan: 0009: “The method includes providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched immunohistochemical (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample; obtaining one or more autofluorescence images of the label-free tissue sample with a fluorescence imaging device; inputting the one or more autofluorescence images of the label-free tissue sample to the trained, deep neural network; and the trained, deep neural network outputting the digitally stained IHC microscopic image of the label-free tissue sample that reveal the features specific to the at least one target biomarker that appears substantially equivalent to a corresponding image of the same label-free tissue sample had it been IHC stained chemically.”;
0038: “The virtual HER2 staining of breast tissue sample 22 was demonstrated by training deep neural network (DNN) models 10 with a dataset of twenty-five (25) breast tissue sections collected from nineteen (19) unique patients, constituting in total 20,910 image patches, each with 1024×1024 pixels. Once a DNN model 10 was trained, it virtually stained the unlabeled tissue sections using their autofluorescence microscopic images 20 captured with DAPI, FITC, TxRed, and Cy5 filter cubes (see Methods section), matching the corresponding bright-field images of the same field-of-views, captured after standard IHC HER2 staining.”; Wherein the autofluorescence images, normalized based on the teachings of Gurcan, were processed by a virtual staining model in order to output a predicted digitally stained IHC microscopic image.), the second imaging device type being different from the first imaging device type (Ozcan: Abstract: “A deep learning-based virtual HER2 IHC staining method uses a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images,”;) and further comprising: receiving a second autofluorescence image of a second tissue sample, wherein: the second autofluorescence image is generated by a first imaging device having the first imaging device type; and the second autofluorescence image comprises the one or more imaging channels (Ozcan: Abstract: “A deep learning-based virtual HER2 IHC staining method uses a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images,”;
0047: “As an ablation study, virtual staining networks 10 that are trained with different autofluorescence input channels by calculating peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were quantitatively compared between the network output and ground truth images (see FIG. 9 b )…These analyses revealed that the performance of the virtual staining network 10 partially degraded with decreasing number of input autofluorescence channels, motivating the use of DAPI, FITC, TxRed, and Cy5 altogether ( FIG. 9 b ).”); determining, using the machine learning model, a first prediction of a first stained tissue sample image based on the one or more normalized imaging channels of the second autofluorescence image; and outputting the first prediction of the first stained tissue sample image (Ozcan: 0009: “The method includes providing a trained, deep neural network that is executed by image processing software using one or more processors of a computing device, wherein the trained, deep neural network is trained with a plurality of matched immunohistochemical (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample; obtaining one or more autofluorescence images of the label-free tissue sample with a fluorescence imaging device; inputting the one or more autofluorescence images of the label-free tissue sample to the trained, deep neural network; and the trained, deep neural network outputting the digitally stained IHC microscopic image of the label-free tissue sample that reveal the features specific to the at least one target biomarker that appears substantially equivalent to a corresponding image of the same label-free tissue sample had it been IHC stained chemically.”;
0038: “The virtual HER2 staining of breast tissue sample 22 was demonstrated by training deep neural network (DNN) models 10 with a dataset of twenty-five (25) breast tissue sections collected from nineteen (19) unique patients, constituting in total 20,910 image patches, each with 1024×1024 pixels. Once a DNN model 10 was trained, it virtually stained the unlabeled tissue sections using their autofluorescence microscopic images 20 captured with DAPI, FITC, TxRed, and Cy5 filter cubes (see Methods section), matching the corresponding bright-field images of the same field-of-views, captured after standard IHC HER2 staining.”; Wherein the normalized autofluorescence images, as taught by Gurcan, were processed by a virtual staining model in order to output a predicted digitally stained IHC microscopic image.).
Ozcan in view of Gurcan does not disclose expressly: further comprising: receiving a second autofluorescence image of a second tissue sample, wherein: the second autofluorescence image is generated by a second imaging device having the second imaging device type; and the second autofluorescence image comprises the one or more imaging channels; determining, using the machine learning model, a first prediction of a first stained tissue sample image based on the one or more normalized imaging channels of the second autofluorescence image (Claim limitation is interpreted according to the rejection of claim 6 under 35 U.S.C. 112(b) disclosed above.); and outputting the first prediction of the first stained tissue sample image.
Klaiman discloses: A method for processing acquired stained, or unstained, images captured by a microscope and generating a predicted virtual stain output image (Klaiman: 0140-0141: “ the received acquired image is input to the MLL. Although many different types of acquired images can be used in various embodiments of the invention, it is important that the type of acquired image used is identical or very similar to the type of images used during the training phase of the MLL. For example, if the acquired image is an autofluorescence image, the MLL is required to have been trained also on autofluorescence images of tissue samples rather than X-ray images. And if the received acquired image depicts a tissue sample having been stained with three marker specific first stains A, B and C, the MLL is required to have been trained also on images of tissue samples having been stained with marker specific first stains A, B C rather than D or E. Next in step 108 , the MLL automatically transforms the acquired image into an output image.”) wherein the method further comprises: receiving a second autofluorescence image of a second tissue sample, wherein: the second autofluorescence image is generated by a second imaging device having a second imaging device type (Klaiman: 0035-0036: “the acquired image is a fluorescence Image of an unstained tissue sample (i.e., an autofluorescence image) and the output image is a virtually generated H&E stained image wherein regions predicted to comprise a biomarker like FAP are highlighted.
According to another embodiment, the acquired image is a multispectral fluorescence image of a tissue sample having been stained with a plurality of specific biomarker stainings, e.g. CD3, CD8, CD4, pan-CK antibody labeled cytokeratines, Ki67 and/or DAPI, and the output image is a virtually generated H&E stained image wherein regions predicted to comprise a biomarker like FAP or cytokeratins (“CK” or “panCK”) are highlighted.”;
0045: “the method further comprises acquiring, by an image acquisition system, the acquired image. The image acquisition system can be, for example, a bright field microscope, a fluorescence microscope or an X-ray microscope…A fluorescence microscope can be used in particular for acquiring images of tissue samples having been stained with one or more biomarker-specific stains consisting of e.g. an antibody that is directly or indirectly coupled to a fluorophore, or for generating autofluorescence images, or for generating images of non-biomarker specific fluorescent stains.”; Wherein the multispectral fluorescence image and the multispectral fluorescence image constitute different images from different device types.); and the second autofluorescence image comprises the one or more imaging channels (Klaiman: 0035-0036: “the acquired image is a fluorescence Image of an unstained tissue sample (i.e., an autofluorescence image) and the output image is a virtually generated H&E stained image wherein regions predicted to comprise a biomarker like FAP are highlighted.
According to another embodiment, the acquired image is a multispectral fluorescence image of a tissue sample having been stained with a plurality of specific biomarker stainings, e.g. CD3, CD8, CD4, pan-CK antibody labeled cytokeratines, Ki67 and/or DAPI, and the output image is a virtually generated H&E stained image wherein regions predicted to comprise a biomarker like FAP or cytokeratins (“CK” or “panCK”) are highlighted.”;
0135: “Irrespective of whether or not the tissue sample depicted in the acquired image is stained or not, the image was acquired under conditions under which the tissue's autofluorescence signal was the most prominent signal captured by the image acquisition system. Thus, it is possible that the acquired image shows a mixture of autofluorescence signal and some staining signals, but is hereby assumed that the most prominent signal is derived from autofluorescence. Autofluorescence images of tissue samples are examples of images captured by a fluorescent microscope.”); determining, using a machine learning model, a first prediction of a first stained tissue sample image based on the imaging channels of the second autofluorescence image; and outputting the first prediction of the first stained tissue sample image (Klaiman: 0140-0141: “ the received acquired image is input to the MLL. Although many different types of acquired images can be used in various embodiments of the invention, it is important that the type of acquired image used is identical or very similar to the type of images used during the training phase of the MLL. For example, if the acquired image is an autofluorescence image, the MLL is required to have been trained also on autofluorescence images of tissue samples rather than X-ray images. And if the received acquired image depicts a tissue sample having been stained with three marker specific first stains A, B and C, the MLL is required to have been trained also on images of tissue samples having been stained with marker specific first stains A, B C rather than D or E. Next in step 108 , the MLL automatically transforms the acquired image into an output image.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the deep neural network model trained to transform autofluorescence images into virtually stained bright-field equivalent images disclosed by Ozcan in view of Gurcan with the machine learning model trained to transform autofluorescence images into virtually stained autofluorescence images taught by Klaiman. The suggestion/motivation for doing so would have been “Irrespective of whether or not the tissue sample depicted in the acquired image is stained or not, the image was acquired under conditions under which the tissue's autofluorescence signal was the most prominent signal captured by the image acquisition system. Thus, it is possible that the acquired image shows a mixture of autofluorescence signal and some staining signals, but is hereby assumed that the most prominent signal is derived from autofluorescence. Autofluorescence images of tissue samples are examples of images captured by a fluorescent microscope.” (Klaiman: 0135; Wherein the mixture of stained images within the acquired images allows for greater flexibility for image acquisition for virtual staining processing.). Further, one skilled in the art could have substituted one known element for another, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ozcan in view of Gurcan with Klaiman to obtain the invention as specified in claim 6.
Regarding claim 7, Ozcan in view of Gurcan and Klaiman discloses: The method of claim 6, further comprising: receiving a third autofluorescence image of a third tissue sample, wherein: the third autofluorescence image is generated by a third imaging device having the second imaging device type; and the third autofluorescence image comprises the one or more imaging channels (Klaiman: 0035-0036: “the acquired image is a fluorescence Image of an unstained tissue sample (i.e., an autofluorescence image)…According to another embodiment, the acquired image is a multispectral fluorescence image of a tissue sample…”;
0045: “the method further comprises acquiring, by an image acquisition system, the acquired image. The image acquisition system can be, for example, a bright field microscope, a fluorescence microscope or an X-ray microscope…A fluorescence microscope can be used in particular for acquiring images of tissue samples having been stained with one or more biomarker-specific stains consisting of e.g. an antibody that is directly or indirectly coupled to a fluorophore, or for generating autofluorescence images, or for generating images of non-biomarker specific fluorescent stains.”.);
determining one or more normalization parameters for a third channel of the one or more imaging channels, the one or more normalization parameters for the third channel based on a third relationship between the first imaging device type and the second imaging device type (Gurcan: Section: 3.1. Color Normalization: “One of the first steps essential for both fluorescent and bright field microscopy image analysis is color and illumination normalization. This process reduces the differences in tissue samples due to variation in staining and scanning conditions. The illumination can be corrected either using calibration targets or estimating the illumination pattern from a series of images by fitting polynomial surfaces [29]. Another approach is to match the histograms of the images. Software that corrects for spectral and spatial illumination variations is becoming a standard package provided by most bright field manufacturers. This is an essential step for algorithms that heavily depend on color space computations.”; Wherein the learned histograms matching of an input image to a target image is based on relationships between their imaging devices.), the second imaging device type being different from the first imaging device type (Ozcan: 0051: “The autofluorescence images 20 of the unlabeled tissue sections were captured using a standard fluorescence microscope 110 (IX-83, Olympus) with a ×40/0.95 NA (UPLSAPO, Olympus) objective lens. Four fluorescent filter cubes, including DAPI (Semrock DAPI-5060C-OFX, EX 377/50 nm, EM 447/60 nm), FITC (Semrock FITC-2024B-OFX, EX 485/20 nm, EM 522/24 nm), TxRed (Semrock TXRED-4040C-OFX, EX 562/40 nm, EM 624/40 nm), and Cy5 (Semrock CY5-4040C-OFX, EX 628/40 nm, EM 692/40 nm) were used to capture the autofluorescence images 20 at different excitation-emission wavelengths. Each autofluorescence image 20 was captured with a scientific complementary metal-oxide-semiconductor (sCMOS) image sensor (ORCA-flash4.0 V2, Hamamatsu Photonics) with an exposure time of 150 ms, 500 ms, 500 ms, and 1000 ms for DAPI, FITC, TxRed, and Cy5 filters, respectively. Images were normalized for the four (4) channels by their respective exposure times. Thus, DAPI images (for training and test) were divided by their exposure time of 150 ms. The other channels were normalized to their respective exposure times. The image acquisition process was controlled by pManager (version 1.4) microscope automation software. After the standard IHC HER2 staining is complete, the bright-field WSIs were acquired using a slide scanner microscope (AxioScan Z1, Zeiss) with a ×20/0.8 NA objective lens (Plan-Apo).”)
(Klaiman: 0035-0036: “the acquired image is a fluorescence Image of an unstained tissue sample (i.e., an autofluorescence image)…According to another embodiment, the acquired image is a multispectral fluorescence image of a tissue sample…”;
0045: “the method further comprises acquiring, by an image acquisition system, the acquired image. The image acquisition system can be, for example, a bright field microscope, a fluorescence microscope or an X-ray microscope…A fluorescence microscope can be used in particular for acquiring images of tissue samples having been stained with one or more biomarker-specific stains consisting of e.g. an antibody that is directly or indirectly coupled to a fluorophore, or for generating autofluorescence images, or for generating images of non-biomarker specific fluorescent stains.”);
applying the one or more normalization parameters for the third channel to the third channel of the third autofluorescence image; and determining, using the machine learning model, a second prediction of a second stained tissue sample image based on the one or more normalized imaging channels of the first autofluorescence image; and outputting the second prediction of the second stained tissue sample image (Klaiman: 0140-0141: “ the received acquired image is input to the MLL. Although many different types of acquired images can be used in various embodiments of the invention, it is important that the type of acquired image used is identical or very similar to the type of images used during the training phase of the MLL. For example, if the acquired image is an autofluorescence image, the MLL is required to have been trained also on autofluorescence images of tissue samples rather than X-ray images. And if the received acquired image depicts a tissue sample having been stained with three marker specific first stains A, B and C, the MLL is required to have been trained also on images of tissue samples having been stained with marker specific first stains A, B C rather than D or E. Next in step 108 , the MLL automatically transforms the acquired image into an output image.”; Wherein the autofluorescence images, normalized as taught by Gurcan, were processed by a virtual staining model in order to output a predicted digitally stained image.).
As per claim(s) 12, arguments made in rejecting claim(s) 5 are analogous.
As per claim(s) 13, arguments made in rejecting claim(s) 6 are analogous.
As per claim(s) 14, arguments made in rejecting claim(s) 7 are analogous.
As per claim(s) 19, arguments made in rejecting claim(s) 6 are analogous.
As per claim(s) 20, arguments made in rejecting claim(s) 7 are analogous.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights