DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
The Office Action has been made issued in response to amendment filed July 14, 2026. Claims 1-11 and 25-37 are pending. Applicant’s arguments have been carefully and respectfully considered in light of the instant amendment, and are not persuasive. Accordingly, this action has been made FINAL.
Claim Objection
Applicant has amended the claims to remove the comma and correct the numbering as suggested by the Examiner. Therefore, the objection is being withdrawn.
Claim Rejections – 35 USC section § 103
Applicant's arguments with respect to claims 1-11 and 25-37 have been considered but are moot in view of the new ground(s) of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-9 and 25-36 are rejected under 35 U.S.C. 103 as being unpatentable over Klaiman (Pub No.: US20210005308) in view of Rivenson et al (NPL titled: Deep learning-based virtual histology staining using autofluorescence of label-free tissue).
Regarding independent claim 1, Klaiman teaches a method of generating a digitally stained immunohistochemical (IHC) microscopic image of a label-free test tissue sample (a digital image of the tissue sample whose pixel intensity values correlate with the amount of a non-biomarker specific stain (e.g. hematoxylin, H&E, or the like) – see [p][0007]), revealing features specific to at least one target biomarker or antigen in the test tissue sample (a trained machine learning logic—MLL. The MLL is a machine learning logic having been trained to (explicitly or implicitly) identify tissue regions predicted to comprise a second biomarker. The method further comprises inputting the received acquired image into the MLL and automatically transforming, by the MLL, the acquired image into an output image – see [p][0012]) the method comprising: providing a deep neural network that is executed by image processing software ([a] “machine learning logic (MLL)” as used herein is a program logic, e.g. a piece of software like a trained neuronal network or a support vector machine or the like that has been trained in a training process and has learned during the training process to perform some predictive and/or data processing tasks based on the provided training data – see [p][0123]) using one or more processors (302 – see Fig 3 and [p][0153]) of a computing device (300 – see Fig 3 and [p][0153]), wherein the trained, deep neural network is trained with a plurality of immunohistochemical (IHC) stained training images or image patches of training tissue samples and their corresponding autofluorescence training images or image patches of the training tissue sample (a digital image of a training tissue sample wherein pixel intensity values of some pixels correlate with the strength of a non-biomarker specific stain (e.g. H&E or hematoxylin) and wherein pixel intensity values of other pixels correlate with the strength of one or more first biomarker specific stains (e.g. a Ki67 specific stain) – see [p][0049]); obtaining one or more autofluorescence images of the label-free tissue sample with a fluorescence imaging device (the acquired image is a fluorescence Image of an unstained tissue sample (i.e., an autofluorescence image) and the output image is a virtually generated H&E stained image wherein regions predicted to comprise a biomarker like FAP are highlighted – see [p][0035]); inputting the one or more autofluorescence images of the label-free tissue sample to the trained, deep neural network (the received acquired image is input to the MLL. Although many different types of acquired images can be used in various embodiments of the invention, it is important that the type of acquired image used is identical or very similar to the type of images used during the training phase of the MLL. For example, if the acquired image is an autofluorescence image – see [p][0140]); and via outputting via deep neural network, the digitally stained IHC microscopic image of the label-free tissue sample that reveal the features specific to the at least one target biomarker or antigen, and that further appears substantially equivalent to a corresponding image of the label-free test tissue sample had it been IHC stained chemically ([t]he output image 206 is a virtual staining image that looks identical or confusingly similar to a bright field image of a tissue sample having been stained with hematoxylin (H), with a Ki67 specific brown stain comprising DAB and with a CD3 specific red stain comprising fastRed – see [p][0144] and he resulting stained specimens are each imaged using an image acquisition system 320 for viewing the detectable signal and acquiring an acquired image 316, such as a digital image of the staining. The images thus obtained are then used by the method of the invention for generating respective output images 318 respectively highlighting a second biomarker of interest for which no biomarker specific stain was applied on the sample before – see [p][0160]).
Ozcan does not teach matched pairs.
However, Ozcan explicitly teaches matched pair (Finally, for the local feature registration we applied an elastic image registration algorithm, which matches the local features of both sets of images (auto-fluorescence vs. brightfield),
by hierarchically matching the corresponding blocks, from large to small (see
Supplementary Fig. S5). The calculated transformation map from this step is finally applied to each bright-field image patch27 – page 9, subsection -Image pre-processing and alignment, [p][003])
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Klaiman of having method of generating a digitally stained immunohistochemical (IHC) microscopic image of a label-free tissue sample, with the teachings of Rivenson of matched pairs.
Wherein having Klaiman matched pairs.
The motivation behind the modification would have been to demonstrate a label-free approach to create a virtually-stained microscopic image using a single wide-field auto-fluorescence image of an unlabeled tissue sample for image generated by the image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image since both Klaiman and Rivenson are methods directed to processing medical images. Wherein Klaiman generates an image by image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image, while Rivenson demonstrate a label-free approach to create a virtually-stained microscopic
image using a single wide-field auto-fluorescence image of an unlabeled tissue sample (Please see Klaiman (Pub No.: US20210005308) [p][0017] and Rivenson et al (NPL titled: Deep learning-based virtual histology staining using autofluorescence of label-free tissue, see Abstract).
Regarding claim 2, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches wherein the deep neural network comprises a convolutional neural network (the neural network is a fully convolutional network, e.g. a network having a U-net architecture – see [p][0063]).
Regarding claim 3, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches wherein the deep neural network is trained using a Generative Adversarial Network (GAN) model (the neural network is a generative adversarial network, e.g. a network having conditional GAN architecture – see [p][0066]).
Regarding claim 4, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches wherein the label-free test tissue sample comprises breast tissue (for example, anti-estrogen receptor antibody (breast cancer) – see [p][0158]).
Regarding claim 5, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches wherein the at least one target biomarker or antigen in the label-free test tissue sample is human epidermal growth factor receptor 2 (HER2) (epidermal growth factor receptor – see [p][0125]).
Regarding claim 6, Klaiman in view of Rivenson teach the method of claim 2, Klaiman teaches wherein the deep neural network is trained using a generator network configured to learn statistical transformation between the plurality of matched pairs of IHC stained and autofluorescence images or image patches of the training tissue sample (the neural network is a generative adversarial network, e.g. a network having conditional GAN architecture – see [p][0066]) and a discriminator network configured to discriminate between a ground truth IHC stained image of the tissue sample and the outputted digitally stained IHC microscopic image of the tissue sample ([t]he discriminator DG implements and “learns” to determine whether or not an image was generated by the inverse generator FG or is an acquired, “real” image of the source domain. All these four blocks participate in the learning process and evaluate and use losses for performing the learning – see [p][0085]).
Regarding claim 8, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches wherein the one or more autofluorescence images of the label-free test sample comprise a plurality of autofluorescence images of the label-free test sample captured at different excitation-emission wavelengths (contrast imaging technology using the difference in absorption of soft X-rays in the water window region (wavelengths: 2.34-4.4 nm, energies: 280-530 eV) by the carbon atom (main element composing the living cell) and the oxygen atom (main element for water) – see [p][0109]).
Regarding claim 9, Klaiman in view of Rivenson teach the method of claim 8, Klaiman teaches, wherein the plurality of autofluorescence images of the label-free test tissue sample captured at different excitation- emission wavelengths comprises autofluorescence images obtained in two or more of the following filter channels: DAPI (see [p][0125]), FITC, TxRed, and Cy5 (see [p][0159]).
Regarding claim 11, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches, wherein the label-free tissue sample comprises a fixed (antibodies adapted to selectively bind to specific proteins – see [p][0158]) or frozen tissue sample.
Regarding claim 25, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches, wherein the label-free tissue sample comprises tissue imaged in vivo (an in-vivo specimen - see [p][0105]).
Regarding claim 26, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches, wherein the plurality of matched pairs of IHC stained and autofluorescence images or image patches of the training tissue samples are subject to a registration process prior to training of the deep neural network, wherein the registration process comprises passing the plurality of matched pairs of IHC stained and autofluorescence images or image patches through a registration neural network model that matches one or both of local styles or local features found in the plurality of matched pairs of IHC stained and autofluorescence images (see [p][0059][0135-0141]).
Regarding claim 27, Klaiman in view of Rivenson teach the method of claim 13, Klaiman teaches, w wherein the registration process further comprises registering the IHC stained images or image patches to respective autofluorescence images or image patches using an elastic registration process (a digital image of the tissue sample whose pixel intensity values correlate (note that correlation is a form of elastic registration) with the amount of a first biomarker specific stain – see [p][0095]).
Regarding claim 28, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches, wherein the digitally stained IHC microscopic image of the label-free test tissue sample is output in real time or near real time after obtaining the one or more autofluorescence images of the label-free test tissue sample (the MLL automatically transforms the acquired image into an output image. The output image highlights tissue regions predicted to comprise the second biomarker – see [p][0141]).
Regarding claim 29, Klaiman in view of Rivenson teach the method of claim 1, Klaiman teaches, wherein the fluorescence imaging device comprises a fluorescence microscope (the acquired image can be an image acquired by a fluorescence microscope (“fluorescence image”) and the output image can be a virtual H&E image highlighting one or more second biomarkers – see [p][0018]).
Regarding independent claim 30, Klaiman teaches system for generating a system for generating a digitally stained immunohistochemical (IHC) microscopic image of a label-free test tissue sample (a digital image of the tissue sample whose pixel intensity values correlate with the amount of a non-biomarker specific stain (e.g. hematoxylin, H&E, or the like) – see [p][0007]), revealing features specific to at least one biomarker or antigen in the test tissue sample, (a trained machine learning logic—MLL. The MLL is a machine learning logic having been trained to (explicitly or implicitly) identify tissue regions predicted to comprise a second biomarker. The method further comprises inputting the received acquired image into the MLL and automatically transforming, by the MLL, the acquired image into an output image – see [p][0012]) the system comprising: a computing device (300 – see Fig 3 and [p][0153]) having image processing software ([a] “machine learning logic (MLL)” as used herein is a program logic, e.g. a piece of software like a trained neuronal network or a support vector machine or the like that has been trained in a training process and has learned during the training process to perform some predictive and/or data processing tasks based on the provided training data – see [p][0123]) executed thereon or thereby, the image processing software comprising a deep neural network that is executed using one or more processors of the computing device, (a trained machine learning logic—MLL. The MLL is a machine learning logic having been trained to (explicitly or implicitly) identify tissue regions predicted to comprise a second biomarker. The method further comprises inputting the received acquired image into the MLL and automatically transforming, by the MLL, the acquired image into an output image – see [p][0012]), wherein the trained, wherein the deep neural network is trained with a plurality of matched pairs of immunohistochemical (IHC) stained training images or image patches of training tissue samples and their corresponding autofluorescence training images or image patches of the training tissue samples (a digital image of a training tissue sample wherein pixel intensity values of some pixels correlate with the strength of a non-biomarker specific stain (e.g. H&E or hematoxylin) and wherein pixel intensity values of other pixels correlate with the strength of one or more first biomarker specific stains (e.g. a Ki67 specific stain) – see [p][0049]), wherein the image processing software configured to receive a one or more autofluorescence images of the label-free test tissue sample (the received acquired image is input to the MLL. Although many different types of acquired images can be used in various embodiments of the invention, it is important that the type of acquired image used is identical or very similar to the type of images used during the training phase of the MLL. For example, if the acquired image is an autofluorescence image – see [p][0140]) and and output the digitally stained IHC microscopic image of the label-free test tissue sample that reveals the features specific to the at least one biomarker or antigen, and that appears substantially equivalent to a corresponding image of the label-free test tissue sample had it been IHC stained chemically ([t]he output image 206 is a virtual staining image that looks identical or confusingly similar to a bright field image of a tissue sample having been stained with hematoxylin (H), with a Ki67 specific brown stain comprising DAB and with a CD3 specific red stain comprising fastRed – see [p][0144] and he resulting stained specimens are each imaged using an image acquisition system 320 for viewing the detectable signal and acquiring an acquired image 316, such as a digital image of the staining. The images thus obtained are then used by the method of the invention for generating respective output images 318 respectively highlighting a second biomarker of interest for which no biomarker specific stain was applied on the sample before – see [p][0160]).
Ozcan does not teach matched pairs.
However, Ozcan explicitly teaches matched pair (Finally, for the local feature registration we applied an elastic image registration algorithm, which matches the local features of both sets of images (auto-fluorescence vs. brightfield),
by hierarchically matching the corresponding blocks, from large to small (see
Supplementary Fig. S5). The calculated transformation map from this step is finally applied to each bright-field image patch27 – page 9, subsection -Image pre-processing and alignment, [p][003])
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Klaiman of having method of generating a digitally stained immunohistochemical (IHC) microscopic image of a label-free tissue sample, with the teachings of Rivenson of matched pairs.
Wherein having Klaiman matched pairs.
The motivation behind the modification would have been to demonstrate a label-free approach to create a virtually-stained microscopic image using a single wide-field auto-fluorescence image of an unlabeled tissue sample for image generated by the image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image since both Klaiman and Rivenson are methods directed to processing medical images. Wherein Klaiman generates an image by image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image, while Rivenson demonstrate a label-free approach to create a virtually-stained microscopic
image using a single wide-field auto-fluorescence image of an unlabeled tissue sample (Please see Klaiman (Pub No.: US20210005308) [p][0017] and Rivenson et al (NPL titled: Deep learning-based virtual histology staining using autofluorescence of label-free tissue, see Abstract).
Regarding claim 31, which corresponds to claim 2 except for reciting a different statutory category of a system. Therefore, the rejection analysis of claim 2 are fully applicable to claim 14.
Regarding claim 32, which corresponds to claim 3 except for reciting a different statutory category of a system. Therefore, the rejection analysis of claim 3 are fully applicable to claim 19.
Regarding claim 33, which corresponds to claim 4 except for reciting a different statutory category of a system. Therefore, the rejection analysis of claim 4 are fully applicable to claim 20.
Regarding claim 34, which corresponds to claim 5 except for reciting a different statutory category of a system. Therefore, the rejection analysis of claim 5 are fully applicable to claim 20.
Regarding claim 35, Klaiman teaches the system of claim 30, wherein the system further comprises a fluorescence microscope configured to obtain the one or more autofluorescence images of the label-free test tissue sample (the MLL automatically transforms the acquired image into an output image. The output image highlights tissue regions predicted to comprise the second biomarker – see [p][0141]).
Regarding claim 36, Klaiman teaches the system of claim 35, wherein the one or more autofluorescence images of the label-free test tissue sample comprises a plurality thereof, and wherein the system further comprises a plurality of filters that are used to obtain a plurality of autofluorescence images of the label-free test tissue sample captured at different excitation-emission wavelengths. (contrast imaging technology using the difference in absorption of soft X-rays in the water window region (wavelengths: 2.34-4.4 nm, energies: 280-530 eV) by the carbon atom (main element composing the living cell) and the oxygen atom (main element for water) – see [p][0109]).
Claims 5 and 37 are rejected under 35 U.S.C. 103 as being unpatentable over
Klaiman (Pub No.: US20210005308) in view of Rivenson et al (NPL titled: Deep learning-based virtual histology staining using autofluorescence of label-free tissue).in view of Ozcan et al (Pub No.: US20240290473A1).
Regarding claim 5, Klaiman in view of Rivenson does not explicitly teach the method of claim 2, wherein the trained, deep neural network comprises an attention-gated neural network.
However, Ozcan explicitly teaches wherein the trained, deep neural network comprises an attention-gated neural network (an attention U-Net structure (encoder—decoder with skip connections and attention gates) – see [p][0076 and Fig 8).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Klaiman in view of Rivenson of having method of generating a digitally stained immunohistochemical (IHC) microscopic image of a label-free tissue sample, with the teachings of Ozcan of having wherein the trained, deep neural network comprises an attention-gated neural network.
Wherein having Klaiman wherein the trained, deep neural network comprises an attention-gated neural network.
The motivation behind the modification would have been to allow a user to rapidly performs in vivo virtual histology of unstained tissue while image generated by the image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image since both Klaiman and Ozcan are methods directed to processing medical images. Wherein Klaiman generates an image by image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image, while Ozcan implements a deep learning-based virtual tissue staining system that rapidly performs in vivo virtual histology of unstained tissue (Please see Klaiman (Pub No.: US20210005308) [p][0017] and Ozcan et al (Pub No.: US20240290473A1), [p][0010]).
Regarding claim 37, Klaiman in view of Rivenson does not explicitly teach the system of claim 18, wherein the trained, deep neural network comprises an attention-gated neural network.
However, Ozcan explicitly teaches wherein the trained, deep neural network comprises an attention-gated neural network (an attention U-Net structure (encoder—decoder with skip connections and attention gates) – see [p][0076 and Fig 8).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Klaiman of having method of generating a digitally stained immunohistochemical (IHC) microscopic image of a label-free tissue sample, with the teachings of Ozcan of having wherein the trained, deep neural network comprises an attention-gated neural network.
Wherein having Klaiman wherein the trained, deep neural network comprises an attention-gated neural network.
The motivation behind the modification would have been to allow a user to rapidly performs in vivo virtual histology of unstained tissue while image generated by the image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image since both Klaiman and Ozcan are methods directed to processing medical images. Wherein Klaiman generates an image by image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image, while Ozcan implements a deep learning-based virtual tissue staining system that rapidly performs in vivo virtual histology of unstained tissue (Please see Klaiman (Pub No.: US20210005308) [p][0017] and Ozcan et al (Pub No.: US20240290473A1), [p][0010]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over
Klaiman (Pub No.: US20210005308) n view of Rivenson et al (NPL titled: Deep learning-based virtual histology staining using autofluorescence of label-free tissue) in view of Elfer et al (NPL titled: DRAQ5andEosin(‘D&E’) as an Analog to Hematoxylin and Eosin for Rapid Fluorescence Histology of Fresh Tissues).
Regarding claim 10, Klaiman in view of Rivenson does not explicitly teach the, wherein the label-free tissue sample comprises a non-fixed or fresh tissue sample.
However, Elfer explicitly teaches wherein the label-free tissue sample comprises a non-fixed or fresh tissue sample (Fresh renal biopsies (n = 3) and prostate biopsies (n = 1) for this study were obtained in accordance with an Institutional ReviewBoard-approved protocol – see section Materials and Methods, subsection Tissue collection and processing, [p][003]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Klaiman in view of Rivenson of having method of generating a digitally stained immunohistochemical (IHC) microscopic image of a label-free tissue sample, with the teachings of Elfer of having wherein the trained, deep neural network comprises an attention-gated neural network.
Wherein having Klaiman wherein the label-free tissue sample comprises a non-fixed or fresh tissue sample
The motivation behind the modification would have been to demonstrate the ability to obtain high-resolution histology-like images of unsectioned, fresh tissues similar to subsequent H&E staining of the tissue while image generated by the image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image since both Klaiman and Elfer are methods directed to processing medical images. Wherein Klaiman generates an image-by-image transformation looks like an image generated by the same type of image acquisition system as used for acquiring the input image, while Elfer demonstrates the ability to obtain high-resolution histology-like images of unsectioned, fresh tissues similar to subsequent H&E staining of the tissue. (Please see Klaiman (Pub No.: US20210005308) [p][0017] and Elfer et al (NPL titled: DRAQ5andEosin(‘D&E’) as an Analog to Hematoxylin and Eosin for Rapid Fluorescence Histology of Fresh Tissues), see Abstract).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDRAE S ALLISON whose telephone number is (571)270-1052. The examiner can normally be reached on Monday-Friday 9am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns, can be reached on (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDRAE S ALLISON/Primary Examiner, Art Unit 2673
September 19, 2026
Prosecution Insights