DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
2. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Oath/Declaration
3. The receipt of Oath/Declaration is acknowledged.
Information Disclosure Statement
4. The information disclosure statement (IDS) submitted on 07/24/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
5. The drawing(s) filed on 07/24/2024 are accepted by the Examiner.
Status of Claims
6. Claims 1-15 are pending in this application.
Claim Rejections - 35 USC § 102
7. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
9. The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
10. Claims 1-13, and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ozcan et al. (US 2023/0030424 A1) hereinafter ‘Ozcan’.
Regarding Claim 1:
Ozcan discloses an image processing system (Ozcan: “A deep learning-based digital/virtual staining method and system enables the creation of digitally/virtually-stained microscopic images from label or stain-free samples” Abstract) comprising one or more processors and one or more storage devices (Ozcan: Fig. 1: “The system includes a computing device 100 that contains one or more processors 102 therein and image processing software 104 that incorporates the trained, deep neural network 10 (e.g., a convolutional neural network as explained herein in one or more embodiments).” ¶ [0043]), wherein the image processing system is configured to:
receive at least one reference image (Ozcan: Figs. 1 and 2 ‘image 20’), wherein each reference image is a microscopy image capturing cells of a biological sample (Ozcan: “FIG. 1 schematically illustrates one embodiment of a system 2 for outputting digitally stained images 40 from an input microscope image 20 of a sample 22.” [0043]), wherein the at least one reference image includes at least one reference labeling directed to a reference cellular compartment of the captured cells (Ozcan: ¶[0070]);
employ a trained deep neural network for processing the at least one reference image to generate a target image (Ozcan: Fig. 2 step 10 described at ¶[0072]), wherein the target image includes a target labeling directed to a target cellular compartment of the captured cells (Ozcan: Fig. 2 ‘output 40’ described at ¶[0078]), wherein the at least one reference labeling comprises a fluorescence labeling (Ozcan: ¶[0070]), wherein the reference cellular compartment is a distributed structure within cells (Ozcan: ¶[0070]), and wherein the target cellular compartment is the cell nucleus (Ozcan: ¶[0078]).
Regarding Claim 2:
Ozcan further disclose the image processing system of claim 1, wherein the image processing system is further configured to process the target image to obtain centroids and shapes of cell nuclei of the captured cells (Ozcan: discloses segmenting the target image in Figs. 16 and 17 ¶[0067-0068; 0063]; Note that shape of a cell is implied by the segmentation such as the claimed centroid).
Regarding Claim 3:
Ozcan further disclose the image processing system of claim 1, wherein the at least one reference labeling is based on cytoskeleton markers or cytosolic markers (Ozcan: “FIG. 16 illustrates a display that shows a Graphical User Interface (GUI) that is used to display an output image 40 from the trained, deep neural network 10 according to one embodiment. In this embodiment, the user is provided a list of tools (e.g., pointer, marker, eraser, loop, highlighter, etc.) that can be used to identify and select certain regions of the output image 40 for virtual staining. For example, the user may use one or more of the tools to select certain areas or regions of the tissue in the output image 40 for virtual staining. In this particular example, three areas are identified by hashed lines (Areas A, B, C) that have been manually selected by the user. The user may be provided with a palette of stains to choose from to stain the regions. For example, the user may be provided stain options to stain the tissue (e.g., Masson's Trichrome, Jones, H&E). The user is then able to select the different Areas for staining with one or more of these stains. This results in a micro-structured output such as that illustrated in FIG. 17. In a separate embodiment, image processing software 104 may be used to automatically identify or segment certain regions of the output image 40. For example, image segmentation and computer-generated mapping may be used to identify certain histological features in the imaged sample 22. For example, cell nuclei, certain cell or tissue types may be automatically identified by the image processing software 104. These automatically identified regions-of-interest in the output image 40 may be manually and/or automatically stained with one or more stains/stain combinations.” ¶[0067]).
Regarding Claim 4:
Ozcan further disclose the image processing system of claim 1, wherein the at least one reference labeling comprises a first fluorescence labeling and a second fluorescence labeling different from the first fluorescence labeling (Ozcan: “The digitally/virtually-stained output images 40 from the trained, deep neural network 10 were compared to the standard histochemical staining images 48 for diagnosing multiple types of conditions on multiple types of tissues, which were either Formalin-Fixed Paraffin-Embedded (FFPE) or frozen sections. The results are summarized in Table 1 below. The analysis of fifteen (15) tissue sections by four board certified pathologists (who were not aware of the virtual staining technique) demonstrated 100% non-major discordance, defined as no clinically significant difference in diagnosis among professional observers. The “time to diagnosis” varied considerably among observers, from an average of 10 seconds-per-image for observer 2 to 276 seconds-per-image for observer 3. However, the intra-observer variability was very minor and tended towards shorter time to diagnosis with the virtually-stained slide images 40 for all the observers except observer 2 which was equal, i.e., ˜10 seconds-per-image for both the virtual slide image 40 and the histology stained slide image 48. These indicate very similar diagnostic utility between the two image modalities.” ¶[0079]).
Regarding Claim 5:
Ozcan further disclose the image processing system of claim 1, wherein the at least one reference image comprises a plurality of reference images, wherein the plurality of reference images represents a series of optical slices parallel to an optical path of a microscope, and wherein the processing the at least one reference image comprises generating a plurality of target images and generating a three-dimensional reconstruction of the target cellular compartment from the plurality of target images (Ozcan: “Following the conclusion of the training phase, the deep neural network 10a can be used to refocus aberrated images from a single defocused image, as demonstrated in FIG. 13, in contrast to standard autofocusing techniques, which require the acquisition of multiple images through multiple depth planes. As seen in FIG. 13 a single defocused image 20d obtained from the microscope 110 is input to the trained, deep neural network 10a and a focused image 20f is generated. This focused image 20f can then be input into the trained, deep neural network 10, in one embodiment as explained herein. Alternatively, the functions of the deep neural networks 10 (virtual staining) may be combined with the autofocusing functionality into a single deep neural network 10a.” ¶[0057]).
Regarding Claim 6:
Ozcan further disclose the image processing system of claim 1, further comprising an image analysis system configured to process the target image based on the target labeling, wherein the processing comprises at least one of: performing a semantic image segmentation to obtain class labels for each pixel in the target image, applying a cell segmentation algorithm on the target image, applying distance transforms and thresholding on the target image, obtaining a count of the captured cells, or identifying a cell type of the captured cells (Ozcan discloses performing a semantic image segmentation to obtain class labels for each pixel in the target image, applying a cell segmentation algorithm on the target image and identifying a cell type of the captured cells described at ¶[0068] Note that applying distance transforms and thresholding on the target image or obtaining a count of the captured cells are well known common practice in the art).
Regarding Claim 7:
Ozcan further disclose the image processing system of claim 1, wherein the image processing system is further configured to process the target image to provide commands to a microscope to least one of: adapt a control flow of the microscope, change imaging modalities used by the microscope, or change a center field of view of the microscope based on the identified cells (Ozcan teaches autofocusing the microscope based on the target image at ¶[0142-0148]).
Regarding Claim 8:
Ozcan further disclose a microscope including the image processing system of claim 1, wherein the microscope is configured to provide the at least one reference image to the image processing system and to display the target image (Ozcan: Fig. 1 “The computing device 100 may be associated with or connected to a monitor or display 106 that is used to display the digitally stained images 40. The display 106 may be used to display a Graphical User Interface (GUI) that is used by the user to display and view the digitally stained images 40. In one embodiment, the user may be able to trigger or toggle manually between multiple different digital/virtual stains for a particular sample 22 using, for example, the GUI. Alternatively, the triggering or toggling between different stains may be done automatically by the computing device 100.” ¶[0043]).
Regarding Claim 9:
Ozcan similarly discloses a computer-implemented method for automated image management for a microscope, the method comprising: using the microscope to obtain at least one reference image; sending the at least one reference image to a server comprising the image processing system of claim 1; receiving the target image sent from the server at the microscope; and controlling settings of the microscope based on processing the target image (Ozcan: See rejection of claim 1; Note that a server is described at ¶[0043] “The computing device 100 may include, as explained herein, a personal computer, laptop, mobile computing device, remote server, or the like, although other computing devices may be used (e.g., devices that incorporate one or more graphic processing units (GPUs)) or other application specific integrated circuits (ASICs). GPUs or ASICs can be used to accelerate training as well as final image output.“).
Regarding Claim 10: (drawn to a method)
The proposed rejection of system claim 1, over Ozcan is similarly cited to reject the steps of the method of claim 10 because these steps occur in the operation of the system as discussed above. Thus, the arguments similar to that presented above for claim 1 are equally applicable to claim 10.
Ozcan: “ a trained, deep neural network 10a is provided that takes an aberrated and/or out-of-focus input image 20 and then outputs a corrected image 20a that substantially matches a focused image of the same field-of-view. A critical step for high-quality and rapid microscopy imaging of e.g., tissue samples 22 is autofocusing. Conventionally, autofocusing is performed using a combination of optical and algorithmic methods. These methods are time consuming, as they image the specimen 22 at multiple focusing depths. Ever-growing demand for higher throughput microscopy, entail more assumptions that are made on the specimen's profile. In other words, one sacrifices the accuracy that is usually obtained by the multiple focal depths acquisition, with the assumption that in adjacent field-of-view, the specimen's profile is uniform. This type of assumption often results image focusing errors. These errors might require the reimaging of the specimen, which is not always possible, for example, in life science experiments. In digital pathology, for example, such focusing errors might prolong the diagnosis of a patient's disease.” ¶[0054].
Regarding Claim 11:
Ozcan further discloses the method of claim 10, wherein the deep neural network is based on a fully convolutional image-to-image neural network, or is based on a visual transformer, or is based on a diffusion model, or is based on an adversarial network (Ozcan: “For example, in one preferred embodiment as is described herein, the trained, deep neural network 10 is trained using a GAN model. In a GAN-trained deep neural network 10, two models are used for training. A generative model is used that captures data distribution while a second model estimates the probability that a sample came from the training data rather than from the generative model. Details regarding GAN may be found in Goodfellow et al., Generative Adversarial Nets., Advances in Neural Information Processing Systems, 27, pp. 2672-2680 (2014), which is incorporated by reference herein. Network training of the deep neural network 10 (e.g., GAN) may be performed the same or different computing device 100. For example, in one embodiment a personal computer may be used to train the GAN although such training may take a considerable amount of time. To accelerate this training process, one or more dedicated GPUs may be used for training. As explained herein, such training and testing was performed on GPUs obtained from a commercially available graphics card. Once the deep neural network 10 has been trained, the deep neural network 10 may be used or executed on a different computing device 110 which may include one with less computational resources used for the training process (although GPUs may also be integrated into execution of the trained deep neural network 10).” ¶[0044]).
Regarding Claim 12:
Ozcan further discloses the method of claim 10, wherein the training tuples further comprise bright field microscopy images and phase contrast microscopy images (Ozcan: “To train the deep neural network 10a, a Generative Adversarial Network (GAN) may be used to perform the virtual focusing. The training dataset is composed of autofluorescence (endogenous fluorophores) images of multiple tissue sections, for multiple excitation and emission wavelengths. In another embodiment, the training images can be other microscope modalities (e.g., brightfield microscope, a super-resolution microscope, a confocal microscope, a light-sheet microscope, a FLIM microscope, a widefield microscope, a darkfield microscope, a structured illumination microscope, a computational microscope, a ptychographic microscope, a synthetic aperture-based microscope, or a total internal reflection microscope, and a phase contrast microscope).: ¶[0058-0059]).
Regarding Claim 13:
Ozcan further discloses the method of claim 10, wherein the method further comprises pre-training the deep neural network on pre-training tuples for predicting another target labeling from another reference labeling, the another target labeling and the another reference labeling being different from the target labeling and the reference labeling (Ozcan: “For example, in one preferred embodiment as is described herein, the trained, deep neural network 10 is trained using a GAN model. In a GAN-trained deep neural network 10, two models are used for training. A generative model is used that captures data distribution while a second model estimates the probability that a sample came from the training data rather than from the generative model. Details regarding GAN may be found in Goodfellow et al., Generative Adversarial Nets., Advances in Neural Information Processing Systems, 27, pp. 2672-2680 (2014), which is incorporated by reference herein. Network training of the deep neural network 10 (e.g., GAN) may be performed the same or different computing device 100. For example, in one embodiment a personal computer may be used to train the GAN although such training may take a considerable amount of time. To accelerate this training process, one or more dedicated GPUs may be used for training. As explained herein, such training and testing was performed on GPUs obtained from a commercially available graphics card. Once the deep neural network 10 has been trained, the deep neural network 10 may be used or executed on a different computing device 110 which may include one with less computational resources used for the training process (although GPUs may also be integrated into execution of the trained deep neural network 10).” ¶[0044]; “The trained neural network 10 is trained, in one embodiment, using lifetime (e.g., decay time) fluorescence images 20 of unstained sample 22, with a paired ground truth image 48, which is the bright-field image of the same field of view after IHC staining. The trained neural network 10 may also be trained, in another embodiment, using a combination of lifetime fluorescence images 20 and fluorescence intensity images 20. Once the neural network 10 has converged (i.e., it is trained), it can be used for the blind inference of new lifetime images 20 from unstained tissue samples 22 and transform or output them to the equivalence of bright-field images 40 of after staining, without any parameter tuning, as illustrated in FIGS. 11A and 11B.
To train the artificial neural network 10, a generative adversarial network (GAN) framework was used to perform virtual staining. The training dataset is composed autofluorescence (endogenous fluorophores) lifetime images 20 of multiple tissue sections 22, for single or multiple excitation and emission wavelengths. The samples 22 are scanned by a standard fluorescence microscope 110 with photon counting capability, that outputs the fluorescence intensity image 201 and lifetime image 20L at each field of view. The tissue samples 22 were also sent to a pathology lab for IHC staining and scanned by a bright-field microscope which was used to generate the ground truth training images 48. The fluorescence lifetime images 20L and the bright-field images 48 of the same field-of-view are paired. The training dataset is composed from thousands of such pairs 20L, 48, which are used as input and output for the training of the network 10, respectively. Typically, an artificial neural network model 10 converges after ˜30 hours on two Nvidia 1080Ti GPUs. Once the neural network 10 converges, the method enables virtual IHC staining of unlabeled tissue section 22 in real time performance, as shown in FIG. 12. Note that deep neural network 10 may be trained with fluorescence lifetime images 20L and fluorescence intensity images 201. This is illustrated in FIG. 11B.” [0052-0053; 0181; 0195]).
Regarding Claim 15: (drawn to a trained machine learning algorithm)
The proposed rejection of system claim 1 and method claim 10, over Ozcan is similarly cited to reject the steps of the algorithm of claim 10 because these steps occur in the operation of the system and method as discussed above. Thus, the arguments similar to that presented above for claims 1 and 10 are equally applicable to claim 15.
Allowable Subject Matter
11. Claim 14 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
12. The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 14:
The prior art disclosed do not fairly suggest the method of claim 13, wherein the deep neural network is based on a variational autoencoder, wherein the method comprises pre-training the deep neural network on the pre-training tuples in a semi-supervised manner.
Conclusion
13. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Stumpe et al. (US 2020/0394825) describes a machine learning predictor model is trained to generate a prediction of the appearance of a tissue sample stained with a special stain such as an IHC stain from an input image that is either unstained or stained with H&E. Training data takes the form of thousands of pairs of precisely aligned images, one of which is an image of a tissue specimen stained with H&E or unstained, and the other of which is an image of the tissue specimen stained with the special stain. The model can be trained to predict special stain images for a multitude of different tissue types and special stain types, in use, an input image, e.g., an H&E image of a given tissue specimen at a particular magnification level is provided to the model and the model generates a prediction of the appearance of the tissue specimen as if it were stained with the special stain. The predicted image is provided to a user and displayed, e.g., on a pathology workstation.
14. Any inquiry concerning this communication or earlier communications from the examiner should be directed to NEIL R MCLEAN whose telephone number is (571)270-1679. The examiner can normally be reached Monday-Thursday, 6AM - 4PM, PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi M Sarpong can be reached at 571.270.3438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NEIL R MCLEAN/Primary Examiner, Art Unit 2681
Prosecution Insights