DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-2, 9-11, 16-17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wu U.S. Patent Application 20190188446 in view of Rephaeli U.S. Patent Application 20190065905.
Regarding claim 10, Wu discloses a system comprising:
a non-transitory computer-readable medium (a non-transitory computer readable storage medium);
one or more processors (data processors) in communication with the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium configured to cause the one or more processors to (paragraph [0010]: a system is provided that includes one or more data processors and a non-transitory computer readable storage medium containing instructions which when executed on the one or more data processors, cause the one or more data processors to perform the actions):
receive a slide image of a target tissue sample of a particular tissue type (paragraph [0034]: An image of the unstained second tissue sample may be accessed at block 220. The image includes a plurality of spectral images of the unstained second tissue sample);
use a first trained machine learning (ML) model to generate virtual immunofluorescence (IF) stains of a first type for the particular tissue type (paragraph [0027]: Some examples of molecular staining methods that may be used to stain the first tissue sample include immunohistochemistry (IHC), immunofluorescence (IF), in situ hybridization (ISH), fluorescent in situ hybridization (FISH), and RNA (f)ISH; paragraph [0030]: At block 205, an image training dataset is accessed... The image training dataset includes a plurality of image pairs, each of which includes a first image 120 of an unstained first tissue sample and a second image 125 of the first tissue sample after staining), wherein
the first trained ML model is trained at least based on a first set of stain images of a plurality of training tissue samples with stains of the first type (paragraph [0032]: The artificial neural network 130 may then be trained by using the image training data set and the parameter set to adjust some or all of the parameters associated with the artificial neurons within the artificial neural network 130, including the weights within the parameter set, at block 215);
generate a virtually stained image of the target tissue sample with the virtual IF stains of the first type using the first trained ML model (paragraph [0034]: The trained artificial neural network 130 then uses the image to generate a virtually stained image of the second tissue sample at block 225. The virtually stained image may be generated based on the parameters of the artificial neural network 130 that are adjusted during the training; paragraph [0027]: Some examples of molecular staining methods that may be used to stain the first tissue sample include immunohistochemistry (IHC), immunofluorescence (IF), in situ hybridization (ISH), fluorescent in situ hybridization (FISH), and RNA (f)ISH); and
cause the virtually stained image of the target tissue sample with the virtual IF stains of the first type to be displayed (paragraph [0034]: The virtually stained image may then be output at block 230. For example, the virtually stained image may be transmitted, stored, and/or displayed on various devices).
Wu discloses all the features with respect to claim 10 as outlined above. However, Wu fails to disclose selecting a trained machine learning (ML) model based on a user input explicitly.
Rephaeli discloses selecting a trained machine learning (ML) model based on a user input (paragraph [0036]: User system 222 may allow a user 224 to send and receive information to assessment system 216... user 224 may specify a level of accuracy for processing... may select certain statistics or parameters for assessment system 216 to use for training, may select the type of machine learning model to use, may indicate portions of the test image data to analyze, and/or may select the form of the output of the assessment).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu’s to select a machine learning model as taught by Rephaeli, to improve detecting flow in images.
Regarding claim 11, Wu as modified by Rephaeli discloses the system of claim 10, wherein the slide image of the target tissue sample comprises an autofluorescence image of the target tissue sample in an unstained condition (Wu’s paragraph [0037]: the hyperspectral autofluorescence images carry information about the molecules within the sample; paragraph [0034]: An image of the unstained second tissue sample may be accessed at block 220. The image includes a plurality of spectral images of the unstained second tissue sample).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu’s to select a machine learning model as taught by Rephaeli, to improve detecting flow in images.
Regarding claim 16, Wu as modified by Rephaeli discloses the system of claim 10, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:
select multiple trained ML models to generate virtual IF stains of multiple types for the particular tissue type based on the user input (Wu’s paragraph [0027]: Some examples of molecular staining methods that may be used to stain the first tissue sample include immunohistochemistry (IHC), immunofluorescence (IF), in situ hybridization (ISH), fluorescent in situ hybridization (FISH), and RNA (f)ISH; paragraph [0030]: At block 205, an image training dataset is accessed... The image training dataset includes a plurality of image pairs, each of which includes a first image 120 of an unstained first tissue sample and a second image 125 of the first tissue sample after staining; Rephaeli’s paragraph [0036]: User system 222 may allow a user 224 to send and receive information to assessment system 216... user 224 may specify a level of accuracy for processing... may select certain statistics or parameters for assessment system 216 to use for training, may select the type of machine learning model to use, may indicate portions of the test image data to analyze, and/or may select the form of the output of the assessment);
generate multiple virtually stained images of the target tissue sample with the virtual IF stains of the multiple types using the multiple trained ML models respectively (Wu’s paragraph [0034]: The trained artificial neural network 130 then uses the image to generate a virtually stained image of the second tissue sample at block 225. The virtually stained image may be generated based on the parameters of the artificial neural network 130 that are adjusted during the training; paragraph [0027]: Some examples of molecular staining methods that may be used to stain the first tissue sample include immunohistochemistry (IHC), immunofluorescence (IF), in situ hybridization (ISH), fluorescent in situ hybridization (FISH), and RNA (f)ISH); and
cause the multiple virtually stained images of the target tissue sample to be displayed (Wu’s paragraph [0034]: The virtually stained image may then be output at block 230. For example, the virtually stained image may be transmitted, stored, and/or displayed on various devices).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu’s to select a machine learning model as taught by Rephaeli, to improve detecting flow in images.
Claim 1 recites the functions of the apparatus recited in claim 10 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 10 applies to the method steps of claim 1.
Claim 2 recites the functions of the apparatus recited in claim 11 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 11 applies to the method steps of claim 2.
Claim 9 recites the functions of the apparatus recited in claim 16 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 16 applies to the method steps of claim 9.
Claim 17 recites the functions of the apparatus recited in claim 10 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 10 applies to the medium steps of claim 17.
Claim 20 recites the functions of the apparatus recited in claim 16 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 16 applies to the medium steps of claim 20.
Claim 3-4, 8, 12-13, 15 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wu U.S. Patent Application 20190188446 in view of Rephaeli U.S. Patent Application 20190065905, and further in view of Rahman U.S. Patent Application 20150118728.
Regarding claim 12, Wu as modified by Rephaeli discloses an image of the target tissue sample with IF stains (Wu’s paragraph [0027]: Some examples of molecular staining methods that may be used to stain the first tissue sample include immunohistochemistry (IHC), immunofluorescence (IF), in situ hybridization (ISH), fluorescent in situ hybridization (FISH), and RNA (f)ISH). However, Wu as modified by Rephaeli fails to disclose PanCK stains.
Rahman discloses PanCK stains (paragraph [0042]: FIG. 15 shows fluorescence images under different filters for cells identification and classification, for Hoechst stains cell nucleus, PE-labelled anti-CD45 antibody stains CD45+ cells (WBCs), and FITC-labelled PanCK antibody stains PanCK+ cells (cancer cells)).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu and Rephaeli’s to stain rare cells as taught by Rahman, to obtain reliable, efficient platform to isolate, enrich and characterize rare cells.
Regarding claim 13, Wu as modified by Rephaeli and Rahman discloses the system of claim 10, wherein the slide image of the target tissue sample comprises an image of the target tissue sample stained with nuclear IF stains (Rahman’s paragraph [0294]: the cells captured onto the micro-slit membrane 1964 were stained with nuclear stain; Wu’s paragraph [0027]: Some examples of molecular staining methods that may be used to stain the first tissue sample include immunohistochemistry (IHC), immunofluorescence (IF), in situ hybridization (ISH), fluorescent in situ hybridization (FISH), and RNA (f)ISH).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu and Rephaeli’s to stain rare cells as taught by Rahman, to obtain reliable, efficient platform to isolate, enrich and characterize rare cells.
Regarding claim 15, Wu as modified by Rephaeli and Rahman discloses the system of claim 10, wherein the first trained ML model is trained based on the first set of stain images of a plurality of training tissue samples with stains of the first type and a second set of stain images of the plurality of training tissue samples with stains of a second type (Rahman’s paragraph [0042]: FIG. 15 shows fluorescence images under different filters for cells identification and classification, for Hoechst stains cell nucleus, PE-labelled anti-CD45 antibody stains CD45+ cells (WBCs), and FITC-labelled PanCK antibody stains PanCK+ cells (cancer cells) (first type); paragraph [0294]: the cells captured onto the micro-slit membrane 1964 were stained with nuclear stain (second type); Wu’s paragraph [0030]: At block 205, an image training dataset is accessed... The image training dataset includes a plurality of image pairs, each of which includes a first image 120 of an unstained first tissue sample and a second image 125 of the first tissue sample after staining; paragraph [0032]: The artificial neural network 130 may then be trained by using the image training data set and the parameter set to adjust some or all of the parameters associated with the artificial neurons within the artificial neural network 130, including the weights within the parameter set, at block 215),
wherein the stains of the first type are associated with a cell-state biomarker for a type of cancer cells, and wherein the stains of the second type are associated with morphological structures of the type of cancer cells (Rahman’s paragraph [0018]: the plurality of magnetic beads couplable to leukocyte specific biomarkers (first type), trapping leukocytes from the sample volume that are coupled to the plurality of magnetic beads at a portion of the input chamber; paragraph [0015]: morphological (second type) separation where size or density is utilized to isolate CTCs from WBCs that overlap with the size of CTCs thus failing to capture the cancer cells that are as small as WBCs).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu and Rephaeli’s to stain rare cells as taught by Rahman, to obtain reliable, efficient platform to isolate, enrich and characterize rare cells.
Claim 3 recites the functions of the apparatus recited in claim 12 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 12 applies to the method steps of claim 3.
Claim 4 recites the functions of the apparatus recited in claim 13 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 13 applies to the method steps of claim 4.
Claim 8 recites the functions of the apparatus recited in claim 15 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 15 applies to the method steps of claim 8.
Claim 19 recites the functions of the apparatus recited in claim 15 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 15 applies to the medium steps of claim 19.
Claim 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Wu U.S. Patent Application 20190188446 in view of Rephaeli U.S. Patent Application 20190065905, and further in view of Shen U.S. Patent Application 20230342940.
Regarding claim 5, Wu as modified by Rephaeli discloses all the features with respect to claim 1 as outlined above. However, Wu as modified by Rephaeli fails to disclose the trained ML model comprises a generative adversarial network.
Shen discloses the trained ML model comprises a generative adversarial network (paragraph [0074]: the trained model may include a Vnet model, a Unet model, a Convolutional Neural Networks (CNN) model, a Recurrent Neural Network (RNN) model, a Generative Adversarial Network (GAN) model, a CycleGAN model, a pix2pix model, or the like, or any combination thereof).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu and Rephaeli’s to use different trained ML models as taught by Shen, to segment structures accurately and effectively.
Regarding claim 6, Wu as modified by Rephaeli and Shen discloses the method of claim 1, wherein the first trained ML model comprises a modified U-Net (Shen’s paragraph [0074]: the trained model may include a Vnet model, a Unet model, a Convolutional Neural Networks (CNN) model, a Recurrent Neural Network (RNN) model, a Generative Adversarial Network (GAN) model, a CycleGAN model, a pix2pix model, or the like, or any combination thereof).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu and Rephaeli’s to use different trained ML models as taught by Shen, to segment structures accurately and effectively.
Claim 7, 14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wu U.S. Patent Application 20190188446 in view of Rephaeli U.S. Patent Application 20190065905, in view of Shen U.S. Patent Application 20230342940, and further in view of Ozcan U.S. Patent Application 20240290473.
Regarding claim 7, Wu as modified by Rephaeli and Shen discloses all the features with respect to claim 6 as outlined above. However, Wu as modified by Rephaeli and Shen fails to disclose the modified U-Net comprises an attention gate module.
Ozcan discloses the modified U-Net comprises an attention gate module (paragraph [0010]: train a deep convolutional neural network (CNN) using structurally-conditioned generative adversarial networks (GAN), together with attention gate modules that process three-dimensional (3D) spatial structure of tissue using 3D convolutions; paragraph [0076]: For the generator network, as shown in FIG. 8A, an attention U-Net structure (encoder—decoder with skip connections and attention gates) was employed to learn the 3D transformation).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu, Rephaeli and Shen’s to use attention gate module as taught by Ozcan, to rapidly transform biopsy images of unstained skin into virtually-stained images.
Regarding claim 14, Wu as modified by Rephaeli, Shen and Ozcan discloses the system of claim 10, wherein the first trained ML model comprises a modified U-Net generative adversarial network (GAN) (Shen’s paragraph [0074]: the trained model may include a Vnet model, a Unet model, a Convolutional Neural Networks (CNN) model, a Recurrent Neural Network (RNN) model, a Generative Adversarial Network (GAN) model, a CycleGAN model, a pix2pix model, or the like, or any combination thereof), and
wherein the modified U-Net GAN comprises an attention gate module (Ozcan’s paragraph [0010]: train a deep convolutional neural network (CNN) using structurally-conditioned generative adversarial networks (GAN), together with attention gate modules that process three-dimensional (3D) spatial structure of tissue using 3D convolutions; paragraph [0076]: For the generator network, as shown in FIG. 8A, an attention U-Net structure (encoder—decoder with skip connections and attention gates) was employed to learn the 3D transformation).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Wu, Rephaeli and Shen’s to use attention gate module as taught by Ozcan, to rapidly transform biopsy images of unstained skin into virtually-stained images.
Claim 18 recites the functions of the apparatus recited in claim 14 as medium steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 14 applies to the medium steps of claim 18.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Yi Yang whose telephone number is (571)272-9589. The examiner can normally be reached on Monday-Friday 9:00 AM-6:00 PM EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/YI YANG/
Primary Examiner, Art Unit 2616
Prosecution Insights