Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 20 objected to because of the following informalities: Line 4 of the claim recites "one or mor data processors" The claim should read "one or more data processors". Appropriate correction is required.
Claim Interpretation
Regarding claim 8, the term “substantially similar” is recited, which is a relative term and is not defined by the claim. The specification provides a standard for ascertaining the requisite degree in [0068], which recites “As used herein, the terms “similarly”, “substantially,” “approximately” and “about” are defined as being largely but not necessarily wholly what is specified (and include wholly what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the term “similarly”, “substantially,” “approximately,” or “about” may be substituted with “within [a percentage] of” what is specified, where the percentage includes 0.1, 1, 5, and 10 percent”. Accordingly, the term “substantially similar” will be interpreted as “largely but not necessarily wholly what is specified”, or as “within a percentage of what is specified”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1, 6-8 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, it recites “A method comprising: receiving a set of real microscopy images representing a plurality of objects of a biological substance, each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images; accessing a plurality of features of the set of real microscopy images and the plurality of objects; and generating, based on the plurality of features, one or more synthetic microscopy images representing the plurality of objects of the biological substance.”
Step 1: With regard to step (1), claim 1 is directed to a method, i.e., to a process, which is one of the statutory categories of inventions.
Step 2A-1: With regard to 2A-1, the first limitation of [1] “receiving a set of real microscopy images representing a plurality of objects of a biological substance, each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images” is directed to a data gathering process without significantly more. For example, the limitation [1] in the context of this claim encompasses a situation in which an input such as a set of images from a microscope can be manually gathered by a person. Similarly, the second limitation of [2] “accessing a plurality of features of the set of real microscopy images and the plurality of objects”, as drafted, is a process that, under its broadest reasonable interpretation, is directed to a mental process that a person is capable of act on it in the mind. For example, the limitation of [2] in the context of this claim encompasses a situation where a human may observe a quantity and size of cells depicted in a microscope image. The third limitation of [3] “generating, based on the plurality of features, one or more synthetic microscopy images representing the plurality of objects of the biological substance”, as drafted, is a process that under its broadest reasonable interpretation, is directed to a process which a person is capable of performing by hand with pen and paper. For example, the limitation of [3] in the context of this claim encompasses a situation where a human may draw a representation of the quantity and size of cells observed in a microscope image, as explained above in reference to limitation [2].
Step 2A-2: With regard to 2A-2, the judicial exception is not integrated into a practical application because the claim only recites a data gathering step without significantly more, and mental processes which do not integrate the abstract idea into a practical application because they do not impose meaningful limits on practicing the abstract idea.
Step 2B: Because the claim falls under (2A), the claim is further evaluated under (2B). The claim herein does not include additional elements that are sufficient to amount to significantly more than a judicial exception because as discussed above with respect to integration of the abstract idea into practical application. The claim is not patent eligible.
Regarding dependent claim 6, the claim recites “The method of claim 1, wherein the biological substance comprises a DNA array, an oligo array, a biological tissue, or an array of cells” which further defines what data is gathered without significantly more.
Regarding dependent claim 7, the claim recites “The method of claim 1, wherein the biological substance comprises a DNA array and the plurality of objects comprises a plurality of DNA nanoballs” which further defines what data is gathered without significantly more.
Regarding dependent claim 8, the claim recites “The method of claim 1, wherein the one or more synthetic microscopy images have substantially similar features to those of the set of real microscopy images” which, as drafted, is a process that, under its broadest reasonable interpretation, is directed to a mental process that a person is capable of act on it in the mind.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 6, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Yao et al (US 12327327 B2, hereinafter Yao) and Smith et al (US 20230281819 A1, hereinafter Smith).
Regarding claim 1, Yao teaches a method comprising: receiving a set of real microscopy images representing a plurality of objects of a biological substance (Claim 1 “receiving a low resolution image of an object, wherein the low resolution image is a fluorescence microscopy image of the object”, Col 7 Line 24-26 “the at least one processor 102 of the device 100 can receive low resolution images from the memory 104 or storage units (not shown) in the device 100.”, Col 7 Line 60-67 “ The object comprises intracellular structures of one or more cells. The intracellular structures comprise one or more microtubules and/or organelles of the one or more cells. The organelles comprise one or more subcellular structures, e.g., nucleus, mitochondria, endoplasmic reticulum, Golgi apparatus, vesicles, vacuoles, etc. of the one or more cells. In some embodiments, the subcellular structures comprise one or more DNA nanorulers”), (Claim 1 “generating an edge map of the low resolution image by an edge extractor, wherein the generating of the edge map comprises extracting the edge map by the edge extractor on a subpixel level based on a radial symmetry of fluorophore in the fluorescence microscopy image, an edge intensity at each subpixel being defined by an extent to which surrounding intensity gradients converge to the subpixel, the edge intensity being weighted by a pixel intensity of the subpixel”, Col 8 Line 45-47 “By determining edges of intracellular structures in the fluorescence microscopy image, features of the intracellular structures are extracted in the edge map 502”); and generating, based on the plurality of features, one or more synthetic microscopy images representing the plurality of objects of the biological substance (Claim 1 “inputting the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”).
Yao fails to explicitly teach each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images. In related field of endeavor Smith teaches each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images (Par 162 “receive a non-synthetic microscopy slide image comprising an object bounding box around a plurality of object pixels; crop out the plurality of object pixels within the object bounding box”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Yao to include each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images as taught by Smith. Doing so would quickly and accurately classify materials within a slide image (Par 28 “improved classification processes that resolve these issues and increase the likelihood that a machine learning algorithm quickly and accurately classifies particles and other materials within a slide image”)
Regarding claim 2, Yao as modified by Smith teaches the method of claim 1. Yao further teaches further comprising: performing a simulation of sequencing biochemistry for the biological substance, wherein the simulation is configured to receive the plurality of features of a real microscopy image of the set of real microscopy images as an input (Col 17 Line 56-60 “FIGS. 10A to 10D depict embodiments where SR images reconstructed based on the SFSRM image processing approach of the present application are utilised for in situ genome sequencing in an interphase human fibroblast nucleus.”, Col 18 Line 6-13 “In this regard, FIG. 10A shows a fluorescence in situ hybridization (FISH) labeling of the genome of the interphase human fibroblast nucleus. FIG. 10B depicts a widefield LR image of the nucleus. FIG. 10C depicts a SR image of the nucleus reconstructed based on the SFSRM image processing approach of the present application. FIG. 10D depicts a table of chromosome sequences by different probe combinations.”);
PNG
media_image1.png
571
789
media_image1.png
Greyscale
and determining, based on the input, a seed intensity for each object of the plurality of objects of the real microscopy image, wherein the seed intensity corresponds to a signal volume for the object (Col 8 Line 24-29 “the edge extractor (not shown) of the device 100 extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image, wherein an edge intensity at each subpixel (x.sub.c, y.sub.c) is defined by an extent θ to which surrounding intensity gradients converge to the subpixel”, as stated in [0066] of the applicants specification, intensity value may be a fluorescence signal).
Regarding claim 3, Yao as modified by Smith teaches the method of claim 2. Yao further teaches further comprising: generating a seed image based on the seed intensity for each object of the plurality of objects, wherein each pixel in the seed image represents the signal volume for the object (Col 8 Line 25-29 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image, wherein an edge intensity at each subpixel (x.sub.c, y.sub.c) is defined by an extent θ to which surrounding intensity gradients converge to the subpixel”, where the edge map corresponds to the seed image).
Regarding claim 6, Yao as modified by Smith teaches the method of claim 1. Yao further teaches wherein the biological substance comprises a DNA array, an oligo array, a biological tissue, or an array of cells (Col 7 Line 60-67 “ The object comprises intracellular structures of one or more cells. The intracellular structures comprise one or more microtubules and/or organelles of the one or more cells. The organelles comprise one or more subcellular structures, e.g., nucleus, mitochondria, endoplasmic reticulum, Golgi apparatus, vesicles, vacuoles, etc. of the one or more cells. In some embodiments, the subcellular structures comprise one or more DNA nanorulers”).
Regarding claim 8, Yao as modified by Smith teaches the method of claim 1. Yao further teaches wherein the one or more synthetic microscopy images have substantially similar features to those of the set of real microscopy images (Col 2 Line 4-8 “the reconstructed image 712 produced by the method 200 for image processing of the present application correctly reconstructs fine structures of the selected region with high similarity to those shown in the GT image 704 and the enlarged image 706”)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Yao and Smith as applied to claim 2 above, and further in view of Ozcan et al (US 11222415 B2, hereinafter Ozcan).
Regarding claim 4, Yao as modified by Smith teaches the method of claim 2. Yao further teaches wherein generating the one or more synthetic microscopy images comprises:
generating a point spread function for the plurality of objects of the real microscopy image based on the plurality of features (Col 8 Line 45-48 “By determining edges of intracellular structures in the fluorescence microscopy image, features of the intracellular structures are extracted in the edge map 502”, Col 9 Line 52-54 “the LR image is degraded from the ground truth image 704 by 200 nm point-spread-function blurring and then downsampling for 10 times”, Col 9 Line 56-57 “Thereafter, an edge map 304 of the low resolution image 302 is generated 322 by an edge extractor (not shown)”); and generating a synthetic image of the one or more synthetic microscopy images based on the signal distribution over the plurality of pixels and the plurality of features (Col 8 Line 25-27 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image”, Col 8 Line 49-52 “input the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”).
Yao as modified by Smith fails to explicitly teach determining a signal distribution over a plurality of pixels by aggregating the point spread function and the seed intensity for each object. In related field of endeavor, Ozcan teaches determining a signal distribution over a plurality of pixels by aggregating the point spread function and the seed intensity for each object (Col 8 Line 50-54 “FIG. 24D is a graph showing the signal (count) as a function of FWHM of PSF (nm) for the network input, network output, and ground truth”, where PSF is point spread function, Col 27 Line 50-56 “Despite the fact that the fluorescent signal from 20 nm beads is rather weak, the deep neural network 10 (trained only with BPAEC samples) successfully picked up the signal from individual nano-beads and blindly improved the resolution to match that of the ground truth 50, as shown in the PSF comparison reported in FIG. 24D.” as stated in [0066] of the applicant’s specification, intensity value may be a fluorescence signal)
PNG
media_image2.png
617
490
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Yao and Smith to include determining a signal distribution over a plurality of pixels by aggregating the point spread function and the seed intensity for each object as taught by Ozcan. Doing so would enhance the output image and show uniform focusing with improved resolution (Col 27 Line 63-67 “The deep network results, on the other hand, once again demonstrate the enhanced DOF of the network output image 40, showing uniform focusing with improved resolution at the network output image 40.”)
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Yao and Smith as applied to claim 1 above, and further in view of Ozcan 2 et al (US 12300006 B2, hereinafter Ozcan 2).
Regarding claim 5, Yao as modified by Smith teaches the method of claim 1. Yao further teaches further comprising: generating a trained machine learning model by training a machine learning model (Col 8 Line 53-54 “the neural network is trained using a multicomponent loss function”, Col 10 Line 58-64 “the frequency loss function 420 receives the reconstructed super resolution image 308 and the ground truth image 310 at steps 330, 332 and compares frequency spectrum differences 424 of the reconstructed super resolution image 308 and the ground truth image 310 in a specified frequency region”).
Yao fails to explicitly teach training a machine learning model to generate intensity values for additional real microscopy images, but in related field of endeavor Ozcan 2 teaches a trained machine learning model to generate intensity values for additional real microscopy images (Col 2 Line 15-19 “a linear approximation that relates the fluorescence intensity of an image to the dye concentration per tissue volume, using empirically determined constants that represent the mean spectral response of various dyes embedded in the tissue”, Col 9 Line 54-59 “outputs or generates a digitally stained or labelled output image 40. The digitally stained output image 40 has “staining” that has been digitally integrated into the stained output image 40 using the trained, deep neural network 10”, as stated in [0066] of the applicant’s specification, intensity value may be a fluorescence signal)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Yao to include training a machine learning model to generate intensity values for additional real microscopy images as taught by Ozcan 2. Doing so would generate virtually stained samples which appear just like samples that have undergone histochemical staining even though no such staining operation was conducted (Col 10 Line 4-7 “This digital or virtual staining of the tissue section sample 22 appears just like the tissue section sample 22 had undergone histochemical staining even though no such staining operation was conducted.”)
Claims 7 is rejected under 35 U.S.C. 103 as being unpatentable over Yao and Smith as applied to claim 1 above, and further in view of Zhang et al (Single-shot structured illumination microscopy, hereinafter Zhang).
Regarding claim 7, Yao and Smith teach the method of claim 1, but fail to explicitly teach wherein the biological substance comprises a DNA array and the plurality of objects comprises a plurality of DNA nanoballs. In related field of endeavor, Zhang teaches wherein the biological substance comprises a DNA array and the plurality of objects comprises a plurality of DNA nanoballs (Zhang Fig. 1, included below, depicts imaging an array of DNA nanoballs and inputting the images to generators in a neural network, Introduction Line 1 “Imaging of self-assembling DNA nanoball (DNB) array”)
PNG
media_image3.png
320
692
media_image3.png
Greyscale
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Yao and Smith to include wherein the biological substance comprises a DNA array and the plurality of objects comprises a plurality of DNA nanoballs as taught by Zhang. Doing so would improve the throughput and measurement speed of DNB samples in the gene sequencing process (Page 2, Paragraph 1 “the significant applications value of the proposed technique in improving the throughput and measurement speed of a single DNB array sample in the gene sequencing process”)
Claims 9, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Yao and Ozcan 2.
Regarding claim 9, Yao teaches a computer-program product tangibly embodied in a non-transitory machine-readable medium, including instructions configured to cause one or more data processors to (Col 7 Line 10-13 “The memory 104 includes computer program code (not shown in FIG. 1) for execution by the at least one processor 102 to perform steps in accordance with a method of image processing”): generate a set of synthetic microscopy images using seed intensities and features extracted or known from a set of real microscopy images, each synthetic microscopy image representing a plurality of objects of a biological substance (Col 7 Line 37-38 “reconstruct super resolution images based on various low resolution images”, Col 8 Line 25-27 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image”, as stated in [0066] of the applicants specification, intensity value may be a fluorescence signal, Col 8 Line 49-52 “input the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”, Col 7 Line 60-65 “The object comprises intracellular structures of one or more cells. The intracellular structures comprise one or more microtubules and/or organelles of the one or more cells. The organelles comprise one or more subcellular structures, e.g., nucleus, mitochondria, endoplasmic reticulum, Golgi apparatus, vesicles, vacuoles, etc. of the one or more cells.”); generate a set of seed images from the seed intensities, wherein each seed image corresponds to a synthetic microscopy image of the set of synthetic microscopy images, and wherein each pixel in the seed image represents a signal volume for an object of the plurality of objects (Col 8 Line 25-29 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image, wherein an edge intensity at each subpixel (x.sub.c, y.sub.c) is defined by an extent θ to which surrounding intensity gradients converge to the subpixel”, as stated in [0066] of the applicants specification, intensity value may be a fluorescence signal); and generate a trained machine learning model by training a machine learning model (Col 8 Line 53-54 “the neural network is trained using a multicomponent loss function”, Col 10 Line 58-64 “the frequency loss function 420 receives the reconstructed super resolution image 308 and the ground truth image 310 at steps 330, 332 and compares frequency spectrum differences 424 of the reconstructed super resolution image 308 and the ground truth image 310 in a specified frequency region”).
Yao fails to explicitly teach training a machine learning model to generate intensity values for additional real microscopy images, but in related field of endeavor Ozcan 2 teaches training a machine learning model to generate intensity values for additional real microscopy images (Col 2 Line 15-19 “a linear approximation that relates the fluorescence intensity of an image to the dye concentration per tissue volume, using empirically determined constants that represent the mean spectral response of various dyes embedded in the tissue”, Col 9 Line 54-59 “outputs or generates a digitally stained or labelled output image 40. The digitally stained output image 40 has “staining” that has been digitally integrated into the stained output image 40 using the trained, deep neural network 10”, as stated in [0066] of the applicant’s specification, intensity value may be a fluorescence signal)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Yao to include training a machine learning model to generate intensity values for additional real microscopy images as taught by Ozcan 2. Doing so would generate virtually stained samples which appear just like samples that have undergone histochemical staining even though no such staining operation was conducted (Col 10 Line 4-7 “This digital or virtual staining of the tissue section sample 22 appears just like the tissue section sample 22 had undergone histochemical staining even though no such staining operation was conducted.”)
Regarding claim 18, Yao as modified by Ozcan 2 teaches The computer-program product of claim 9. Yao further teaches wherein the biological substance comprises a DNA array, an oligo array, a biological tissue, or an array of cells (Col 7 Line 60-67 “ The object comprises intracellular structures of one or more cells. The intracellular structures comprise one or more microtubules and/or organelles of the one or more cells. The organelles comprise one or more subcellular structures, e.g., nucleus, mitochondria, endoplasmic reticulum, Golgi apparatus, vesicles, vacuoles, etc. of the one or more cells. In some embodiments, the subcellular structures comprise one or more DNA nanorulers”).
Regarding claim 20, Yao teaches A system comprising: one or more data processors; and a non-transitory computer readable medium storing instructions which, when executed on the one or mor data processors, cause the one or more data processors to (Col 7 Line 10-13 “The memory 104 includes computer program code (not shown in FIG. 1) for execution by the at least one processor 102 to perform steps in accordance with a method of image processing”): generate a set of synthetic microscopy images using seed intensities and features extracted or known from a set of real microscopy images, each synthetic microscopy image representing a plurality of objects of a biological substance (Col 7 Line 37-38 “reconstruct super resolution images based on various low resolution images”, Col 8 Line 25-27 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image”, Col 8 Line 49-52 “input the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”, Col 7 Line 60-65 “The object comprises intracellular structures of one or more cells. The intracellular structures comprise one or more microtubules and/or organelles of the one or more cells. The organelles comprise one or more subcellular structures, e.g., nucleus, mitochondria, endoplasmic reticulum, Golgi apparatus, vesicles, vacuoles, etc. of the one or more cells.”); generate a set of seed images from the seed intensities, wherein each seed image corresponds to a synthetic microscopy image of the set of synthetic microscopy images, and wherein each pixel in the seed image represents a signal volume for an object of the plurality of objects (Col 8 Line 25-29 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image, wherein an edge intensity at each subpixel (x.sub.c, y.sub.c) is defined by an extent θ to which surrounding intensity gradients converge to the subpixel”, as stated in [0066] of the applicants specification, intensity value may be a fluorescence signal); and generate a trained machine learning model by training a machine learning model (Col 8 Line 53-54 “the neural network is trained using a multicomponent loss function”, Col 10 Line 58-64 “the frequency loss function 420 receives the reconstructed super resolution image 308 and the ground truth image 310 at steps 330, 332 and compares frequency spectrum differences 424 of the reconstructed super resolution image 308 and the ground truth image 310 in a specified frequency region”).
Yao fails to explicitly teach training a machine learning model to generate intensity values for additional real microscopy images, but in related field of endeavor Ozcan 2 teaches training a machine learning model to generate intensity values for additional real microscopy images (Col 2 Line 15-19 “a linear approximation that relates the fluorescence intensity of an image to the dye concentration per tissue volume, using empirically determined constants that represent the mean spectral response of various dyes embedded in the tissue”, Col 9 Line 54-59 “outputs or generates a digitally stained or labelled output image 40. The digitally stained output image 40 has “staining” that has been digitally integrated into the stained output image 40 using the trained, deep neural network 10”, as stated in [0066] of the applicant’s specification, intensity value may be a fluorescence signal)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Yao to include training a machine learning model to generate intensity values for additional real microscopy images as taught by Ozcan 2. Doing so would generate virtually stained samples which appear just like samples that have undergone histochemical staining even though no such staining operation was conducted (Col 10 Line 4-7 “This digital or virtual staining of the tissue section sample 22 appears just like the tissue section sample 22 had undergone histochemical staining even though no such staining operation was conducted.”)
Claims 10-17 are rejected under 35 U.S.C. 103 as being unpatentable over Yao and Ozcan 2 as applied to claim 9 above, and further in view of Smith.
Regarding claim 10, Yao and Ozcan 2 teach the computer-program product of claim 9. Yao further teaches further including instructions configured to cause the one or more data processors to: input a real microscopy image into the trained machine learning model, the real microscopy image depicting an additional plurality of objects (Col 7 Line 51-52 “the low resolution image is a fluorescence microscopy image of the object”, Col 8 Line 49-52 “input the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”, Col 7 Line 60-65 “The object comprises intracellular structures of one or more cells. The intracellular structures comprise one or more microtubules and/or organelles of the one or more cells. The organelles comprise one or more subcellular structures, e.g., nucleus, mitochondria, endoplasmic reticulum, Golgi apparatus, vesicles, vacuoles, etc. of the one or more cells.”); receive, (Col 8 Line 25-29 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image, wherein an edge intensity at each subpixel (x.sub.c, y.sub.c) is defined by an extent θ to which surrounding intensity gradients converge to the subpixel”); and generate a simulated microscopy image corresponding to the real microscopy image based on the output (Col 8 Line 49-52 “input the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”).
In related field of endeavor, Smith teaches a trained machine learning model generating an output representing a seed intensity for each object of the additional plurality of objects in the real microscopy image (Par 32 “the classification system may generate a heat map for an image indicating which regions of the image include diverse types of particles”, Par 120 “a network or machine learning algorithm 1802 (which may also be referred to as a hypothesis), may be trained and used for identifying and classifying or detecting particles in an image”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Yao and Ozcan 2 to include a machine learning model generating an output representing a seed intensity for each object of the additional plurality of objects in the real microscopy image as taught by Smith. Doing so would increase the accuracy of particle and material classification in a microscopy slide images (Par 28 “improved classification processes that resolve these issues and increase the likelihood that a machine learning algorithm quickly and accurately classifies particles and other materials within a slide image”).
Regarding claim 11, Yao as modified by Ozcan 2 and Smith teach the computer-program product of claim 10. Yao further teaches further including instructions configured to cause the one or more data processors to: determine a difference between the real microscopy image and the simulated microscopy image (Claim 2 “inputting the reconstructed super resolution image of the object and a ground true image of the object to the multicomponent loss function to quantify differences between the reconstructed super resolution image and the ground true image”).
Regarding claim 12, Yao as modified by Ozcan 2 and Smith teach the computer-program product of claim 11. Yao further teaches further including instructions configured to cause the one or more data processors to: in response to determining the difference, determine a set of features to use to generate subsequent simulated microscopy images (Claim 3 “inputting the quantified differences between the reconstructed super resolution image and the ground true image to the neural network for subsequent training to optimize the neural network”).
Regarding claim 13, Yao as modified by Ozcan 2 and Smith teaches the computer-program product of claim 11, wherein the trained machine learning model is a first trained machine learning model. Ozcan 2 further teaches wherein the computer-program product further includes instructions configured to cause the one or more data processors to: in response to determining the difference, input the simulated microscopy image into a second trained machine learning model (Col 17 Line 18-20 “the virtually de-stained microscopic image 86 of the sample 12 (i.e., the sample 12 that is to be tested or imaged) is input to the second trained, deep neural network 10″”); and receive, from the second trained machine learning model, a result of an adjusted simulated microscopy image corresponding to the real microscopy image (Col 17 Line 20-25 “The second trained, deep neural network 10″ then outputs a stained or labeled microscopic image 92 of the sample 12 that is substantially equivalent to a corresponding image of the same sample 12 obtained with a different chemical stain.”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Yao, Ozcan 2, and Smith to include in response to determining the difference, input the simulated microscopy image into a second trained machine learning model and receive, from the second trained machine learning model, a result of an adjusted simulated microscopy image corresponding to the real microscopy image as taught by Ozcan 2. Doing so would provide a model for virtually destaining and restaining an image (Col 6 Line 66 – Col 7 Line 4 “to virtually de-stain images obtained from a microscope (e.g., stained images to de-stained images). Also illustrated is an optional machine learning-based operation to re-stain the image with a different chemical stain from that obtained with the microscope.”)
Regarding claim 14, Yao as modified by Ozcan 2 and Smith teaches the computer-program product of claim 9. Yao further teaches further including instructions configured to cause the one or more data processors to: generate the set of synthetic microscopy images by: receiving the set of real microscopy images representing the plurality of objects of the biological substance (Claim 1 “receiving a low resolution image of an object, wherein the low resolution image is a fluorescence microscopy image of the object”, Col 7 Line 24-26 “the at least one processor 102 of the device 100 can receive low resolution images from the memory 104 or storage units (not shown) in the device 100.”), (Claim 1 “generating an edge map of the low resolution image by an edge extractor, wherein the generating of the edge map comprises extracting the edge map by the edge extractor on a subpixel level based on a radial symmetry of fluorophore in the fluorescence microscopy image, an edge intensity at each subpixel being defined by an extent to which surrounding intensity gradients converge to the subpixel, the edge intensity being weighted by a pixel intensity of the subpixel”, Col 8 Line 45-47 “By determining edges of intracellular structures in the fluorescence microscopy image, features of the intracellular structures are extracted in the edge map 502”); and generating, based on the plurality of features, one or more synthetic microscopy images representing the plurality of objects of the biological substance (Claim 1 “inputting the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”).
Yao fails to explicitly teach each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images. In related field of endeavor Smith teaches each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images (Par 162 “receive a non-synthetic microscopy slide image comprising an object bounding box around a plurality of object pixels; crop out the plurality of object pixels within the object bounding box”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Yao to include each object of the plurality of objects corresponding to one or more pixels of the set of real microscopy images as taught by Smith. Doing so would quickly and accurately classify materials within a slide image (Par 28 “improved classification processes that resolve these issues and increase the likelihood that a machine learning algorithm quickly and accurately classifies particles and other materials within a slide image”)
Regarding claim 15, Yao as modified by Ozcan 2 and Smith teaches the computer-program product of claim 14. Yao further teaches further including instructions configured to cause the one or more data processors to: performing a simulation of sequencing biochemistry for the biological substance, wherein the simulation is configured to receive the plurality of features of a real microscopy image of the set of real microscopy images as an input (Col 17 Line 56-60 “FIGS. 10A to 10D depict embodiments where SR images reconstructed based on the SFSRM image processing approach of the present application are utilised for in situ genome sequencing in an interphase human fibroblast nucleus.”, Col 18 Line 6-13 “In this regard, FIG. 10A shows a fluorescence in situ hybridization (FISH) labeling of the genome of the interphase human fibroblast nucleus. FIG. 10B depicts a widefield LR image of the nucleus. FIG. 10C depicts a SR image of the nucleus reconstructed based on the SFSRM image processing approach of the present application. FIG. 10D depicts a table of chromosome sequences by different probe combinations.”); and determining, based on the input, a seed intensity for each object of the plurality of objects of the real microscopy image, wherein the seed intensity corresponds to a signal volume for the object (Col 8 Line 24-29 “the edge extractor (not shown) of the device 100 extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image, wherein an edge intensity at each subpixel (x.sub.c, y.sub.c) is defined by an extent θ to which surrounding intensity gradients converge to the subpixel”, as stated in [0066] of the applicants specification, intensity value may be a fluorescence signal).
Regarding claim 16, Yao as modified by Ozcan 2 and Smith teaches the computer-program product of claim 15. Yao further teaches further including instructions configured to cause the one or more data processors to: generate a seed image based on the seed intensity for each object of the plurality of objects, wherein each pixel in the seed image represents the signal volume for the object (Col 8 Line 25-29 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image, wherein an edge intensity at each subpixel (x.sub.c, y.sub.c) is defined by an extent θ to which surrounding intensity gradients converge to the subpixel”).
Claims 17 is rejected under 35 U.S.C. 103 as being unpatentable over Yao, Ozcan 2, and Smith as applied to claim 15 above, and further in view of Ozcan.
Regarding claim 17, Yao as modified by Ozcan 2 and Smith teaches the computer-program product of claim 15. Yao further teaches wherein generating the one or more synthetic microscopy images comprises: generating a point spread function for the plurality of objects of the real microscopy image based on the plurality of features (Col 8 Line 45-48 “By determining edges of intracellular structures in the fluorescence microscopy image, features of the intracellular structures are extracted in the edge map 502”, Col 9 Line 52-54 “the LR image is degraded from the ground truth image 704 by 200 nm point-spread-function blurring and then downsampling for 10 times”, Col 9 Line 56-57 “Thereafter, an edge map 304 of the low resolution image 302 is generated 322 by an edge extractor (not shown)”); and generating a synthetic image of the one or more synthetic microscopy images based on the signal distribution over the plurality of pixels and the plurality of features (Col 8 Line 25-27 “extracts the edge map 502 on a subpixel level based on a radial symmetry 504 of fluorophore in the fluorescence microscopy image”, Col 8 Line 49-52 “input the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object”).
Yao fails to explicitly teach determining a signal distribution over a plurality of pixels by aggregating the point spread function and the seed intensity for each object. In related field of endeavor, Ozcan teaches determining a signal distribution over a plurality of pixels by aggregating the point spread function and the seed intensity for each object (Col 8 Line 50-54 “FIG. 24D is a graph showing the signal (count) as a function of FWHM of PSF (nm) for the network input, network output, and ground truth”, where PSF is point spread function, Col 27 Line 50-56 “Despite the fact that the fluorescent signal from 20 nm beads is rather weak, the deep neural network 10 (trained only with BPAEC samples) successfully picked up the signal from individual nano-beads and blindly improved the resolution to match that of the ground truth 50, as shown in the PSF comparison reported in FIG. 24D.” as stated in [0066] of the applicant’s specification, intensity value may be a fluorescence signal)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Yao and Smith to include determining a signal distribution over a plurality of pixels by aggregating the point spread function and the seed intensity for each object as taught by Ozcan. Doing so would enhance the output image and show uniform focusing with improved resolution (Col 27 Line 63-67 “The deep network results, on the other hand, once again demonstrate the enhanced DOF of the network output image 40, showing uniform focusing with improved resolution at the network output image 40.”)
Claims 19 is rejected under 35 U.S.C. 103 as being unpatentable over Yao and Ozcan 2 as applied to claim 9 above, and further in view of Zhang.
Regarding claim 19, Yao as modified by Ozcan 2 teaches the computer-program product of claim 9, but fails to explicitly teach wherein the biological substance comprises a DNA array and the plurality of objects comprises a plurality of DNA nanoballs. In related field of endeavor, Zhang teaches wherein the biological substance comprises a DNA array and the plurality of objects comprises a plurality of DNA nanoballs (Zhang Fig. 1, included below, depicts imaging an array of DNA nanoballs and inputting the images to generators in a neural network, Introduction Line 1 “Imaging of self-assembling DNA nanoball (DNB) array”)
PNG
media_image3.png
320
692
media_image3.png
Greyscale
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Yao and Smith to include wherein the biological substance comprises a DNA array and the plurality of objects comprises a plurality of DNA nanoballs as taught by Zhang. Doing so would improve the throughput and measurement speed of DNB samples in the gene sequencing process (Page 2, Paragraph 1 “the significant applications value of the proposed technique in improving the throughput and measurement speed of a single DNB array sample in the gene sequencing process”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN PATRICK GOCO whose telephone number is (571)272-5872. The examiner can normally be reached M-Th, 7:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.P.G./ Examiner, Art Unit 2611
/KEE M TUNG/ Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights