DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
On page 8, line 7, “as well as well as the light frequencies” should read “as well as the light frequencies”.
Appropriate correction is required.
Claim Objections
Claim 1 objected to because of the following informalities:
In Claim 1, line 6, the limitation of “the presence” lacks antecedent basis.
In Claim 10, line 21, the limitation of “the accuracy” lacks antecedent basis.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 14, 22, and 39 are rejected under 35 U.S.C. 103 as being unpatentable over Hennig et al. (US Pub No 2017/0052106), hereinafter Hennig, in view of Wu et al. (US Pub No 2024/0193734), hereinafter Wu.
As to Claim 1, Hennig teaches method of classifying cytometric image data (see paragraph [0005], “a computer-implemented method for the label-free classification of cells using image cytometry is provided”), the method comprising,
receiving unclassified cytometric image data (see paragraph [0005], “receives as an input one or more images of a cell obtained from an image cytometer”),
wherein the cytometric image data comprises a plurality of images corresponding to image channels (see paragraph [0090], “As disclosed herein, the method uses the images acquired from an imaging flow cytometer, such as the brightfield and/or darkfield images, to assign cells to certain cell classes, such as cells in various phases of the cell cycle or by cell type”, where brightfield images and darkfield images correspond to different image channels),
and applying a model to the modulated cytometric image data to classify the cytometric image data (see paragraph [0017], “Using machine learning based on the features extracted from the brightfield images only (neglecting the stains) the cells could be classified in particular phases of the cell cycle stage correctly with 89.3% accuracy”),
wherein the model is trained to estimate the presence of a particle belonging to a first category of particles in the cytometric image data (see paragraph [0037], “Using the extracted features, a cell shown in the one or more images is classified using a classifier that has been trained on a control sample to recognize and classify the cells in the images based on the features and values derived therefrom”, and see paragraph [0091], “In the tests disclosed herein, a label-free way was developed to measure important cell cycle phenotypes including a continuous property (a cell's DNA content, from which G1, S and G2 phases can be estimated) and discrete phenotypes (whether a cell was in each phase of mitosis: prophase, anaphase, metaphase, and telophase)”).
Hennig teaches that cytometric image data may be modulated before a model is applied (see paragraph [0060], “In block 210, the raw image can be optionally subject to preprocessing, for example to remove artifacts or skip images that do not contain usable images for subsequent analysis”). However, Hennig fails to explicitly teach that the images are iteratively modulated. Wu teaches an image enhancement method (see abstract) in which images can be iteratively filtered (see paragraph [0006], “performing iterative detail enhancement processing on the original image by utilizing at least one filter function comprising iteratively-updated filter parameters”). Wu is combinable with Hennig since both are from the analogous field of image analysis. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the iterative image enhancement taught by Wu with the method of classifying cytometric image data taught by Hennig. The motivation for doing so would be to improve image enhancement by performing image specific enhancements. Wu teaches in paragraph [0004], “In the current image quality enhancement solution, the parameters used for enhancing the image quality of any input image are fixed. However, owing to the differences in contrast, saturation, color, etc., between different images, if the image qualities of different images are enhanced with the same set of parameters, it is highly possible to lead to inferior image quality enhancement, and it is impossible to adaptively adjust the parameters and enhance the image qualities for different input images.” Thus, it would have been obvious to combine the iterative image enhancement taught by Wu with the method taught by Hennig in order to obtain the invention as claimed in Claim 1.
As to Claim 2, Hennig in view of Wu teaches the method according to claim 1, wherein applying the model to cytometric image data to classify the cytometric image data comprises obtaining an estimate of the presence of a particle belonging to the first category of particles in the cytometric image data (see Hennig, Fig 6C-6H, where classifications estimates are shown in a bar graph).
As to Claim 14, Hennig in view of Wu wherein teaches modulating a plurality of aspects of a first image of the plurality of images of the cytometric image data (see Hennig, paragraph [0060], “In block 210, the raw image can be optionally subject to preprocessing, for example to remove artifacts”, and see paragraph [0125], “Next, the brightfield images were segmented without using any stains, but by smoothing the images (CellProfiler module ‘Smooth’ with a Gaussian Filter) followed by an edge detection (CellProfiler module ‘EnhanceEdges’ with Sobel edge-finding) and by applying a threshold (CellProfiler module ‘ApplyThreshold’ with the MCT thresholding method and binary output))”, where smoothing and edge enhancing are examples of modulations),
and applying the model to the modulated image data (see Hennig, paragraph [0037]).
As to Claim 22, Hennig in view of Wu teaches identifying modulated image data corresponding to an estimate of the presence of a particle belonging to a first category of particles in the cytometric image data (see Hennig, paragraph [0125], “Next, the brightfield images were segmented without using any stains, but by smoothing the images (CellProfiler module ‘Smooth’ with a Gaussian Filter) followed by an edge detection (CellProfiler module ‘EnhanceEdges’ with Sobel edge-finding) and by applying a threshold (CellProfiler module ‘ApplyThreshold’ with the MCT thresholding method and binary output). The obtained objects were closed (CellProfiler module ‘Morph’ with the ‘close’ operation) and use them to identify the cells on the grid sites (CellProfiler module ‘IdentifyPrimaryObjects’)”. Thus, the image was modulated multiple times, and then the presence of cells was later identified.
As to Claim 39, Hennig in view of Wu teaches a that is trained to estimate the presence of a particle belonging to a first category of particles in the cytometric image data using one or more of: an unsupervised learning technique, a semi- supervised learning technique, a supervised learning technique or a round robin training technique (see Hennig, paragraph [0013], “These features are then input for supervised machine learning, namely classification and regression”).
Claims 3-11 and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Hennig et al. (US Pub No 2017/0052106), hereinafter Hennig, in view of Wu et al. (US Pub No 2024/0193734), hereinafter Wu, and further in view of Kumar (US Pub No 2018/0247195), hereinafter Kumar.
As to Claim 3, Hennig in view of Wu fails to explicitly teach obtaining a confidence score associated with the estimate of the presence of a particle belonging to the first category of particles in the cytometric image data. However, Kumar teaches an artificial neural network which classifies biological data (see abstract) and obtains a confidence score (see paragraph [0213], “If a first neural network analyzing a first test data sample has an output of, for example, confidence of non-tumor-bearing=40% and confidence of cancer=60%, and a second neural network, analyzing the same test data sample has an output of confidence of non-tumor-bearing=99% and confidence of cancer=1%, the second network is more likely to be correct, because it is detecting a feature or features in that particular test data sample that it generalizes better on than the first network”). Kumar is combinable with Hennig and Wu since all three are from the analogous field of image analysis. Thus, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention to combine the confidence score taught by Kumar with the teachings of Hennig and Wu. The motivation for doing so would be to be to allow for the user to determine how reliable the output of the neural network is. Kumar teaches in paragraph [0213] that a higher confidence is more likely to be correct. Thus, it would have been obvious to combine the confidence score taught by Kumar with the teachings of Hennig and Wu in order to obtain the invention as claimed in Claim 3.
As to Claim 4, Hennig in view of Wu teaches that the model can be applied multiple times (see Hennig, paragraph [0158], “To prevent overfitting the data and to fix the stopping criterion for the applied boosting algorithms, a five-fold internal cross-validation was performed. To this end, we split up the training set into an internal-training (consisting of 80% of the cells in the training set) and an internal-validation (20% of the cells in the training set) set. The algorithm was trained on the internal-training set with up to 6,000 decision trees. The DNA content/cell cycle phase of the inner-validation set and evaluate the quality of the prediction as a function of the used amount of decision trees was predicted”). However, Hennig fails to explicitly teach obtaining a plurality of estimates and confidences scores for each iteration. However, Kumar teaches a model that can produce outputs corresponding to each iteration (see paragraph [0095], “The difference between the target and the set of input samples is calculated, and the ANN is modified using the back-propagation algorithm to cause the output to more closely approximate the desired target value. After a large number of training iterations, the ANN output will closely match the desired target for each sample in the input training set”). Kumar further teaches that multiple confidence scores can be output with the estimates (see paragraph [0106], “The input layers can be presented to a summing layer 609 and then to the output layer 610 of the artificial neural network to provide confidences in the classification of the sample, as normal 611 or as having cancer 612”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the confidence scores taught by Kumar with the teachings of Hennig and Wu. The motivation for doing so would be to allow the user to easily determine the reliability of the output of each iteration of the neural network. Kumar teaches in paragraph [0213], “The higher value represents the classification result, and its value indicates how “confident” the neural network is in the result. If a first neural network analyzing a first test data sample has an output of, for example, confidence of non-tumor-bearing=40% and confidence of cancer=60%, and a second neural network, analyzing the same test data sample has an output of confidence of non-tumor-bearing=99% and confidence of cancer=1%, the second network is more likely to be correct”. Thus, it would have been obvious to combine the confidences taught by Kumar with the teachings of Hennig and Wu in order to obtain the invention as claimed in Claim 4.
As to Claim 5, Hennig in view of Wu and Kumar teaches that a higher confidence score corresponds to a greater likelihood that the estimate of the presence of a particle belonging to the first category of particles in the cytometric image data is accurate (see paragraph [0213] of Kumar, “If a first neural network analyzing a first test data sample has an output of, for example, confidence of non-tumor-bearing=40% and confidence of cancer=60%, and a second neural network, analyzing the same test data sample has an output of confidence of non-tumor-bearing=99% and confidence of cancer=1%, the second network is more likely to be correct, because it is detecting a feature or features in that particular test data sample that it generalizes better on than the first network.”).
The reasons to combine are similar to those mentioned in Claim 3 for reliability purposes and also apply to the rejection of the claims listed below that use Kumar to teach confidence scores.
As to Claim 6, Hennig in view of Wu and Kumar teaches wherein a lower confidence score corresponds to a lower likelihood that the estimate of the presence of a particle belonging to the first category of particles in the cytometric image data is accurate (see paragraph [0213], “If a first neural network analyzing a first test data sample has an output of, for example, confidence of non-tumor-bearing=40% and confidence of cancer=60%, and a second neural network, analyzing the same test data sample has an output of confidence of non-tumor-bearing=99% and confidence of cancer=1%, the second network is more likely to be correct, because it is detecting a feature or features in that particular test data sample that it generalizes better on than the first network”). Thus, it is understood that the lower confidence score produced by the first neural network corresponds to a lower likelihood that the estimate is accurate.
As to Claim 7, Hennig in view of Wu and Kumar teaches the method according to claim 3, wherein iteratively modulating aspects of at least one image and applying the model to the modulated cytometric image data comprises iteratively
applying an image modulation operation to the at least one image of the plurality of images of the cytometric image data (see Hennig paragraph [0060] and Wu paragraph [0006]),
applying the model to the unclassified cytometric image data comprising at least one modulated image (see Hennig paragraph [0017]),
obtaining a confidence score as a result of applying the model (see Kumar, paragraph [0213]),
and comparing the confidence score to a reference confidence score (see Kumar paragraph [0213], “If a first neural network analyzing a first test data sample has an output of, for example, confidence of non-tumor-bearing=40% and confidence of cancer=60%, and a second neural network, analyzing the same test data sample has an output of confidence of non-tumor-bearing=99% and confidence of cancer=1%, the second network is more likely to be correct, because it is detecting a feature or features in that particular test data sample that it generalizes better on than the first network”, where the confidence score of one neural network is compared to a reference score generated by another neural network).
As to Claim 8, Hennig in view of Kumar fails to teach wherein the reference confidence score comprises a confidence score corresponding to applying the model to the unclassified cytometric image data without any image modulation. Kumar teaches comparing two confidence scores generated by comparing the outputs of two neural networks (see paragraph [0213]). However, Wu teaches an image quality evaluation score that can be determined by comparing an input, unenhanced image to an enhanced image. (see Wu, paragraph [0006], “and continuing to perform the detail enhancement processing on the input image by utilizing at least one filter function comprising the filter parameters of the (i+1)th iteration, until a difference between the quality evaluation score of the input image and a quality evaluation score of a detail-enhanced image of the (i+1)th iteration satisfies the iteration condition”, where the input image has no image enhancements or modulations done). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the confidence comparison taught by Kumar with the reference, unenhanced image comparison taught by Wu. The motivation for doing so would be to improve image enhancement by performing image specific enhancements. Wu teaches in paragraph [0004], “In the current image quality enhancement solution, the parameters used for enhancing the image quality of any input image are fixed. However, owing to the differences in contrast, saturation, color, etc., between different images, if the image qualities of different images are enhanced with the same set of parameters, it is highly possible to lead to inferior image quality enhancement, and it is impossible to adaptively adjust the parameters and enhance the image qualities for different input images.” Thus, it would have been obvious to combine the iterative image enhancement taught by Wu with the teachings taught by Hennig and Kunar in order to obtain the invention as claimed in Claim 8.
As to Claim 9, Hennig in view of Wu and Kumar teaches determining a subsequent image modulation operation based on the results of comparing the confidence score to the reference confidence score (see Wu, paragraph [0006], “and continuing to perform the detail enhancement processing on the input image by utilizing at least one filter function comprising the filter parameters of the (i+1) iteration, until a difference between the quality evaluation score of the input image and a quality evaluation score of a detail-enhanced image of the (i+1) iteration satisfies the iteration condition”, where image enhancement continues until the difference between reference score and the current score is satisfies a condition).
As to Claim 10, Hennig in view of Wu and Kumar teaches repeatedly iterating to improve the accuracy of the estimate of the presence of a particle belonging to the first category of particles in the cytometric image data (see Kumar, paragraph [0095], “The difference between the target and the set of input samples is calculated, and the ANN is modified using the back-propagation algorithm to cause the output to more closely approximate the desired target value. After a large number of training iterations, the ANN output will closely match the desired target for each sample in the input training set”).
As to Claim 11, Hennig in view of Wu and Kumar teaches wherein iteratively modulating aspects of at least one image and applying the model to the modulated image data comprises repeatedly iterating to improve the confidence score associated with each estimate of the presence of a particle belonging to the first category of particles in the cytometric image data. Kumar teaches in paragraph [0017], “In some embodiments, the method improves a performance characteristic of the artificial neural network in detecting a condition in the biological sample”, where a confidence score is a performance characteristic of a neural network. Furthermore, Kumar teaches iterating to improve the model (see paragraph [0095], “After a large number of training iterations, the ANN output will closely match the desired target for each sample in the input training set”).
As to Claim 29, Hennig in view of Wu and Kumar teaches further updating the model based on the results of iteratively applying the model to the modulated image data (see Kumar paragraph [0020], “updating the artificial neural network by iteratively performing steps (a) and (b) with a biological data sample from at least one additional biological sample”).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Hennig et al. (US Pub No 20170052106), hereinafter Hennig, in view of Wu et al. (US Pub No 2024/0193734), hereinafter Wu, and further in view of Kumar (US Pub No 2018/0247195), and further in view of Ota et al. (US Pub No 2018/0327699 ), hereinafter Ota.
As to Claim 12, Hennig in view of Wu and Kumar teaches wherein iteratively modulating aspects of at least one image and applying the model to the modulated image data comprises modulating aspects of at least one image to improve the confidence score associated with each estimate of the presence of a particle belonging to the first category of particles in the cytometric image data (see Kumar paragraph [0017], “In some embodiments, the method improves a performance characteristic of the artificial neural network in detecting a condition in the biological sample”, where a confidence score is a performance characteristic of a neural network, and see paragraph [0095], “After a large number of training iterations, the ANN output will closely match the desired target for each sample in the input training set”). However, Kumar fails to explicitly teach that the confidence score is optimized. However, Ota teaches that a model that classifies cells (see paragraph [0091]) that may by optimized (see paragraph [0090], “The analysis device preferably optimizes a classification algorithm of the analysis unit 11 using machine learning”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the optimization taught by Ota with the confidence scores taught by Kumar and the teachings of Hennig and Wu. The motivation for doing so would be to improve the detection of cells. Ota teaches in paragraph [0091], “By performing machine learning, it becomes possible to detect the presence or absence of specific cells extremely efficiently and rapidly.” Thus, it would have been obvious to combine the optimization taught by Ota with the teachings of Hennig, Wu, and Kumar in order to obtain the invention as claimed in Claim 12.
Claims 13 is rejected under 35 U.S.C. 103 as being unpatentable over Hennig et al. (US Pub No 20170052106), in view of Wu et al. (US Pub No 2024/0193734), further in view of Kumar (US Pub No 2018/0247195), further in view of Ota et al. (US Pub No 2018/0327699 ), and further in view of Van Leeuwen et al. of (US Pub No 2021/0166076), hereinafter Van Leeuwen.
As to Claim 13, Hennig in view of Wu, Kumar, and Ota fails to teach he method wherein optimizing the confidence score associated with each estimate of the presence of a particle belonging to the first category of particles in the cytometric image data comprises finding a local maximum of the confidence score. However, Van Leeuwen teaches a method of classifying microscopic image data (see abstract) that includes determining a probability data that indicates a probability of the presence of a particle belonging to the first class (see paragraph [0006], “For one or more of the object classes and for each of the pixel groups, the probabilistic group classification data are indicative of a probability that the respective group shows at least a portion of an object of the respective object class”, where probability is similar to the confidence). Furthermore, Van Leeuwen teaches a local maximum of the probabilities can be determined (see paragraph [0064], “A local maximum of the probability values may be determined”). Van Leeuwen is combinable with Hennig, Wu, Kumar, and Ota since all four are from the analogous field of image analysis. Thus, it would have been obvious to one of ordinary skill in the art before the effective fling date of the claimed invention to combine the local maximum taught by Van Leeuwen with the teachings of Hennig, Wu, Kumar, and Ota, the motivation for doing so would be to better identify pixel clusters that are associated with the presence of a particle. Van Leeuwen teaches in paragraph [0065], “Additionally or alternatively, further algorithms may be applied for forming the pixel groups, which may include but are not limited to: active contour operations, watershed operations, level set operations, and maximally stable extremal regions. These algorithms may use the identified local maxima in probability.” Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Van Leeuwen with the teachings of Hennig, Wu, Kumar, and Ota in order to obtain the invention as claimed in Claim 13.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Hennig et al. (US Pub No 20170052106), in view of Wu et al. (US Pub No 2024/0193734), and further in view of Ota et al. (US Pub No 2018/0327699 ).
As to Claim 16, Hennig in view of Wu fails to teach wherein iteratively modulating aspects of at least one image and applying the model to the modulated image data comprises applying a simulated annealing technique. However, Ota teaches a model that can analyze cells (see paragraph [0090]), and can use the annealing technique (see paragraph [0121], “For example, during cell imaging based on GMI, the optical structure can be optimized with a well-known machine learning and optimization algorithm. Well-known machine learning and optimization algorithms include evolutionary algorithms and simulated annealing”). Ota is combinable with Hennig and Wu since all three are from the analogous field of image analysis. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the simulated annealing taught by Ota with the teachings of Hennig and Wu. The motivation for doing so would be to optimize cell imaging as taught by Ota in paragraph [0090]. Thus, it would have been obvious to combine the simulated annealing method taught by Ota with the teachings of Hennig and Wu in order to obtain the invention as claimed in Claim 16.
Claim 43 is rejected under 35 U.S.C. 103 as being unpatentable over Hennig et al. (US Pub No 20170052106), in view of Wu et al. (US Pub No 2024/0193734), and further in view of Van Leeuwen et al. of (US Pub No 2021/0166076).
As to Claim 43, Hennig in view of Wu fails to explicitly teach normalizing aspects of the cytometric image data. However, Van Leeuwen teaches a method of classifying microscopic image data teaches includes normalizing image data by applying a mean filter (see paragraph [0102]), “An ideal mean filter is to replace each pixel in the image with a mean of each pixel and its surrounding pixels (for example, 4 neighboring pixels or 8 neighboring pixels)”, where the examiner has interpreted ‘normalizing’ to mean adjusting the intensity of pixels in an image). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the normalization taught by Van Leeuwen with the teachings of Henning and Wu. The motivation for doing so would be to reduce any present in the raw image data. Van Leeuwen teaches in paragraph [0102], “The mean filter is a typical linear filter algorithm. In consideration of a frequency domain, the mean filter is a low-pass filter, to remove high-frequency signals, thereby assisting in eliminating sharp noise from the image”. Thus, it would have been obvious to combine the normalization taught by Van Leeuwen with the teachings of Hennig and Wu in order to obtain the invention as claimed in Claim 43.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOUMYA THOMAS whose telephone number is (571)272-8639. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.T./ Examiner, Art Unit 2664
/JENNIFER MEHMOOD/ Supervisory Patent Examiner, Art Unit 2664