Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because several figures (e.g., Fig. 1, Fig. 4) contain figure keys which are based on color, but the current figures do not provide any color and therefore the reader cannot correctly interpret the figure keys. Appropriate correction is required such that any reader can clearly visualize and understand the attached figures.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is
directed to an abstract idea without reciting elements that amount to significant more than
the abstract idea. The rationale for this rejection, under MPEP § 2106, for this finding is
explained below.
Step 1: Under step 1, the claims are analyzed to determine if the claim is directed to a
process, machine, article of manufacture, or composition of matter. For the claims in question,
claims 1-10 and claim 20 are directed towards a method, and claims 11-19 are directed towards a machine.
Step 2A, Prong 1: Under step 2A, prong 1, the claims are evaluated to determine if the
claim recites a judicial exception, which includes the laws of nature, physical phenomena, or an
abstract idea.
For independent claim 1 (and corresponding independent claim 11), the limitations relating to registering a plurality of pairs of images, and consequently receiving an annotation identifying a functional unit on the registered image are directed towards a mental process. Given an anatomical image and a molecular image, an individual can reasonably align (i.e., register) the two images such that they spatially correspond to each other, and furthermore can utilize the aligned images to then annotate (i.e., identify and label either mentally or utilizing paper and pen) the image for a functional unit present on the image (i.e., an individual can label the cell nucleus or cell wall in an image containing cells).
For independent claim 20, the limitation related to segmenting a morphological feature in an anatomical image is directed towards a mental process. Similarly to the reasoning provided for claims 1 and 11, given an anatomical image an individual can produce a segmentation (i.e., generate an outline around a morphological feature) in the image either mentally or by utilizing paper and pencil.
Step 2A, Prong 2: Under step 2A, prong 2, the claims are evaluated to determine
whether the claim as a whole integrates the recited judicial exception into a practical application
of the exception (see MPEP 2106.04(d)). The examiner notes that MPEP 2106.05(a)-(c) and (e)
generally concern limitations that are indicative of integration, whereas 2106.05(f)-(h) generally
concern limitations that are not indicative of integration.
In regards to claims 1-19, the limitations as described above are directed towards a mental process. The additional limitations of the independent claims relating to receiving images and producing a dataset to train a model are considered extra-solution activity and/or are broadly recited and generally linked to applying machine learning models to images, and do not constitute integration into a practical application or significantly more (see MPEP 2106.05(g), (h)). With regards to the dependent claims, the limitations are broadly recited and either further disclose mental processes (i.e., utilizing a corrective model to generate annotations) and/or are generally linked to applying machine learning models to images and do not constitute integration into a practical application or significantly more.
In regards to claim 20, the limitations as described above are directed towards a mental process. The additional limitations of deploying a model, receiving an image and inputting the image into the model are considered extra-solution activity and/or are broadly recited and generally linked to applying machine learning models to images, and do not constitute integration into a practical application or significantly more.
The examiner emphasizes MPEP 2106.05(a), which states that a limitation is indicative of integration into a practical application if the limitation identifies a manner in which an improvement is explicitly and specifically achieved and recited in the claims. The current claim language all are recited at a high level of generality which do not serve to integrate the limitations in view of MPEP 2106.05(f), and furthermore nothing precludes the current limitations from being interpreted under the mental processes grouping.
Step 2B: Under step 2B, the claims are evaluated as a whole to determine if it amounts to
significantly more than the recited exception (i.e., whether any additional element, or
combination of additional elements, adds an inventive concept to the claim). The considerations
of step 2A, prong 2 and step 2B overlap, but differ in that 2B also requires considering the claim
as a whole/combination of limitations, and with reference to MPEP 2106.05(d) whether the
claims feature any “specific limitation(s) other than what is well - understood, routine,
conventional activity in the field” (WURC). The examiner asserts that, even when considered in
combination, the additional elements of claims 1-20 represent mere instruction to apply a mental
process (registering and annotating anatomical images with the aide of molecular images) at a high level of generality that is generally linked to the field of applying machine learning techniques for annotating/labeling images, and therefore does not provide a specifically recited inventive concept.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 7-10, and 17-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lee et al. (“Deep Learning of Histopathology Images at the Single Cell Level”, DOI: 10.3389/frai.2021.754641, Publication Year: 2021; hereinafter “Lee”).
Regarding Claim 1, Lee discloses a method comprising (see “An Overview of Generating Weakly Cell-Level Annotation Using IHC Stained Images”):
receiving a plurality of anatomical images (IHC-Based Cell-Level Annotation, Lee discloses obtaining H&E stained images.);
receiving a plurality of corresponding molecular images, wherein each of a plurality of pairs of corresponding anatomical and molecular images captures a respective biological specimen (IHC-Based Cell-Level Annotation, Lee discloses obtaining adjacent (i.e., respective) immunohistochemistry slides.);
registering each of the plurality of pairs of corresponding anatomical and molecular images (Image Registration and Normalization, Lee discloses performing image registration on the adjacent images.);
for each of the plurality of pairs of corresponding anatomical and molecular images, receiving at least one annotation on an anatomical image that is registered to its corresponding molecular image, wherein the at least one annotation identifies a functional unit of interest within the anatomical image (Cell Boundary Detection and IHC Intensity Level Acquisition, Lee discloses utilizing software to annotate the registered H&E images with the cell boundaries.); and
creating a dataset comprising a plurality of annotated anatomical images, wherein the dataset is used to train a machine learning model, wherein the machine learning model is configured for multi-class functional unit segmentation (Reformatting the Annotation Dataset Into Common Objects in Context challenge Format, Divide Images Into Patches/Train + Test Dataset Generation, Training Machine Learning Classifier, Lee discloses utilizing the labeled images to train a machine learning model to segment objects of various classes.).
Claim 11 is the system claim corresponding to claim 1, and is similarly rejected (the Examiner notes section “Divide Images Into Patches/Train + Test Dataset Generation” which notes the usage of GPUs to perform image processing. The Examiner asserts that the claimed GPUs are utilized in a computing device which includes the claimed processor and memory of the claimed system.).
Regarding Claim 7, Lee discloses the method of claim 1, wherein the plurality of anatomical images are histological stained images (IHC-Based Cell-Level Annotation, Lee discloses utilizing H&E stained images.).
Claim 17 is the system claim corresponding to claim 7, and is similarly rejected.
Regarding Claim 8, Lee discloses the method of claim 1, wherein the plurality of corresponding molecular images are immunofluorescence (IF) images (IHC-Based Cell-Level Annotation, Lee discloses utilizing immunohistochemistry stained images.).
Claim 18 is the system claim corresponding to claim 8, and is similarly rejected.
Regarding Claim 9, Lee discloses the method of claim 1, wherein the machine learning model is a deep learning model (Training Machine Learning Classifier, Lee discloses training a deep learning model.).
Regarding Claim 10, Lee discloses the method of claim 1, wherein the machine learning model is a convolutional neural network (Training Machine Learning Classifier, Lee discloses training a Mask R-CNN model.).
Regarding Claim 19, Lee discloses the system of claim 11, wherein the machine learning model is a deep learning model or a convolutional neural network (Training Machine Learning Classifier, Lee discloses training a deep learning model, which further includes convolutional neural networks such as a Mask R-CNN model.).
Regarding Claim 20, Lee discloses a method comprising:
deploying a trained machine learning model (Training Machine Learning Classifier, Evaluation of the Results, Lee discloses training a machine learning model and then testing the model on test data.);
receiving an anatomical image (Divide Images Into Patches/Train + Test Dataset Generation, Evaluation of the Results, Lee discloses obtaining a training dataset and a test dataset.);
inputting the anatomical image into the trained deployed machine learning model (Evaluation of the Results, Lee discloses evaluating the trained machine learning model on the test dataset.); and
segmenting, using the trained deployed machine learning model, a subvisual or supervisual morphological feature in the anatomical image (Evaluation of the Results, Visualization and Obtaining Biological Insights, Lee discloses classifying and segmenting cells in an image.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2 and 12 are rejected as being unpatentable over Lee in view of Komura et al. (“Restaining-based annotation for cancer histology segmentation to overcome annotation-related limitations among pathologists”, DOI: 10.1016/j.patter.2023.100688, Publication Year: 2023; hereinafter “Komura”) in view of Della Mea et al. (“Preliminary results from a crowdsourcing experiment in immunohistochemistry, DOI: 10.1186/1746-1596-9-S1-S6, Publication Year: 2014; hereinafter “Della Mea”).
Regarding Claim 2, Lee discloses the method of claim 1.
Lee does not explicitly disclose further comprising annotating, by a layperson, the plurality of anatomical images using the plurality of corresponding molecular images as a guide.
Komura discloses further comprising (Fig. 1, Dataset generation workflow, Komura discloses registering anatomical H&E stained images based on the corresponding molecular IHC stained images (i.e., using the IHC stained images as a guide). Additionally, the Examiner notes that the registration process leads to the generation of a segmentation mask.).
Lee and Komura are considered to be analogous to the claimed invention as they are in the same field of applying deep learning image processing methods to process anatomical images. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lee by incorporating the specific disclosure of Komura which utilizes molecular IHC stained images as a guide for registering and further segmenting. The motivation for this combination being the ability to visualize the general structure of the biological specimen based on the H&E stain while also being able to identify specific target regions in the image through the IHC staining.
Lee in view of Komura does not explicitly disclose annotating, by a lay person, the plurality of anatomical images (italicized for context).
Della Mea discloses annotating, by a lay person, the plurality of anatomical images (italicized for context) (Figs. 1-2, Della Mea discloses utilizing crowdsourcing (i.e., from a lay person) to aide in the annotation of regions in an anatomical image.)
Lee, Komura, and Della Mea are considered to be analogous to the claimed invention as they are in the same field of analyzing and labeling/annotating anatomical images. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lee in view of Komura such that the image guidance based on anatomical and molecular images, as taught by Lee in view of Komura, is provided as input and then further annotated based on the crowdsourcing techniques disclosed by Della Mea. The motivation for this combination being the ability to reduce the burden on manual annotations performed by experts while also increasing the sample size of annotations which can be obtained (see Background, Della Mea).
Claim 12 is the system claim corresponding to claim 2, and is similarly rejected.
Claims 3-6 and 13-16 are rejected as being unpatentable over Lee in view of Bouteldja et al. (“Deep Learning-Based Segmentation and Quantification in Experimental Kidney Histopathology”, DOI: 10.1681/ASN.2020050597, Publication Year: 2020; hereinafter “Bouteldja”).
Regarding Claim 3, Lee discloses the method of claim 1.
Lee does not explicitly disclose further comprising evaluating the at least one annotation using a corrective machine learning model.
Bouteldja discloses further comprising evaluating the at least one annotation using a corrective machine learning model (Data Quality and Quantity, Bouteldja discloses utilizing an initial segmentation network (i.e., a corrective machine learning model) to help generate pre-annotations.).
Lee and Bouteldja are considered to be analogous to the claimed invention as they are in the same field of utilizing deep learning methods analyzing and labeling/annotating anatomical images. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lee such that it incorporates Bouteldja’s initial segmentation network as a corrective machine learning model. The motivation for this combination being the ability to aide manual annotators and consequently help improve accuracy of the annotations being made.
Claim 13 is the system claim corresponding to claim 3, and is similarly rejected.
Regarding Claim 4, Lee in view of Bouteldja teach the method of claim 3, wherein evaluating the at least one annotation using the corrective machine learning model comprises:
providing an unannotated anatomical image into the corrective machine learning model (Data Quality and Quantity, Bouteldja discloses inputting an unannotated image into an initial segmentation network.);
receiving, from the corrective machine learning model, a corrected annotated anatomical image (Data Quality and Quantity, Bouteldja discloses obtaining a prediction from the initial segmentation network as a “preannotation”.); and
comparing the anatomical image including the at least one annotation to the corrected annotated anatomical image (Data Quality and Quantity, Bouteldja discloses utilizing the preannotation to help facilitate the manual annotation effort performed by human annotators.).
Claim 14 is the system claim corresponding to claim 4, and is similarly rejected.
Regarding Claim 5, Lee in view of Bouteldja teach the method of claim 4, further comprising adjusting the at least one annotation on the anatomical image based on a comparison of the anatomical image including the at least one annotation to the corrected annotated anatomical image (Data Quality and Quantity, Bouteldja discloses utilizing the “preannotations” to aide human annotators. The Examiner asserts that if there is a difference (i.e., a comparison) between the annotation made by the human annotator and preannotation, an adjustment is made to the annotation applied to the image.).
Claim 15 is the system claim corresponding to claim 5, and is similarly rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kang et al. (“Development and operation of a digital platform for sharing pathology image data”, DOI: 10.1186/s12911-021-01466-1, Publication Year: 2021)
Lawson et al. (“Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays, DOI: 10.1038/bjc.2016.404, Publication Year: 2016)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PROMOTTO TAJRIAN ISLAM whose telephone number is (703)756-5584. The examiner can normally be reached Monday - Friday 8:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PROMOTTO TAJRIAN ISLAM/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights