DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-50 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-31 of U.S. Patent No. 12,198,300. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant claims are broader versions of the patented claims which generally replace “Fourier ptychographic digital refocusing” with a broader category of “substantially uniformly focused images generated using computational microscopy” or “all-in-focus analysis image of the cytology specimen using a computational microscopy digital refocusing procedure”.
In addition, the prior art rejections, references and claim mappings below are hereby incorporated by reference to demonstrate obviousness of other claimed features relative to the patenting claim features.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 28, 29, 32 and 33 are rejected under 35 U.S.C. 103 as being unpatentable over Engel (US 2021/0217190 A1) and Tosun (US 2020/0294231 A1)
Claim 28
In regards to claim 28, Engel discloses a method of computational refocusing-assisted deep learning {see abstract, Fig. 1, [0002], [0022] and cites below including light field camera 8 that generates an image that may be computationally refocused and generates a uniformly focused image using computational microscopy digital refocusing, [0025]-[0035]}, the method comprising:
(a) generating a segmented image of an analysis image of a specimen being imaged using a machine learning model trained by a first training dataset, the first training dataset comprising one or more substantially uniformly focused images generated using a computational microscopy digital refocusing procedure,
{see cell segmentation using the light field image (substantially uniformly focused image generated using a computational microscopy digital refocusing procedure) [0025]-[0035] with cell segmentation in [0033]-[0035], [0071], [0074], [0165], [0169], [0183], [0192]. Further as to training and machine learning model see [0141]-[0144], [0157], [0163]-[0169], [0185]-[0193] including neural networks and Deep CNNs}; and
(b) automatedly identifying one or more portions of interest in the analysis image based on the one or more boundaries of known portions of interest on the segmented image {see [0007]-[0009], [0016], [0031]-[0038], [0052]-[0054], [0139], [0166], [0173], [0181], [0187] that automatically classifies (identifies) the segmented portions of interest}.
Although Engel segments a digital microscopy image using a machine learning model trained by a first training dataset, the first training dataset comprising one or more substantially uniformly focused images generated using a computational microscopy digital refocusing procedure (light field imaging), Engel does not specifically describe the conventional machine learning training procedure, particularly the use of annotations to demarcate one or more boundaries of known portions of interest on the segmented image.
Tosun is an analogous reference from the same field of applying machine learning to detect classify cells. See abstract, background and cites below.
Tosun also teaches
generating a segmented image of an analysis image of a specimen being imaged using a machine learning model trained by a first training dataset, the first training dataset comprising one or more images generated using a computational microscopy digital refocusing procedure, the one or more images annotated to demarcate one or more boundaries of known portions of interest on the segmented image
{see [0016], [0023], [0094], [0103]-[0104], [0133]-[0134] segmenting a WSI (whole slide image) to define an enclosed region ROI (region of interest. Further as to annotating (labeling, ground truth labeling) see [0024], [0032], [0105], [0113], [0135]-[0136] which also includes a user affirming the ground truth labels (annotations) assigned to the ROIs}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Engel which already segments a digital microscopy image using a machine learning model trained by a first training dataset, the first training dataset comprising one or more substantially uniformly focused images generated using a computational microscopy digital refocusing procedure (light field imaging) such that the machine learning model includes a conventional machine learning training procedures including the use of annotations to demarcate one or more boundaries of known portions of interest on the segmented image as taught by Tosun because Tosun motivates such annotation because “ground truth labelling is necessary for machine learning training in [0135], because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 29
In regards to claim 29, Engel discloses wherein: (i) the analysis image is also generated using the computational microscopy digital refocusing procedure {see above cites for claim 28}; or
(ii) the analysis image is of a pathology slide, and the one or more portions of interest of the analysis image identified in (b) comprise one or more
Claim 32
In regards to claim 32, Engel is not relied upon to disclose but Tosun teaches wherein the one or more boundaries of known portions of interest on the segmented image comprise a visual indication on the segmented image determined based on a probability of a pixel associated with the visual indication exceeding a threshold compared to another pixel, the probability of the pixel produced by the machine learning model {see above mapping of claim 28 including [0024], [0032], [0105], [0113], [0135]-[0136] in which a user affirms the ground truth labels (annotations) assigned to the ROIs via a highlighted (green) segmentation boundary 804 wherein the segmentation that produces the highlighted boundary 804 is performed via a neural network which includes the broadly recited “probability”}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Engel which already segments a digital microscopy image using a machine learning model trained by a first training dataset, the first training dataset comprising one or more substantially uniformly focused images generated using a computational microscopy digital refocusing procedure (light field imaging) such that the machine learning model includes wherein the one or more boundaries of known portions of interest on the segmented image comprise a visual indication on the segmented image determined based on a probability of a pixel associated with the visual indication exceeding a threshold compared to another pixel, the probability of the pixel produced by the machine learning model as taught by Tosun because Tosun motivates such highlighting to permit a pathologist to confirm the segmentation boundary thereby increasing the accuracy of the segmentation, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 33
In regards to claim 33, Engel discloses wherein the segmented image comprises one or more demarcated groups of pixels, each demarcated group of pixels classified as respective one or more categories of cells {see [0007]-[0009], [0016], [0031]-[0038], [0052]-[0054], [0139], [0166], [0173], [0181], [0187] that automatically classifies (identifies) the segmented portions of interest such that the segmented image comprises demarcated groups of pixels each being classified as a respective category of cells}.
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over Engel and Tosun as applied to claim 28 above, and further in view of Georgescu (US 20190206056 A1).
Claim 30
In regards to claim 30, Engel classifies cells from a pathology slide image as per above but is not relied upon to disclose the one or more portions of interest of the analysis image identified in (b) comprise one or more tumor cells; and the method further comprises: (c) determining a percentage area coverage of tumor cells in the pathology slide based on the one or more tumor cells identified in (b); and (d) automatedly generating a diagnostic indicator based on the percentage area coverage of tumor cells determined
Georgescu is an analogous reference from the same field of applying machine learning to detect classify cells. See abstract, background and cites below.
Georgescu also teaches the one or more portions of interest of the analysis image identified in (b) comprise one or more tumor cells {see [0010]-[0011]; and the method further comprises
determining a percentage area coverage of the tumor cells in the pathology slide based on the tumor cells identified in (b) {see [0010]-[0011] teaching that the College of American Pathologists recommends evaluating tumor containing areas by performing image analysis to outline the tumor areas and compute a percentage positivity}, and
d) automatedly generating a diagnostic indicator based on the percentage area coverage determined {see the scoring algorithm in [0035]-[0036], Fig. 5, [0129]-[0141].
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Engel’s cell classification such that it
the one or more portions of interest of the analysis image identified in (b) comprise one or more tumor cells, determines a percentage area coverage of the tumor cells in the pathology slide based on the tumor cells identified in (b), and automatedly generates a diagnostic indicator based on the percentage area coverage determined
as taught by Georgescu because the College of American Pathologists recommends/motivates doing so in [0010]-[0011], because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 34 is rejected under 35 U.S.C. 103 as being unpatentable over Engel and Tosun as applied to claim 28 above, and further in view of Kumar (US 9739783 B1).
Claim 34
In regards to claim 33, Engel is not relied upon to disclose wherein the respective one or more categories of cells comprise benign cells, tumor cells, or a combination thereof.
Kumar is an analogous reference from the same field of analyzing cells and includes obtaining an analysis image of the specimen {Fig. 16, camera, fluorescence microscope, column 29, lines 36-57, column 30, lines 43-57, column 31, lines 41-60, column 31, lines 51-column 34, line 6 and particularly column 32, lines 51-60 for extended depth of field/ all-in-focus/substantially uniformly focused}; generating a representative image of the analysis image obtained based on a machine learning model {see column 34, line 29—column 35, line 63 in which the analysis image of a specimen is fed into a CNN/DNN (Convolutional Neural Network/Deep Neural Network) to generated a representative image of the analysis image in which the CNN/DNN has been trained using a machine learning model trained by a first training dataset (labeled training database). See also column 37, lines 9-27 for CNN/DNN training.}; and automatedly identifying (segmenting) one or more points of interest in the specimen based on the representative image {see column 37, lines 27—column 38, line 21 discussing classification (automatically identifying tumor classifications};
Kumar also teaches wherein the respective one or more categories of cells comprise benign cells, tumor cells, or a combination thereof {see column 37, lines 27—column 38, line 21 discussing classification (automatically identifying) tumor classifications (abnormalities). Further as to pathology slide see column 40, lines 25-29 and column 55, lines 13-15 and 65-67 and column 61, line 39-55}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Engel which already segments a digital microscopy image using a machine learning model trained by a first training dataset, the first training dataset comprising one or more substantially uniformly focused images generated using a computational microscopy digital refocusing procedure (light field imaging), wherein the segmented image comprises one or more demarcated groups of pixels, each demarcated group of pixels classified as respective one or more categories of cells such that wherein the respective one or more categories of cells comprise benign cells, tumor cells, or a combination thereof as taught by Kumar because doing so increases the diagnostic capabilities of the cell analysis and classification system of Engel, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 35-37 and 50 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Engel (US 2021/0217190 A1).
Independent Claim 35
In regards to claim 35, Engel discloses a method for analyzing a cytology specimen {see abstract, Fig. 1, [0002], [0022] including automated analyzing system and methods for analyzing and classifying cytology specimens (cells)}, the method comprising:
obtaining an all-in-focus analysis image of the cytology specimen using a computational microscopy digital refocusing procedure
{see abstract, Fig. 1, [0002], [0019]-[0022] and cites below including light field camera 8 that obtains an all-in-focus image using a computational microscopy digital refocusing procedure [0025]-[0035]};
generating a segmented image of the all-in-focus analysis image based on a machine learning model
{see cell segmentation using the light field image (all-in-focus image using a computational microscopy digital refocusing procedure) [0025]-[0035] with cell segmentation in [0033]-[0035], [0071], [0074], [0165], [0169], [0183], [0192]. Further as to machine learning model see [0141]-[0144], [0157], [0163]-[0167], [0185]-[0189] including neural networks and Deep CNNs}; and
automatedly identifying one or more points of interest in the cytology specimen based on one or more demarcation lines on the segmented image, wherein the one or more demarcation lines on the segmented image correspond to the one or more points of interest in the cytology specimen
{see [0007]-[0009], [0016], [0031]-[0038], [0052]-[0054], [0139], [0166], [0173], [0181], [0187] that automatically classifies (identifies) the segmented portions of interest};
wherein the machine learning model is trained by at least:
one or more all-in-focus training images generated by the computational microscopy digital refocusing procedure; and at least one training segmented image indicative of positions of points of interest in the one or more all-in-focus training images
{see [0141]-[0144], [0157], [0163]-[0169], [0185]-[0193]}.
Claim 36
In regards to claim 36, Engel discloses wherein the one or more points of interest in the cytology specimen comprise one or more abnormalities and/or one or more spatial relationships {See above. See also [0074], [0138], [0139], [0161]-[0174] including classifying cell type, fine cell diagnostics for pathological cells, establishment of ground truth data and resultant identification of rare pathologies}.
Claim 37
In regards to claim 37, Engel discloses generating a diagnostic indicator based on the one or more abnormalities and/or the one or more spatial relationships {See above. See also [0074], [0138], [0139], [0161]-[0174], [0185]-[0186] including generating diagnostic indicators (e.g. stains, classifications, etc.) indicating classification of cell type, fine cell diagnostics for pathological cells, identification of rare pathologies and display of diagnostic results 2D, 3D and VR [0193]-[0199]}.
Independent Claim 50
In regards to claim 50, Engel discloses an apparatus for identifying abnormalities in a specimen {See Fig. 1, [0002], [0022] and cites below for apparatus including light field camera 8 for identifying abnormalities in a cell specimen [0074], [0138], [0139], [0161]-[0174] including classifying cell type, fine cell diagnostics for pathological cells, establishment of ground truth data and resultant identification of rare pathologies}, the apparatus comprising:
a machine learning model; one or more processor apparatus configured to operate the machine learning model {machine learning model see [0141]-[0144], [0157], [0163]-[0167], [0185]-[0189] including neural networks and Deep CNNs }; and
a non-transitory computer-readable apparatus coupled to the one or more processor apparatus and comprising a storage medium, the storage medium comprising a plurality of instructions configured to, when executed by the one or more processor apparatus, cause the one or more processor apparatus to {see the implementation using “computer learning” in [0166], [0186] that implements neural networks using a computer which has a processor and storage medium};
obtain an analysis image of the specimen {see abstract, Fig. 1, [0002], [0019]-[0022] and cites below including light field camera 8 that obtains an all-in-focus image (substantially uniformly focused image) using a computational microscopy digital refocusing procedure [0025]-[0035]};
generate a segmented image of the analysis image obtained of the specimen using the machine learning model
{see cell segmentation using the light field image (substantially uniformly focused image generated using a computational microscopy digital refocusing procedure) [0025]-[0035] with cell segmentation in [0033]-[0035], [0071], [0074], [0165], [0169], [0183], [0192]. Further as to the machine learning model see [0141]-[0144], [0157], [0163]-[0167], [0185]-[0189] including neural networks and Deep CNNs},
the machine learning model trained by (i) generation of one or more convolutional representations of at least one substantially uniformly focused training image obtained using a computational microscopy digital refocusing procedure, and (ii) generation of at least one training segmented image based on the one or more convolutional representations of the at least one substantially uniformly focused training image
{see [0141]-[0144], [0157], [0163]-[0167], [0185]-[0189] including deep CNN (convolutional neural network) that generated convolutional representations of the training images which include segmented images (substantially uniformly focused images from the light field camera)}; and
based on one or more demarcated boundaries, in the segmented image, of one or more image segments determined to correspond to one or more abnormalities, automatedly identify the one or more abnormalities in the specimen
{see [0007]-[0009], [0016], [0031]-[0038], [0052]-[0054], [0139], [0166], [0173], [0181], [0187] that automatically classifies (identifies) the segmented portions of interest. Further as to abnormalities see [0074], [0138], [0139], [0161]-[0174], [0185]-[0186] including generating diagnostic indicators (e.g. stains, classifications, etc.) indicating classification of cell type, fine cell diagnostics for pathological cells, identification of rare pathologies and display of diagnostic results 2D, 3D and VR [0193]-[0199]}.
Claim 51 is rejected under 35 U.S.C. 103 as being unpatentable over Engel and Kumar.
Claim 51
In regards to claim 51, Engel discloses machine learning and training of a machine learning model but does not discuss the conventional details of wherein the machine learning model has further been trained by (i) determination of a performance metric based at least on an intersection of the at least one training segmented image with respect to a ground truth image.
Kumar teaches wherein the machine learning model has further been trained by (i) determination of a performance metric based at least on an intersection of the at least one training segmented image with respect to a ground truth image {see column 34, line 2 9—column 35, line 63 and column 37, line 9—column 38, line 3}
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Engel which already segments a digital microscopy image using a machine learning model trained by a first training dataset, the first training dataset comprising one or more substantially uniformly focused images generated using a computational microscopy digital refocusing procedure (light field imaging) as well as machine learning and training of a machine learning model such that the conventional details of machine learning model are utilized including the model being trained by (i) determination of a performance metric based at least on an intersection of the at least one training segmented image with respect to a ground truth image as taught by Kumar because such performance metrics cause the model to converge on a useful and beneficial results, because doing so increases the diagnostic capabilities of the cell analysis and classification system of Engel, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Allowable Subject Matter
Claims 31 are potentially allowable as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the double patenting rejection were overcome. Claims 38-49 would also be allowable if the double patenting rejection thereof were overcome by, e.g., filing an eTD.
In regards to claim 31, none of the prior art of record discloses or suggests the one or more substantially uniformly focused images of the first training dataset are used to train an encoder-decoder network and are of a first pathology slide taken from a first portion of a body; the analysis image is of a second pathology slide taken from a second portion of the body; and the method further comprises training the encoder-decoder network by applying one or more weights associated with the first training dataset to a second training dataset corresponding to the second pathology slide taken from the second portion of the body in combination with the limitations of base claim 28.
In regards to independent claim 38, none of the prior art of record discloses or suggests a method for identifying one or more points of interest in a specimen, the method comprising:
obtaining an analysis image of the specimen;
generating a segmented image of the analysis image obtained based on a machine learning model; and automatedly identifying the one or more points of interest in the specimen based on one or more demarcation boundaries of the one or more points of interest on the segmented image;
wherein the machine learning model comprises an encoder-decoder network trained by at least:
receiving at least one substantially uniformly focused training image determined based on digitally refocused images at different lateral positions; and
generating at least one training segmented image indicative of positions of points of interest in the at least one substantially uniformly focused training image.
Claims 39-49 are allowable due to their dependency upon claim 38.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael R Cammarata whose telephone number is (571)272-0113. The examiner can normally be reached M-Th 7am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ROBERT CAMMARATA/Primary Examiner, Art Unit 2667
Prosecution Insights