DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 11 and 12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Li et al. (WO 2020/200087).
Regarding claim 1, Li et al. discloses a method comprising using at least one computer processor to:
receive one or more color fundus photographs of a patient (“In some embodiments, the machine learning framework is used to analyze ophthalmic images generated using at least one ophthalmic medical imaging technique selected from optical coherence tomography (OCT), color fundus photography of the retina (CFP)” at paragraph 0060, line 7);
apply a machine-learning classifier having been trained using a dataset of CFPs of a patient cohort that have been classified according to their glaucoma status, to classify the received CFPs of the patient to thereby diagnose whether the patient has glaucoma (“In certain embodiments, a CNN is first trained on this dataset to develop features at its lower layers that are important for discriminating objects. In further embodiments, a second network is created that copies the parameters and structure of the first network, but with the final layer(s) optionally re-structured as needed for a new task. In certain embodiments, these final layer(s) are configured to perform the classification of retinal images. Thus, in some embodiments, the second network uses the first network to seed its structure. This allows training to continue on the new, but related task. In some embodiments, the first network is trained using labeled images comprising non-domain images (e.g. images not labeled with the final desired classification such as glaucoma), and the second network is trained using labeled images comprising domain images (e.g. images classified as having or not having glaucoma) to complete the training, allowing for high accuracy diagnosis of ophthalmic disorders and/or conditions” at paragraph 0058, line 6).
Regarding claim 11, Li et al. discloses a method wherein the machine-learning classifier comprises a deep learning model (“The machine-learning classifier may comprise a convolutional neural network” at paragraph 0006, line 1).
Regarding claim 12, Li et al. discloses a method wherein the deep learning model comprises convolutional neural networks (“The machine-learning classifier may comprise a convolutional neural network” at paragraph 0006, line 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3, 7 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Li et al., Hart et al. (US 20220319708) and Fumero (“Rim-one dl: A unified retinal image database for assessing glaucoma using deep learning”).
Regarding claim 3, Li et al. discloses a method wherein the machine-learning classifier comprises a segmentation module based on segmentation of anatomical structures including retinal vessels from the received CFPs (“The neural network architecture used to achieve vessel segmentation was derived from the U-net architecture” at paragraph 0126, line 1).
Li et al. does not explicitly disclose segmentation of anatomical structures including macula, optic cup and optic disk from the received CFPs.
Hart et al. teaches a method in the same field of endeavor of ophthalmic image analysis, wherein the machine-learning classifier comprises a segmentation module based on segmentation of anatomical structures including macula, and optic disk from the received CFPs (“For example, the graders may manually segment features within the ophthalmic images. According to some examples, the entity may identify landmarks within the ophthalmic images. For instance, the manual graders may further identify at least one of a macula, an optic disc (OD), or a fovea in each of multiple images” at paragraph 0088, third to last sentence).
Fumero teaches a method in the same field of endeavor of machine-learning based glaucoma detection wherein the machine-learning classifier comprises a segmentation module based on segmentation of anatomical structures including optic cup and optic disk from the received CFPs (”In addition to having a preliminary diagnosis, include also manual reference segmentations of the disc and cup” at item 4 in introduction).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include the macula, optic cup and optic disk for segmentation as taught by Hart et al. and Fumero using a segmentation module of Li et al. as these features assist in localization of relevant landmarks and are relevant to determination of glaucoma (see Fumero et al. at Introduction, in discussion of Figure 1).
Regarding claim 7, the Li et al., Hart et al. and Fumero et al. combination discloses a method wherein the segmentation module has been trained by manual annotations or segmentations of the anatomical structures including retinal vessels, macula, optic cup and optic disk independently (“For example, the graders may manually segment features within the ophthalmic images. According to some examples, the entity may identify landmarks within the ophthalmic images. For instance, the manual graders may further identify at least one of a macula, an optic disc (OD), or a fovea in each of multiple images” Hart et al. at paragraph 0088, third to last sentence; ”In addition to having a preliminary diagnosis, include also manual reference segmentations of the disc and cup” Fumero at item 4 in introduction; disclosure of Li et al. in paragraph 0143 implies that the expert annotated images are available for training, including the segmentation network; “In addition, training images with portions depicting a single feature that is a sign of a rare disease may be easier to obtain than whole training images that depict the rare disease. Accordingly, it may be easier and more efficient to train various ML models described herein than global image-based ML models” Hart et al. at paragraph 0008, line 6).
Regarding claim 13, Li et al. discloses a method wherein the machine-learning classifier comprises segmenting the anatomical structures including retinal vessels of the patient's CFPs using a U-net architecture (“The neural network architecture used to achieve vessel segmentation was derived from the U-net architecture” at paragraph 0126, line 1).
Li et al. does not explicitly disclose segmentation of anatomical structures including macula, optic cup and optic disk from the received CFPs.
Hart et al. teaches a method in the same field of endeavor of ophthalmic image analysis, wherein the machine-learning classifier comprises segmenting the anatomical structures including macula, and optic disk of the patient's CFPs using a U-net architecture (“For example, the graders may manually segment features within the ophthalmic images. According to some examples, the entity may identify landmarks within the ophthalmic images. For instance, the manual graders may further identify at least one of a macula, an optic disc (OD), or a fovea in each of multiple images” at paragraph 0088, third to last sentence; “Examples of ML models that can be used to implement the presence analyzer 304, the location analyzer 306, the amount analyzer 308 include one or more NNs, such as CNNs (e.g., U-Net) and/or transformer-based models” at paragraph 0060, second to last sentence).
Fumero teaches a method in the same field of endeavor of machine-learning based glaucoma detection wherein the machine-learning classifier comprises segmenting the anatomical structures including optic cup and optic disk of the patient's CFPs (”In addition to having a preliminary diagnosis, include also manual reference segmentations of the disc and cup” at item 4 in introduction).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include the macula, optic cup and optic disk for segmentation as taught by Hart et al. and Fumero using a segmentation module of Li et al. as these features assist in localization of relevant landmarks and are relevant to determination of glaucoma (see Fumero et al. at Introduction, in discussion of Figure 1).
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Li et al. and Garnavi et al (US 20170270653).
Li et al. discloses a method wherein the received one or more CFPs of the patient is obtained from a fundus image of the patient (“In some embodiments, the machine learning framework is used to analyze ophthalmic images generated using at least one ophthalmic medical imaging technique selected from optical coherence tomography (OCT), color fundus photography of the retina (CFP)” at paragraph 0060, line 7).
Li et al. does not explicitly disclose that the images is captured by a smart phone.
Garnavi et al. teaches a method in the same field of endeavor of retinal image processing, wherein the received one or more CFPs of the patient is obtained from a fundus image of the patient captured by a smart phone (“images are acquired by the patient using mobile devices” at paragraph 0091, line 11).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a mobile device to provide images as taught by Garnavi et al. for the system of Li et al. as “Teleophthalmology is particularly relevant for locations which have limited access to clinical specialists and resources” (Garnavi et al. at paragraph 0086, line 1), thereby allowing the system to be accessible to those in remote or rural settings.
Claim(s) 4 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Li et al., Hart et al. and Fumero et al. as applied to claim 3 above, and further in view of Martins et al. (“Offline computer-aided diagnosis for Glaucoma detection using fundus images targeted at mobile devices”).
The Li et al., Hart et al. and Fumero et al. combination discloses a method as described in claim 3 above.
The Li et al., Hart et al. and Fumero et al. combination does not explicitly disclose that the machine-learning classifier further comprises a diagnostic module which generates a glaucomatous probability score.
Martins et al. teaches a method in the same field of endeavor image-based glaucoma detection, wherein the machine-learning classifier further comprises a diagnostic module which generates a glaucomatous probability score (“To obtain a Glaucoma confidence level, a classification network, entitled GFI-C, was created using MobileNetV2 [16] feature extrac- tor as a backbone” at section 3.5, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a probability quantification as taught by Martins et al. for the system of the Li et al., Hart et al. and Fumero et al. combination to reflect the confidence of glaucoma being present in the analyzed image.
Claim(s) 2 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Li et al. and Lee et al. (“Predicting Glaucoma Development With Longitudinal Deep Learning Predictions From Fundus Photographs”)
Li et al. discloses a method comprising using at least one computer processor to:
receive one or more color fundus photographs of a patient (“In some embodiments, the machine learning framework is used to analyze ophthalmic images generated using at least one ophthalmic medical imaging technique selected from optical coherence tomography (OCT), color fundus photography of the retina (CFP)” at paragraph 0060, line 7);
apply a machine-learning classifier having been trained using a dataset of CFPs of a patient cohort regarding glaucoma development of each of the patients in the cohort, to predict a likelihood of glaucoma incidence or progression for the patient in the future (“In certain embodiments, a CNN is first trained on this dataset to develop features at its lower layers that are important for discriminating objects. In further embodiments, a second network is created that copies the parameters and structure of the first network, but with the final layer(s) optionally re-structured as needed for a new task. In certain embodiments, these final layer(s) are configured to perform the classification of retinal images. Thus, in some embodiments, the second network uses the first network to seed its structure. This allows training to continue on the new, but related task. In some embodiments, the first network is trained using labeled images comprising non-domain images (e.g. images not labeled with the final desired classification such as glaucoma), and the second network is trained using labeled images comprising domain images (e.g. images classified as having or not having glaucoma) to complete the training, allowing for high accuracy diagnosis of ophthalmic disorders and/or conditions” at paragraph 0058, line 6; “In some aspects, the systems, methods, and software disclosed herein provide an improvement in the field of software-based image data processing whereby retinal images are used to detect or predict the development of an ophthalmic or systemic disease, disorder, or condition at a future time point” at paragraph 0035, line 1).
Li et al. does not explicitly disclose a longitudinal patient cohort regarding glaucoma development of each of the patients in the cohort over a period of time.
Lee et al. teaches a method in the same field of endeavor of machine-learning based glaucoma detection comprising using at least one computer processor to:
apply a machine-learning classifier having been trained using a dataset of CFPs (“the M2M model is a convolutional neural network trained to predict global RNFL thickness measurements using color fundus photographs” at page 87, right column, “Deep Learning Machine to Machine Predictions”, line 4) of a longitudinal patient cohort regarding glaucoma development of each of the patients in the cohort over a period of time, to predict a likelihood of glaucoma incidence or progression for the patient in the future (“In brief, this analysis involves joint evaluation of a longitudinal submodel and a survival submodel, which are tied together by sharing random effects. The longitudinal submodel was composed of a linear mixed-effects model, which accounts for measurement error by including a random error term and assumes random slopes and random intercepts, allowing different rates of change for each eye” at page 88, left column, Statistical Analysis, line 10; “Trajectory of machine to machine–predicted retinal nerve fiber layer thicknesses from fundus photographs over time (longitudinal response) for non-converters and converters” at page 89, Figure 1 description).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include a longitudinal patient cohort as taught by Lee et al. in the training of Li et al. as “ability to continuously update predictions and use longitudinal changes in the acquired data makes our model a useful clinical tool in monitoring for glaucomatous change and improve risk prediction” (Lee et al. at page 92, right column, line 1).
Claim(s) 5, 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Li et al. and Lee et al. as applied to claim 2 above, and further in view of Hart et al. and Fumero et al.
Regarding claim 5, the Li et al. and Lee et al. combination discloses a method wherein the machine-learning classifier comprises a segmentation module based on segmentation of anatomical structures including retinal vessels from the received CFPs (“The neural network architecture used to achieve vessel segmentation was derived from the U-net architecture” Li et al. at paragraph 0126, line 1).
The Li et al. and Lee et al. combination does not explicitly disclose segmentation of anatomical structures including macula, optic cup and optic disk from the received CFPs.
Hart et al. teaches a method in the same field of endeavor of ophthalmic image analysis, wherein the machine-learning classifier comprises a segmentation module based on segmentation of anatomical structures including macula, and optic disk from the received CFPs (“For example, the graders may manually segment features within the ophthalmic images. According to some examples, the entity may identify landmarks within the ophthalmic images. For instance, the manual graders may further identify at least one of a macula, an optic disc (OD), or a fovea in each of multiple images” at paragraph 0088, third to last sentence).
Fumero teaches a method in the same field of endeavor of machine-learning based glaucoma detection wherein the machine-learning classifier comprises a segmentation module based on segmentation of anatomical structures including optic cup and optic disk from the received CFPs (”In addition to having a preliminary diagnosis, include also manual reference segmentations of the disc and cup” at item 4 in introduction).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include the macula, optic cup and optic disk for segmentation as taught by Hart et al. and Fumero using a segmentation module of the Li et al. and Lee et al. combination as these features assist in localization of relevant landmarks and are relevant to determination of glaucoma (see Fumero et al. at Introduction, in discussion of Figure 1).
Regarding claim 6, the Li et al., Lee et al., Hart et al. and Fumero et al. combination discloses a method wherein the machine-learning classifier further comprises a prediction module which produces a risk score of glaucoma incidence or progression in the future for the patient (“In some aspects, the systems, methods, and software disclosed herein provide an improvement in the field of software-based image data processing whereby retinal images are used to detect or predict the development of an ophthalmic or systemic disease, disorder, or condition at a future time point” Li et al. at paragraph 0035, line 1; “The primary purpose of our study was to assess whether M2M’s predictions of baseline and longitudinal RNFL thicknesses from fundus photographs could predict risk of conversion in glaucoma suspects. We used joint longitudinal survival models21 to investigate the relationship between baseline and longitudinal predictions of RNFL thicknesses and the risk of developing visual field loss in glaucoma suspects” Lee et al. at page 89, left column, Statistical Analysis, line 1).
Regarding claim 14, the Li et al., Lee et al., Hart et al. and Fumero et al. combination discloses a method wherein the segmentation module has been trained by manual annotations or segmentations of the anatomical structures including retinal vessels, macula, optic cup and optic disk independently (“For example, the graders may manually segment features within the ophthalmic images. According to some examples, the entity may identify landmarks within the ophthalmic images. For instance, the manual graders may further identify at least one of a macula, an optic disc (OD), or a fovea in each of multiple images” Hart et al. at paragraph 0088, third to last sentence; ”In addition to having a preliminary diagnosis, include also manual reference segmentations of the disc and cup” Fumero at item 4 in introduction; disclosure of Li et al. in paragraph 0143 implies that the expert annotated images are available for training, including the segmentation network; “In addition, training images with portions depicting a single feature that is a sign of a rare disease may be easier to obtain than whole training images that depict the rare disease. Accordingly, it may be easier and more efficient to train various ML models described herein than global image-based ML models” Hart et al. at paragraph 0008, line 6).
Claim(s) 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Li et al. and Lee et al. as applied to claim 2 above, and further in view of Peng et al. (US 20210357696).
Regarding claim 9, the Li et al. and Lee et al. combination discloses a method as described in claim 2 above.
The Li et al. and Lee et al. combination does not explicitly disclose that the dataset of CFPs the longitudinal patient cohort has been stratified into low-risk and high-risk groups in glaucoma incidence or progression.
Peng et al. teaches a method in the same field of endeavor of machine-learning based glaucoma detection, wherein the dataset of CFPs the patient cohort has been stratified into low-risk and high-risk groups in glaucoma incidence or progression (“For example, in the case of glaucoma, the set of scores may include a score for no glaucoma, mild or early-stage glaucoma, moderate-stage glaucoma, and severe-stage glaucoma, with the score for each stage representing the likelihood that the corresponding stage will be the stage of glaucoma for the patient at the future time” at paragraph 0124; “In others of these implementations, the set of risk scores includes a respective score for each of multiple risk levels, e.g., low, medium, and high, for the health event, with each risk score representing a likelihood that the corresponding risk level is the current risk level of the health event occurring” at paragraph 0147, last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to stratify the cohort of the Li et al. and Lee et al. combination using the risk levels as taught by Peng et al. to further characterize the disease severity and progression for proper treatment.
Regarding claim 10, the Li et al., Lee et al. and Peng et al. combination discloses a method further comprising: using at least one computer processor to classify the patient as belonging to a low-risk or a high-risk group for glaucoma incidence or progression in the future (“For example, in the case of glaucoma, the set of scores may include a score for no glaucoma, mild or early-stage glaucoma, moderate-stage glaucoma, and severe-stage glaucoma, with the score for each stage representing the likelihood that the corresponding stage will be the stage of glaucoma for the patient at the future time” Peng et al. at paragraph 0124; “In some aspects, the systems, methods, and software disclosed herein provide an improvement in the field of software-based image data processing whereby retinal images are used to detect or predict the development of an ophthalmic or systemic disease, disorder, or condition at a future time point” Li et al. at paragraph 0035, line 1; “The primary purpose of our study was to assess whether M2M’s predictions of baseline and longitudinal RNFL thicknesses from fundus photographs could predict risk of conversion in glaucoma suspects. We used joint longitudinal survival models21 to investigate the relationship between baseline and longitudinal predictions of RNFL thicknesses and the risk of developing visual field loss in glaucoma suspects” Lee et al. at page 89, left column, Statistical Analysis, line 1).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Xiong et al. (US 12,156,697) is pertinent as disclosing a machine-learning based glaucoma detection utilizing manually segmented training images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672
Prosecution Insights