DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/18/2026 has been considered by the examiner.
Response to Arguments
Applicant’s arguments (see remarks), filed 08/18/2026, with respect to claims 1-20, have been fully considered but are respectfully unpersuasive.
Applicant argues on page 9 “These disclosures do not provide the claimed paired disease presence and disease severity probabilities for each disease-specific model, nor do they provide the claimed adjustment of both probabilities for one disease based on corresponding probabilities for a different disease“.
In response, the Office respectfully finds this argument unpersuasive. Based on the breadth of the claim language, the prior art by SURESH et al. (US 20240032784 A1) explicitly teaches processing a plurality of disease presence probabilities and a plurality of disease severity probabilities determined by the plurality of second machine learning models with a third machine learning model to determine a plurality of disease indicators based on relationships between a plurality of diseases and their respective severities identified by the plurality of second machine learning models (Fig. 4. Paragraph [0027]-SURESH discloses the severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. In an exemplary embodiment, the severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct). The confidence score may be a percentage), the processing comprising adjusting, for each disease of the plurality of diseases, the disease presence probability and the disease severity probability based on a disease presence probability and a disease severity probability for at least one different disease of the plurality of diseases (Fig. 4. Paragraph [0039]-SURESH discloses the output component 155 receives the local features 230 and the global features 235 and generates a diagnostic output 165. The output component 155 concatenates the local features 230 and global features 235 to obtain at least one ophthalmic image 175 with one or more disease indications 180. The at least one ophthalmic image 175 may be a retinal image, and the one or more disease indications 180 may include one or more segmented disease regions within the retinal image. Further in paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease. The output component 155 can use a neural network (e.g., 2D or 3D CNN) or a vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235. Please also read paragraph [0027-0028, 0036, 0044-0047]).
Although SURESH explicitly teaches processing the retinal image by a plurality of second machine learning models (Fig. 3-4, #410, #420 and #304-1-k called a deep learning model and a CNN, respectively. Paragraph [0043]. In paragraph [0037]-SURESH discloses the extraction component 145 processes the spectral information 205 and the spatial information 210 separately using the local feature detection tool 220 and the global feature detection tool 225. The local feature detection tool 220 and the global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform the respective local feature detection and global feature detection (wherein the visualization tool #215 uses one or more CNNS #304-1-k to process spectral information #205, and deep learning models are each used for local feature extraction, global feature extraction, and generation of a probability map for diseases. Please also read paragraph [0044 and 0048]), wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease and 2) a disease severity probability for the disease (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. Each “word” may associate a label (corresponding to a local feature 230) with a description of a retinal disease symptom (e.g., seeing flashes of light, blurry vision, reduced central or peripheral vision, sudden loss of vision, change in color perception) or a particular retinal disease (e.g., retinal tear, retinal detachment, diabetic retinopathy, macular hole, etc.). The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease);
SURESH is silent on at a first time, determining that the retinal image reflects the abnormal condition of the retina and in response to determining that the retinal image reflects the abnormal condition of the retina: processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease; and at a second time, determining that the retinal image reflects the normal condition of the retina and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image.
However, BHUIYAN explicitly teaches at a first time, determining that the retinal image (Fig. 3, #132 and #134 called a red-free fundus image and a color fundus image, respectively. Paragraph [0064]-BHUIYAN discloses a multimodal color Fundus (CF) and red-free (RF) image analysis platform for AMD screening and prediction system is illustrated in FIG. 3. The system includes two modules. First, a module for automated screening of individuals for early-stage AMD 110 is provided (see also, FIG. 4). Second, a module for prediction of individuals at risk of developing late AMD in the near future 120 is provided. In paragraph [0066]-BHUIYAN discloses the screening module 110 receives patient data 130. Patient data 130 includes, e.g., red-free (RF) fundus data 132, color fundus (CF) data 134, and socio-demographic data 136. In paragraph [0068]-BHUIYAN discloses the AMD screening system 100 utilizes machine-learning-based algorithms for extraction of features (e.g., AMD pathologies) and fusion from multimodal imaging and deep convolution neural network (wherein the system may preprocess, segment, select and/or fuse RF and CF features to generate normalized and prominent features with potential for AMD pathology)) reflects the abnormal condition of the retina (Fig. 1. Paragraph [0072]-BHUIYAN discloses Deep learning and image analysis are applied (step 470) to identify the normal/healthy individuals, and AMD suspect individuals. As the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”) and in response to determining that the retinal image reflects the abnormal condition of the retina (Fig. 5. Paragraph [0073]-BHUYIAN discloses if an individual determined to be not “at risk” for AMD, the individual is advised to return in one year, as shown in FIG. 3 (Step 142). If an individual is determined to be “at risk” for AMD 140, the individual's data is then processed by the AMD incidence prediction module 120, which includes feature and/or longitudinal image analysis, e.g., to be provided to an ophthalmologist for a high confidence diagnosis. Following the analysis, treatment 160 and/or follow up 162 are prescribed. Further, an image is taken for progression analysis 164. In paragraph [0074]-BHUIYAN discloses once an individual is identified as “at risk” by the AMD Suspect screening module 110, a prediction score 144 for the individual for developing late AMD in near future is computed by the AMD incidence prediction module 120. The pathology quantification is performed utilizing the CF images 134 and RF images 132 to find the prediction score of that individual for developing late AMD):
processing the retinal image by a plurality of second machine learning models (Fig. 5. Paragraph [0075]-BHUIYAN discloses a deep learning model is used to predict the individual at risk of AMD. In paragraph [0078]-BHUYIAN discloses a method for drusen quantification during AMD incidence prediction 120 is illustrated in FIG. 8. The quantification provides “early,” “intermediate,” or “late” AMD stage classification along with categories 1-9 (wherein a machine-learning method and graph-based method are applied for drusen quantification)), wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease (Fig. 5. Paragraph [0134]-BHUIYAN discloses scoring scale is developed considering four pathologies, e.g., drusen area, and three pigment abnormalities (Increased Pigmentation, Depigmentation, and Geographic Atrophy). Deep learning-based classification system has been built for each of the four pathologies. Further in paragraph [0133]-BHUIYAN discloses for AMD severity level categorization and fuzzy weighted score generation, the categorized AMD severity levels are: No AMD (severity level 1), Early AMD (severity level 2), Intermediate AMD (severity level 3) and Advanced AMD (severity level 4). Then the probability for each severity levels is generated. In paragraph [0135]-BHUIYAN discloses combining the result of the four classifiers and using the same protocol as AREDS, nine probabilities for an image (for an eye) are obtained. The nine probabilities of eye1 are referred to as: a, b, c, d, e, f, g, h, and i. In paragraph [0136]-BHUIYAN discloses these are the probabilities that an image falls in the particular severity scale level 1-9); and
at a second time, determining that the retinal image (Fig. 3, #132 and #134 called a red-free fundus image and a color fundus image, respectively. Paragraph [0064]) reflects the normal condition of the retina (Fig. 1. Paragraph [0068]-BHUYIAN discloses the AMD screening system 100 utilizes machine-learning-based algorithms for extraction of features (e.g., AMD pathologies) and fusion from multimodal imaging and deep convolution neural network. In paragraph [0069]-BHUYIAN discloses FIG. 4 illustrates the components of the AMD suspect screening module 110 described above. At Step 410, CF images (or RF or fused CF and RF images) are segmented for prominent region selection; then elastic registration is applied on the segmented CF images (or RF or fused CF and RF images) to find the corresponding positions of the potential AMD pathology. In paragraph [0071]-BHUYIAN discloses a preprocessing step can be used to generate the normalized image and prominent features. In paragraph [0072]-BHUIYAN discloses deep learning and image analysis are applied (step 470) to identify the normal/healthy individuals, and AMD suspect individuals. As the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”) and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image (Fig. 1. Paragraph [0072]-BHUIYAN discloses as the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”. In paragraph [0073]-BHUYIAN discloses if an individual determined to be not “at risk” for AMD, the individual is advised to return in one year, as shown in FIG. 3 (Step 142). If an individual is determined to be “at risk” for AMD 140, the individual's data is then processed by the AMD incidence prediction module 120, which includes feature and/or longitudinal image analysis, e.g., to be provided to an ophthalmologist for a high confidence diagnosis. Following the analysis, treatment 160 and/or follow up 162 are prescribed (wherein an image is taken for progression analysis 164 when there is not a high risk score)).
Applicant argues on page 9 “The additional references applied to claims 2-6, 9-14, and 19-20 do not cure these deficiencies “.
In response, the Office respectfully finds this argument unpersuasive for the reasons stated above and below.
Applicant argues on page 9 “Accordingly, the applied combinations do not render obvious independent claims 1 and 18. Claims 2-17 and 19-20 depend directly or indirectly from claims 1 or 18 and include the same amended limitations as well as additional patentable features. Applicant respectfully requests withdrawal of the rejections under 35 U.S.C. § 103 “
In response, the Office respectfully finds this argument unpersuasive for the reasons stated above and below.
Furthermore, in response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, SURESH teaches processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease and 2) a disease severity probability for the disease”. As discussed further above, SURESH teaches using multiple machine learning models for both disease detection and severity. For instance, at paragraph [0052], SURESH states “the diagnostic tool uses a first deep learning model to extract a set of local features (e.g., local features 230) from the spectral information and uses a second deep learning model to extract a set of global features (e.g., global features 235) from the spatial information.” Local features represent local features while global features represent disease severity (please see para. [0048]). Nevertheless, BHUYIAN teaches the entirety of the limitation based on the breadth of the claim language as described further above. Specifically, BHUYIAN teaches a graph-based machine learning model and leveraging multiple prediction models for identifying and quantifying the risk and severity of pathologies, including disease specific machine learning models for symptomology/pathology/severity (please see paragraph [0064-0075, 0126-0127 and 0134-0136].
Applicant argues on page 10 “Applicant respectfully submits that Applicant does not necessarily agree with the characterization and assessments of the dependent claims made by the Examiner, and Applicant believes that each claim is patentable on its own merits. Applicant respectfully submits that the dependent claims incorporate by reference all the limitations of the claim to which they refer and include their own patentable features and are therefore in condition for allowance. Therefore, Applicant respectfully requests the withdrawal of all claim rejections and prompts allowance of the claims “.
In response, the Office respectfully finds this argument unpersuasive for the reasons stated above and below.
Applicant argues on page 10 “Therefore, Applicant respectfully requests the withdrawal of all claim rejections and prompts allowance of the claims “.
In response, the Office respectfully finds this argument unpersuasive for the reasons stated above and below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-8 and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over SURESH et al. (US 20240032784 A1), hereinafter referenced as SURESH in view of BHUIYAN et al. (US 20200242763 A1), hereinafter referenced as BHUIYAN.
Regarding claim 1, SURESH explicitly teaches a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors cause the one or more processors to perform a method for disease detection using retinal images (Fig. 1. Paragraph [0019]-SURESH discloses FIG. 1 illustrates an example ophthalmic system 100 for analyzing multiple spectral information (e.g., MSI/HSI information). The ophthalmic system 100 includes an imaging system 150, a computing system 130, a computing system 190, and a display 170 (wherein a computing system may be a mobile computer, server, etc.). Please also see Fig. 7 and read paragraph [0058-0059]), the method comprising:
processing (Fig. 1. Paragraph [0021]-SURESH discloses the computing system 130 includes a diagnostic tool 135, which is configured for analyzing multiple spectral information for ophthalmology applications. The diagnostic tool 135 includes an analysis component 140, an extraction component 145, and an output component 155. The diagnostic tool 135 receives input data 110 from the imaging system 150 and generates ophthalmic information 120, based on an evaluation of the input data 110 (wherein the diagnostic tool separates input data into spectral and spatial data, and the extraction component #125 contains a visualization tool #215, a local feature detection tool #220 and a global feature detection tool #225)) a retinal image (Fig. 1, #110 called input data. Paragraph [0020]. Please also read paragraph [0021]) of a retina (Fig. 1. Paragraph [0020]-SURESH discloses the imaging system 150 can capture multiple spectral information associated with a target (e.g., tissues/structures of the eye 160)) with a first machine learning model (Fig. 3-4, #410, #420 and #304-1-k called a deep learning model and a CNN, respectively. Paragraph [0043]. In paragraph [0037]-SURESH discloses the extraction component 145 processes the spectral information 205 and the spatial information 210 separately using the local feature detection tool 220 and the global feature detection tool 225. The local feature detection tool 220 and the global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform the respective local feature detection and global feature detection (wherein the visualization tool #215 uses one or more CNNS #304-1-k to process spectral information #205, and deep learning models are each used for local feature extraction, global feature extraction, and generation of a probability map for diseases. Please also read paragraph [0044 and 0048]) to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina (Fig. 4. Paragraph [0024]-SURESH discloses the diagnostic tool 135 receives diagnostic criteria 195. In paragraph [0025]-SURESH discloses the ophthalmic information 120 includes an interpretative image 125 and a diagnostic output 165. In paragraph [0026]-SURESH discloses the diagnostic output 165 may include information associated with a prediction of at least one disease of the patient's eye 160, based on an analysis of the input data 110 (wherein the diagnostic output is based on local and global features). In paragraph [0027]-SURESH discloses the severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease (wherein the severity score or ocular disease prediction may be ranked from 0 to 100). Please also read paragraph [0048]);
processing a plurality of disease presence probabilities and a plurality of disease severity probabilities determined by the plurality of second machine learning models with a third machine learning model to determine a plurality of disease indicators based on relationships between a plurality of diseases and their respective severities identified by the plurality of second machine learning models (Fig. 4. Paragraph [0027]-SURESH discloses the severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. In an exemplary embodiment, the severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct). The confidence score may be a percentage), the processing comprising adjusting, for each disease of the plurality of diseases, the disease presence probability and the disease severity probability based on a disease presence probability and a disease severity probability for at least one different disease of the plurality of diseases (Fig. 4. Paragraph [0039]-SURESH discloses the output component 155 receives the local features 230 and the global features 235 and generates a diagnostic output 165. The output component 155 concatenates the local features 230 and global features 235 to obtain at least one ophthalmic image 175 with one or more disease indications 180. The at least one ophthalmic image 175 may be a retinal image, and the one or more disease indications 180 may include one or more segmented disease regions within the retinal image. Further in paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease. The output component 155 can use a neural network (e.g., 2D or 3D CNN) or a vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235. Please also read paragraph [0027-0028, 0036, 0044-0047]).
processing a plurality of disease presences and a plurality of disease severities determined by the plurality of second machine learning models (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. Each “word” may associate a label (corresponding to a local feature 230) with a description of a retinal disease symptom (e.g., seeing flashes of light, blurry vision, reduced central or peripheral vision, sudden loss of vision, change in color perception) or a particular retinal disease (e.g., retinal tear, retinal detachment, diabetic retinopathy, macular hole, etc.). The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease) with a third machine learning model to determine a plurality of disease indicators (Fig. 4. Paragraph [0039]-SURESH discloses the output component 155 receives the local features 230 and the global features 235 and generates a diagnostic output 165. The output component 155 concatenates the local features 230 and global features 235 to obtain at least one ophthalmic image 175 with one or more disease indications 180. The at least one ophthalmic image 175 may be a retinal image, and the one or more disease indications 180 may include one or more segmented disease regions within the retinal image. Please also read paragraph [0027-0028]) based on relationships between a plurality of diseases and their respective severities identified by the plurality of second machine learning models (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 can use a neural network (e.g., 2D or 3D CNN) or a vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235);
outputting the plurality of disease indicators (Fig. 4, #180 called disease indications. Paragraph [0027 and 0039]. In paragraph [0039]-SURESH discloses the output component 155 receives the local features 230 and the global features 235 and generates a diagnostic output 165. The output component 155 concatenates the local features 230 and global features 235 to obtain at least one ophthalmic image 175 with one or more disease indications 180. The at least one ophthalmic image 175 may be a retinal image, and the one or more disease indications 180 may include one or more segmented disease regions within the retinal image. Please also read paragraph [0026-0028]); and
outputting the retinal image (Fig. 4, #175 and #125 called an ophthalmic image and an interpretative image. Paragraph [0028 and 0039]. In paragraph [0028]-SURESH discloses in an exemplary diagnostic output 165, a first ophthalmic image 175-1 may have a first disease indication 180-1 of a first retinal disease (e.g., one or more segmented regions in the first ophthalmic image 175-1 that indicate the first retinal disease) and a severity score 185-1 for the first retinal disease based on the first disease indication 180-1; a second ophthalmic image 175-2 may have a second disease indication 180-2 of a second retinal disease (e.g., one or more segmented regions in the second ophthalmic image 175-2 that indicate the second retinal disease) and a severity score 185-2 for the second retinal disease based on the second disease indication 180-2; and so on. In paragraph [0039]-SURESH discloses the output component 155 may provide the interpretative image 125 (received from the visualization tool 215) to a user (e.g., via imaging system 150 and/or display 170). The interpretative image 125 may enable the user to interpret the multiple spectral information obtained from the imaging system 150 in a more effective and/or easier manner when making a disease diagnosis); and
Although SURESH explicitly teaches processing the retinal image by a plurality of second machine learning models (Fig. 3-4, #410, #420 and #304-1-k called a deep learning model and a CNN, respectively. Paragraph [0043]. In paragraph [0037]-SURESH discloses the extraction component 145 processes the spectral information 205 and the spatial information 210 separately using the local feature detection tool 220 and the global feature detection tool 225. The local feature detection tool 220 and the global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform the respective local feature detection and global feature detection (wherein the visualization tool #215 uses one or more CNNS #304-1-k to process spectral information #205, and deep learning models are each used for local feature extraction, global feature extraction, and generation of a probability map for diseases. Please also read paragraph [0044 and 0048]), wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease and 2) a disease severity probability for the disease (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. Each “word” may associate a label (corresponding to a local feature 230) with a description of a retinal disease symptom (e.g., seeing flashes of light, blurry vision, reduced central or peripheral vision, sudden loss of vision, change in color perception) or a particular retinal disease (e.g., retinal tear, retinal detachment, diabetic retinopathy, macular hole, etc.). The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease);
SURESH fails to teach at a first time, determining that the retinal image reflects the abnormal condition of the retina and in response to determining that the retinal image reflects the abnormal condition of the retina: processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease; and at a second time, determining that the retinal image reflects the normal condition of the retina and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image.
However, BHUIYAN explicitly teaches at a first time, determining that the retinal image (Fig. 3, #132 and #134 called a red-free fundus image and a color fundus image, respectively. Paragraph [0064]-BHUIYAN discloses a multimodal color Fundus (CF) and red-free (RF) image analysis platform for AMD screening and prediction system is illustrated in FIG. 3. The system includes two modules. First, a module for automated screening of individuals for early-stage AMD 110 is provided (see also, FIG. 4). Second, a module for prediction of individuals at risk of developing late AMD in the near future 120 is provided. In paragraph [0066]-BHUIYAN discloses the screening module 110 receives patient data 130. Patient data 130 includes, e.g., red-free (RF) fundus data 132, color fundus (CF) data 134, and socio-demographic data 136. In paragraph [0068]-BHUIYAN discloses the AMD screening system 100 utilizes machine-learning-based algorithms for extraction of features (e.g., AMD pathologies) and fusion from multimodal imaging and deep convolution neural network (wherein the system may preprocess, segment, select and/or fuse RF and CF features to generate normalized and prominent features with potential for AMD pathology)) reflects the abnormal condition of the retina (Fig. 1. Paragraph [0072]-BHUIYAN discloses Deep learning and image analysis are applied (step 470) to identify the normal/healthy individuals, and AMD suspect individuals. As the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”) and in response to determining that the retinal image reflects the abnormal condition of the retina (Fig. 5. Paragraph [0073]-BHUYIAN discloses if an individual determined to be not “at risk” for AMD, the individual is advised to return in one year, as shown in FIG. 3 (Step 142). If an individual is determined to be “at risk” for AMD 140, the individual's data is then processed by the AMD incidence prediction module 120, which includes feature and/or longitudinal image analysis, e.g., to be provided to an ophthalmologist for a high confidence diagnosis. Following the analysis, treatment 160 and/or follow up 162 are prescribed. Further, an image is taken for progression analysis 164. In paragraph [0074]-BHUIYAN discloses once an individual is identified as “at risk” by the AMD Suspect screening module 110, a prediction score 144 for the individual for developing late AMD in near future is computed by the AMD incidence prediction module 120. The pathology quantification is performed utilizing the CF images 134 and RF images 132 to find the prediction score of that individual for developing late AMD):
processing the retinal image by a plurality of second machine learning models (Fig. 5. Paragraph [0075]-BHUIYAN discloses a deep learning model is used to predict the individual at risk of AMD. In paragraph [0078]-BHUYIAN discloses a method for drusen quantification during AMD incidence prediction 120 is illustrated in FIG. 8. The quantification provides “early,” “intermediate,” or “late” AMD stage classification along with categories 1-9 (wherein a machine-learning method and graph-based method are applied for drusen quantification)), wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease (Fig. 5. Paragraph [0134]-BHUIYAN discloses scoring scale is developed considering four pathologies, e.g., drusen area, and three pigment abnormalities (Increased Pigmentation, Depigmentation, and Geographic Atrophy). Deep learning-based classification system has been built for each of the four pathologies. Further in paragraph [0133]-BHUIYAN discloses for AMD severity level categorization and fuzzy weighted score generation, the categorized AMD severity levels are: No AMD (severity level 1), Early AMD (severity level 2), Intermediate AMD (severity level 3) and Advanced AMD (severity level 4). Then the probability for each severity levels is generated. In paragraph [0135]-BHUIYAN discloses combining the result of the four classifiers and using the same protocol as AREDS, nine probabilities for an image (for an eye) are obtained. The nine probabilities of eye1 are referred to as: a, b, c, d, e, f, g, h, and i. In paragraph [0136]-BHUIYAN discloses these are the probabilities that an image falls in the particular severity scale level 1-9); and
at a second time, determining that the retinal image (Fig. 3, #132 and #134 called a red-free fundus image and a color fundus image, respectively. Paragraph [0064]) reflects the normal condition of the retina (Fig. 1. Paragraph [0068]-BHUYIAN discloses the AMD screening system 100 utilizes machine-learning-based algorithms for extraction of features (e.g., AMD pathologies) and fusion from multimodal imaging and deep convolution neural network. In paragraph [0069]-BHUYIAN discloses FIG. 4 illustrates the components of the AMD suspect screening module 110 described above. At Step 410, CF images (or RF or fused CF and RF images) are segmented for prominent region selection; then elastic registration is applied on the segmented CF images (or RF or fused CF and RF images) to find the corresponding positions of the potential AMD pathology. In paragraph [0071]-BHUYIAN discloses a preprocessing step can be used to generate the normalized image and prominent features. In paragraph [0072]-BHUIYAN discloses deep learning and image analysis are applied (step 470) to identify the normal/healthy individuals, and AMD suspect individuals. As the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”) and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image (Fig. 1. Paragraph [0072]-BHUIYAN discloses as the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”. In paragraph [0073]-BHUYIAN discloses if an individual determined to be not “at risk” for AMD, the individual is advised to return in one year, as shown in FIG. 3 (Step 142). If an individual is determined to be “at risk” for AMD 140, the individual's data is then processed by the AMD incidence prediction module 120, which includes feature and/or longitudinal image analysis, e.g., to be provided to an ophthalmologist for a high confidence diagnosis. Following the analysis, treatment 160 and/or follow up 162 are prescribed (wherein an image is taken for progression analysis 164 when there is not a high risk score)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina; outputting the plurality of disease indicators; outputting the retinal image, with the teachings of BHUYIAN of having at a first time, determining that the retinal image reflects the abnormal condition of the retina and in response to determining that the retinal image reflects the abnormal condition of the retina: processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease; and at a second time, determining that the retinal image reflects the normal condition of the retina and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image.
Wherein SURESH’s non-transitory computer readable storage medium having at a first time, determining that the retinal image reflects the abnormal condition of the retina and in response to determining that the retinal image reflects the abnormal condition of the retina: processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease; and at a second time, determining that the retinal image reflects the normal condition of the retina and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and BHUYIAN concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while BHUYIAN’s provides systems and methods that improve machine learning performance and the automated screening of early and late stage Age-related Macular Degeneration (AMD). Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and BHUYIAN et al. (US 20200242763 A1), Abstract and Paragraph [0063-0068 and 0125].
Regarding claim 7, SURESH in view of BHUYIAN explicitly teaches the non-transitory computer readable storage medium of claim 1, SURESH further teaches wherein each machine learning model of the plurality of second machine learning models (Fig. 3-4, #410, #420 and #304-1-k called a deep learning model and a CNN, respectively. Paragraph [0043]. Further in paragraph [0036]-SURESH discloses the extraction component 145 includes a visualization tool 215, a local feature detection tool 220, and a global feature detection tool 225. In [0037]-SURESH discloses the extraction component 145 processes the spectral information 205 and the spatial information 210 separately using the local feature detection tool 220 and the global feature detection tool 225. The local feature detection tool 220 and the global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform the respective local feature detection and global feature detection. In paragraph [0045]-SURESH discloses the local feature detection tool 220 uses the deep learning model 410 to extract one or more local features 230 from the spectral information 205. In paragraph [0047]-SURESH discloses the global feature detection tool 225 receives spatial information 210 and uses a deep learning model 420 to extract the global features 235 (wherein the visualization tool 215 includes CNNs 304 1-K to process spectral data to generate an interpretative image and the output component 155 may also use a neural network (e.g., 2D or 3D CNN) or a vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235)) includes a disease detection branch and a disease severity branch that is separate from the disease detection branch (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. Each “word” may associate a label (corresponding to a local feature 230) with a description of a retinal disease symptom (e.g., seeing flashes of light, blurry vision, reduced central or peripheral vision, sudden loss of vision, change in color perception) or a particular retinal disease (e.g., retinal tear, retinal detachment, diabetic retinopathy, macular hole, etc.). The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease (wherein the local and global feature detection is performed by two separate machine learning networks, the local feature detection detects diseases and the global feature detection detects either a disease pattern or disease severity)).
Regarding claim 8, SURESH in view of BHUIYAN explicitly teaches the non-transitory computer readable storage medium of claim 7, SURESH further teaches wherein the disease detection branch is configured to determine a disease presence probability for the presence of the disease (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. Each “word” may associate a label (corresponding to a local feature 230) with a description of a retinal disease symptom (e.g., seeing flashes of light, blurry vision, reduced central or peripheral vision, sudden loss of vision, change in color perception) or a particular retinal disease (e.g., retinal tear, retinal detachment, diabetic retinopathy, macular hole, etc.). Further in paragraph [0026]-SURESH discloses the diagnostic output 165 may include information associated with a prediction of at least one disease of the patient's eye 160, based on an analysis of the input data 110. Further in paragraph [0027]-SURESH discloses the severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease), and wherein the disease severity branch is configured to determine a disease severity (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease. Further in paragraph [0027]-SURESH discloses the severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct)).
Regarding claim 15, SURESH in view of BHUIYAN explicitly teaches the non-transitory computer readable storage medium of claim 1, SURESH further teaches wherein outputting the plurality of disease indicators comprises outputting probabilities of the plurality of disease severities (Fig. 1. Paragraph [0027]-SURESH discloses the disease indication(s) 180 is a segmented region(s) of the respective ophthalmic image 175 that is associated with a particular ocular disease (e.g., retinal disease). The severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. The severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct).
Regarding claim 16, SURESH in view of BHUIYAN explicitly teaches the non-transitory computer readable storage medium of claim 15, SURESH further teaches wherein outputting probabilities of the plurality of disease severities comprises outputting an ordered representation of probabilities of the plurality of disease severities (Fig. 1. Paragraph [0027]-SURESH discloses the disease indication(s) 180 is a segmented region(s) of the respective ophthalmic image 175 that is associated with a particular ocular disease (e.g., retinal disease). The severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. The severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct).
Regarding claim 17, SURESH in view of BHUIYAN explicitly teaches the non-transitory computer readable storage medium of claim 15, SURESH further teaches wherein outputting probabilities of the plurality of disease severities comprises outputting a level of severity for a plurality of diseases based on a likelihood or frequency of disease occurrence (Fig. 1. Paragraph [0027]-SURESH discloses the disease indication(s) 180 is a segmented region(s) of the respective ophthalmic image 175 that is associated with a particular ocular disease (e.g., retinal disease). The severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. The severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct).
Regarding claim 18, SURESH explicitly teaches a method for disease detection using retinal images (Fig. 1. Paragraph [0019]-SURESH discloses FIG. 1 illustrates an example ophthalmic system 100 for analyzing multiple spectral information (e.g., MSI/HSI information). The ophthalmic system 100 includes an imaging system 150, a computing system 130, a computing system 190, and a display 170 (wherein a computing system may be a mobile computer, server, etc.). Please also see Fig. 7 and read paragraph [0058-0059]), the method comprising:
processing (Fig. 1. Paragraph [0021]-SURESH discloses the computing system 130 includes a diagnostic tool 135, which is configured for analyzing multiple spectral information for ophthalmology applications. The diagnostic tool 135 includes an analysis component 140, an extraction component 145, and an output component 155. The diagnostic tool 135 receives input data 110 from the imaging system 150 and generates ophthalmic information 120, based on an evaluation of the input data 110 (wherein the diagnostic tool separates input data into spectral and spatial data, and the extraction component #125 contains a visualization tool #215, a local feature detection tool #220 and a global feature detection tool #225)) a retinal image (Fig. 1, #110 called input data. Paragraph [0020]. Please also read paragraph [0021]) of a retina (Fig. 1. Paragraph [0020]-SURESH discloses the imaging system 150 can capture multiple spectral information associated with a target (e.g., tissues/structures of the eye 160)) with a first machine learning model (Fig. 3-4, #410, #420 and #304-1-k called a deep learning model and a CNN, respectively. Paragraph [0043]. In paragraph [0037]-SURESH discloses the extraction component 145 processes the spectral information 205 and the spatial information 210 separately using the local feature detection tool 220 and the global feature detection tool 225. The local feature detection tool 220 and the global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform the respective local feature detection and global feature detection (wherein the visualization tool #215 uses one or more CNNS #304-1-k to process spectral information #205, and deep learning models are each used for local feature extraction, global feature extraction, and generation of a probability map for diseases. Please also read paragraph [0044 and 0048]) to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina (Fig. 4. Paragraph [0024]-SURESH discloses the diagnostic tool 135 receives diagnostic criteria 195. In paragraph [0025]-SURESH discloses the ophthalmic information 120 includes an interpretative image 125 and a diagnostic output 165. In paragraph [0026]-SURESH discloses the diagnostic output 165 may include information associated with a prediction of at least one disease of the patient's eye 160, based on an analysis of the input data 110 (wherein the diagnostic output is based on local and global features). In paragraph [0027]-SURESH discloses the severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease (wherein the severity score is the ocular disease prediction and/or disease severity, and it may be ranked from 0 to 100). Please also read paragraph [0048]);
processing a plurality of disease presence probabilities and a plurality of disease severity probabilities determined by the plurality of second machine learning models with a third machine learning model to determine a plurality of disease indicators based on relationships between a plurality of diseases and their respective severities identified by the plurality of second machine learning models (Fig. 4. Paragraph [0027]-SURESH discloses the severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. In an exemplary embodiment, the severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct). The confidence score may be a percentage), the processing comprising adjusting, for each disease of the plurality of diseases, the disease presence probability and the disease severity probability based on a disease presence probability and a disease severity probability for at least one different disease of the plurality of diseases (Fig. 4. Paragraph [0039]-SURESH discloses the output component 155 receives the local features 230 and the global features 235 and generates a diagnostic output 165. The output component 155 concatenates the local features 230 and global features 235 to obtain at least one ophthalmic image 175 with one or more disease indications 180. The at least one ophthalmic image 175 may be a retinal image, and the one or more disease indications 180 may include one or more segmented disease regions within the retinal image. Further in paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease. The output component 155 can use a neural network (e.g., 2D or 3D CNN) or a vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235. Please also read paragraph [0027-0028, 0036, 0044-0047]);
outputting the plurality of disease indicators (Fig. 4, #180 called disease indications. Paragraph [0027 and 0039]. In paragraph [0039]-SURESH discloses the output component 155 receives the local features 230 and the global features 235 and generates a diagnostic output 165. The output component 155 concatenates the local features 230 and global features 235 to obtain at least one ophthalmic image 175 with one or more disease indications 180. The at least one ophthalmic image 175 may be a retinal image, and the one or more disease indications 180 may include one or more segmented disease regions within the retinal image. Please also read paragraph [0026-0028]); and
outputting the retinal image (Fig. 4, #175 and #125 called an ophthalmic image and an interpretative image. Paragraph [0028 and 0039]. In paragraph [0028]-SURESH discloses in an exemplary diagnostic output 165, a first ophthalmic image 175-1 may have a first disease indication 180-1 of a first retinal disease (e.g., one or more segmented regions in the first ophthalmic image 175-1 that indicate the first retinal disease) and a severity score 185-1 for the first retinal disease based on the first disease indication 180-1; a second ophthalmic image 175-2 may have a second disease indication 180-2 of a second retinal disease (e.g., one or more segmented regions in the second ophthalmic image 175-2 that indicate the second retinal disease) and a severity score 185-2 for the second retinal disease based on the second disease indication 180-2; and so on. In paragraph [0039]-SURESH discloses the output component 155 may provide the interpretative image 125 (received from the visualization tool 215) to a user (e.g., via imaging system 150 and/or display 170). The interpretative image 125 may enable the user to interpret the multiple spectral information obtained from the imaging system 150 in a more effective and/or easier manner when making a disease diagnosis); and
Although SURESH explicitly teaches processing the retinal image by a plurality of second machine learning models (Fig. 3-4, #410, #420 and #304-1-k called a deep learning model and a CNN, respectively. Paragraph [0043]. In paragraph [0037]-SURESH discloses the extraction component 145 processes the spectral information 205 and the spatial information 210 separately using the local feature detection tool 220 and the global feature detection tool 225. The local feature detection tool 220 and the global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform the respective local feature detection and global feature detection (wherein the visualization tool #215 uses one or more CNNS #304-1-k to process spectral information #205, and deep learning models are each used for local feature extraction, global feature extraction, and generation of a probability map for diseases. Please also read paragraph [0044 and 0048]), wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease and 2) a disease severity probability for the disease (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data. Each “word” may associate a label (corresponding to a local feature 230) with a description of a retinal disease symptom (e.g., seeing flashes of light, blurry vision, reduced central or peripheral vision, sudden loss of vision, change in color perception) or a particular retinal disease (e.g., retinal tear, retinal detachment, diabetic retinopathy, macular hole, etc.). The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection. The global features 235 may be used to determine that a group of “words” (that collectively indicate a set of retinal disease symptoms) is indicative of a disease diagnosis, which includes at least one of: (i) a particular retinal disease or (ii) a severity of the retinal disease);
SURESH fail to teach at a first time, determining that the retinal image reflects the abnormal condition of the retina and in response to determining that the retinal image reflects the abnormal condition of the retina: processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease; and at a second time, determining that the retinal image reflects the normal condition of the retina and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image.
However, BHUIYAN explicitly teaches at a first time, determining that the retinal image (Fig. 3, #132 and #134 called a red-free fundus image and a color fundus image, respectively. Paragraph [0064]-BHUIYAN discloses a multimodal color Fundus (CF) and red-free (RF) image analysis platform for AMD screening and prediction system is illustrated in FIG. 3. The system includes two modules. First, a module for automated screening of individuals for early-stage AMD 110 is provided (see also, FIG. 4). Second, a module for prediction of individuals at risk of developing late AMD in the near future 120 is provided. In paragraph [0066]-BHUIYAN discloses the screening module 110 receives patient data 130. Patient data 130 includes, e.g., red-free (RF) fundus data 132, color fundus (CF) data 134, and socio-demographic data 136. In paragraph [0068]-BHUIYAN discloses the AMD screening system 100 utilizes machine-learning-based algorithms for extraction of features (e.g., AMD pathologies) and fusion from multimodal imaging and deep convolution neural network (wherein the system may preprocess, segment, select and/or fuse RF and CF features to generate normalized and prominent features with potential for AMD pathology)) reflects the abnormal condition of the retina (Fig. 1. Paragraph [0072]-BHUIYAN discloses Deep learning and image analysis are applied (step 470) to identify the normal/healthy individuals, and AMD suspect individuals. As the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”) and in response to determining that the retinal image reflects the abnormal condition of the retina (Fig. 5. Paragraph [0073]-BHUYIAN discloses if an individual determined to be not “at risk” for AMD, the individual is advised to return in one year, as shown in FIG. 3 (Step 142). If an individual is determined to be “at risk” for AMD 140, the individual's data is then processed by the AMD incidence prediction module 120, which includes feature and/or longitudinal image analysis, e.g., to be provided to an ophthalmologist for a high confidence diagnosis. Following the analysis, treatment 160 and/or follow up 162 are prescribed. Further, an image is taken for progression analysis 164. In paragraph [0074]-BHUIYAN discloses once an individual is identified as “at risk” by the AMD Suspect screening module 110, a prediction score 144 for the individual for developing late AMD in near future is computed by the AMD incidence prediction module 120. The pathology quantification is performed utilizing the CF images 134 and RF images 132 to find the prediction score of that individual for developing late AMD):
processing the retinal image by a plurality of second machine learning models (Fig. 5. Paragraph [0075]-BHUIYAN discloses a deep learning model is used to predict the individual at risk of AMD. In paragraph [0078]-BHUYIAN discloses a method for drusen quantification during AMD incidence prediction 120 is illustrated in FIG. 8. The quantification provides “early,” “intermediate,” or “late” AMD stage classification along with categories 1-9 (wherein a machine-learning method and graph-based method are applied for drusen quantification)), wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease (Fig. 5. Paragraph [0134]-BHUIYAN discloses scoring scale is developed considering four pathologies, e.g., drusen area, and three pigment abnormalities (Increased Pigmentation, Depigmentation, and Geographic Atrophy). Deep learning-based classification system has been built for each of the four pathologies. Further in paragraph [0133]-BHUIYAN discloses for AMD severity level categorization and fuzzy weighted score generation, the categorized AMD severity levels are: No AMD (severity level 1), Early AMD (severity level 2), Intermediate AMD (severity level 3) and Advanced AMD (severity level 4). Then the probability for each severity levels is generated. In paragraph [0135]-BHUIYAN discloses combining the result of the four classifiers and using the same protocol as AREDS, nine probabilities for an image (for an eye) are obtained. The nine probabilities of eye1 are referred to as: a, b, c, d, e, f, g, h, and i. In paragraph [0136]-BHUIYAN discloses these are the probabilities that an image falls in the particular severity scale level 1-9); and
at a second time, determining that the retinal image (Fig. 3, #132 and #134 called a red-free fundus image and a color fundus image, respectively. Paragraph [0064]) reflects the normal condition of the retina (Fig. 1. Paragraph [0068]-BHUYIAN discloses the AMD screening system 100 utilizes machine-learning-based algorithms for extraction of features (e.g., AMD pathologies) and fusion from multimodal imaging and deep convolution neural network. In paragraph [0069]-BHUYIAN discloses FIG. 4 illustrates the components of the AMD suspect screening module 110 described above. At Step 410, CF images (or RF or fused CF and RF images) are segmented for prominent region selection; then elastic registration is applied on the segmented CF images (or RF or fused CF and RF images) to find the corresponding positions of the potential AMD pathology. In paragraph [0071]-BHUYIAN discloses a preprocessing step can be used to generate the normalized image and prominent features. In paragraph [0072]-BHUIYAN discloses deep learning and image analysis are applied (step 470) to identify the normal/healthy individuals, and AMD suspect individuals. As the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”) and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image, wherein the method is performed by one or more processors (Fig. 1. Paragraph [0072]-BHUIYAN discloses as the output of the analysis (480), an individual is determined to be either “at risk” or not “at risk.”. In paragraph [0073]-BHUYIAN discloses if an individual determined to be not “at risk” for AMD, the individual is advised to return in one year, as shown in FIG. 3 (Step 142). If an individual is determined to be “at risk” for AMD 140, the individual's data is then processed by the AMD incidence prediction module 120, which includes feature and/or longitudinal image analysis, e.g., to be provided to an ophthalmologist for a high confidence diagnosis. Following the analysis, treatment 160 and/or follow up 162 are prescribed (wherein an image is taken for progression analysis 164 when there is not a high risk score)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH of having a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina; processing a plurality of disease presences and a plurality of disease severities determined by the plurality of second machine learning models with a third machine learning model to determine a plurality of disease indicators based on relationships between a plurality of diseases and their respective severities identified by the plurality of second machine learning models; outputting the plurality of disease indicators; outputting the retinal image, with the teachings of BHUYIAN of having at a first time, determining that the retinal image reflects the abnormal condition of the retina and in response to determining that the retinal image reflects the abnormal condition of the retina: processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease; and at a second time, determining that the retinal image reflects the normal condition of the retina and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image, wherein the method is performed by one or more processors.
Wherein SURESH’s method having at a first time, determining that the retinal image reflects the abnormal condition of the retina and in response to determining that the retinal image reflects the abnormal condition of the retina: processing the retinal image by a plurality of second machine learning models, wherein each machine learning model of the plurality of second machine learning models is configured to output 1) a disease presence probability for a disease different from any other machine learning model of the plurality of second machine learning models and 2) a disease severity probability for the disease; and at a second time, determining that the retinal image reflects the normal condition of the retina and in response to determining that the retinal image reflects the normal condition of the retina, outputting the retinal image, wherein the method is performed by one or more processors.
The motivation behind the modification would have been to obtain a method that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and BHUYIAN concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while BHUYIAN’s provides systems and methods that improve machine learning performance and the automated screening of early and late stage Age-related Macular Degeneration (AMD). Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and BHUYIAN et al. (US 20200242763 A1), Abstract and Paragraph [0063-0068 and 0125].
Claims 2-3 and 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over SURESH et al. (US 20240032784 A1), hereinafter referenced as SURESH in view of BHUIYAN et al. (US 20200242763 A1), hereinafter referenced as BHUIYAN and in further view of PARK et al. (US 20220189012 A1), hereinafter referenced as PARK.
Regarding claim 2, SURESH in view of BHUYIAN explicitly teaches the non-transitory computer readable storage medium of claim 1, SURESH fails to explicitly teach wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern, and wherein the third machine learning model comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern.
However, PARK explicitly teaches wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern (Fig. 3. Paragraph [0062]-PARK discloses a new result interpretation method reflecting the hierarchical characteristic of medical terms is used to train the disease inference unit 210. Most diseases including eye diseases have a hierarchical structure and are exclusive at a lower level. That is, they have a characteristic of being overlapped at a parent node and exclusive at a child node. Being overlapped means that a person may have several diseases at the same time, and classification of diseases is hierarchical, and sub-classification of a diagnosed disease is exclusive. That is, labeling by learning data of the disease inference unit 210 is performed based on an overlapping characteristic indicating that a person may have several diseases at the same time, a hierarchical characteristic indicating that the category of a disease is hierarchically classified, and exclusiveness indicating that sub-classifications of a diagnosed lesion are mutually exclusive. Please also read paragraph [0063-0066]), and wherein the third machine learning model (Fig. 2, called GLEM. Paragraph [0039]-PARK discloses as shown in Fig. 2, the present invention is largely configured of a trunk module 100, a branch module 200, and a final diagnosis unit 300, and the branch module 200 includes a disease inference unit 210, a location search unit 220, a key lesion finder 230, and a small-sized lesion finder 240. Hereinafter, a deep learning architecture system for automatic fundus image reading of the present invention is named as Grem (wherein FIG. 1 depicts HydraNet, which is another view of the basic deep learning architecture of GREM). In paragraph [0041]-PARK discloses the trunk module 100 is an architecture that combines common parts in a plurality of convolutional neural network (CNN) architectures into one part (wherein the trunk module 100 is a common layer for extracting features of a fundus image using a convolutional neural network (CNN)). In paragraph [0055]-PARK discloses the disease inference unit 210 performs a function of inferring a disease category by looking at the entire picture) comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern (Fig. 3. Paragraph [0068]-PARK discloses the location search unit 220 is a layer for finding the optic nerve head (ONH), classifying a blind spot ratio, and searching for locations of the optic nerve disc and the macula. In paragraph [0071]-PARK discloses the key lesion finder 230 finds a key lesion, which is a component constituting a disease. In paragraph [0078]-PARK discloses the small-sized lesion finder 240 is a branch designed to separately detect a very small but very important lesion from a fundus image. In paragraph [0080]-PARK discloses the Glem architecture of the present invention is based on HydraNet having four branch modules 200, and it is a layer that determines and outputs a final diagnosis name by integrating the outputs of the four branch modules 200. This layer uses a random forest (RF) algorithm for each disease name. That is, it is a structure in which there is one RF having existence 1 or non-existence 0 of a disease as an output for each of N diseases. Please also read paragraph [0063-0066]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of PARK of having wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern, and wherein the third machine learning model comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern, and wherein the third machine learning model comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and PARK concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while PARK’s provides automatic fundus image reading, and a deep learning architecture for automatic fundus image reading, which are capable of minimizing the amount of data required for learning by training and reading artificial intelligence in a manner similar to that of an ophthalmologist who acquires medical knowledge. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and PARK et al. (US 20220189012 A1), Abstract and Paragraph [0035-0043].
Regarding claim 3, SURESH in view of BHUYIAN and in further view of PARK explicitly teach the non-transitory computer readable storage medium of claim 2, SURESH in view of PARK fails to explicitly teach wherein the graph-based machine learning model comprises conditional random fields.
However, BHUIYAN explicitly teaches wherein the graph-based machine learning model (Fig. 1. Paragraph [0078]-BHUIYAN discloses a method for drusen quantification during AMD incidence prediction 120 is illustrated in FIG. 8. The quantification provides “early,” “intermediate,” or “late” AMD stage classification along with categories 1-9 discussed herein. A machine-learning method 804 and a graph-based method 806 for drusen quantification are applied to the retinal color and red-free image data 802, which are then merged to determine the drusen regions (Step 808)) comprises conditional random fields (Fig. 1. Paragraph [0122]-BHUIYAN discloses the support vector machine is utilized with a conditional random field to find the individuals as AMD suspect. In paragraph [0132]-BHUIYAN discloses in the deep ConvNet model, following the feature optimization by PCA—the dense layer is utilized to perform the final classification. The dense layer can be replaced with a Conditional Random Field to predict the individual at risk of developing late AMD).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN and in further view of PARK of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of BHUYIAN of having wherein the graph-based machine learning model comprises conditional random fields.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the graph-based machine learning model comprises conditional random fields.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and BHUYIAN concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while BHUYIAN’s provides systems and methods that improve machine learning performance and the automated screening of early and late stage Age-related Macular Degeneration (AMD). Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and BHUYIAN et al. (US 20200242763 A1), Abstract and Paragraph [0063-0068 and 0125].
Regarding claim 5, SURESH in view of BHUIYAN explicitly teaches the non-transitory computer readable storage medium of claim 1, SURESH is silent on wherein the first machine learning model comprises a binary classification convolutional neural network configured to differentiate between abnormal and normal conditions of the retina.
However, PARK explicitly teaches wherein the first machine learning model (Fig. 2. Paragraph [0039]-PARK discloses as shown in Fig. 2, the present invention is largely configured of a trunk module 100, a branch module 200, and a final diagnosis unit 300, and the branch module 200 includes a disease inference unit 210, a location search unit 220, a key lesion finder 230, and a small-sized lesion finder 240. Hereinafter, a deep learning architecture system for automatic fundus image reading of the present invention is named as Grem (wherein FIG. 1 depicts HydraNet, which is another view of the basic deep learning architecture of GREM). In paragraph [0041]-PARK discloses the trunk module 100 is an architecture that combines common parts in a plurality of convolutional neural network (CNN) architectures into one part (wherein the trunk module 100 is a common layer for extracting features of a fundus image using a convolutional neural network (CNN)). In paragraph [0055]-PARK discloses the disease inference unit 210 performs a function of inferring a disease category by looking at the entire picture) comprises a binary classification convolutional neural network configured to differentiate between abnormal and normal conditions of the retina (Fig. 1. Paragraph [0058]-PARK discloses the activation function of the last Dense2 layer uses a sigmoid function to independently score a value between 0 and 1 for each disease category. Since it is not guaranteed that a person has only one disease, scoring is independently performed using a value between 0 and 1. The number of output values is N.sub.1 in the Dense2 layer).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of PARK of having wherein the first machine learning model comprises a binary classification convolutional neural network configured to differentiate between abnormal and normal conditions of the retina.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the first machine learning model comprises a binary classification convolutional neural network configured to differentiate between abnormal and normal conditions of the retina.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and PARK concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while PARK’s provides automatic fundus image reading, and a deep learning architecture for automatic fundus image reading, which are capable of minimizing the amount of data required for learning by training and reading artificial intelligence in a manner similar to that of an ophthalmologist who acquires medical knowledge. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and PARK et al. (US 20220189012 A1), Abstract and Paragraph [0035-0043].
Regarding claim 6, SURESH in view of BHUYIAN explicitly teaches the non-transitory computer readable storage medium of claim 1, SURESH fails to explicitly teach wherein the plurality of second machine learning models comprises a plurality of multi-branch disease-specific convolutional neural networks.
However, PARK explicitly teaches wherein the plurality of second machine learning models (Fig. 2. Paragraph [0039]-PARK discloses as shown in Fig. 2, the present invention is largely configured of a trunk module 100, a branch module 200, and a final diagnosis unit 300, and the branch module 200 includes a disease inference unit 210, a location search unit 220, a key lesion finder 230, and a small-sized lesion finder 240. Hereinafter, a deep learning architecture system for automatic fundus image reading of the present invention is named as Grem (wherein FIG. 1 depicts HydraNet, which is another view of the basic deep learning architecture of GREM). In paragraph [0041]-PARK discloses the trunk module 100 is an architecture that combines common parts in a plurality of convolutional neural network (CNN) architectures into one part (wherein the trunk module 100 is a common layer for extracting features of a fundus image using a convolutional neural network (CNN)). In paragraph [0055]-PARK discloses the disease inference unit 210 performs a function of inferring a disease category by looking at the entire picture) comprises a plurality of multi-branch disease-specific convolutional neural networks (Fig. 2, called GREM. Paragraph [0045]-PARK discloses a section 110 is an architecture that connects any one branch module 200 among a plurality of branch modules 200 to the trunk module 100. One branch module 200 and the trunk module 100 are combined to form one section 110 for each disease).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of PARK of having wherein the plurality of second machine learning models comprises a plurality of multi-branch disease-specific convolutional neural networks.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the plurality of second machine learning models comprises a plurality of multi-branch disease-specific convolutional neural networks.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and PARK concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while PARK’s provides automatic fundus image reading, and a deep learning architecture for automatic fundus image reading, which are capable of minimizing the amount of data required for learning by training and reading artificial intelligence in a manner similar to that of an ophthalmologist who acquires medical knowledge. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and PARK et al. (US 20220189012 A1), Abstract and Paragraph [0035-0043].
Claims 4 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over SURESH et al. (US 20240032784 A1), hereinafter referenced as SURESH in view of BHUIYAN et al. (US 20200242763 A1), hereinafter referenced as BHUIYAN and in further view of PARK et al. (US 20220189012 A1), hereinafter referenced as PARK and in further view of O’CONNOR et al. (US 20190286661 A1), hereinafter referenced as O’CONNOR.
Regarding claim 4, SURESH in view of BHUYIAN and in further view of PARK explicitly teach the non-transitory computer readable storage medium of claim 2, SURESH fails to explicitly teach wherein the plurality of disease indicators is determined by: generating a plurality of nodes, wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseasesdisease severity
However, PARK explicitly teaches wherein the plurality of disease indicators is determined (Fig. 2. Paragraph [0039]-PARK discloses as shown in FIG. 2, the present invention is largely configured of a trunk module 100, a branch module 200, and a final diagnosis unit 300. Hereinafter, a deep learning architecture system for automatic fundus image reading of the present invention is named as Grem (wherein one branch module 200 and the trunk module 100 are combined to form one section 110 for each disease, and the branch module 200 includes a disease inference unit 210, a location search unit 220, a key lesion finder 230, and a small-sized lesion finder 240). In paragraph [0080]-PARK discloses Glem architecture is based on HydraNet having four branch modules 200, and it is a layer that determines and outputs a final diagnosis name by integrating the outputs of the four branch modules 200 (wherein the output is an existence 1 or non-existence 0 for N diseases)) by:
generating a plurality of nodes (Fig. 3. Paragraph [0062]-PARK discloses a new result interpretation method reflecting the hierarchical characteristic of medical terms is used to train the disease inference unit 210. Most diseases including eye diseases have a hierarchical structure and are exclusive at a lower level. That is, they have a characteristic of being overlapped at a parent node and exclusive at a child node. Being overlapped means that a person may have several diseases at the same time, and classification of diseases is hierarchical, and sub-classification of a diagnosed disease is exclusive. That is, labeling by learning data of the disease inference unit 210 is performed based on an overlapping characteristic indicating that a person may have several diseases at the same time, a hierarchical characteristic indicating that the category of a disease is hierarchically classified, and exclusiveness indicating that sub-classifications of a diagnosed lesion are mutually exclusive), wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseasesdisease severity (Fig. 3. Paragraph [0055]-PARK discloses the disease inference unit 210 performs a function of inferring a disease category. In paragraph [0057]-PARK discloses N.sub.1 is the number of diseases in the disease inference unit 210) (wherein the node. In paragraph [0058]-PARK discloses the activation function of the last Dense2 layer uses a sigmoid function to independently score a value between 0 and 1 for each disease category. Since it is not guaranteed that a person has only one disease, scoring is independently performed using a value between 0 and 1. The number of output values is N.sub.1 in the Dense2 layer. Please also read paragraph [0061 and 0063-0066]);
determining one or more neighboring nodes for each node of the plurality of nodes based on the disease co-occurrence pattern (Fig. 3. Paragraph [0065]-PARK discloses the number of outputs of the disease inference unit 210 of the present invention is equal to the number of final child nodes. The number of outputs of the last Dense2 layer of the disease inference unit 210 is 6 of final child nodes (N3, N5, N6, N7, N9, N10) shown in green. In the doctor's charting (ground truth), there are cases in which only parent nodes (N2, N8) are classified according to the degree of confidence in diagnosis, or when it is further certain, there are cases in which even the sub-classifications are completed and diagnose is performed below the level. Only parent nodes (N2, N8) may be classified according to the degree of confidence in diagnosis, or when it is further certain, there are cases in which even the sub-classifications are completed and diagnose is performed below the level. Please also read paragraph [0061-0064 and 0066]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of PARK of having wherein the plurality of disease indicators is determined by: generating a plurality of nodes, wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseases and a plurality of disease severity probabilities for the plurality of diseases; determining one or more neighboring nodes for each node of the plurality of nodes based on the disease co-occurrence pattern.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the plurality of disease indicators is determined by: generating a plurality of nodes, wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseases and a plurality of disease severity probabilities for the plurality of diseases; determining one or more neighboring nodes for each node of the plurality of nodes based on the disease co-occurrence pattern.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and PARK concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while PARK’s provides automatic fundus image reading, and a deep learning architecture for automatic fundus image reading, which are capable of minimizing the amount of data required for learning by training and reading artificial intelligence in a manner similar to that of an ophthalmologist who acquires medical knowledge. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and PARK et al. (US 20220189012 A1), Abstract and Paragraph [0035-0043].
SURESH in view of BHUYIAN and in further view of PARK fail to explicitly teach and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes, the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities.
However, O’CONNOR explicitly teaches and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases (Fig. 1. Paragraph [0046]-O’CONNOR discloses a node mapping sub-system 150 can perform a targeted or general query pertaining to a graph model to determine whether the graph model includes a node having a criteria-group constraint corresponding to the detected constraint. In paragraph [0048]-O’CONNOR discloses a node generation sub-system 160 can generate a new node that corresponds with a criteria group including the detected constraint. Further edge generation sub-system 155 can generate a plurality of edges, each of which can be configured such that one end of the edge connects to the new node and another end of the edge connects to another node (e.g., another new node or an existing node in the graph model). Node generation sub-system 160 generates multiple new nodes to represent the detected constraint, and edge generation sub-system 155 can generate a plurality of nodes to integrate the multiple new nodes into the graph model. A constraint may indicate that a criteria group is satisfied when a given attribute is equal to any of three specified values (wherein a constraint may be represented by one or more nodes)) based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes (Fig. 1. Paragraph [0040]-O’CONNOR discloses the one or more starting nodes may correspond to (for example) a disease diagnosis or a symptom occurrence and/or one or more other factors (e.g., procedure history, demographic characteristic, family history, genetic-mutation characteristic, etc) (wherein nodes and/or criteria may be a stage or disease progression). In paragraph [0041]-O’CONNOR discloses edges and nodes connected to the starting node can be evaluated to determine whether the entity matches a corresponding particular inclusion criterion, avoids a corresponding particular exclusion criterion and/or meets a given combination of multiple criteria as defined by logical operator (wherein for each branch, criteria represented along the branch can be iteratively (e.g., and conditionally) evaluated and a weighted score can be generated based on the criteria/criterion being satisfied, the probability that a criteria/criterion is satisfied, the extent to which a criteria/criterion, an estimate of the criteria/criterion being satisfied, the extent to which the criteria/criterion was not completely evaluated due to a lack of pertinent entity data points). Please also read paragraph [0031, 0043, 0046-0056 and 0104]), the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities (Fig. 1. Paragraph [0056]-O’CONNOR discloses the data imputation sub-system 180 can assess trajectories that intersect individual nodes to update each of one, more or all nodes in a graph model to be associated with a (e.g., new or updated) probability that reflects a likelihood that (for example) satisfaction of a criteria group associated with the node is predictive of one or both of: reaching an end node that terminates a path along which the node is positioned and being selected for an investigatory event associated with an end node that terminates a path along which the node is positioned. Further data imputation sub-system 180 can assess trajectories that extend along paths that intersect individual nodes to update each of one, more or all nodes in a graph model to be associated with a (e.g., new or updated) probability that reflects a likelihood that (for example) one or more attribute-value characterizations and/or one or more results of assessment of criteria group(s) of one or more nodes positioned closer to a starting node along a path is predictive of a result of an evaluation of a criteria group of the node. Thus, the graph model may be a probabilistic graph model. Please also read paragraph [0024 and 0037]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN and in further view of PARK of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of O’CONNOR of having and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes, the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities.
Wherein SURESH’s non-transitory computer readable storage medium having and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes, the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the detection, identification and visualization of eye-related diseases, since both SURESH and O’CONNOR concern machine learning and the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while O’CONNOR’s provides methods and systems disclosed that improve identification of diseases by using graph structures and parent-child node configurations, which facilitate data integration across corresponding investigatory events. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and O’CONNOR et al. (US 20190286661 A1), Abstract and Paragraph [0003-0005, 0031, 0037 and 0048-0051].
Regarding claim 19, SURESH in view of BHUYIAN explicitly teaches the method of claim 18, SURESH fails to explicitly teach wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern, wherein the third machine learning model comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern, and wherein the plurality of disease indicators is determined by: generating a plurality of nodes, wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseases and a plurality of disease severity probabilities for the plurality of diseases; determining one or more neighboring nodes for each node of the plurality of nodes based on the disease co-occurrence pattern.
However, PARK explicitly teaches wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern (Fig. 3. Paragraph [0062]-PARK discloses a new result interpretation method reflecting the hierarchical characteristic of medical terms is used to train the disease inference unit 210. Most diseases including eye diseases have a hierarchical structure and are exclusive at a lower level. That is, they have a characteristic of being overlapped at a parent node and exclusive at a child node. Being overlapped means that a person may have several diseases at the same time, and classification of diseases is hierarchical, and sub-classification of a diagnosed disease is exclusive. That is, labeling by learning data of the disease inference unit 210 is performed based on an overlapping characteristic indicating that a person may have several diseases at the same time, a hierarchical characteristic indicating that the category of a disease is hierarchically classified, and exclusiveness indicating that sub-classifications of a diagnosed lesion are mutually exclusive. Please also read paragraph [0063-0066]), wherein the third machine learning model (Fig. 2, called GLEM. Paragraph [0039]-PARK discloses as shown in Fig. 2, the present invention is largely configured of a trunk module 100, a branch module 200, and a final diagnosis unit 300, and the branch module 200 includes a disease inference unit 210, a location search unit 220, a key lesion finder 230, and a small-sized lesion finder 240. Hereinafter, a deep learning architecture system for automatic fundus image reading of the present invention is named as Grem (wherein FIG. 1 depicts HydraNet, which is another view of the basic deep learning architecture of GREM). In paragraph [0041]-PARK discloses the trunk module 100 is an architecture that combines common parts in a plurality of convolutional neural network (CNN) architectures into one part (wherein the trunk module 100 is a common layer for extracting features of a fundus image using a convolutional neural network (CNN)). In paragraph [0055]-PARK discloses the disease inference unit 210 performs a function of inferring a disease category by looking at the entire picture) comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern, and wherein the plurality of disease indicators is determined (Fig. 3. Paragraph [0068]-PARK discloses the location search unit 220 is a layer for finding the optic nerve head (ONH), classifying a blind spot ratio, and searching for locations of the optic nerve disc and the macula. In paragraph [0071]-PARK discloses the key lesion finder 230 finds a key lesion, which is a component constituting a disease. In paragraph [0078]-PARK discloses the small-sized lesion finder 240 is a branch designed to separately detect a very small but very important lesion from a fundus image. In paragraph [0080]-PARK discloses the Glem architecture of the present invention is based on HydraNet having four branch modules 200, and it is a layer that determines and outputs a final diagnosis name by integrating the outputs of the four branch modules 200. This layer uses a random forest (RF) algorithm for each disease name. That is, it is a structure in which there is one RF having existence 1 or non-existence 0 of a disease as an output for each of N diseases. Please also read paragraph [0063-0066]) by:
generating a plurality of nodes, wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseases and a plurality of disease severity probabilities for the plurality of diseases (Fig. 3. Paragraph [0055]-PARK discloses the disease inference unit 210 performs a function of inferring a disease category. In paragraph [0057]-PARK discloses N.sub.1 is the number of diseases in the disease inference unit 210) (wherein N.sub.X represents nodes in a graph structure and each node is defined by a disease category and/or stage of disease)). In paragraph [0058]-PARK discloses the activation function of the last Dense2 layer uses a sigmoid function to independently score a value between 0 and 1 for each disease category. Since it is not guaranteed that a person has only one disease, scoring is independently performed using a value between 0 and 1. The number of output values is N.sub.1 in the Dense2 layer. Please also read paragraph [0061 and 0063-0066]);
determining one or more neighboring nodes for each node of the plurality of nodes based on the disease co-occurrence pattern (Fig. 3. Paragraph [0065]-PARK discloses the number of outputs of the disease inference unit 210 of the present invention is equal to the number of final child nodes. The number of outputs of the last Dense2 layer of the disease inference unit 210 is 6 of final child nodes (N3, N5, N6, N7, N9, N10) shown in green. In the doctor's charting (ground truth), there are cases in which only parent nodes (N2, N8) are classified according to the degree of confidence in diagnosis, or when it is further certain, there are cases in which even the sub-classifications are completed and diagnose is performed below the level. Only parent nodes (N2, N8) may be classified according to the degree of confidence in diagnosis, or when it is further certain, there are cases in which even the sub-classifications are completed and diagnose is performed below the level. Please also read paragraph [0061-0064 and 0066]);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of having a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of PARK of having wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern, wherein the third machine learning model comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern, and wherein the plurality of disease indicators is determined by: generating a plurality of nodes, wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseases and a plurality of disease severity probabilities for the plurality of diseases; determining one or more neighboring nodes for each node of the plurality of nodes based on the disease co-occurrence pattern.
Wherein SURESH’s method having wherein the relationships between the plurality of diseases comprises a disease co-occurrence pattern, wherein the third machine learning model comprises a graph-based machine learning model configured to determine the plurality of disease indicators using the disease co-occurrence pattern, and wherein the plurality of disease indicators is determined by: generating a plurality of nodes, wherein each node of the plurality of nodes is based on a plurality of disease presence probabilities for the plurality of diseases and a plurality of disease severity probabilities for the plurality of diseases; determining one or more neighboring nodes for each node of the plurality of nodes based on the disease co-occurrence pattern.
The motivation behind the modification would have been to obtain a method that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and PARK concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while PARK’s provides automatic fundus image reading, and a deep learning architecture for automatic fundus image reading, which are capable of minimizing the amount of data required for learning by training and reading artificial intelligence in a manner similar to that of an ophthalmologist who acquires medical knowledge. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and PARK et al. (US 20220189012 A1), Abstract and Paragraph [0035-0043].
SURESH in view of BHUYIAN and in further view of PARK fail to explicitly teach and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes, the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities.
However, O’CONNOR explicitly teaches and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases (Fig. 1. Paragraph [0046]-O’CONNOR discloses a node mapping sub-system 150 can perform a targeted or general query pertaining to a graph model to determine whether the graph model includes a node having a criteria-group constraint corresponding to the detected constraint. In paragraph [0048]-O’CONNOR discloses a node generation sub-system 160 can generate a new node that corresponds with a criteria group including the detected constraint. Further edge generation sub-system 155 can generate a plurality of edges, each of which can be configured such that one end of the edge connects to the new node and another end of the edge connects to another node (e.g., another new node or an existing node in the graph model). Node generation sub-system 160 generates multiple new nodes to represent the detected constraint, and edge generation sub-system 155 can generate a plurality of nodes to integrate the multiple new nodes into the graph model. A constraint may indicate that a criteria group is satisfied when a given attribute is equal to any of three specified values (wherein a constraint may be represented by one or more nodes)) based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes (Fig. 1. Paragraph [0040]-O’CONNOR discloses the one or more starting nodes may correspond to (for example) a disease diagnosis or a symptom occurrence and/or one or more other factors (e.g., procedure history, demographic characteristic, family history, genetic-mutation characteristic, etc) (wherein nodes and/or criteria may be a stage or disease progression). In paragraph [0041]-O’CONNOR discloses edges and nodes connected to the starting node can be evaluated to determine whether the entity matches a corresponding particular inclusion criterion, avoids a corresponding particular exclusion criterion and/or meets a given combination of multiple criteria as defined by logical operator (wherein for each branch, criteria represented along the branch can be iteratively (e.g., and conditionally) evaluated and a weighted score can be generated based on the criteria/criterion being satisfied, the probability that a criteria/criterion is satisfied, the extent to which a criteria/criterion, an estimate of the criteria/criterion being satisfied, the extent to which the criteria/criterion was not completely evaluated due to a lack of pertinent entity data points). Please also read paragraph [0031, 0043, 0046-0056 and 0104]), the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities (Fig. 1. Paragraph [0056]-O’CONNOR discloses the data imputation sub-system 180 can assess trajectories that intersect individual nodes to update each of one, more or all nodes in a graph model to be associated with a (e.g., new or updated) probability that reflects a likelihood that (for example) satisfaction of a criteria group associated with the node is predictive of one or both of: reaching an end node that terminates a path along which the node is positioned and being selected for an investigatory event associated with an end node that terminates a path along which the node is positioned. Further data imputation sub-system 180 can assess trajectories that extend along paths that intersect individual nodes to update each of one, more or all nodes in a graph model to be associated with a (e.g., new or updated) probability that reflects a likelihood that (for example) one or more attribute-value characterizations and/or one or more results of assessment of criteria group(s) of one or more nodes positioned closer to a starting node along a path is predictive of a result of an evaluation of a criteria group of the node. Thus, the graph model may be a probabilistic graph model. Please also read paragraph [0024 and 0037]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN and in further view of PARK of having a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of O’CONNOR of having and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes, the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities.
Wherein SURESH’s method having and adjusting a disease presence probability and a disease severity probability for each disease of the plurality of diseases based on the plurality of disease presence probabilities and the plurality of disease severity probabilities for neighboring nodes, the plurality of disease indicators comprising the adjusted disease presence probabilities and the adjusted disease severity probabilities.
The motivation behind the modification would have been to obtain a method that improves the detection, identification and visualization of eye-related diseases, since both SURESH and O’CONNOR concern machine learning and the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while O’CONNOR’s provides methods and systems disclosed that improve identification of diseases by using graph structures and parent-child node configurations, which facilitate data integration across corresponding investigatory events. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and O’CONNOR et al. (US 20190286661 A1), Abstract and Paragraph [0003-0005, 0031, 0037 and 0048-0051].
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over SURESH et al. (US 20240032784 A1), hereinafter referenced as SURESH in view of BHUIYAN et al. (US 20200242763 A1), hereinafter referenced as BHUIYAN and in further view of SEHANOBISH et al. (US 20230161978 A1), hereinafter referenced as SEHANOBISH.
Regarding claim 9, SURESH in view of BHUIYAN explicitly teaches the non-transitory computer readable storage medium of claim 7, SURESH fails to explicitly teach wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
However, SEHANOBISH explicitly teaches wherein the disease detection branch (Fig. 3, #340 and #360 called a multi-task NER engine and deep learning classifier, respectively. Paragraph [0083]) and the disease severity branch (Fig. 3, #340 called a multi-task NER engine. Paragraph [0025]-SEHANOBISH discloses a deep learning architecture may learn a hierarchy of features. In paragraph [0083]-SEHANOBISH discloses the multi-task NER engine 340 and/or the multi-task machine learning architecture 300 of FIG. 3 can be implemented using a multitasking BERT model. The systems and techniques can implement multi-task NER engine 340 based on applying a separate classifier head (e.g., linear layer) to a pre-trained BERT-based model for each pathology classification task. In paragraph [0084]-SEHANOBISH discloses multi-task NER engine 340 can be implemented by applying four separate classifier heads (e.g., four linear layers) to a pre-trained BERT model or a pre-trained ClinicalBERT model. Each classifier head can be associated with a respective one of the four different pathology classification tasks or categories. Please also read paragraph [0043]) are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch (Fig. 3. Paragraph [0087]-SEHANOBISH discloses each classifier head (e.g., of the four classifier heads) can be trained based on a corresponding cross-entropy loss. NER engine 340 can determine a set of pathology severity logits that are indicative of a predicted severity of an identified pathology classification. The set of pathology severity logits can be generated and provided as output by the classifier head that is associated with the identified pathology classification. Accordingly, the cross-entropy loss for a given classifier head can be determined as the cross-entropy loss between the predicted pathology severity logits generated by the given classifier head and the corresponding ground truth targets/labels for the input text 305 (e.g., thereby resulting in four separate cross-entropy losses). The respective cross-entropy losses determined for each pathology-specific classifier head can be used to obtain a joint loss. In paragraph [0088]-SEHANOBISH discloses the joint loss of Eq. (1) can be used to allow the gradients to be back-propagated through the whole model, and the four classifier heads can be trained jointly to yield a trained multitasking BERT model, e.g., by fine-tuning the parameters of the pre-trained BERT or RADBERT model used as input. In other words, the four classifier heads included in multi-task NER engine 340 can be jointly trained using backpropagation based on the joint loss function of Eq. (1)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of SEHANOBISH of having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of diseases, since both SURESH and SEHANOBISH concern machine learning and the identification of diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while SEHANOBISH provides an improved multi-task learning (MTL)-based machine learning network that can generate a plurality of sets of features, each set of features generated based on a particular sentence group and indicative of pathology severity predictions determined for the anatomical location associated with the particular sentence group. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and SEHANOBISH et al. (US 20230161978 A1), Abstract and Paragraph [0065-0070, 0087-0094 and 0106].
Claims 10-13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over SURESH et al. (US 20240032784 A1), hereinafter referenced as SURESH in view of BHUIYAN et al. (US 20200242763 A1), hereinafter referenced as BHUIYAN and in further view of SEHANOBISH et al. (US 20230161978 A1), hereinafter referenced as SEHANOBISH and in further view of PARK et al. (US 20220189012 A1), hereinafter referenced as PARK.
Regarding claim 10, SURESH in view of BHUYIAN and in further view of SEHANOBISH explicitly teaches the non-transitory computer readable storage medium of claim 9, although SURESH explicitly teaches the third machine learning model (Fig. 3-4, #410, #420 and #304-1-k called a deep learning model and a CNN, respectively. Paragraph [0043]. In paragraph [0037]-SURESH discloses the extraction component 145 processes the spectral information 205 and the spatial information 210 separately using the local feature detection tool 220 and the global feature detection tool 225. The local feature detection tool 220 and the global feature detection tool 225 may use one or more deep learning techniques (or models) (e.g., 2D CNN, 3D CNN, autoencoder, etc.) to perform the respective local feature detection and global feature detection (wherein the visualization tool #215 uses one or more CNNS #304-1-k to process spectral information #205, and deep learning models are each used for local feature extraction, global feature extraction, and generation of a probability map for diseases. Please also read paragraph [0044 and 0048]).
SURESH fails to explicitly teach wherein: the third machine learning model is configured to minimize a loss function that measures a discrepancy between a predicted co-occurrence pattern and an actual co-occurrence pattern, the loss function for the third machine learning model comprising a co-occurrence loss function.
However, PARK explicitly teaches wherein: the machine learning model (Fig. 2, called GREM. Paragraph [0039]-PARK discloses as shown in Fig. 2, the present invention is largely configured of a trunk module 100, a branch module 200, and a final diagnosis unit 300, and the branch module 200 includes a disease inference unit 210, a location search unit 220, a key lesion finder 230, and a small-sized lesion finder 240. Hereinafter, a deep learning architecture system for automatic fundus image reading of the present invention is named as Grem) is configured to minimize a loss function that measures a discrepancy between a predicted co-occurrence pattern and an actual co-occurrence pattern, the loss function for the third machine learning model comprising a co-occurrence loss function (Fig. 2. Paragraph [0057]-PARK discloses N.sub.1 is the number of diseases in the disease inference unit 210. In paragraph [0058]-PARK discloses the activation function of the last Dense2 layer uses a sigmoid function to independently score a value between 0 and 1 for each disease category. In paragraph [0059]-PARK discloses a first loss (loss.sub.b1) generated when the section 110 connecting the disease inference unit 210 and the trunk module 100 is trained is calculated by [Equation 1]. The loss function for training the disease inference unit 210 generally follows a widely used sum of squared error. In paragraph [0060]-PARK discloses P1.sub.i is the probability of a disease to belong to the i-th category through training, which is output as a value between 0 and 1. Please also read paragraph [0063-0066]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN and in further view of SEHANOBISH of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of PARK of having wherein: the third machine learning model is configured to minimize a loss function that measures a discrepancy between a predicted co-occurrence pattern and an actual co-occurrence pattern, the loss function for the third machine learning model comprising a co-occurrence loss function.
Wherein SURESH’s non-transitory computer readable storage medium having wherein: the third machine learning model is configured to minimize a loss function that measures a discrepancy between a predicted co-occurrence pattern and an actual co-occurrence pattern, the loss function for the third machine learning model comprising a co-occurrence loss function.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and PARK concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while PARK’s provides automatic fundus image reading, and a deep learning architecture for automatic fundus image reading, which are capable of minimizing the amount of data required for learning by training and reading artificial intelligence in a manner similar to that of an ophthalmologist who acquires medical knowledge. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and PARK et al. (US 20220189012 A1), Abstract and Paragraph [0035-0043].
Regarding claim 11, SURESH in view of BHUIYAN and in further view of SEHANOBISH and in further view of PARK explicitly teaches the non-transitory computer readable storage medium of claim 10, SURESH in view of BHUYIAN fails to explicitly teaches wherein: the loss function of the disease detection branch comprises a binary cross-entropy loss function and the loss function of the disease severity branch comprises a categorical cross-entropy loss function.
However, SEHANOBISH explicitly teaches wherein: the loss function of the disease detection branch (Fig. 3, #340 and #360 called a multi-task NER engine and deep learning classifier, respectively. Paragraph [0078]-SEHANOBISH discloses NER engine 340 can be implemented as BERT-based (e.g., ClinicalBERT-based) binary classifier. In paragraph [0083]-SEHANOBISH discloses the multi-task NER engine 340 and/or the multi-task machine learning architecture 300 of FIG. 3 can be implemented using a multitasking BERT model. The systems and techniques can implement multi-task NER engine 340 based on applying a separate classifier head (e.g., linear layer) to a pre-trained BERT-based model for each pathology classification task. In paragraph [0084]-SEHANOBISH discloses multi-task NER engine 340 can be implemented by applying four separate classifier heads (e.g., four linear layers) to a pre-trained BERT model or a pre-trained ClinicalBERT model. Each classifier head can be associated with a respective one of the four different pathology classification tasks or categories) comprises a binary cross-entropy loss function and the loss function of the disease severity branch (Fig. 3, #340 called a multi-task NER engine. Paragraph [0083]) comprises a categorical cross-entropy loss function (Fig. 1. Paragraph [0087]-SEHANOBISH discloses each classifier head (e.g., of the four classifier heads) can be trained based on a corresponding cross-entropy loss. NER engine 340 can determine a set of pathology severity logits that are indicative of a predicted severity of an identified pathology classification. The set of pathology severity logits can be generated and provided as output by the classifier head that is associated with the identified pathology classification. The cross-entropy loss for a given classifier head can be determined as the cross-entropy loss between the predicted pathology severity logits generated by the given classifier head and the corresponding ground truth targets/labels for the input text 305 (e.g., thereby resulting in four separate cross-entropy losses). The respective cross-entropy losses determined for each pathology-specific classifier head can be used to obtain a joint loss (wherein multi-task NER engine 340 can be jointly trained using backpropagation based on the joint loss function of Eq. (1)). Therefore, it would have been obvious to one of ordinary skill in the art to specifically designate a binary cross entropy function and a categorical cross-entropy loss for each branch. SEHANOBISH discloses using multiple cross-entropy functions for disease detection/severity losses. Both binary cross entropy and a categorical cross-entropy are commonly used functions and sub-types of a broader class of cross-entropy functions. Furthermore, these functions are best suited for disease detection and disease severity given detection is typically a binary classification (i.e. a probability from 0 to 1) and severity involves multiple categories (e.g. stages). Thus, it would be obvious to specifically select a binary cross entropy function for detection and a categorical cross entropy function for severity. This would improve the performance of a system for screening retinal diseases by ensuring the losses are correctly represented in the machine learning models).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of and in further view of SEHANOBISH and in further view of PARK of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of SEHANOBISH of having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of diseases, since both SURESH and SEHANOBISH concern machine learning and the identification of diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while SEHANOBISH provides an improved multi-task learning (MTL)-based machine learning network that can generate a plurality of sets of features, each set of features generated based on a particular sentence group and indicative of pathology severity predictions determined for the anatomical location associated with the particular sentence group. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and SEHANOBISH et al. (US 20230161978 A1), Abstract and Paragraph [0065-0070, 0087-0094 and 0106].
Regarding claim 12, SURESH in view of BHUIYAN and in further view of SEHANOBISH and in further view of PARK explicitly teaches the non-transitory computer readable storage medium of claim 10, SURESH in view of BHUYIAN fails to explicitly teach wherein the method further comprises minimizing a total loss determined based on a weighted sum of weights for each of the loss functions and updated weights determined based on a validation performance for each of the loss functions.
However, SEHANOBISH explicitly teaches wherein the method further comprises minimizing a total loss (Fig. 3. Paragraph [0087]-SEHANOBISH discloses each classifier head (e.g., of the four classifier heads) can be trained based on a corresponding cross-entropy loss. NER engine 340 can determine a set of pathology severity logits that are indicative of a predicted severity of an identified pathology classification. The set of pathology severity logits can be generated and provided as output by the classifier head that is associated with the identified pathology classification. Accordingly, the cross-entropy loss for a given classifier head can be determined as the cross-entropy loss between the predicted pathology severity logits generated by the given classifier head and the corresponding ground truth targets/labels for the input text 305 (e.g., thereby resulting in four separate cross-entropy losses). The respective cross-entropy losses determined for each pathology-specific classifier head can be used to obtain a joint loss. In paragraph [0088]-SEHANOBISH discloses the joint loss of Eq. (1) can be used to allow the gradients to be back-propagated through the whole model, and the four classifier heads can be trained jointly to yield a trained multitasking BERT model, e.g., by fine-tuning the parameters of the pre-trained BERT or RADBERT model used as input. In other words, the four classifier heads included in multi-task NER engine 340 can be jointly trained using backpropagation based on the joint loss function of Eq. (1)) determined based on a weighted sum of weights for each of the loss functions and updated weights determined based on a validation performance for each of the loss functions (Fig. 3. Paragraph [0091]-SEHANOBISH discloses in existing and/or conventional fine-tuning operations, both the new top layer weights and the original weights of a given machine learning network are updated. For example, existing fine-tuning operations may be performed by updating the new top layer weights associated with the four classifier heads in the example described above, and by updating the original weights from the pre-trained BERT model (e.g., ClinicalBERT, RADBERT, etc.). Please also read paragraph [0033-0034]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of and in further view of SEHANOBISH and in further view of PARK of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of SEHANOBISH of having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of diseases, since both SURESH and SEHANOBISH concern machine learning and the identification of diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while SEHANOBISH provides an improved multi-task learning (MTL)-based machine learning network that can generate a plurality of sets of features, each set of features generated based on a particular sentence group and indicative of pathology severity predictions determined for the anatomical location associated with the particular sentence group. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and SEHANOBISH et al. (US 20230161978 A1), Abstract and Paragraph [0065-0070, 0087-0094 and 0106].
Regarding claim 13, SURESH in view of BHUIYAN and in further view of SEHANOBISH and in further view of PARK explicitly teaches the non-transitory computer readable storage medium of claim 12, SURESH fails to explicitly teach wherein the updated weights are determined based on backpropagating an error signal corresponding to each of the loss functions to minimize the error signal.
However, SEHANOBISH explicitly teaches wherein the updated weights are determined based on backpropagating an error signal corresponding to each of the loss functions to minimize the error signal (Fig. 3. Paragraph [0087]-SEHANOBISH discloses each classifier head (e.g., of the four classifier heads) can be trained based on a corresponding cross-entropy loss. NER engine 340 can determine a set of pathology severity logits that are indicative of a predicted severity of an identified pathology classification. The set of pathology severity logits can be generated and provided as output by the classifier head that is associated with the identified pathology classification. Accordingly, the cross-entropy loss for a given classifier head can be determined as the cross-entropy loss between the predicted pathology severity logits generated by the given classifier head and the corresponding ground truth targets/labels for the input text 305 (e.g., thereby resulting in four separate cross-entropy losses). The respective cross-entropy losses determined for each pathology-specific classifier head can be used to obtain a joint loss. In paragraph [0088]-SEHANOBISH discloses the joint loss of Eq. (1) can be used to allow the gradients to be back-propagated through the whole model, and the four classifier heads can be trained jointly to yield a trained multitasking BERT model, e.g., by fine-tuning the parameters of the pre-trained BERT or RADBERT model used as input. In other words, the four classifier heads included in multi-task NER engine 340 can be jointly trained using backpropagation based on the joint loss function of Eq. (1)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN of and in further view of SEHANOBISH and in further view of PARK of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of SEHANOBISH of having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of diseases, since both SURESH and SEHANOBISH concern machine learning and the identification of diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while SEHANOBISH provides an improved multi-task learning (MTL)-based machine learning network that can generate a plurality of sets of features, each set of features generated based on a particular sentence group and indicative of pathology severity predictions determined for the anatomical location associated with the particular sentence group. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and SEHANOBISH et al. (US 20230161978 A1), Abstract and Paragraph [0065-0070, 0087-0094 and 0106].
Regarding claim 20, SURESH in view of BHUYIAN explicitly teach the method of claim 18, SURESH further teaches wherein:
the disease detection branch (Fig. 3. Paragraph [0045]-SURESH discloses the local feature detection tool 220 uses the deep learning model 410 to extract one or more local features 230 from the spectral information 205. In paragraph [0047]-SURESH discloses the global feature detection tool 225 receives spatial information 210 and uses a deep learning model 420 to extract the global features 235 (wherein the local feature detection tool 220 and global feature detection tool 225 separately extract features using one or more deep learning techniques (or models), a visualization tool 215 includes multiple convolutional neural networks (i.e. CNNs 304 1-K) to process spectral data to generate an interpretative image, and the output component 155 may also use a third neural network or vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235)) is configured to determine a disease presence probability for the presence of the disease (Fig. 4. Paragraph [0048]-SURESH discloses the output component 155 may use the labels corresponding to the local features 230 to establish “words” to describe the spectral data (wherein each “word” may associate a label (corresponding to a local feature 230) with a description of a retinal disease/symptom). The output component 155 may then use the global features 235 to combine the “words” into one or more “sentences,” which can be used to evaluate a disease pattern or perform early detection (wherein the pattern may include (i) a particular retinal disease or (ii) a severity of the retinal disease, and the disease indications may be associated with a score). In paragraph [0027]-SURESH discloses the severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct). Please also read paragraph [0048]);
the disease severity branch (Fig. 3. Paragraph [0047]-SURESH discloses the global feature detection tool 225 receives spatial information 210 and uses a deep learning model 420 to extract the global features 235 (wherein the local feature detection tool 220 and global feature detection tool 225 separately extract features using one or more deep learning techniques (or models), a visualization tool 215 includes multiple convolutional neural networks (i.e. CNNs 304 1-K) to process spectral data to generate an interpretative image, and the output component 155 may also use a third neural network or vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235)) is configured to determine a disease severity probability for the severity of the disease (Fig. 4. Paragraph [0027]-SURESH discloses the severity score 185 includes (i) an indication (or prediction) of an ocular disease for the disease indication 180 (e.g., segmented region) within the ophthalmic image 175 and (ii) an indication of the severity of the ocular disease. The severity score 185 includes a ranking. The ranking may be based on a predefined scale (e.g., 0 to 10, 0 to 100), where one end of the scale is associated with a lowest severity and the other end of the scale is associated with a highest severity. The severity score 185 includes a confidence score (e.g., an indication of the likelihood that the disease prediction is correct). Further in paragraph [0048]-SURESH discloses the output component 155 can use a neural network (e.g., 2D or 3D CNN) or a vision transformer to generate a probability map of disease onset, based on the local features 230 and global features 235. The output component 155 can segment choroidal neovascularization structures from the retinal image to indicate the presence of age related macular degeneration); and
SURESH fails to explicitly teach each machine learning model of the plurality of second machine learning models includes a disease detection branch and a disease severity branch that is separate from the disease detection branch.
However, PARK explicitly teaches each machine learning model of the plurality of second machine learning models (Fig. 2. Paragraph [0039]-PARK discloses as shown in FIG. 2, the present invention is largely configured of a trunk module 100, a branch module 200, and a final diagnosis unit 300. Hereinafter, a deep learning architecture system for automatic fundus image reading of the present invention is named as Grem. In paragraph [0042]-PARK discloses the trunk module 100 is a common layer for extracting features of a fundus image using a convolutional neural network (CNN). In paragraph [0045]-PARK discloses a section 110 is an architecture that connects any one branch module 200 among a plurality of branch modules 200 to the trunk module 100. One branch module 200 and the trunk module 100 are combined to form one section 110 for each disease) includes a disease detection branch and a disease severity branch that is separate from the disease detection branch (Fig. 3. Paragraph [0049]-PARK discloses as shown in FIG. 2, the branch module 200 is configured of a disease inference unit 210, a location search unit 220, a key lesion finder 230, and a small-sized lesion finder 240. In paragraph [0055]-PARK discloses the disease inference unit 210 corresponds to a category classifier for classifying a category by looking at the entire fundus image. In paragraph [0071]-PARK discloses the key lesion finder 230 finds a key lesion, which is a component constituting a disease. In paragraph [0078]-PARK discloses the small-sized lesion finder 240 is a branch designed to separately detect a very small but very important lesion from a fundus image (wherein the output for each lesion is a label name, X coordinate, Y coordinate, confidence level for each lesion). In paragraph [0080]-PARK discloses Glem architecture is based on HydraNet having four branch modules 200, and it is a layer that determines and outputs a final diagnosis name by integrating the outputs of the four branch modules 200 (wherein the output includes existence 1 or non-existence 0 for N diseases));
SURESH fails to explicitly teach the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
However, SEHANOBISH explicitly teaches the disease detection branch and the disease severity branch (Fig. 3, #360 called a multi-task NER engine. Paragraph [0083]. Further in paragraph [0025]-SEHANOBISH discloses a deep learning architecture may learn a hierarchy of features. In paragraph [0083]-SEHANOBISH discloses the multi-task NER engine 340 and/or the multi-task machine learning architecture 300 of FIG. 3 can be implemented using a multitasking BERT model. The systems and techniques can implement multi-task NER engine 340 based on applying a separate classifier head (e.g., linear layer) to a pre-trained BERT-based model for each pathology classification task. In paragraph [0084]-SEHANOBISH discloses multi-task NER engine 340 can be implemented by applying four separate classifier heads (e.g., four linear layers) to a pre-trained BERT model or a pre-trained ClinicalBERT model. Each classifier head can be associated with a respective one of the four different pathology classification tasks or categories. Please also read paragraph [0043 and 0087]) are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch (Fig. 3. Paragraph [0087]-SEHANOBISH discloses each classifier head (e.g., of the four classifier heads) can be trained based on a corresponding cross-entropy loss. NER engine 340 can determine a set of pathology severity logits that are indicative of a predicted severity of an identified pathology classification. The set of pathology severity logits can be generated and provided as output by the classifier head that is associated with the identified pathology classification. The cross-entropy loss for a given classifier head can be determined as the cross-entropy loss between the predicted pathology severity logits generated by the given classifier head and the corresponding ground truth targets/labels for the input text 305 (e.g., thereby resulting in four separate cross-entropy losses). The respective cross-entropy losses determined for each pathology-specific classifier head can be used to obtain a joint loss. In paragraph [0088]-SEHANOBISH discloses the joint loss of Eq. (1) can be used to allow the gradients to be back-propagated through the whole model, and the four classifier heads can be trained jointly to yield a trained multitasking BERT model, e.g., by fine-tuning the parameters of the pre-trained BERT or RADBERT model used as input. The four classifier heads included in multi-task NER engine 340 can be jointly trained using backpropagation based on the joint loss function of Eq. (1)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUYIAN and in further view of PARK of having a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of SEHANOBISH of having the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
Wherein SURESH’s method having the disease detection branch and the disease severity branch are configured to minimize a loss function using backpropagation to reduce a combination of the loss function for the disease detection branch and the loss function of the disease severity branch.
The motivation behind the modification would have been to obtain a method that improves the performance of learning systems as well as the detection, identification and visualization of diseases, since both SURESH and SEHANOBISH concern machine learning and the identification of diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while SEHANOBISH provides an improved multi-task learning (MTL)-based machine learning network that can generate a plurality of sets of features, each set of features generated based on a particular sentence group and indicative of pathology severity predictions determined for the anatomical location associated with the particular sentence group. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and SEHANOBISH et al. (US 20230161978 A1), Abstract and Paragraph [0065-0070, 0087-0094 and 0106].
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over SURESH et al. (US 20240032784 A1), hereinafter referenced as SURESH in view of BHUIYAN et al. (US 20200242763 A1), hereinafter referenced as BHUIYAN and in further view of BURLINA et al. (US 20150265144 A1), hereinafter referenced as BURLINA.
Regarding claim 14, SURESH in view of BHUIYAN explicitly teaches the non-transitory computer readable storage medium of claim 1, SURESH fails to explicitly teach wherein the method further comprises: optimizing the plurality of second machine learning models using backpropagation.
However, BHUYIAN explicitly teaches wherein the method further comprises: optimizing the plurality of second machine learning models using backpropagation (Fig. 3. Paragraph [0079]-BHUYIAN discloses at step 812, the Backpropagation artificial neural network (ANN) (524) is used to train the AMD prediction incidence module 120).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of BHUYIAN of having wherein the method further comprises: optimizing the plurality of second machine learning models using backpropagation.
Wherein SURESH’s non-transitory computer readable storage medium having wherein the method further comprises: optimizing the plurality of second machine learning models using backpropagation.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems as well as the detection, identification and visualization of eye-related diseases, since both SURESH and BHUYIAN concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while BHUYIAN’s provides systems and methods that improve machine learning performance and the automated screening of early and late stage Age-related Macular Degeneration (AMD). Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and BHUYIAN et al. (US 20200242763 A1), Abstract and Paragraph [0063-0068 and 0125].
SURESH in view of BHUYIAN fail to explicitly teach and optimizing the third machine learning model using an expectation maximization.
However, BURLINA explicitly teaches and optimizing the learning model (Fig. 3. Paragraph [0009]-BURLINA discloses a method of detecting, and classifying severity of, a retinal disease using retinal images (wherein the method includes generating reference data concerning occurrences of key image features of retinal diseases and severities, processing an individual’s retinal image to identify occurrences of image features, determining which of the identified occurrences correspond to the key image features of the reference data, calculating the number of occurrences of each of key image features, and determining the likelihood retinal disease or development of disease based on a comparison of the number of occurrences with reference data. In paragraph [0039]-BURLINA discloses the entire corpus of available images and their associated category labels is used for training and testing. As is standard in machine learning applications, a N-fold cross validation approach is used. Then, a random forest classifier is trained using the training dataset. The random forest algorithm uses the consensus of a large number of weak (only slightly better than chance) binary decision trees to classify the testing images into different severity classes. In paragraph [0040]-BURLINA discloses FIG. 4 is a flowchart illustrating an embodiment of a training phase) using an expectation maximization (Fig. 3. Paragraph [0049]-BURLINA discloses generating the reference data can include processing the plurality of expert-classified retinal images to identify occurrences of each of a plurality of distinguishable reference image features, identifying key image features corresponding to the plurality of expert-classified retinal images, and computing a frequency of occurrence of each of the key image features corresponding to each classification of the expert-classified retinal images. The identifying the key image features can use Expectation maximization).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SURESH in view of BHUIYAN of having a non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for disease detection using retinal images, the method comprising: processing a retinal image of a retina with a first machine learning model to determine whether the retinal image reflects an abnormal condition or a normal condition of the retina, with the teachings of BURLINA of having and optimizing the third machine learning model using an expectation maximization.
Wherein SURESH’s non-transitory computer readable storage medium having and optimizing the third machine learning model using an expectation maximization.
The motivation behind the modification would have been to obtain a non-transitory computer readable storage medium that improves the performance of learning systems for the detection, identification and visualization of eye-related diseases, since both SURESH and BURLINA concern machine learning and retinal image screening for the identification of eye-related diseases. Wherein SURESH provide improved systems, devices, and techniques for analyzing MSI/HSI data for ophthalmology applications, such as ocular disease detection, while BURLINA’s provides improved systems and methods for automatically detecting and classifying the severity of retinal diseases. Please see SURESH et al. (US 20240032784 A1), Abstract and Paragraph [0016-0017] and BURLINA et al. (US 20150265144 A1), Abstract and Paragraph [0029].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
JIA et al. (US 20210153738 A1)- Methods and systems for identifying CNV membranes and vasculature in images obtained using noninvasive imaging techniques are described. An example method includes generating, by a first model based on at least one image of a retina of a subject, a membrane mask indicating a location of a CNV membrane in the retina. The method further includes generating, by a second model based on the membrane mask and the at least one image, a vasculature mask of the retina of the subject, the vasculature mask indicating CNV vascularization in the retina....................... Please see Fig. 2-4 and 7 and read para. [0067-0074]. Abstract.
Fukushima et al. (US 20220175325 A1)- An information processing apparatus disclosed in the present specification includes: estimation means for estimating a subject's risk of developing a disease using a learned model that has learned a relationship between a feature obtained from a fundus image and a risk of developing the disease, which is evaluated from the feature; and correction means for correcting the estimated risk of developing the disease based on the subject's biological information...................... Please see Fig. 3 and 5. Abstract.
GUPTA et al. (US 20190096111 A1)- Techniques for automating the generation and analysis of fundus drawings are described. Captured images undergo image processing to extract information about image features. Fundus images are generated and recommended labels for the fundus drawing are generated. Fundus drawings can be analyzed and undergo textual processing to extract existing labels. Machine learning models and co-occurrence analysis can be applied to collections of fundus images and drawings to gather information about commonly associated labels, label locations, and user information. The most frequently used labels associated with the image can be identified to improve recommendations and personalize labels........................ Please see Fig. 1-5 and para. [0026-0031]. Abstract.
ZHANG et al. (US 20220075955 A1)- Provided are a semantic classification method and apparatus, a neural network training method and apparatus and a storage medium. The semantic classification method includes: inputting a first remark relating to a first object; extracting a first common representation vector for representing a common representation in the first remark by processing the first remark using a common representation extractor; extracting a first single representation vector for representing a single representation in the first remark by processing the first remark using a first representation extractor; obtaining a first representation vector by splicing the first common representation vector and the first single representation vector; and obtaining a semantic classification of the first remark by processing the first representation vector using a first semantic classifier; where the common representation includes an intention representation which is used to remark on both the first object and a second object, the second object is an associated remarked object different from the first object, and the single representation in the first remark includes an intention representation which is only used to remark on the first object......................... Please see Fig. 1-5, and read para. [0171-0183]. Abstract.
BOYD et al. (US 20220207729 A1)- Computer systems and computer-implemented methods for performing classification, detection, and/or prediction based on processing of ocular images obtained from various imaging modalities are disclosed. Use of delayed near-infrared analysis (DNIRA) as one of the imaging modality is also disclosed........................ Please see Fig. 5, 8 and 11, and read para. [0243, 0299, 0302, 0309, 0313, 0329, 0425-0428, 0566-0567]. Abstract.
CHANG et al. (US 20230157533 A1)- The present invention relates to computer-implemented method for assessing a level of activity, including presence or an absence, of a disease in at least one eye of a patient, wherein the disease is a neovascular ocular disease. The method comprises the steps of receiving, via one or more input elements, a set of input patient data corresponding to the patient and comprising retinal images of the patient; applying a first algorithm for imaging data analysis to the retinal images to identify values of e anatomical variables of the patient's eye; applying a second algorithm to the values of anatomical variables identified, and to distinct clinical, non-image derived input patient data comprised in the set of input patient data, in order to consequently make an assessment of the level of activity of the disease in the eye of the patient, and/or of the progression or regression of the disease with respect to a level of activity formerly determined. Based on the assessment, a disease activity score is generated and output corresponding to the level of activity of the disease. The assessment of disease activity is used to adjust a dosing regimen of a drug for treatment of the patient's eye disease........................... Please see Fig. 2 and 4-7. Abstract.
WANG et al. (US 20220039768 A1)- Embodiments of the disclosure provide methods and systems for predicting a disease condition from images of a patient. The exemplary system may include a communication interface configured to receive a sequence of images acquired of the patient by an image acquisition device. The sequence of images are acquired at a sequence of prior time points during progression of a disease. The system may further include at least one processor, configured to determine regions of interest corresponding to the sequence of prior time points based on the sequence of images. The at least one processor also applies a progressive condition prediction network to the regions of interest to predict a disease condition at a future time point during the progression of the disease. The progressive condition prediction network includes a forward path for predicting the disease condition based on the regions of interest and disease conditions at the sequence of prior time points. The at least one processor further provides a diagnostic output based on the predicted disease condition at the future time point...................... Please see Fig. 4-6, 9-10 and para. [0071-0078] Abstract.
WU et al. (US 20220012890 A1)- An automated method for segmentation includes steps of receiving at a computing device an input image representing at least one surface and performing by the computing device image segmentation on the input image based on a graph surface segmentation model with deep learning. The deep learning may be used to parameterize the graph surface segmentation model.......................... Please see Fig. 2 and 5 and read para. [0067, 0073, 0078, 0139-155]. Abstract.
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Aaron Bonansinga whose telephone number is (703) 756-5380 The examiner can normally be reached on Monday-Friday, 9:00 a.m. - 6:00 p.m. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Chineyere Wills-Burns can be reached by phone at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON TIMOTHY BONANSINGA/Examiner, Art Unit 2673 /CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights