DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-13, 18 are rejected under 35 U.S.C. 103 as not being patentable over DE SISTERNES (US 20230140881 A1, Date Filed: 2021-04-28), and in view of Reisman (US 10117568 B2), and further in view of YANG (WO 2022120044 A1).
Re Claim 1, DE SISTERNES discloses a computer-implemented method of automatically predicting progression of age-related macular degeneration (see DE SISTERNES: e.g., --A system and method for use with optical coherence tomography (OCT) data to identify a target pathology extracts multiple pathology-characteristic images from the OCT data. The extracted pathology-characteristic images may include a mixture of OCT structural images (including retinal layer thickness information) and OCT angiography images. Optionally, other pathology-characteristic images and data maps (mapped to corresponding positions in the OCT data), such as fundus images and visual field test maps may be accessed as additional pathology-characteristic images. Each pathology-characteristic image defines a different image channel (e.g., “color channel”) per pixel in a composite, channel-coded image, which is then used to train a neural network to search for the target pathology in OCT data. The trained neural network may then receive new composite, channel-coded image and identify/segment the target pathology within the new channel-coded image.--, in abstract, and, --a method of analyzing optical coherence tomography data to identify a target pathology. More specifically, it is directed to analyzing optical coherence tomography data to identify Geographic Atrophy (GA)… [0002] Age-related macular degeneration (AMD) is an eye disease most common in the older population and arising from damage to the macula that leads to loss of central vision. Some patients with age-related macular degeneration (AMD) develop geographic atrophy (GA), which refers to regions of the retina where cells waste away and die. Geographic Atrophy (GA) is a condition of the macula present at the advanced stages of non-exudative macular degeneration. GA has a characteristic appearance resulting from the loss of the photoreceptor layer, retinal pigment epithelium (RPE), and choriocapillaris. GA typically first appears in the parafoveal location and progresses around the fovea and then through the fovea with loss of central visual acuity….characterization and monitoring macular regions affected by GA is fundamental for patient diagnosis, monitoring and management as well as for treatment research purposes.--, in [0002], and, --GA may result in a progressive loss of vision, particularly central vision. However, GA may start with loss of vision outside the central area, and progress toward the center over time. Thus, it is advantageous to incorporate information from visual field test results FV. A visual field test is a method of measuring an individual's entire scope of vision, e.g., their central and peripheral (side) vision. Visual field testing is a way to map the visual fields of each eye individually and can detect blind spots (scotomas) as well as more subtle areas of dim vision. A campimeter, or “perimeter,” is a dedicated machine/device/system that applies a visual field test to a patient. A more in-depth discussion of perimeters and visual field testing is provided below. All, or select parts of a visual field test (such as the VF gray scale or numerical gray scale mapped to corresponding retinal locations) may be incorporated into the present multi-channel composite image 27.--, in [0055]-[0060], and [0105]-[0107]); the method comprising:
receiving, by an image analysis computing system, optical coherence tomography data (OCT data) (see DE SISTERNES: e.g., ----A system and method for use with optical coherence tomography (OCT) data to identify a target pathology extracts multiple pathology-characteristic images from the OCT data. The extracted pathology-characteristic images may include a mixture of OCT structural images (including retinal layer thickness information) and OCT angiography images. Optionally, other pathology-characteristic images and data maps (mapped to corresponding positions in the OCT data), such as fundus images and visual field test maps may be accessed as additional pathology-characteristic images. Each pathology-characteristic image defines a different image channel (e.g., “color channel”) per pixel in a composite, channel-coded image, which is then used to train a neural network to search for the target pathology in OCT data. The trained neural network may then receive new composite, channel-coded image and identify/segment the target pathology within the new channel-coded image.--, in abstract);
determining, by the image analysis computing system, an optical attenuation coefficient for each pixel of the OCT data to create optical attenuation coefficient data (OAC data) corresponding to the OCT data (see DE SISTERNES: e.g., --The OCT data 21 may include OCT structural data and/or OCTA flow data, and the extracted metrics may include OCT-based metrics extracted from the OCT structural data, such as retinal layer thicknesses, distances of a specific A-scan to a specific retinal structure (e.g., distance to the fovea center), layer integrity (e.g., the loss of a specific layer), sub-RPE reflectivity, inner RPE reflectivity, overall retinal thickness, and/or the optical attenuation coefficient (OAC).
[0049] The optical attenuation coefficient (OAC) is an optical property of a medium that determines how the power of a coherent light beam propagating through the (e.g., turbid) medium (e.g., tissue) is attenuated along its path due to scattering and absorption. The irradiance (power per unit area) of the coherent light beam that propagates through a (e.g., homogeneous) medium is given by Lambert-Beer's law: L(z)=L.sub.0e.sup.−μz, where L(z) is the irradiance of the beam after traveling through the medium over a distance z, L.sub.0 is the irradiance of the incident light beam and μ is the optical attenuation coefficient. Large attenuation coefficients result in a quick and exponential decline of the irradiance of the coherent light beam with depth. Because the OAC is an optical property of the medium, determining the OAC provides information on the composition of this medium. Applicants propose that providing the OAC (per A-scan) as one of the extracted metrics may be beneficial identify specific pathologies (e.g., GA), particularly since it can be indicative of the current state (e.g., light attenuating state) of tissued at specific A-scan positions. An example of how the OAC may be determined/calculated is provided in “Depth-Resolved Model-Based Reconstruction of Attenuation Coefficients in Optical Coherence Tomography”, by K. A. Vermeer et al., Biomedical Optics Express, Vol. 5, Issue 1, pp. 322-337 (2014). A discussion of previous applications of OAC may be found in “In Vivo Tissue Injury Mapping Using Optical Coherence Tomography Based Methods”, by Utka Baran et al., Applied Optics, Vol. 54, No. 21, Jul. 20, 2015.--, in [0048]-[0050]; and, --[0060] FIG. 6 illustrates a general workflow of the present invention, including the U-net architecture of the proof of concept implementation. As shown, OCT Data 21 is accessed/acquired and multiple pathology characteristic images PCI's are defined from the OCT data 21, as described above. For example, the pathology characteristic images PCI's may include a sub-RPE reflectivity image, an inner RPE reflectivity image, Retinal thickness, and/or optical attenuation coefficient (OAC) as described above in reference to FIGS. 3 and 4. Optionally, other pathology characteristic image may also be used, such an en face images/maps of specific retinal layer thicknesses or layer integrity, OCTA flow images (such as flow at, or within the vicinity of, the choriocapillaris), an/or other pathology characteristic data as described above in reference to FIG. 5. the pathology characteristic images are then combined into a different (e.g., color or monochrome value) channels of a channel-coded image 27, which is then submitted to machine learning model 29, which is herein implemented using a U-Net architecture. A discussion of a U-Net architecture is provided below.--, in [0059]-[0061]);
determining, by the image analysis computing system, an area exhibiting geographic atrophy based on at least one of the OCT data and the OAC data (see DE SISTERNES: e.g., --The OCT data 21 may include OCT structural data and/or OCTA flow data, and the extracted metrics may include OCT-based metrics extracted from the OCT structural data, such as retinal layer thicknesses, distances of a specific A-scan to a specific retinal structure (e.g., distance to the fovea center), layer integrity (e.g., the loss of a specific layer), sub-RPE reflectivity, inner RPE reflectivity, overall retinal thickness, and/or the optical attenuation coefficient (OAC).
[0049] The optical attenuation coefficient (OAC) is an optical property of a medium that determines how the power of a coherent light beam propagating through the (e.g., turbid) medium (e.g., tissue) is attenuated along its path due to scattering and absorption. The irradiance (power per unit area) of the coherent light beam that propagates through a (e.g., homogeneous) medium is given by Lambert-Beer's law: L(z)=L.sub.0e.sup.−μz, where L(z) is the irradiance of the beam after traveling through the medium over a distance z, L.sub.0 is the irradiance of the incident light beam and μ is the optical attenuation coefficient. Large attenuation coefficients result in a quick and exponential decline of the irradiance of the coherent light beam with depth. Because the OAC is an optical property of the medium, determining the OAC provides information on the composition of this medium. Applicants propose that providing the OAC (per A-scan) as one of the extracted metrics may be beneficial identify specific pathologies (e.g., GA), particularly since it can be indicative of the current state (e.g., light attenuating state) of tissued at specific A-scan positions. An example of how the OAC may be determined/calculated is provided in “Depth-Resolved Model-Based Reconstruction of Attenuation Coefficients in Optical Coherence Tomography”, by K. A. Vermeer et al., Biomedical Optics Express, Vol. 5, Issue 1, pp. 322-337 (2014). A discussion of previous applications of OAC may be found in “In Vivo Tissue Injury Mapping Using Optical Coherence Tomography Based Methods”, by Utka Baran et al., Applied Optics, Vol. 54, No. 21, Jul. 20, 2015.--, in [0048]-[0050]; and, --[0060] FIG. 6 illustrates a general workflow of the present invention, including the U-net architecture of the proof of concept implementation. As shown, OCT Data 21 is accessed/acquired and multiple pathology characteristic images PCI's are defined from the OCT data 21, as described above. For example, the pathology characteristic images PCI's may include a sub-RPE reflectivity image, an inner RPE reflectivity image, Retinal thickness, and/or optical attenuation coefficient (OAC) as described above in reference to FIGS. 3 and 4. Optionally, other pathology characteristic image may also be used, such an en face images/maps of specific retinal layer thicknesses or layer integrity, OCTA flow images (such as flow at, or within the vicinity of, the choriocapillaris), an/or other pathology characteristic data as described above in reference to FIG. 5. the pathology characteristic images are then combined into a different (e.g., color or monochrome value) channels of a channel-coded image 27, which is then submitted to machine learning model 29, which is herein implemented using a U-Net architecture. A discussion of a U-Net architecture is provided below.--, in [0059]-[0061]; also see: and, --a method of analyzing optical coherence tomography data to identify a target pathology. More specifically, it is directed to analyzing optical coherence tomography data to identify Geographic Atrophy (GA)… [0002] Age-related macular degeneration (AMD) is an eye disease most common in the older population and arising from damage to the macula that leads to loss of central vision. Some patients with age-related macular degeneration (AMD) develop geographic atrophy (GA), which refers to regions of the retina where cells waste away and die. Geographic Atrophy (GA) is a condition of the macula present at the advanced stages of non-exudative macular degeneration. GA has a characteristic appearance resulting from the loss of the photoreceptor layer, retinal pigment epithelium (RPE), and choriocapillaris. GA typically first appears in the parafoveal location and progresses around the fovea and then through the fovea with loss of central visual acuity….characterization and monitoring macular regions affected by GA is fundamental for patient diagnosis, monitoring and management as well as for treatment research purposes.--, in [0002], and, --GA may result in a progressive loss of vision, particularly central vision. However, GA may start with loss of vision outside the central area, and progress toward the center over time. Thus, it is advantageous to incorporate information from visual field test results FV. A visual field test is a method of measuring an individual's entire scope of vision, e.g., their central and peripheral (side) vision. Visual field testing is a way to map the visual fields of each eye individually and can detect blind spots (scotomas) as well as more subtle areas of dim vision. A campimeter, or “perimeter,” is a dedicated machine/device/system that applies a visual field test to a patient. A more in-depth discussion of perimeters and visual field testing is provided below. All, or select parts of a visual field test (such as the VF gray scale or numerical gray scale mapped to corresponding retinal locations) may be incorporated into the present multi-channel composite image 27.--, in [0055]-[0060], and [0105]-[0107]);
DE SISTERNES however does not explicitly disclose measuring, by the image analysis computing system, one or more attributes within an adjacent area that is adjacent to the area exhibiting geographic atrophy,
Reisman discloses measuring, by the image analysis computing system, one or more attributes within an adjacent area that is adjacent to the area exhibiting geographic atrophy (see Reisman: e.g., Fig. 9A, Fig. 9B. and, -- processing ophthalmic image data for detecting geographical atrophy is desired that overcomes the above limitations and deficiencies of the current systems and methods.
(20) According to one example, a method of processing ophthalmic image data comprises obtaining optical coherence tomography (OCT) data of a subject's eye, the OCT data including data beyond the retinal pigment epithelium (RPE) of a subject's eye and intensities of an imaging signal as the imaging signal is backscattered by each of a plurality of tissue layers in the subject's eye; determining a ratio of the intensities of the backscattered imaging signal, the ratio being the intensity of a first portion of the backscattered imaging signal corresponding to at least a portion of a retinal layer with respect to the intensity of a second portion of the backscattered imaging signal corresponding to at least a portion of a sub-RPE layer; determining a representative value based at least in part on the determined ratio; and phenotyping or classifying the subject based at least in part on the determined representative value.
(21) In various embodiments of the above example, the OCT data is captured at a plurality of times, a ratio and representative value are determined for the OCT data captured at each of the plurality of times, and the subject is phenotyped or classified based at least in part on a change in the determined ratios and/or representative values at the plurality of times; the change in the representative values and/or ratios is determined by performing a statistical measurement between the representative values and/or ratios at a first time of the plurality of times and the representative values and/or ratios and a second time of the plurality of times; the representative values and/or ratios at a first time of the plurality of times are registered with representative values and/or ratios at a second time of the plurality of times; the subject is phenotyped or classified based at least in part on detecting spot-like regions or statistical anomalies in representative values and/or ratios determined for OCT data obtained for a plurality of axial scans; the spot-like regions are determined by comparing spatial standard deviations or pattern deviations to a threshold level; the method further comprises applying a curve fitting technique to model a boundary traversing the bottom of the RPE based on pixels determined to correspond to the RPE, and selecting the sub-RPE region based at least in part on the smooth boundary; the representative value is selected from the group consisting of: attenuation coefficients, integrated attenuation, or a monotonic or near-monotonic proxy measurement; the method further comprises identifying diseased areas of the subject's eye based at least in part on the determined ratio or representative values; the method further comprises: identifying regions of the subject's eye based at least in part on a change in the determined ratios and/or representative values at the plurality of times; the imaging signal has a center wavelength of at least 1 μm; and/or the retinal layer, portion of the retinal layer, combination of retinal layers, sub-RPE layer, portion of a sub-RPE layer, or combination of sub-RPE layers is determined using polarization sensitive optical coherence tomography (PS-OCT).
(22) According to another example, a method of processing ophthalmic image data comprises obtaining optical coherence tomography (OCT) data of a subject's eye for each of a plurality of axial scans, the OCT data including data beyond the retinal pigment epithelium (RPE) of a subject's eye and intensities of an imaging signal as the imaging signal is backscattered by each of a plurality of tissue layers in the subject's eye; determining a ratio of the intensities of the backscattered imaging signal, the ratio being the intensity of a first portion of the backscattered imaging signal corresponding to at least a portion of a retinal layer with respect to the intensity of a second portion of the backscattered imaging signal corresponding to at least a portion of a sub-RPE layer, for each of the plurality of axial scans; determining a representative value based at least in part on the determined ratio for each of the plurality of axial scans; generating a map of the representative values, ratios, the intensities of the first portion of the backscattered imaging signal, or the intensities of the second portion of the backscattered imaging signal; and phenotyping or classifying the subject based at least in part on the generated map.
(23) In various embodiments of the above example, the OCT data is captured at a plurality of times, the map is generated for the OCT data captured at each of the plurality of times, and the subject is phenotyped or classified based at least in part on a change in the generated maps at the plurality of times; the change in the generated maps is determined by performing a statistical measurement between the representative values, ratios, intensities of the first portion, and/or intensities of the second portion at a first time of the plurality of times and the representative values and/or ratios and a second time of the plurality of times; the generated map at a first time of the plurality of times is registered with a generated map at a second time of the plurality of times; the subject is phenotyped or classified based at least in part on detecting spot-like regions or statistical anomalies in the generated maps; the spot-like regions are determined by comparing spatial standard deviations or pattern deviations to a threshold level; the method further comprises applying a curve fitting technique to model a boundary traversing the bottom of the RPE based on pixels determined to correspond to the RPE, and selecting the sub-RPE region based at least in part on the smooth boundary; the representative value is selected from the group consisting of: attenuation coefficients, integrated attenuation, or a monotonic or near-monotonic proxy measurement; the method further comprises: identifying diseased areas of the subject's eye based at least in part on the generated map; the method further comprises: identifying regions of the subject's eye based at least in part on changes between the generated maps at the plurality of times; the step of identifying diseased areas of the subject's eye further comprises: generating seeds of diseased areas, removing outliers of the generated seeds, growing a region encompassed by the generated seeds that were not removed, refining a contour of the grown regions, identifying an area inside the contour as diseased, and outputting the generated map with a contour around the regions identified as diseased or outputting a binary mask of the regions identified as diseased; the step of generating the seeds of diseased areas is performed by removing noise from the generated map and applying a thresholding technique on the generated map; the thresholding technique comprises finding an Otsu threshold of the generated map, comparing the Otsu threshold with a pre-set value, and selecting an intensity threshold based on the comparison, wherein the seeds of diseased areas are generated using pixels of the map that have intensities lower than the selected intensity threshold; the step of removing outliers is performed by grouping connected seed components and applying a distance analysis on the generated seeds; the imaging signal has a center wavelength of at least 1 μm; and/or the retinal layer, portion of the retinal layer, combination of retinal layers, sub-RPE layer, portion of a sub-RPE layer, or combination of sub-RPE layers is determined using polarization sensitive optical coherence tomography (PS-OCT).--, in line 6, col. 4 through line 5, col. 6;
also see: -- (15) Traditional imaging modalities, fundus imaging and fundus autofluorescence imaging, have been used to detect GA. In fundus imaging, GA is defined as a sharply demarcated area exhibiting an apparent absence of the RPE, with visible choroidal vessels and no neovascular AMD. Fundus autofluorescence imaging is based on the autofluorescence properties of AMD-related compounds, such as lipofuscin, that build up in RPE cells. Fundus autofluorescence imaging is probably the most widely applied technique with respect to GA detection at present.
(16) In an emerging OCT technique, GA is associated with increased OCT signal intensities in the choroidal region (i.e., outer to the Bruch's membrane), which arises from the absence of the RPE, other parts of the outer retina, and possibly the choriocapillaris. The RPE and choriocapillaris are two tissue layers, hyperreflective in OCT scans, that normally cause the incident light to scatter, thus partially preventing deeper transmission of light (and therefore OCT signal) into the choroid. OCT allows cross-sectional visualization that permits image readers to characterize microstructural alterations in the different laminae of the retina. Using only one type of scan for documenting both en face and cross-sectional images of the retina, it can therefore provide more detailed insight in retinal alterations of GA patients than fundus autofluorescence imaging…. As a technique, this will serve to increase measurement variability and reduce methodological sensitivity/specificity. For the sub-RPE slab technique, the complexity of the OCT signal in the choroid can lead to some degree of randomness in the integrated signal and resulting analysis. The inner choroid, including the choriocapillaris and Sattler's layer, includes many high intensity pixels, but with great variation both in intensity and spatially. The outer choroid, consisting of Haller's layer, comprises many pixels of lower intensity corresponding to large blood vessels, but the size (both in terms of width and thickness) and spacing of such vessels can vary widely. --, in lines 4-61, col. 3;
also see: --, the enlargement of GA is characterized by the progressive loss of the outer hyperreflective bands corresponding with the RPE/Bruch’s membrane complex and by thinning of the ONL with subsequent approach of the outer plexiform layer toward Bruch’s membrane. There appears to be a high degree of variability in
how the borders of GA change--, in right col. of page 792, and Fig. 1, and Fig. 2, and, -- Since image intensity varies from scan to scan, comparison of MI values among different scans requires the comparison to be gauged relative to the values within each scan. The relationship in MI between the lesions and their surroundings is fairly consistent, so for these purposes the value of the MI at each pixel was assessed relative to the values inside and outside of the boundaries of the GA lesions for each case: MI relative=(MI-Moutside)/(Minside-Moutside), where Minside and Moutside represent the mean values of MI observed inside and outside the boundaries of the GA lesions for the MI en face image from that case, with the boundaries of the GA being determined from the sub-RPE slab en face image from that same OCT scan.
To assess the ability to predict growth at a specific location in the margin by thresholding the MI image, a receiver operating characteristic (ROC) analysis was performed, analyzing locations of progression and nonprogression with their values of MI relative. Because the MI often is decreased at the fovea, this analysis also was repeated with the 1-mm diameter circle around the foveal center excluded. To assess the ability to predict the rate of growth outside the margin, the mean value of MI relative within the 180 µm margin of the GA lesions was calculated, and these mean values
for all the subjects were correlated with the observed 52-week changes in the square root of the lesion area. This square root– area growth rate has been shown previously to be less sensitive than the area growth rate to differences in baseline lesion
area.3 To assess the ability to distinguish between subjects with high and low growth rates, the subjects also were separated into high- and low-MI groups,--, in right col. of 794, as disclosed in Paul F. Stetson’s “OCT Minimum Intensity as a Predictor of Geographic Atrophy Enlargement”; Paul F. Stetson, Zohar Yehoshua, Carlos Alexandre A. Garcia Filho, Renata Portella Nunes, Giovanni Gregori and Philip J. Rosenfeld; Investigative Ophthalmology & Visual Science; Copyright 2014, The Association for Research in Vision and Ophthalmology, Inc.; Published Feb. 2014/vol. 55/No. 2. cited by Reisman, and herein fully incorporated);
DE SISTERNES and Reisman are combinable as they are in the same field of endeavor: analyzing optical coherence tomography data to identify Geographic Atrophy (GA), and diagnosis/detection of Age-related macular degeneration (AMD) eye diseases. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify DE SISTERNES’s method using Reisman’s teachings by including measuring, by the image analysis computing system, one or more attributes within an adjacent area that is adjacent to the area exhibiting geographic atrophy to DE SISTERNES’s analyzing optical coherence tomography data to identify Geographic Atrophy (GA) in order to identifying diseased areas of the subject's eye based at least in part on the generated map and identifying regions of the subject's eye based at least in part on changes between the generated maps at the plurality of times (see Reisman: e.g., Fig. 9A, Fig. 9B, and in lines 4-61, col. 3, and in line 6, col. 4 through line 5, col. 6);
DE SISTERNES as modified by Reisman however still do not explicitly disclose determining, by the image analysis computing system, a predicted enlargement rate based on the one or more attributes within the adjacent area;
YANG discloses determining, by the image analysis computing system, a predicted enlargement rate based on the one or more attributes within the adjacent area (see YANG: e.g., -- [0003] Age-related macular degeneration (AMD) is a leading cause of vision loss in patients 50 years or older. Geographic atrophy (GA) is one of two advanced stages of AMD and is characterized by progressive and irreversible loss of choriocapillaries, retinal pigment epithelium (RPE), and photoreceptors. GA progression varies between patients and currently, no Food and Drug Administration (FDA) accepted treatment for preventing or slowing down the progression of GA exists. Therefore, predicting GA progression in individual patients may be important to researching GA and developing an effective treatment. Currently, the diagnosis and monitoring of GA lesion enlargement may be performed using fundus autofluorescence (FAF) images that are obtained by confocal scanning laser ophthalmoscopy (cSLO). This type of imaging technology, which shows topographic mapping of lipofuscin in RPE, can be used to measure the change in GA lesions over time. Further, FAF images may be used to predict the GA growth rate.--, in [0003]; and, -- to predict GA growth over time using imaging modalities like CFP, FAF, NIR, OCT and OCTA. In general, the GA growth rate has been found to be linear. A recent study on CFP indicated that GA growth rate was strongly correlated to lesion perimeter. Findings from studies on FAF have suggested that lesion shape-descriptive features, surrounding abnormal autofluorescence patterns and previous progression rate were prognostic of GA lesion enlargement. A study using FAF and NIR images showed RPD to be highly predictive of GA lesion growth. A predictive model based on extracted features from OCT volumes demonstrated the ability to predict where GA is likely to grow. Another study on OCT indicated the presence of outer-retinal tubulation may be associated with slower lesion growth. Further, a study on OCTA showed that choriocapillaris flow void could be a precursor to GA lesion growth.--, in [0161]; and, -- The FAF and OCT images included in this study were captured using the same vendor’s devices (Heidelberg Engineering, Inc., Germany), had similar ART values and were of high quality as required for eligibility screening by a central reading center. In various embodiments, more interpretive CNN models or additional explainability techniques can be used to gain insights on the decision-making process, which would help further understand the pathophysiology of GA lesion enlargement and possibly identify new imaging biomarkers. For example, lesion shape features as well as other image extracted features can be explicitly incorporated into statistical models for GA growth rate prediction to increase interpretability and ease of implementation in clinical trial analysis. In addition, images from other modalities (e.g., OCTA, scotopic microperimetric sensitivity) may provide additional predictive value and can be added into the input data for model training as well.
[0190] In summary, the feasibility of utilizing baseline FAF and/or OCT images to predict individual GA lesion area and growth rates using a multi-task deep learning approach is demonstrated.--, in [0189]-[0190]);
DE SISTERNES (as modified by Reisman) and YANG are combinable as they are in the same field of endeavor: analyzing optical coherence tomography data to identify Geographic Atrophy (GA), and diagnosis/detection of Age-related macular degeneration (AMD) eye diseases. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify DE SISTERNES (as modified by Reisman)’s method using YANG’s teachings by including determining, by the image analysis computing system, a predicted enlargement rate based on the one or more attributes within the adjacent area to DE SISTERNES (as modified by Reisman)’s characterization and monitoring macular regions affected by GA in order to predict GA progression in individual patients that is important to researching GA and developing an effective treatment (see YANG: e.g., in [0003], [0161], and [0189]-[0190]).
Re Claim 2, DE SISTERNES as modified by Reisman and YANG further disclose providing, by the image analysis computing system, the predicted enlargement rate for use in at least one of a diagnosis, a determination of an appropriate treatment, and an evaluation of an applied treatment (see YANG: e.g., -- [0003] Age-related macular degeneration (AMD) is a leading cause of vision loss in patients 50 years or older. Geographic atrophy (GA) is one of two advanced stages of AMD and is characterized by progressive and irreversible loss of choriocapillaries, retinal pigment epithelium (RPE), and photoreceptors. GA progression varies between patients and currently, no Food and Drug Administration (FDA) accepted treatment for preventing or slowing down the progression of GA exists. Therefore, predicting GA progression in individual patients may be important to researching GA and developing an effective treatment. Currently, the diagnosis and monitoring of GA lesion enlargement may be performed using fundus autofluorescence (FAF) images that are obtained by confocal scanning laser ophthalmoscopy (cSLO). This type of imaging technology, which shows topographic mapping of lipofuscin in RPE, can be used to measure the change in GA lesions over time. Further, FAF images may be used to predict the GA growth rate.--, in [0003]; and, -- to predict GA growth over time using imaging modalities like CFP, FAF, NIR, OCT and OCTA. In general, the GA growth rate has been found to be linear. A recent study on CFP indicated that GA growth rate was strongly correlated to lesion perimeter. Findings from studies on FAF have suggested that lesion shape-descriptive features, surrounding abnormal autofluorescence patterns and previous progression rate were prognostic of GA lesion enlargement. A study using FAF and NIR images showed RPD to be highly predictive of GA lesion growth. A predictive model based on extracted features from OCT volumes demonstrated the ability to predict where GA is likely to grow. Another study on OCT indicated the presence of outer-retinal tubulation may be associated with slower lesion growth. Further, a study on OCTA showed that choriocapillaris flow void could be a precursor to GA lesion growth.--, in [0161]; and, -- The FAF and OCT images included in this study were captured using the same vendor’s devices (Heidelberg Engineering, Inc., Germany), had similar ART values and were of high quality as required for eligibility screening by a central reading center. In various embodiments, more interpretive CNN models or additional explainability techniques can be used to gain insights on the decision-making process, which would help further understand the pathophysiology of GA lesion enlargement and possibly identify new imaging biomarkers. For example, lesion shape features as well as other image extracted features can be explicitly incorporated into statistical models for GA growth rate prediction to increase interpretability and ease of implementation in clinical trial analysis. In addition, images from other modalities (e.g., OCTA, scotopic microperimetric sensitivity) may provide additional predictive value and can be added into the input data for model training as well.
[0190] In summary, the feasibility of utilizing baseline FAF and/or OCT images to predict individual GA lesion area and growth rates using a multi-task deep learning approach is demonstrated.--, in [0189]-[0190]);
Re Claim 3, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring one or more attributes within the adjacent area that is adjacent to the area exhibiting geographic atrophy includes measuring a distance between a retinal pigment epithelium (RPE) and a Bruch's membrane (BM) within the adjacent area (see Reisman: e.g., -- (7) Clinical trials to evaluate new therapies for non-neovascular AMD require reliable, accurate, and simple means of monitoring GA size and progression. Accurately monitoring GA progression can also help to better understand the pathogenesis of GA and AMD in general, as numerous aspects of AMD pathogenesis are not particularly well understood at present.
(8) In choroideremia, the choriocapillaris (small capillary vessels in the choroid just outer to the Bruch's membrane), the RPE, and photoreceptors (in the later stages of disease) degenerate leading to lost visual function over time. As with GA, a visible thinning of the RPE can often be observed in affected areas in OCT scans. Additionally, due to decreased attenuation of the RPE layer (primarily of the RPE complex, though also of the choriocapillaris and possibly other tissue features, such as the photoreceptors), the signal in the choroid and beyond (e.g., the sclera) appears relatively bright in OCT scans.
(9) Retinitis pigmentosa is a progressive retinal disease that affects the photoreceptors resulting in a severe loss of vision. Atrophy can be observed in the outer segments (OS) of receptor and other layers, such as the outer nuclear layer (ONL). Usually the thinning of the OS layer precedes changes in other receptor layers. In the case of RP, the visible thinning of the OS, possibly the RPE, ONL, and total retinal thickness can be observed with the NFL layer intact or even thicker.
(10) There are four main processes in age-related macular degeneration pathogenesis, which preferentially affects the macula. In the first, lipofuscin formation, RPE metabolic insufficiency associated with aging leads to progressive accumulation of lipofuscin granules (a roughly even mixture of lipids and proteins) in the RPE. This is also related to failure to clear some metabolites from outer segment phagocytosis from the RPE. A lipofuscin component known as A2E is known to be a cytotoxic molecule, capable of generating free radicals, damaging DNA, etc.
(11) Next, drusen formation is the result of extracellular deposits collecting between the RPE and Bruch's membrane. While most elderly individuals have a small number of “hard” drusen, the presence of numerous “hard” or “soft” drusen (especially the soft variety, which are typically larger in area), particularly when accompanied by pigment changes, is thought to be an early indicator of AMD. Drusen formation is also thought to relate to inflammatory processes as well as CFH gene allele Y402H.--, in lines 4-49, col. 2; and,
also see: -- (15) Traditional imaging modalities, fundus imaging and fundus autofluorescence imaging, have been used to detect GA. In fundus imaging, GA is defined as a sharply demarcated area exhibiting an apparent absence of the RPE, with visible choroidal vessels and no neovascular AMD. Fundus autofluorescence imaging is based on the autofluorescence properties of AMD-related compounds, such as lipofuscin, that build up in RPE cells. Fundus autofluorescence imaging is probably the most widely applied technique with respect to GA detection at present.
(16) In an emerging OCT technique, GA is associated with increased OCT signal intensities in the choroidal region (i.e., outer to the Bruch's membrane), which arises from the absence of the RPE, other parts of the outer retina, and possibly the choriocapillaris. The RPE and choriocapillaris are two tissue layers, hyperreflective in OCT scans, that normally cause the incident light to scatter, thus partially preventing deeper transmission of light (and therefore OCT signal) into the choroid. OCT allows cross-sectional visualization that permits image readers to characterize microstructural alterations in the different laminae of the retina. Using only one type of scan for documenting both en face and cross-sectional images of the retina, it can therefore provide more detailed insight in retinal alterations of GA patients than fundus autofluorescence imaging…. As a technique, this will serve to increase measurement variability and reduce methodological sensitivity/specificity. For the sub-RPE slab technique, the complexity of the OCT signal in the choroid can lead to some degree of randomness in the integrated signal and resulting analysis. The inner choroid, including the choriocapillaris and Sattler's layer, includes many high intensity pixels, but with great variation both in intensity and spatially. The outer choroid, consisting of Haller's layer, comprises many pixels of lower intensity corresponding to large blood vessels, but the size (both in terms of width and thickness) and spacing of such vessels can vary widely. --, in lines 4-61, col. 3).
Re Claim 4, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring the distance between the RPE and the BM includes identifying a pixel above the BM having a maximum optical attenuation coefficient value (see Reisman: e.g., -- pixel intensity integrations as a fixed-depth integration (such that integration occurs over a fixed number of pixels, corresponding to a constant depth) for the outer layer calculation. However, it should be noted that for the outer reference layer, it is mathematically equivalent to take either the sum over a fixed depth or to take the mean pixel intensity over either a fixed or variable depth. Calculating over a variable depth may involve making a modeling approximation, but the calculation results can still be expected to convey physical meaning. Averaging over a variable depth may allow the calculation methodology to increase the degree of averaging (i.e., include more pixels in the mean calculation), which can serve to reduce the effects of Gaussian and/or speckle noise. Therefore, even though it may not be ideal from a physical equation point of view, the reduction in noise may outweigh physical concerns. Furthermore, other calculation schemes could be both practical and beneficial, considering the complex nature of choroidal signal in OCT images. For example, taking the maximum intensity projection or an average of the n maximum values within a region of interest might in some circumstances yield more consistent image processing results than simple integrations. Likewise, taking the median or some arbitrary quantile of the pixel signal intensity levels could also be performed. Similar variations to the inner layer combination calculations can be performed. For example, instead of integrating the signal, the mean pixel value can be calculated. It should be noted that if the inner layer calculation depth were to vary from A-scan to A-scan, this might reduce the accuracy of the ratio's physical meaning, but the calculation results could still be a useful towards identifying regions of geographic atrophy.--, in line 31, col. 9, through line62, col. 62).
Re Claim 5, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring the one or more attributes within the adjacent area includes determining a mean and a standard deviation of the measured distance between the RPE and the BM within the adjacent area (see Reisman: e.g., -- pixel intensity integrations as a fixed-depth integration (such that integration occurs over a fixed number of pixels, corresponding to a constant depth) for the outer layer calculation. However, it should be noted that for the outer reference layer, it is mathematically equivalent to take either the sum over a fixed depth or to take the mean pixel intensity over either a fixed or variable depth. Calculating over a variable depth may involve making a modeling approximation, but the calculation results can still be expected to convey physical meaning. Averaging over a variable depth may allow the calculation methodology to increase the degree of averaging (i.e., include more pixels in the mean calculation), which can serve to reduce the effects of Gaussian and/or speckle noise. Therefore, even though it may not be ideal from a physical equation point of view, the reduction in noise may outweigh physical concerns. Furthermore, other calculation schemes could be both practical and beneficial, considering the complex nature of choroidal signal in OCT images. For example, taking the maximum intensity projection or an average of the n maximum values within a region of interest might in some circumstances yield more consistent image processing results than simple integrations. Likewise, taking the median or some arbitrary quantile of the pixel signal intensity levels could also be performed. Similar variations to the inner layer combination calculations can be performed. For example, instead of integrating the signal, the mean pixel value can be calculated. It should be noted that if the inner layer calculation depth were to vary from A-scan to A-scan, this might reduce the accuracy of the ratio's physical meaning, but the calculation results could still be a useful towards identifying regions of geographic atrophy….The individual ratios may then be combined by taking the sum, mean, median, quantile or other similar statistical function. Another similar method envisioned involves calculating a ratio for every combination of pixels in the inner and outer regions. The sum of every ratio for each pixel in the outer region can then be calculated to find a single ratio value for each outer region pixel. Finally, the sum, mean, median, or similar statistical function can be calculated for each of the single ratio values, which can optionally be scaled by a desired constant.--, in line 31, col. 9, through line62, col. 62).
Re Claim 6, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring one or more attributes within the adjacent area that is adjacent to the area exhibiting geographic atrophy includes measuring an outer retinal layer thickness within the adjacent area (see DE SISTERNES: e.g., --A system and method for use with optical coherence tomography (OCT) data to identify a target pathology extracts multiple pathology-characteristic images from the OCT data. The extracted pathology-characteristic images may include a mixture of OCT structural images (including retinal layer thickness information) and OCT angiography images. Optionally, other pathology-characteristic images and data maps (mapped to corresponding positions in the OCT data), such as fundus images and visual field test maps may be accessed as additional pathology-characteristic images. Each pathology-characteristic image defines a different image channel (e.g., “color channel”) per pixel in a composite, channel-coded image, which is then used to train a neural network to search for the target pathology in OCT data. The trained neural network may then receive new composite, channel-coded image and identify/segment the target pathology within the new channel-coded image.--, in abstract, and, --a method of analyzing optical coherence tomography data to identify a target pathology. More specifically, it is directed to analyzing optical coherence tomography data to identify Geographic Atrophy (GA)… [0002] Age-related macular degeneration (AMD) is an eye disease most common in the older population and arising from damage to the macula that leads to loss of central vision. Some patients with age-related macular degeneration (AMD) develop geographic atrophy (GA), which refers to regions of the retina where cells waste away and die. Geographic Atrophy (GA) is a condition of the macula present at the advanced stages of non-exudative macular degeneration. GA has a characteristic appearance resulting from the loss of the photoreceptor layer, retinal pigment epithelium (RPE), and choriocapillaris. GA typically first appears in the parafoveal location and progresses around the fovea and then through the fovea with loss of central visual acuity….characterization and monitoring macular regions affected by GA is fundamental for patient diagnosis, monitoring and management as well as for treatment research purposes.--, in [0002], and, --The OCT data 21 may include OCT structural data and/or OCTA flow data, and the extracted metrics may include OCT-based metrics extracted from the OCT structural data, such as retinal layer thicknesses, distances of a specific A-scan to a specific retinal structure (e.g., distance to the fovea center), layer integrity (e.g., the loss of a specific layer), sub-RPE reflectivity, inner RPE reflectivity, overall retinal thickness, and/or the optical attenuation coefficient (OAC).
[0049] The optical attenuation coefficient (OAC) is an optical property of a medium that determines how the power of a coherent light beam propagating through the (e.g., turbid) medium (e.g., tissue) is attenuated along its path due to scattering and absorption. The irradiance (power per unit area) of the coherent light beam that propagates through a (e.g., homogeneous) medium is given by Lambert-Beer's law: L(z)=L.sub.0e.sup.−μz, where L(z) is the irradiance of the beam after traveling through the medium over a distance z, L.sub.0 is the irradiance of the incident light beam and μ is the optical attenuation coefficient. Large attenuation coefficients result in a quick and exponential decline of the irradiance of the coherent light beam with depth. Because the OAC is an optical property of the medium, determining the OAC provides information on the composition of this medium. Applicants propose that providing the OAC (per A-scan) as one of the extracted metrics may be beneficial identify specific pathologies (e.g., GA), particularly since it can be indicative of the current state (e.g., light attenuating state) of tissued at specific A-scan positions. An example of how the OAC may be determined/calculated is provided in “Depth-Resolved Model-Based Reconstruction of Attenuation Coefficients in Optical Coherence Tomography”, by K. A. Vermeer et al., Biomedical Optics Express, Vol. 5, Issue 1, pp. 322-337 (2014). A discussion of previous applications of OAC may be found in “In Vivo Tissue Injury Mapping Using Optical Coherence Tomography Based Methods”, by Utka Baran et al., Applied Optics, Vol. 54, No. 21, Jul. 20, 2015.--, in [0048]-[0050]; and, --[0060] FIG. 6 illustrates a general workflow of the present invention, including the U-net architecture of the proof of concept implementation. As shown, OCT Data 21 is accessed/acquired and multiple pathology characteristic images PCI's are defined from the OCT data 21, as described above. For example, the pathology characteristic images PCI's may include a sub-RPE reflectivity image, an inner RPE reflectivity image, Retinal thickness, and/or optical attenuation coefficient (OAC) as described above in reference to FIGS. 3 and 4. Optionally, other pathology characteristic image may also be used, such an en face images/maps of specific retinal layer thicknesses or layer integrity, OCTA flow images (such as flow at, or within the vicinity of, the choriocapillaris), an/or other pathology characteristic data as described above in reference to FIG. 5. the pathology characteristic images are then combined into a different (e.g., color or monochrome value) channels of a channel-coded image 27, which is then submitted to machine learning model 29, which is herein implemented using a U-Net architecture. A discussion of a U-Net architecture is provided below.--, in [0059]-[0061]).
Re Claim 7, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring the outer retinal layer thickness within the adjacent area includes:
determining a location of the retinal pigment epithelium (RPE) by identifying a pixel above the BM having a maximum optical coefficient value (see Reisman: e.g., -- pixel intensity integrations as a fixed-depth integration (such that integration occurs over a fixed number of pixels, corresponding to a constant depth) for the outer layer calculation. However, it should be noted that for the outer reference layer, it is mathematically equivalent to take either the sum over a fixed depth or to take the mean pixel intensity over either a fixed or variable depth. Calculating over a variable depth may involve making a modeling approximation, but the calculation results can still be expected to convey physical meaning. Averaging over a variable depth may allow the calculation methodology to increase the degree of averaging (i.e., include more pixels in the mean calculation), which can serve to reduce the effects of Gaussian and/or speckle noise. Therefore, even though it may not be ideal from a physical equation point of view, the reduction in noise may outweigh physical concerns. Furthermore, other calculation schemes could be both practical and beneficial, considering the complex nature of choroidal signal in OCT images. For example, taking the maximum intensity projection or an average of the n maximum values within a region of interest might in some circumstances yield more consistent image processing results than simple integrations. Likewise, taking the median or some arbitrary quantile of the pixel signal intensity levels could also be performed. Similar variations to the inner layer combination calculations can be performed. For example, instead of integrating the signal, the mean pixel value can be calculated. It should be noted that if the inner layer calculation depth were to vary from A-scan to A-scan, this might reduce the accuracy of the ratio's physical meaning, but the calculation results could still be a useful towards identifying regions of geographic atrophy….The individual ratios may then be combined by taking the sum, mean, median, quantile or other similar statistical function. Another similar method envisioned involves calculating a ratio for every combination of pixels in the inner and outer regions. The sum of every ratio for each pixel in the outer region can then be calculated to find a single ratio value for each outer region pixel. Finally, the sum, mean, median, or similar statistical function can be calculated for each of the single ratio values, which can optionally be scaled by a desired constant.--, in line 31, col. 9, through line62, col. 62; also see: -- (7) Clinical trials to evaluate new therapies for non-neovascular AMD require reliable, accurate, and simple means of monitoring GA size and progression. Accurately monitoring GA progression can also help to better understand the pathogenesis of GA and AMD in general, as numerous aspects of AMD pathogenesis are not particularly well understood at present.
(8) In choroideremia, the choriocapillaris (small capillary vessels in the choroid just outer to the Bruch's membrane), the RPE, and photoreceptors (in the later stages of disease) degenerate leading to lost visual function over time. As with GA, a visible thinning of the RPE can often be observed in affected areas in OCT scans. Additionally, due to decreased attenuation of the RPE layer (primarily of the RPE complex, though also of the choriocapillaris and possibly other tissue features, such as the photoreceptors), the signal in the choroid and beyond (e.g., the sclera) appears relatively bright in OCT scans.
(9) Retinitis pigmentosa is a progressive retinal disease that affects the photoreceptors resulting in a severe loss of vision. Atrophy can be observed in the outer segments (OS) of receptor and other layers, such as the outer nuclear layer (ONL). Usually the thinning of the OS layer precedes changes in other receptor layers. In the case of RP, the visible thinning of the OS, possibly the RPE, ONL, and total retinal thickness can be observed with the NFL layer intact or even thicker.
(10) There are four main processes in age-related macular degeneration pathogenesis, which preferentially affects the macula. In the first, lipofuscin formation, RPE metabolic insufficiency associated with aging leads to progressive accumulation of lipofuscin granules (a roughly even mixture of lipids and proteins) in the RPE. This is also related to failure to clear some metabolites from outer segment phagocytosis from the RPE. A lipofuscin component known as A2E is known to be a cytotoxic molecule, capable of generating free radicals, damaging DNA, etc.
(11) Next, drusen formation is the result of extracellular deposits collecting between the RPE and Bruch's membrane. While most elderly individuals have a small number of “hard” drusen, the presence of numerous “hard” or “soft” drusen (especially the soft variety, which are typically larger in area), particularly when accompanied by pigment changes, is thought to be an early indicator of AMD. Drusen formation is also thought to relate to inflammatory processes as well as CFH gene allele Y402H.--, in lines 4-49, col. 2; and,
also see: -- (15) Traditional imaging modalities, fundus imaging and fundus autofluorescence imaging, have been used to detect GA. In fundus imaging, GA is defined as a sharply demarcated area exhibiting an apparent absence of the RPE, with visible choroidal vessels and no neovascular AMD. Fundus autofluorescence imaging is based on the autofluorescence properties of AMD-related compounds, such as lipofuscin, that build up in RPE cells. Fundus autofluorescence imaging is probably the most widely applied technique with respect to GA detection at present.
(16) In an emerging OCT technique, GA is associated with increased OCT signal intensities in the choroidal region (i.e., outer to the Bruch's membrane), which arises from the absence of the RPE, other parts of the outer retina, and possibly the choriocapillaris. The RPE and choriocapillaris are two tissue layers, hyperreflective in OCT scans, that normally cause the incident light to scatter, thus partially preventing deeper transmission of light (and therefore OCT signal) into the choroid. OCT allows cross-sectional visualization that permits image readers to characterize microstructural alterations in the different laminae of the retina. Using only one type of scan for documenting both en face and cross-sectional images of the retina, it can therefore provide more detailed insight in retinal alterations of GA patients than fundus autofluorescence imaging…. As a technique, this will serve to increase measurement variability and reduce methodological sensitivity/specificity. For the sub-RPE slab technique, the complexity of the OCT signal in the choroid can lead to some degree of randomness in the integrated signal and resulting analysis. The inner choroid, including the choriocapillaris and Sattler's layer, includes many high intensity pixels, but with great variation both in intensity and spatially. The outer choroid, consisting of Haller's layer, comprises many pixels of lower intensity corresponding to large blood vessels, but the size (both in terms of width and thickness) and spacing of such vessels can vary widely. --, in lines 4-61, col. 3);
determining a location of an inner boundary of an outer plexiform layer (OPL) (see DE SISTERNES: e.g., --[0096] A variety of ways to create B-scans are known in the art including but not limited to: along the horizontal or x-direction, along the vertical or y-direction, along the diagonal of x and y, or in a circular or spiral pattern. B-scans may be in the x-z dimensions but may be any cross-sectional image that includes the z-dimension. An example OCT B-scan image of a normal retina of a human eye is illustrated in FIG. 13. An OCT B-scan of the retinal provides a view of the structure of retinal tissue. For illustration purposes, FIG. 13 identifies various canonical retinal layers and layer boundaries. The identified retinal boundary layers include (from top to bottom): the inner limiting membrane (ILM) Lyer1, the retinal nerve fiber layer (BNFL or NFL) Layr2, the ganglion cell layer (GCL) Layr3, the inner plexiform layer (IPL) Layr4, the inner nuclear layer (INL) Layr5, the outer plexiform layer (OPL) Layr6, the outer nuclear layer (ONL) Layr7, the junction between the outer segments (OS) and inner segments (IS) (indicated by reference character Layr8) of the photoreceptors, the external or outer limiting membrane (ELM or OLM) Layr9, the retinal pigment epithelium (RPE) Layr10, and the Bruch's membrane (BM) Layr11.--, in [0096]); and
measuring the distance between the location of the RPE and the OPL within the adjacent area (see DE SISTERNES: e.g., --[0096] A variety of ways to create B-scans are known in the art including but not limited to: along the horizontal or x-direction, along the vertical or y-direction, along the diagonal of x and y, or in a circular or spiral pattern. B-scans may be in the x-z dimensions but may be any cross-sectional image that includes the z-dimension. An example OCT B-scan image of a normal retina of a human eye is illustrated in FIG. 13. An OCT B-scan of the retinal provides a view of the structure of retinal tissue. For illustration purposes, FIG. 13 identifies various canonical retinal layers and layer boundaries. The identified retinal boundary layers include (from top to bottom): the inner limiting membrane (ILM) Lyer1, the retinal nerve fiber layer (BNFL or NFL) Layr2, the ganglion cell layer (GCL) Layr3, the inner plexiform layer (IPL) Layr4, the inner nuclear layer (INL) Layr5, the outer plexiform layer (OPL) Layr6, the outer nuclear layer (ONL) Layr7, the junction between the outer segments (OS) and inner segments (IS) (indicated by reference character Layr8) of the photoreceptors, the external or outer limiting membrane (ELM or OLM) Layr9, the retinal pigment epithelium (RPE) Layr10, and the Bruch's membrane (BM) Layr11.--, in [0096]).
Re Claim 8, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring the one or more attributes within the adjacent area includes determining a mean and a standard deviation of the outer retinal layer thickness within the adjacent area (see Reisman: e.g., Fig. 9A, Fig. 9B. and, -- processing ophthalmic image data for detecting geographical atrophy is desired that overcomes the above limitations and deficiencies of the current systems and methods.
(20) According to one example, a method of processing ophthalmic image data comprises obtaining optical coherence tomography (OCT) data of a subject's eye, the OCT data including data beyond the retinal pigment epithelium (RPE) of a subject's eye and intensities of an imaging signal as the imaging signal is backscattered by each of a plurality of tissue layers in the subject's eye; determining a ratio of the intensities of the backscattered imaging signal, the ratio being the intensity of a first portion of the backscattered imaging signal corresponding to at least a portion of a retinal layer with respect to the intensity of a second portion of the backscattered imaging signal corresponding to at least a portion of a sub-RPE layer; determining a representative value based at least in part on the determined ratio; and phenotyping or classifying the subject based at least in part on the determined representative value.
(21) In various embodiments of the above example, the OCT data is captured at a plurality of times, a ratio and representative value are determined for the OCT data captured at each of the plurality of times, and the subject is phenotyped or classified based at least in part on a change in the determined ratios and/or representative values at the plurality of times; the change in the representative values and/or ratios is determined by performing a statistical measurement between the representative values and/or ratios at a first time of the plurality of times and the representative values and/or ratios and a second time of the plurality of times; the representative values and/or ratios at a first time of the plurality of times are registered with representative values and/or ratios at a second time of the plurality of times; the subject is phenotyped or classified based at least in part on detecting spot-like regions or statistical anomalies in representative values and/or ratios determined for OCT data obtained for a plurality of axial scans; the spot-like regions are determined by comparing spatial standard deviations or pattern deviations to a threshold level; the method further comprises applying a curve fitting technique to model a boundary traversing the bottom of the RPE based on pixels determined to correspond to the RPE, and selecting the sub-RPE region based at least in part on the smooth boundary; the representative value is selected from the group consisting of: attenuation coefficients, integrated attenuation, or a monotonic or near-monotonic proxy measurement; the method further comprises identifying diseased areas of the subject's eye based at least in part on the determined ratio or representative values; the method further comprises: identifying regions of the subject's eye based at least in part on a change in the determined ratios and/or representative values at the plurality of times; the imaging signal has a center wavelength of at least 1 μm; and/or the retinal layer, portion of the retinal layer, combination of retinal layers, sub-RPE layer, portion of a sub-RPE layer, or combination of sub-RPE layers is determined using polarization sensitive optical coherence tomography (PS-OCT).
(22) According to another example, a method of processing ophthalmic image data comprises obtaining optical coherence tomography (OCT) data of a subject's eye for each of a plurality of axial scans, the OCT data including data beyond the retinal pigment epithelium (RPE) of a subject's eye and intensities of an imaging signal as the imaging signal is backscattered by each of a plurality of tissue layers in the subject's eye; determining a ratio of the intensities of the backscattered imaging signal, the ratio being the intensity of a first portion of the backscattered imaging signal corresponding to at least a portion of a retinal layer with respect to the intensity of a second portion of the backscattered imaging signal corresponding to at least a portion of a sub-RPE layer, for each of the plurality of axial scans; determining a representative value based at least in part on the determined ratio for each of the plurality of axial scans; generating a map of the representative values, ratios, the intensities of the first portion of the backscattered imaging signal, or the intensities of the second portion of the backscattered imaging signal; and phenotyping or classifying the subject based at least in part on the generated map.
(23) In various embodiments of the above example, the OCT data is captured at a plurality of times, the map is generated for the OCT data captured at each of the plurality of times, and the subject is phenotyped or classified based at least in part on a change in the generated maps at the plurality of times; the change in the generated maps is determined by performing a statistical measurement between the representative values, ratios, intensities of the first portion, and/or intensities of the second portion at a first time of the plurality of times and the representative values and/or ratios and a second time of the plurality of times; the generated map at a first time of the plurality of times is registered with a generated map at a second time of the plurality of times; the subject is phenotyped or classified based at least in part on detecting spot-like regions or statistical anomalies in the generated maps; the spot-like regions are determined by comparing spatial standard deviations or pattern deviations to a threshold level; the method further comprises applying a curve fitting technique to model a boundary traversing the bottom of the RPE based on pixels determined to correspond to the RPE, and selecting the sub-RPE region based at least in part on the smooth boundary; the representative value is selected from the group consisting of: attenuation coefficients, integrated attenuation, or a monotonic or near-monotonic proxy measurement; the method further comprises: identifying diseased areas of the subject's eye based at least in part on the generated map; the method further comprises: identifying regions of the subject's eye based at least in part on changes between the generated maps at the plurality of times; the step of identifying diseased areas of the subject's eye further comprises: generating seeds of diseased areas, removing outliers of the generated seeds, growing a region encompassed by the generated seeds that were not removed, refining a contour of the grown regions, identifying an area inside the contour as diseased, and outputting the generated map with a contour around the regions identified as diseased or outputting a binary mask of the regions identified as diseased; the step of generating the seeds of diseased areas is performed by removing noise from the generated map and applying a thresholding technique on the generated map; the thresholding technique comprises finding an Otsu threshold of the generated map, comparing the Otsu threshold with a pre-set value, and selecting an intensity threshold based on the comparison, wherein the seeds of diseased areas are generated using pixels of the map that have intensities lower than the selected intensity threshold; the step of removing outliers is performed by grouping connected seed components and applying a distance analysis on the generated seeds; the imaging signal has a center wavelength of at least 1 μm; and/or the retinal layer, portion of the retinal layer, combination of retinal layers, sub-RPE layer, portion of a sub-RPE layer, or combination of sub-RPE layers is determined using polarization sensitive optical coherence tomography (PS-OCT).--, in line 6, col. 4 through line 5, col. 6).
Re Claim 9, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring the one or more attributes within the adjacent area includes measuring choriocapillaris flow deficits within the adjacent area(see DE SISTERNES: e.g., --A system and method for use with optical coherence tomography (OCT) data to identify a target pathology extracts multiple pathology-characteristic images from the OCT data. The extracted pathology-characteristic images may include a mixture of OCT structural images (including retinal layer thickness information) and OCT angiography images. Optionally, other pathology-characteristic images and data maps (mapped to corresponding positions in the OCT data), such as fundus images and visual field test maps may be accessed as additional pathology-characteristic images. Each pathology-characteristic image defines a different image channel (e.g., “color channel”) per pixel in a composite, channel-coded image, which is then used to train a neural network to search for the target pathology in OCT data. The trained neural network may then receive new composite, channel-coded image and identify/segment the target pathology within the new channel-coded image.--, in abstract, and, --a method of analyzing optical coherence tomography data to identify a target pathology. More specifically, it is directed to analyzing optical coherence tomography data to identify Geographic Atrophy (GA)… [0002] Age-related macular degeneration (AMD) is an eye disease most common in the older population and arising from damage to the macula that leads to loss of central vision. Some patients with age-related macular degeneration (AMD) develop geographic atrophy (GA), which refers to regions of the retina where cells waste away and die. Geographic Atrophy (GA) is a condition of the macula present at the advanced stages of non-exudative macular degeneration. GA has a characteristic appearance resulting from the loss of the photoreceptor layer, retinal pigment epithelium (RPE), and choriocapillaris. GA typically first appears in the parafoveal location and progresses around the fovea and then through the fovea with loss of central visual acuity….characterization and monitoring macular regions affected by GA is fundamental for patient diagnosis, monitoring and management as well as for treatment research purposes.--, in [0002], and, --GA may result in a progressive loss of vision, particularly central vision. However, GA may start with loss of vision outside the central area, and progress toward the center over time. Thus, it is advantageous to incorporate information from visual field test results FV. A visual field test is a method of measuring an individual's entire scope of vision, e.g., their central and peripheral (side) vision. Visual field testing is a way to map the visual fields of each eye individually and can detect blind spots (scotomas) as well as more subtle areas of dim vision. A campimeter, or “perimeter,” is a dedicated machine/device/system that applies a visual field test to a patient. A more in-depth discussion of perimeters and visual field testing is provided below. All, or select parts of a visual field test (such as the VF gray scale or numerical gray scale mapped to corresponding retinal locations) may be incorporated into the present multi-channel composite image 27.--, in [0055]-[0060], and [0105]-[0107] also see Reisman: e.g., -- (15) Traditional imaging modalities, fundus imaging and fundus autofluorescence imaging, have been used to detect GA. In fundus imaging, GA is defined as a sharply demarcated area exhibiting an apparent absence of the RPE, with visible choroidal vessels and no neovascular AMD. Fundus autofluorescence imaging is based on the autofluorescence properties of AMD-related compounds, such as lipofuscin, that build up in RPE cells. Fundus autofluorescence imaging is probably the most widely applied technique with respect to GA detection at present.
(16) In an emerging OCT technique, GA is associated with increased OCT signal intensities in the choroidal region (i.e., outer to the Bruch's membrane), which arises from the absence of the RPE, other parts of the outer retina, and possibly the choriocapillaris. The RPE and choriocapillaris are two tissue layers, hyperreflective in OCT scans, that normally cause the incident light to scatter, thus partially preventing deeper transmission of light (and therefore OCT signal) into the choroid. OCT allows cross-sectional visualization that permits image readers to characterize microstructural alterations in the different laminae of the retina. Using only one type of scan for documenting both en face and cross-sectional images of the retina, it can therefore provide more detailed insight in retinal alterations of GA patients than fundus autofluorescence imaging…. As a technique, this will serve to increase measurement variability and reduce methodological sensitivity/specificity. For the sub-RPE slab technique, the complexity of the OCT signal in the choroid can lead to some degree of randomness in the integrated signal and resulting analysis. The inner choroid, including the choriocapillaris and Sattler's layer, includes many high intensity pixels, but with great variation both in intensity and spatially. The outer choroid, consisting of Haller's layer, comprises many pixels of lower intensity corresponding to large blood vessels, but the size (both in terms of width and thickness) and spacing of such vessels can vary widely. --, in lines 4-61, col. 3).
Re Claim 10, DE SISTERNES as modified by Reisman and YANG further disclose wherein measuring one or more attributes within the adjacent area that is adjacent to the area exhibiting geographic atrophy includes measuring the one or more attributes within:
a 1-degree rim region that extends from 0 µm to 300 µm outside the area exhibiting geographic atrophy (see DE SISTERNES: e.g., -- IG. 4B shows a channel-coded image (multi-channel composite image) in accord with the present invention, where each image channel (e.g., color channel) embodies a different pathology-specific characteristic (embodies a different metric group). In the present example, the red channel (or medium gray in a black-and-white, monochrome image) comprises sub-RPE reflectivity data. To gather metrics for the red channel, a 300 μm slab is defined outer to the RPE layer and into the choroid vicinity, with surface limits specified between the RPE layer plus offsets of 50 μm and 350 μm, respectively. The OCT signal within this slab is filtered for noise removal and then processed so the signal at each A-scan location has a constantly decreasing function with increase of depth, filling “valleys” in the signal. That is, for each particular pixel in an A-scan, the value is set to be as the highest value recorded in such A-scan from the considered pixel to increasing depths within the defined slab limits. This operation is set to eliminate lower value signal originated from the presence of choroidal blood vessels. The resulting data is projected into an en face image by averaging the pixel values within the slab limit definition for each A-scan. The resulting values of the en face image are then normalized to be in the range between 0 and 1. The goal of this slab is to characterize the increased reflectivity present in the choroid in GA regions.
[0052] In the present example, the green channel (e.g., light gray in a black-and-white, monochrome image) encompasses inner RPE reflectivity. To gather metrics for the green channel, a 20 μm slab is defined inner to the RPE-Fit layer (an estimation of the Bruch's membrane curvature set at the level of the RPE centerline), with surface limits specified between the RPE-Fit layer and offsets of minus 50 μm and minus 30 μm, respectively…[0053] In the present example, the blue channel (or dark gray in a black-and-white, monochrome image) encompasses retinal thickness. To gather metrics for the blue channel, the distance between the ILM layer and the RPE-Fit layer (retinal thickness) is measured for each A-scan location and projected into an en face image. The recorded values are then scaled in an inverted linear operation to take values from 0 to 1 so that a retinal thickness of 100 μm takes the value of 1 and a thickness of 350 μm takes the value of 0. The goal of this slab is to characterize the localized regions of retinal thinning and collapse characteristic of GA presence.--, in [0051]-[0052]);
an additional 1-degree rim region that extends from 300 µm outside the area exhibiting geographic atrophy to 600 µm outside the area exhibiting geographic atrophy (see DE SISTERNES: e.g., -- IG. 4B shows a channel-coded image (multi-channel composite image) in accord with the present invention, where each image channel (e.g., color channel) embodies a different pathology-specific characteristic (embodies a different metric group). In the present example, the red channel (or medium gray in a black-and-white, monochrome image) comprises sub-RPE reflectivity data. To gather metrics for the red channel, a 300 μm slab is defined outer to the RPE layer and into the choroid vicinity, with surface limits specified between the RPE layer plus offsets of 50 μm and 350 μm, respectively. The OCT signal within this slab is filtered for noise removal and then processed so the signal at each A-scan location has a constantly decreasing function with increase of depth, filling “valleys” in the signal. That is, for each particular pixel in an A-scan, the value is set to be as the highest value recorded in such A-scan from the considered pixel to increasing depths within the defined slab limits. This operation is set to eliminate lower value signal originated from the presence of choroidal blood vessels. The resulting data is projected into an en face image by averaging the pixel values within the slab limit definition for each A-scan. The resulting values of the en face image are then normalized to be in the range between 0 and 1. The goal of this slab is to characterize the increased reflectivity present in the choroid in GA regions.
[0052] In the present example, the green channel (e.g., light gray in a black-and-white, monochrome image) encompasses inner RPE reflectivity. To gather metrics for the green channel, a 20 μm slab is defined inner to the RPE-Fit layer (an estimation of the Bruch's membrane curvature set at the level of the RPE centerline), with surface limits specified between the RPE-Fit layer and offsets of minus 50 μm and minus 30 μm, respectively….[0053] In the present example, the blue channel (or dark gray in a black-and-white, monochrome image) encompasses retinal thickness. To gather metrics for the blue channel, the distance between the ILM layer and the RPE-Fit layer (retinal thickness) is measured for each A-scan location and projected into an en face image. The recorded values are then scaled in an inverted linear operation to take values from 0 to 1 so that a retinal thickness of 100 μm takes the value of 1 and a thickness of 350 μm takes the value of 0. The goal of this slab is to characterize the localized regions of retinal thinning and collapse characteristic of GA presence.--, in [0051]-[0053]);
a 2-degree rim region that extends from 0 µm to 600 µm outside the area exhibiting geographic atrophy (see DE SISTERNES: e.g., -- IG. 4B shows a channel-coded image (multi-channel composite image) in accord with the present invention, where each image channel (e.g., color channel) embodies a different pathology-specific characteristic (embodies a different metric group). In the present example, the red channel (or medium gray in a black-and-white, monochrome image) comprises sub-RPE reflectivity data. To gather metrics for the red channel, a 300 μm slab is defined outer to the RPE layer and into the choroid vicinity, with surface limits specified between the RPE layer plus offsets of 50 μm and 350 μm, respectively. The OCT signal within this slab is filtered for noise removal and then processed so the signal at each A-scan location has a constantly decreasing function with increase of depth, filling “valleys” in the signal. That is, for each particular pixel in an A-scan, the value is set to be as the highest value recorded in such A-scan from the considered pixel to increasing depths within the defined slab limits. This operation is set to eliminate lower value signal originated from the presence of choroidal blood vessels. The resulting data is projected into an en face image by averaging the pixel values within the slab limit definition for each A-scan. The resulting values of the en face image are then normalized to be in the range between 0 and 1. The goal of this slab is to characterize the increased reflectivity present in the choroid in GA regions.
[0052] In the present example, the green channel (e.g., light gray in a black-and-white, monochrome image) encompasses inner RPE reflectivity. To gather metrics for the green channel, a 20 μm slab is defined inner to the RPE-Fit layer (an estimation of the Bruch's membrane curvature set at the level of the RPE centerline), with surface limits specified between the RPE-Fit layer and offsets of minus 50 μm and minus 30 μm, respectively.--, in [0051]-[0052];
Also see YANG: e.g., --[0073] Resized and normalized FAF/OCT images can be used as input (e.g., fused input) in lesion area analytical system 114. For example, as described in further detail below, FAF images can be resized to 512x512 pixels and normalized between 0 and 1. For OCT volumes (e.g., 3D images), pre-processing can be performed prior to using the images. For example, a histogram matching can be applied first to calibrate differences in image intensity between B -scans, then each B-scan can be flattened along Bruch’s membrane (BM). As a non-limiting example, three en-face maps, averaged over full depth, above-BM, and sub-BM depths can be combined as a three-channel input (e.g., fused input). In various embodiments, OCT pre-processing may include general image contrast improvement (or adjustment) with or without volume flattening. The flattening can be along any layers, such as an internal limiting membrane (ILM) of the retina. Alternately or in addition, OCT cross-sectional images can be integrated into image input channels. As further provided below in detail, both above-BM and sub-BM depth can be set at 100 pixels (390 pm), whereas the en-face maps can be resized to 512x512 pixels and normalized between 0 and 1. Setting pixel dimensions and normalized intensity to between 0 and 1 for any two or more sources of imaging data (e.g., any two of the FAF images 110, the OCT images 112, and/or the IR images 113) enable fusing of the aforementioned imaging data prior to feeding them into the lesion area analytical system 114 and/or neural network system 118, or one of both modules of lesion area detection module 122 and lesion area computation module 124.--, in [0073]);
a region that extends from 600 µm outside the area exhibiting geographic atrophy to an edge of the OAC data (see DE SISTERNES: e.g., -- IG. 4B shows a channel-coded image (multi-channel composite image) in accord with the present invention, where each image channel (e.g., color channel) embodies a different pathology-specific characteristic (embodies a different metric group). In the present example, the red channel (or medium gray in a black-and-white, monochrome image) comprises sub-RPE reflectivity data. To gather metrics for the red channel, a 300 μm slab is defined outer to the RPE layer and into the choroid vicinity, with surface limits specified between the RPE layer plus offsets of 50 μm and 350 μm, respectively. The OCT signal within this slab is filtered for noise removal and then processed so the signal at each A-scan location has a constantly decreasing function with increase of depth, filling “valleys” in the signal. That is, for each particular pixel in an A-scan, the value is set to be as the highest value recorded in such A-scan from the considered pixel to increasing depths within the defined slab limits. This operation is set to eliminate lower value signal originated from the presence of choroidal blood vessels. The resulting data is projected into an en face image by averaging the pixel values within the slab limit definition for each A-scan. The resulting values of the en face image are then normalized to be in the range between 0 and 1. The goal of this slab is to characterize the increased reflectivity present in the choroid in GA regions.
[0052] In the present example, the green channel (e.g., light gray in a black-and-white, monochrome image) encompasses inner RPE reflectivity. To gather metrics for the green channel, a 20 μm slab is defined inner to the RPE-Fit layer (an estimation of the Bruch's membrane curvature set at the level of the RPE centerline), with surface limits specified between the RPE-Fit layer and offsets of minus 50 μm and minus 30 μm, respectively….[0053] In the present example, the blue channel (or dark gray in a black-and-white, monochrome image) encompasses retinal thickness. To gather metrics for the blue channel, the distance between the ILM layer and the RPE-Fit layer (retinal thickness) is measured for each A-scan location and projected into an en face image. The recorded values are then scaled in an inverted linear operation to take values from 0 to 1 so that a retinal thickness of 100 μm takes the value of 1 and a thickness of 350 μm takes the value of 0. The goal of this slab is to characterize the localized regions of retinal thinning and collapse characteristic of GA presence.--, in [0051]-[0053]); and
a region that extends from the area exhibiting geographic atrophy to the edge of the OAC data (see DE SISTERNES: e.g., -- IG. 4B shows a channel-coded image (multi-channel composite image) in accord with the present invention, where each image channel (e.g., color channel) embodies a different pathology-specific characteristic (embodies a different metric group). In the present example, the red channel (or medium gray in a black-and-white, monochrome image) comprises sub-RPE reflectivity data. To gather metrics for the red channel, a 300 μm slab is defined outer to the RPE layer and into the choroid vicinity, with surface limits specified between the RPE layer plus offsets of 50 μm and 350 μm, respectively. The OCT signal within this slab is filtered for noise removal and then processed so the signal at each A-scan location has a constantly decreasing function with increase of depth, filling “valleys” in the signal. That is, for each particular pixel in an A-scan, the value is set to be as the highest value recorded in such A-scan from the considered pixel to increasing depths within the defined slab limits. This operation is set to eliminate lower value signal originated from the presence of choroidal blood vessels. The resulting data is projected into an en face image by averaging the pixel values within the slab limit definition for each A-scan. The resulting values of the en face image are then normalized to be in the range between 0 and 1. The goal of this slab is to characterize the increased reflectivity present in the choroid in GA regions.
[0052] In the present example, the green channel (e.g., light gray in a black-and-white, monochrome image) encompasses inner RPE reflectivity. To gather metrics for the green channel, a 20 μm slab is defined inner to the RPE-Fit layer (an estimation of the Bruch's membrane curvature set at the level of the RPE centerline), with surface limits specified between the RPE-Fit layer and offsets of minus 50 μm and minus 30 μm, respectively….[0053] In the present example, the blue channel (or dark gray in a black-and-white, monochrome image) encompasses retinal thickness. To gather metrics for the blue channel, the distance between the ILM layer and the RPE-Fit layer (retinal thickness) is measured for each A-scan location and projected into an en face image. The recorded values are then scaled in an inverted linear operation to take values from 0 to 1 so that a retinal thickness of 100 μm takes the value of 1 and a thickness of 350 μm takes the value of 0. The goal of this slab is to characterize the localized regions of retinal thinning and collapse characteristic of GA presence.--, in [0051]-[0053]).
Re Claim 11, DE SISTERNES as modified by Reisman and YANG further disclose wherein determining the predicted enlargement rate based on the one or more attributes within the adjacent area includes providing the one or more attributes to a multiple linear regression model (see Yang: e.g., -- lesion area analytical system 114 may include lesion area detection module 122 and lesion area computation module 124. Lesion area analytical system 114, via lesion area detection module 122 and/or lesion area computation module 124, can use FAF images and/or OCT volumes to predict individual GA area and growth rates. Computational module 124, in various embodiments, can utilize a neural network system to predict individual GA area and growth rates. The predictions can be performed via lesion area analytical system 114 using screening images, for example, of image input 109, which may include any or all of FAF images 110, OCT images 112 and IR images 113. In various embodiments, GA growth rate (e.g., mm.sup.2/year if annualized) can be derived from a linear model fitted using all available FAF measurements based on accumulated FAF and OCT imaging, which can also be done longitudinally, e.g., every 24 weeks over 2 years.
[0072] In accordance with various embodiments, GA growth rate prediction can be formulated as a regression task. For example, using a neural network system 118 having three multi-task convolutional neural networks (CNNs) can be trained with multi-modal imaging data (e.g., a combination of FAF and OCT images) as input to predict (e.g., simultaneously), via lesion area detection module 122 and lesion area computation module 124, the GA lesion area and GA growth rate (e.g., annualized). In various embodiments, a linear model based on baseline GA lesion features, lesion area, lesion distance to fovea, lesion contiguity (unifocal/multifocal), and low luminance deficit (LLD) to derive GA growth rate prediction, can serve as a reference model to benchmark performance.--, in [0071]-[0072]; [0152], and, --[0166] GA growth rate prediction was formulated as a regression task. Three multi-task convolutional neural networks (CNNs) were trained with baseline FAF-only, OCT-only, and multi-modal (a combination of FAF and OCT images) images as input to simultaneously predict the baseline GA lesion area and annualized GA growth rate. A linear model based on baseline GA lesion features, lesion area, lesion distance to fovea, lesion contiguity (unifocal/multifocal), and low luminance deficit (FED) to derive GA growth rate prediction was used as a reference model to benchmark performance. Baseline visit images provide more prognostic information for GA disease progression than a linear model based on baseline GA lesion features and LLD alone. Furthermore, a multi-modal approach (see Figure 8B) gives more insight into disease progression and outperform a single modality approach (see Figure 8A).--, in [0166], and [0173]; and, --In comparison, a previously developed benchmark model using a linear regression model based on baseline visit GA lesion area, lesion distance to fovea, lesion contiguity and LLD showed an R.sup.2 value of 0.16 (0.10 - 0.23) for GA growth rate predictions on the same holdout dataset (Table 3).--, in [0177]).
Re Claim 12, DE SISTERNES as modified by Reisman and YANG further disclose wherein providing the one or more attributes to the multiple linear regression model includes providing a measured distance between a retinal pigment epithelium (RPE) and a Bruch's membrane (BM) within the adjacent area and a measured choriocapillaris flow deficit within the adjacent area to the multiple linear regression model (see Yang: e.g., -- lesion area analytical system 114 may include lesion area detection module 122 and lesion area computation module 124. Lesion area analytical system 114, via lesion area detection module 122 and/or lesion area computation module 124, can use FAF images and/or OCT volumes to predict individual GA area and growth rates. Computational module 124, in various embodiments, can utilize a neural network system to predict individual GA area and growth rates. The predictions can be performed via lesion area analytical system 114 using screening images, for example, of image input 109, which may include any or all of FAF images 110, OCT images 112 and IR images 113. In various embodiments, GA growth rate (e.g., mm.sup.2/year if annualized) can be derived from a linear model fitted using all available FAF measurements based on accumulated FAF and OCT imaging, which can also be done longitudinally, e.g., every 24 weeks over 2 years.
[0072] In accordance with various embodiments, GA growth rate prediction can be formulated as a regression task. For example, using a neural network system 118 having three multi-task convolutional neural networks (CNNs) can be trained with multi-modal imaging data (e.g., a combination of FAF and OCT images) as input to predict (e.g., simultaneously), via lesion area detection module 122 and lesion area computation module 124, the GA lesion area and GA growth rate (e.g., annualized). In various embodiments, a linear model based on baseline GA lesion features, lesion area, lesion distance to fovea, lesion contiguity (unifocal/multifocal), and low luminance deficit (LLD) to derive GA growth rate prediction, can serve as a reference model to benchmark performance.--, in [0071]-[0072]; [0152], and, --[0166] GA growth rate prediction was formulated as a regression task. Three multi-task convolutional neural networks (CNNs) were trained with baseline FAF-only, OCT-only, and multi-modal (a combination of FAF and OCT images) images as input to simultaneously predict the baseline GA lesion area and annualized GA growth rate. A linear model based on baseline GA lesion features, lesion area, lesion distance to fovea, lesion contiguity (unifocal/multifocal), and low luminance deficit (FED) to derive GA growth rate prediction was used as a reference model to benchmark performance. Baseline visit images provide more prognostic information for GA disease progression than a linear model based on baseline GA lesion features and LLD alone. Furthermore, a multi-modal approach (see Figure 8B) gives more insight into disease progression and outperform a single modality approach (see Figure 8A).--, in [0166], and [0173]; and, --In comparison, a previously developed benchmark model using a linear regression model based on baseline visit GA lesion area, lesion distance to fovea, lesion contiguity and LLD showed an R.sup.2 value of 0.16 (0.10 - 0.23) for GA growth rate predictions on the same holdout dataset (Table 3).--, in [0177]).
Re Claim 13, DE SISTERNES as modified by Reisman and YANG further disclose wherein providing the one or more attributes to the multiple linear regression model further includes providing a measured outer retinal layer thickness within the adjacent area to the multiple linear regression model (see Yang: e.g., -- lesion area analytical system 114 may include lesion area detection module 122 and lesion area computation module 124. Lesion area analytical system 114, via lesion area detection module 122 and/or lesion area computation module 124, can use FAF images and/or OCT volumes to predict individual GA area and growth rates. Computational module 124, in various embodiments, can utilize a neural network system to predict individual GA area and growth rates. The predictions can be performed via lesion area analytical system 114 using screening images, for example, of image input 109, which may include any or all of FAF images 110, OCT images 112 and IR images 113. In various embodiments, GA growth rate (e.g., mm.sup.2/year if annualized) can be derived from a linear model fitted using all available FAF measurements based on accumulated FAF and OCT imaging, which can also be done longitudinally, e.g., every 24 weeks over 2 years.
[0072] In accordance with various embodiments, GA growth rate prediction can be formulated as a regression task. For example, using a neural network system 118 having three multi-task convolutional neural networks (CNNs) can be trained with multi-modal imaging data (e.g., a combination of FAF and OCT images) as input to predict (e.g., simultaneously), via lesion area detection module 122 and lesion area computation module 124, the GA lesion area and GA growth rate (e.g., annualized). In various embodiments, a linear model based on baseline GA lesion features, lesion area, lesion distance to fovea, lesion contiguity (unifocal/multifocal), and low luminance deficit (LLD) to derive GA growth rate prediction, can serve as a reference model to benchmark performance.--, in [0071]-[0072]; [0152], and, --[0166] GA growth rate prediction was formulated as a regression task. Three multi-task convolutional neural networks (CNNs) were trained with baseline FAF-only, OCT-only, and multi-modal (a combination of FAF and OCT images) images as input to simultaneously predict the baseline GA lesion area and annualized GA growth rate. A linear model based on baseline GA lesion features, lesion area, lesion distance to fovea, lesion contiguity (unifocal/multifocal), and low luminance deficit (FED) to derive GA growth rate prediction was used as a reference model to benchmark performance. Baseline visit images provide more prognostic information for GA disease progression than a linear model based on baseline GA lesion features and LLD alone. Furthermore, a multi-modal approach (see Figure 8B) gives more insight into disease progression and outperform a single modality approach (see Figure 8A).--, in [0166], and [0173]; and, --In comparison, a previously developed benchmark model using a linear regression model based on baseline visit GA lesion area, lesion distance to fovea, lesion contiguity and LLD showed an R.sup.2 value of 0.16 (0.10 - 0.23) for GA growth rate predictions on the same holdout dataset (Table 3).--, in [0177]).
Re Claim 18, DE SISTERNES as modified by Reisman and YANG further disclose wherein determining the area exhibiting geographic atrophy based on at least one of the OCT data and the OAC data includes:
extracting a subRPE slab from the OCT data to generate an en face OCT image (see DE SISTERNES: e.g., --A system and method for use with optical coherence tomography (OCT) data to identify a target pathology extracts multiple pathology-characteristic images from the OCT data. The extracted pathology-characteristic images may include a mixture of OCT structural images (including retinal layer thickness information) and OCT angiography images. Optionally, other pathology-characteristic images and data maps (mapped to corresponding positions in the OCT data), such as fundus images and visual field test maps may be accessed as additional pathology-characteristic images. Each pathology-characteristic image defines a different image channel (e.g., “color channel”) per pixel in a composite, channel-coded image, which is then used to train a neural network to search for the target pathology in OCT data. The trained neural network may then receive new composite, channel-coded image and identify/segment the target pathology within the new channel-coded image.--, in abstract, and, --a method of analyzing optical coherence tomography data to identify a target pathology. More specifically, it is directed to analyzing optical coherence tomography data to identify Geographic Atrophy (GA)… [0002] Age-related macular degeneration (AMD) is an eye disease most common in the older population and arising from damage to the macula that leads to loss of central vision. Some patients with age-related macular degeneration (AMD) develop geographic atrophy (GA), which refers to regions of the retina where cells waste away and die. Geographic Atrophy (GA) is a condition of the macula present at the advanced stages of non-exudative macular degeneration. GA has a characteristic appearance resulting from the loss of the photoreceptor layer, retinal pigment epithelium (RPE), and choriocapillaris. GA typically first appears in the parafoveal location and progresses around the fovea and then through the fovea with loss of central visual acuity….characterization and monitoring macular regions affected by GA is fundamental for patient diagnosis, monitoring and management as well as for treatment research purposes.--, in [0002]; --The OCT data 21 may include OCT structural data and/or OCTA flow data, and the extracted metrics may include OCT-based metrics extracted from the OCT structural data, such as retinal layer thicknesses, distances of a specific A-scan to a specific retinal structure (e.g., distance to the fovea center), layer integrity (e.g., the loss of a specific layer), sub-RPE reflectivity, inner RPE reflectivity, overall retinal thickness, and/or the optical attenuation coefficient (OAC).
[0049] The optical attenuation coefficient (OAC) is an optical property of a medium that determines how the power of a coherent light beam propagating through the (e.g., turbid) medium (e.g., tissue) is attenuated along its path due to scattering and absorption. The irradiance (power per unit area) of the coherent light beam that propagates through a (e.g., homogeneous) medium is given by Lambert-Beer's law: L(z)=L.sub.0e.sup.−μz, where L(z) is the irradiance of the beam after traveling through the medium over a distance z, L.sub.0 is the irradiance of the incident light beam and μ is the optical attenuation coefficient. Large attenuation coefficients result in a quick and exponential decline of the irradiance of the coherent light beam with depth. Because the OAC is an optical property of the medium, determining the OAC provides information on the composition of this medium. Applicants propose that providing the OAC (per A-scan) as one of the extracted metrics may be beneficial identify specific pathologies (e.g., GA), particularly since it can be indicative of the current state (e.g., light attenuating state) of tissued at specific A-scan positions. An example of how the OAC may be determined/calculated is provided in “Depth-Resolved Model-Based Reconstruction of Attenuation Coefficients in Optical Coherence Tomography”, by K. A. Vermeer et al., Biomedical Optics Express, Vol. 5, Issue 1, pp. 322-337 (2014). A discussion of previous applications of OAC may be found in “In Vivo Tissue Injury Mapping Using Optical Coherence Tomography Based Methods”, by Utka Baran et al., Applied Optics, Vol. 54, No. 21, Jul. 20, 2015.--, in [0048]-[0050]; and, --[0060] FIG. 6 illustrates a general workflow of the present invention, including the U-net architecture of the proof of concept implementation. As shown, OCT Data 21 is accessed/acquired and multiple pathology characteristic images PCI's are defined from the OCT data 21, as described above. For example, the pathology characteristic images PCI's may include a sub-RPE reflectivity image, an inner RPE reflectivity image, Retinal thickness, and/or optical attenuation coefficient (OAC) as described above in reference to FIGS. 3 and 4. Optionally, other pathology characteristic image may also be used, such an en face images/maps of specific retinal layer thicknesses or layer integrity, OCTA flow images (such as flow at, or within the vicinity of, the choriocapillaris), an/or other pathology characteristic data as described above in reference to FIG. 5. the pathology characteristic images are then combined into a different (e.g., color or monochrome value) channels of a channel-coded image 27, which is then submitted to machine learning model 29, which is herein implemented using a U-Net architecture. A discussion of a U-Net architecture is provided below.--, in [0059]-[0061]); and at least one of:
providing the en face OCT image to a machine learning model trained to detect areas exhibiting geographic atrophy within en face OCT images (see DE SISTERNES: e.g., -- IG. 4B shows a channel-coded image (multi-channel composite image) in accord with the present invention, where each image channel (e.g., color channel) embodies a different pathology-specific characteristic (embodies a different metric group). In the present example, the red channel (or medium gray in a black-and-white, monochrome image) comprises sub-RPE reflectivity data. To gather metrics for the red channel, a 300 μm slab is defined outer to the RPE layer and into the choroid vicinity, with surface limits specified between the RPE layer plus offsets of 50 μm and 350 μm, respectively. The OCT signal within this slab is filtered for noise removal and then processed so the signal at each A-scan location has a constantly decreasing function with increase of depth, filling “valleys” in the signal. That is, for each particular pixel in an A-scan, the value is set to be as the highest value recorded in such A-scan from the considered pixel to increasing depths within the defined slab limits. This operation is set to eliminate lower value signal originated from the presence of choroidal blood vessels. The resulting data is projected into an en face image by averaging the pixel values within the slab limit definition for each A-scan. The resulting values of the en face image are then normalized to be in the range between 0 and 1. The goal of this slab is to characterize the increased reflectivity present in the choroid in GA regions.
[0052] In the present example, the green channel (e.g., light gray in a black-and-white, monochrome image) encompasses inner RPE reflectivity. To gather metrics for the green channel, a 20 μm slab is defined inner to the RPE-Fit layer (an estimation of the Bruch's membrane curvature set at the level of the RPE centerline), with surface limits specified between the RPE-Fit layer and offsets of minus 50 μm and minus 30 μm, respectively….[0053] In the present example, the blue channel (or dark gray in a black-and-white, monochrome image) encompasses retinal thickness. To gather metrics for the blue channel, the distance between the ILM layer and the RPE-Fit layer (retinal thickness) is measured for each A-scan location and projected into an en face image. The recorded values are then scaled in an inverted linear operation to take values from 0 to 1 so that a retinal thickness of 100 μm takes the value of 1 and a thickness of 350 μm takes the value of 0. The goal of this slab is to characterize the localized regions of retinal thinning and collapse characteristic of GA presence.--, in [0051]-[0053]); and
presenting the en face OCT image to a user to receive manual annotations of areas exhibiting geographic atrophy within the en face OCT image (see DE SISTERNES: e.g., -- IG. 4B shows a channel-coded image (multi-channel composite image) in accord with the present invention, where each image channel (e.g., color channel) embodies a different pathology-specific characteristic (embodies a different metric group). In the present example, the red channel (or medium gray in a black-and-white, monochrome image) comprises sub-RPE reflectivity data. To gather metrics for the red channel, a 300 μm slab is defined outer to the RPE layer and into the choroid vicinity, with surface limits specified between the RPE layer plus offsets of 50 μm and 350 μm, respectively. The OCT signal within this slab is filtered for noise removal and then processed so the signal at each A-scan location has a constantly decreasing function with increase of depth, filling “valleys” in the signal. That is, for each particular pixel in an A-scan, the value is set to be as the highest value recorded in such A-scan from the considered pixel to increasing depths within the defined slab limits. This operation is set to eliminate lower value signal originated from the presence of choroidal blood vessels. The resulting data is projected into an en face image by averaging the pixel values within the slab limit definition for each A-scan. The resulting values of the en face image are then normalized to be in the range between 0 and 1. The goal of this slab is to characterize the increased reflectivity present in the choroid in GA regions.
[0052] In the present example, the green channel (e.g., light gray in a black-and-white, monochrome image) encompasses inner RPE reflectivity. To gather metrics for the green channel, a 20 μm slab is defined inner to the RPE-Fit layer (an estimation of the Bruch's membrane curvature set at the level of the RPE centerline), with surface limits specified between the RPE-Fit layer and offsets of minus 50 μm and minus 30 μm, respectively….[0053] In the present example, the blue channel (or dark gray in a black-and-white, monochrome image) encompasses retinal thickness. To gather metrics for the blue channel, the distance between the ILM layer and the RPE-Fit layer (retinal thickness) is measured for each A-scan location and projected into an en face image. The recorded values are then scaled in an inverted linear operation to take values from 0 to 1 so that a retinal thickness of 100 μm takes the value of 1 and a thickness of 350 μm takes the value of 0. The goal of this slab is to characterize the localized regions of retinal thinning and collapse characteristic of GA presence.--, in [0051]-[0053]).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 21 are rejected under 35 U.S.C. 102 (a)(2) as being anticipated by DE SISTERNES (US 20230140881 A1, Date Filed: 2021-04-28).
Re Claim 21, DE SISTERNES discloses a computer-implemented method of automatically detecting an area of an eye exhibiting geographic atrophy (see DE SISTERNES: e.g., --A system and method for use with optical coherence tomography (OCT) data to identify a target pathology extracts multiple pathology-characteristic images from the OCT data. The extracted pathology-characteristic images may include a mixture of OCT structural images (including retinal layer thickness information) and OCT angiography images. Optionally, other pathology-characteristic images and data maps (mapped to corresponding positions in the OCT data), such as fundus images and visual field test maps may be accessed as additional pathology-characteristic images. Each pathology-characteristic image defines a different image channel (e.g., “color channel”) per pixel in a composite, channel-coded image, which is then used to train a neural network to search for the target pathology in OCT data. The trained neural network may then receive new composite, channel-coded image and identify/segment the target pathology within the new channel-coded image.--, in abstract, and, --a method of analyzing optical coherence tomography data to identify a target pathology. More specifically, it is directed to analyzing optical coherence tomography data to identify Geographic Atrophy (GA)… [0002] Age-related macular degeneration (AMD) is an eye disease most common in the older population and arising from damage to the macula that leads to loss of central vision. Some patients with age-related macular degeneration (AMD) develop geographic atrophy (GA), which refers to regions of the retina where cells waste away and die. Geographic Atrophy (GA) is a condition of the macula present at the advanced stages of non-exudative macular degeneration. GA has a characteristic appearance resulting from the loss of the photoreceptor layer, retinal pigment epithelium (RPE), and choriocapillaris. GA typically first appears in the parafoveal location and progresses around the fovea and then through the fovea with loss of central visual acuity….characterization and monitoring macular regions affected by GA is fundamental for patient diagnosis, monitoring and management as well as for treatment research purposes.--, in [0002], and, --GA may result in a progressive loss of vision, particularly central vision. However, GA may start with loss of vision outside the central area, and progress toward the center over time. Thus, it is advantageous to incorporate information from visual field test results FV. A visual field test is a method of measuring an individual's entire scope of vision, e.g., their central and peripheral (side) vision. Visual field testing is a way to map the visual fields of each eye individually and can detect blind spots (scotomas) as well as more subtle areas of dim vision. A campimeter, or “perimeter,” is a dedicated machine/device/system that applies a visual field test to a patient. A more in-depth discussion of perimeters and visual field testing is provided below. All, or select parts of a visual field test (such as the VF gray scale or numerical gray scale mapped to corresponding retinal locations) may be incorporated into the present multi-channel composite image 27.--, in [0055]-[0060], and [0105]-[0107]), the method comprising:
receiving, by an image analysis computing system, optical coherence tomography data (OCT data) (see DE SISTERNES: e.g., ----A system and method for use with optical coherence tomography (OCT) data to identify a target pathology extracts multiple pathology-characteristic images from the OCT data. The extracted pathology-characteristic images may include a mixture of OCT structural images (including retinal layer thickness information) and OCT angiography images. Optionally, other pathology-characteristic images and data maps (mapped to corresponding positions in the OCT data), such as fundus images and visual field test maps may be accessed as additional pathology-characteristic images. Each pathology-characteristic image defines a different image channel (e.g., “color channel”) per pixel in a composite, channel-coded image, which is then used to train a neural network to search for the target pathology in OCT data. The trained neural network may then receive new composite, channel-coded image and identify/segment the target pathology within the new channel-coded image.--, in abstract);
determining, by the image analysis computing system, an optical attenuation coefficient for each pixel of the OCT data to create optical attenuation coefficient data (OAC data) corresponding to the OCT data (see DE SISTERNES: e.g., --The OCT data 21 may include OCT structural data and/or OCTA flow data, and the extracted metrics may include OCT-based metrics extracted from the OCT structural data, such as retinal layer thicknesses, distances of a specific A-scan to a specific retinal structure (e.g., distance to the fovea center), layer integrity (e.g., the loss of a specific layer), sub-RPE reflectivity, inner RPE reflectivity, overall retinal thickness, and/or the optical attenuation coefficient (OAC).
[0049] The optical attenuation coefficient (OAC) is an optical property of a medium that determines how the power of a coherent light beam propagating through the (e.g., turbid) medium (e.g., tissue) is attenuated along its path due to scattering and absorption. The irradiance (power per unit area) of the coherent light beam that propagates through a (e.g., homogeneous) medium is given by Lambert-Beer's law: L(z)=L.sub.0e.sup.−μz, where L(z) is the irradiance of the beam after traveling through the medium over a distance z, L.sub.0 is the irradiance of the incident light beam and μ is the optical attenuation coefficient. Large attenuation coefficients result in a quick and exponential decline of the irradiance of the coherent light beam with depth. Because the OAC is an optical property of the medium, determining the OAC provides information on the composition of this medium. Applicants propose that providing the OAC (per A-scan) as one of the extracted metrics may be beneficial identify specific pathologies (e.g., GA), particularly since it can be indicative of the current state (e.g., light attenuating state) of tissued at specific A-scan positions. An example of how the OAC may be determined/calculated is provided in “Depth-Resolved Model-Based Reconstruction of Attenuation Coefficients in Optical Coherence Tomography”, by K. A. Vermeer et al., Biomedical Optics Express, Vol. 5, Issue 1, pp. 322-337 (2014). A discussion of previous applications of OAC may be found in “In Vivo Tissue Injury Mapping Using Optical Coherence Tomography Based Methods”, by Utka Baran et al., Applied Optics, Vol. 54, No. 21, Jul. 20, 2015.--, in [0048]-[0050]; and, --[0060] FIG. 6 illustrates a general workflow of the present invention, including the U-net architecture of the proof of concept implementation. As shown, OCT Data 21 is accessed/acquired and multiple pathology characteristic images PCI's are defined from the OCT data 21, as described above. For example, the pathology characteristic images PCI's may include a sub-RPE reflectivity image, an inner RPE reflectivity image, Retinal thickness, and/or optical attenuation coefficient (OAC) as described above in reference to FIGS. 3 and 4. Optionally, other pathology characteristic image may also be used, such an en face images/maps of specific retinal layer thicknesses or layer integrity, OCTA flow images (such as flow at, or within the vicinity of, the choriocapillaris), an/or other pathology characteristic data as described above in reference to FIG. 5. the pathology characteristic images are then combined into a different (e.g., color or monochrome value) channels of a channel-coded image 27, which is then submitted to machine learning model 29, which is herein implemented using a U-Net architecture. A discussion of a U-Net architecture is provided below.--, in [0059]-[0061]);
determining, by the image analysis computing system, an area exhibiting geographic atrophy based on the OAC data (see DE SISTERNES: e.g., --The OCT data 21 may include OCT structural data and/or OCTA flow data, and the extracted metrics may include OCT-based metrics extracted from the OCT structural data, such as retinal layer thicknesses, distances of a specific A-scan to a specific retinal structure (e.g., distance to the fovea center), layer integrity (e.g., the loss of a specific layer), sub-RPE reflectivity, inner RPE reflectivity, overall retinal thickness, and/or the optical attenuation coefficient (OAC).
[0049] The optical attenuation coefficient (OAC) is an optical property of a medium that determines how the power of a coherent light beam propagating through the (e.g., turbid) medium (e.g., tissue) is attenuated along its path due to scattering and absorption. The irradiance (power per unit area) of the coherent light beam that propagates through a (e.g., homogeneous) medium is given by Lambert-Beer's law: L(z)=L.sub.0e.sup.−μz, where L(z) is the irradiance of the beam after traveling through the medium over a distance z, L.sub.0 is the irradiance of the incident light beam and μ is the optical attenuation coefficient. Large attenuation coefficients result in a quick and exponential decline of the irradiance of the coherent light beam with depth. Because the OAC is an optical property of the medium, determining the OAC provides information on the composition of this medium. Applicants propose that providing the OAC (per A-scan) as one of the extracted metrics may be beneficial identify specific pathologies (e.g., GA), particularly since it can be indicative of the current state (e.g., light attenuating state) of tissued at specific A-scan positions. An example of how the OAC may be determined/calculated is provided in “Depth-Resolved Model-Based Reconstruction of Attenuation Coefficients in Optical Coherence Tomography”, by K. A. Vermeer et al., Biomedical Optics Express, Vol. 5, Issue 1, pp. 322-337 (2014). A discussion of previous applications of OAC may be found in “In Vivo Tissue Injury Mapping Using Optical Coherence Tomography Based Methods”, by Utka Baran et al., Applied Optics, Vol. 54, No. 21, Jul. 20, 2015.--, in [0048]-[0050]; and, --[0060] FIG. 6 illustrates a general workflow of the present invention, including the U-net architecture of the proof of concept implementation. As shown, OCT Data 21 is accessed/acquired and multiple pathology characteristic images PCI's are defined from the OCT data 21, as described above. For example, the pathology characteristic images PCI's may include a sub-RPE reflectivity image, an inner RPE reflectivity image, Retinal thickness, and/or optical attenuation coefficient (OAC) as described above in reference to FIGS. 3 and 4. Optionally, other pathology characteristic image may also be used, such an en face images/maps of specific retinal layer thicknesses or layer integrity, OCTA flow images (such as flow at, or within the vicinity of, the choriocapillaris), an/or other pathology characteristic data as described above in reference to FIG. 5. the pathology characteristic images are then combined into a different (e.g., color or monochrome value) channels of a channel-coded image 27, which is then submitted to machine learning model 29, which is herein implemented using a U-Net architecture. A discussion of a U-Net architecture is provided below.--, in [0059]-[0061]; also see: --a method of analyzing optical coherence tomography data to identify a target pathology. More specifically, it is directed to analyzing optical coherence tomography data to identify Geographic Atrophy (GA)… [0002] Age-related macular degeneration (AMD) is an eye disease most common in the older population and arising from damage to the macula that leads to loss of central vision. Some patients with age-related macular degeneration (AMD) develop geographic atrophy (GA), which refers to regions of the retina where cells waste away and die. Geographic Atrophy (GA) is a condition of the macula present at the advanced stages of non-exudative macular degeneration. GA has a characteristic appearance resulting from the loss of the photoreceptor layer, retinal pigment epithelium (RPE), and choriocapillaris. GA typically first appears in the parafoveal location and progresses around the fovea and then through the fovea with loss of central visual acuity….characterization and monitoring macular regions affected by GA is fundamental for patient diagnosis, monitoring and management as well as for treatment research purposes.--, in [0002], and, --GA may result in a progressive loss of vision, particularly central vision. However, GA may start with loss of vision outside the central area, and progress toward the center over time. Thus, it is advantageous to incorporate information from visual field test results FV. A visual field test is a method of measuring an individual's entire scope of vision, e.g., their central and peripheral (side) vision. Visual field testing is a way to map the visual fields of each eye individually and can detect blind spots (scotomas) as well as more subtle areas of dim vision. A campimeter, or “perimeter,” is a dedicated machine/device/system that applies a visual field test to a patient. A more in-depth discussion of perimeters and visual field testing is provided below. All, or select parts of a visual field test (such as the VF gray scale or numerical gray scale mapped to corresponding retinal locations) may be incorporated into the present multi-channel composite image 27.--, in [0055]-[0060], and [0105]-[0107]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 22 are rejected under 35 U.S.C. 103 as being patentable over DE SISTERNES, and in view of Reisman (US 10117568 B2).
Re Claim 22, DE SISTERNES however does not explicitly disclose,
Reisman discloses
as modified by Reisman and YANG further disclose
Re Claim 16, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 17, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 15, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 16, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 17, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 15, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 16, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 17, DE SISTERNES as modified by Reisman and YANG further disclose
Re Claim 11, claim 11 is the corresponding system claims to claim 1 respectively. Thus, claim 11 is rejected for the similar reasons as for claim 1. Furthermore, DE SISTERNES as modified by YANG further disclose a machine learning system for training a learning model that converts a domain of a medical image which is input, and generates a generated image of a different domain (see DE SISTERNES: e.g., --a method, system, and transitory or non-transitory computer readable medium are provided for training a model to generate a synthetic computed tomography (sCT) image from a cone-beam computed tomography (CBCT) image, comprising: receiving a CBCT image of a subject as an input of a generative model; and training the generative model, via first and second paths, in a generative adversarial network (GAN) to process the CBCT image to provide first and second synthetic computed tomography (sCT) images corresponding to the CBCT image as outputs of the generative model, the first path comprising a first set of one or more deformable offset layers and a first set of one or more convolution layers, the second path comprising the first set of the one or more convolution layers without the first set of the one or more deformable offset layers.
[0017] In some implementations, the GAN is trained using a cycle generative adversarial network (CycleGAN) comprising the generative model and a discriminative model, wherein the generative model is a first generative model and the discriminative model is a first discriminative model, further comprising: training a second generative model to process produced first and second sCT images as inputs and provide first and second cycle-CBCT images as outputs via third and fourth paths, respectively, the third path comprising a second set of the one or more deformable offset layers and a second set of the one or more convolution layers, the fourth path comprising the second set of the one or more convolution layers without the second set of the one or more deformable offset layers; and training a second discriminative model to classify the first cycle-CBCT image as a synthetic or a real CBCT image.
[0018] In some implementations, the CycleGAN comprises first and second portions to train the first generative model, further comprising: obtaining a training CBCT image that is paired with a real CT image; transmitting the training CBCT image to the input of the first generative model via the first and second paths to output the first and second synthetic CT images; receiving the first synthetic CT image at the input of the first discriminative model;--, in [0016]-[0018], and see Fig. 4, and, --training and use of a generative adversarial network adapted for generating a sCT image from a received CBCT image--, in [0024]; and, --the single generator is trained to convert the CBCT image appearance to CT image in a way that removes artefacts in original CBCT images and converts to the correct CT numbers while, at the same time, being trained based on some level of structure deformation. When the shape distribution or other feature distribution in CT images domain have large amount of differences compared to the original CBCT images domain--, [0036], and, --Radiotherapy system 100 may use a GAN to generate sCT images from a received CBCT image. The sCT image may represent an improved CBCT image with sharp-edge looking features that are akin to real CT images. Radiotherapy system 100 may thus produce sCT type of images for medical analysis in real time using lower quality CBCT images that are captured of a region of a subject.--, in [0038]-[0040], and, the radiotherapy processing computing system 110 may obtain image data 152 from the image data source 150 (e.g., CBCT images)….computing system 110 may instruct a CBCT device to obtain an image of a target region of a subject (e.g., a brain region). Computing system 110 may store the image data in storage device 116 with an associated indication of a time and target region captured by the CBCT image.--, in [0046]-[0047]; --0052] In an example, the image data 152 may include one or more MRI image (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric Mill, 4D cine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI),…etc.,--, in [0052]-[0053], and, -- a true CBCT image 602 is received and provided to multiple deformable offset layers 660A in a first path. The CBCT image 602 passes through the deformable offset layers 660A in an interleaved manner with convolution blocks in the convolution blocks 661A…..first generation result 612 is an sCT image produced with offset layers and second generation result 614 is an sCT image produced without offset layers. The result 612 that includes the sCT image produced with the offset layers is provided to the first discriminator model 630 for the CT domain while result 614 is not provided to the first discriminator model 630.
[0119] Referring back to FIG. 6A first generation results 612 (e.g., sCT image) may also be concurrently provided to the second generator model 608 together with the-second generation results 614 via third and fourth paths, respectively.--, in [0118]-0119]).
Re Claim 12, claim 12 is the corresponding medium claims to claim 1 respectively. Thus, claim 12 is rejected for the similar reasons as for claim 1. Furthermore, DE SISTERNES as modified by YANG further disclose non-transitory, computer-readable tangible recording medium on which a program for causing, when read by a computer, the computer to execute the method of generating a trained model according to claim 1 (see DE SISTERNES: e.g., --a method, system, and transitory or non-transitory computer readable medium are provided for training a model to generate a synthetic computed tomography (sCT) image from a cone-beam computed tomography (CBCT) image, comprising: receiving a CBCT image of a subject as an input of a generative model; and training the generative model, via first and second paths, in a generative adversarial network (GAN) to process the CBCT image to provide first and second synthetic computed tomography (sCT) images corresponding to the CBCT image as outputs of the generative model, the first path comprising a first set of one or more deformable offset layers and a first set of one or more convolution layers, the second path comprising the first set of the one or more convolution layers without the first set of the one or more deformable offset layers.
[0017] In some implementations, the GAN is trained using a cycle generative adversarial network (CycleGAN) comprising the generative model and a discriminative model, wherein the generative model is a first generative model and the discriminative model is a first discriminative model, further comprising: training a second generative model to process produced first and second sCT images as inputs and provide first and second cycle-CBCT images as outputs via third and fourth paths, respectively, the third path comprising a second set of the one or more deformable offset layers and a second set of the one or more convolution layers, the fourth path comprising the second set of the one or more convolution layers without the second set of the one or more deformable offset layers; and training a second discriminative model to classify the first cycle-CBCT image as a synthetic or a real CBCT image.
[0018] In some implementations, the CycleGAN comprises first and second portions to train the first generative model, further comprising: obtaining a training CBCT image that is paired with a real CT image; transmitting the training CBCT image to the input of the first generative model via the first and second paths to output the first and second synthetic CT images; receiving the first synthetic CT image at the input of the first discriminative model;--, in [0016]-[0018], and see Fig. 4, and, --training and use of a generative adversarial network adapted for generating a sCT image from a received CBCT image--, in [0024]; and, --the single generator is trained to convert the CBCT image appearance to CT image in a way that removes artefacts in original CBCT images and converts to the correct CT numbers while, at the same time, being trained based on some level of structure deformation. When the shape distribution or other feature distribution in CT images domain have large amount of differences compared to the original CBCT images domain--, [0036], and, --Radiotherapy system 100 may use a GAN to generate sCT images from a received CBCT image. The sCT image may represent an improved CBCT image with sharp-edge looking features that are akin to real CT images. Radiotherapy system 100 may thus produce sCT type of images for medical analysis in real time using lower quality CBCT images that are captured of a region of a subject.--, in [0038]-[0040], and, the radiotherapy processing computing system 110 may obtain image data 152 from the image data source 150 (e.g., CBCT images)….computing system 110 may instruct a CBCT device to obtain an image of a target region of a subject (e.g., a brain region). Computing system 110 may store the image data in storage device 116 with an associated indication of a time and target region captured by the CBCT image.--, in [0046]-[0047]; --0052] In an example, the image data 152 may include one or more MRI image (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric Mill, 4D cine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI),…etc.,--, in [0052]-[0053], and, -- a true CBCT image 602 is received and provided to multiple deformable offset layers 660A in a first path. The CBCT image 602 passes through the deformable offset layers 660A in an interleaved manner with convolution blocks in the convolution blocks 661A…..first generation result 612 is an sCT image produced with offset layers and second generation result 614 is an sCT image produced without offset layers. The result 612 that includes the sCT image produced with the offset layers is provided to the first discriminator model 630 for the CT domain while result 614 is not provided to the first discriminator model 630.
[0119] Referring back to FIG. 6A first generation results 612 (e.g., sCT image) may also be concurrently provided to the second generator model 608 together with the-second generation results 614 via third and fourth paths, respectively.--, in [0118]-0119]).
Re Claim 13, claim 13 is the corresponding apparatus claims to claim 1 respectively. Thus, claim 13 is rejected for the similar reasons as for claim 1. Furthermore, DE SISTERNES as modified by YANG further disclose medical image processing apparatus comprising:
a second storage device that stores a first trained model which is the trained first generator trained by implementing the method of generating a trained model according to claim 1 (see DE SISTERNES: e.g., --a method, system, and transitory or non-transitory computer readable medium are provided for training a model to generate a synthetic computed tomography (sCT) image from a cone-beam computed tomography (CBCT) image, comprising: receiving a CBCT image of a subject as an input of a generative model; and training the generative model, via first and second paths, in a generative adversarial network (GAN) to process the CBCT image to provide first and second synthetic computed tomography (sCT) images corresponding to the CBCT image as outputs of the generative model, the first path comprising a first set of one or more deformable offset layers and a first set of one or more convolution layers, the second path comprising the first set of the one or more convolution layers without the first set of the one or more deformable offset layers.
[0017] In some implementations, the GAN is trained using a cycle generative adversarial network (CycleGAN) comprising the generative model and a discriminative model, wherein the generative model is a first generative model and the discriminative model is a first discriminative model, further comprising: training a second generative model to process produced first and second sCT images as inputs and provide first and second cycle-CBCT images as outputs via third and fourth paths, respectively, the third path comprising a second set of the one or more deformable offset layers and a second set of the one or more convolution layers, the fourth path comprising the second set of the one or more convolution layers without the second set of the one or more deformable offset layers; and training a second discriminative model to classify the first cycle-CBCT image as a synthetic or a real CBCT image.
[0018] In some implementations, the CycleGAN comprises first and second portions to train the first generative model, further comprising: obtaining a training CBCT image that is paired with a real CT image; transmitting the training CBCT image to the input of the first generative model via the first and second paths to output the first and second synthetic CT images; receiving the first synthetic CT image at the input of the first discriminative model;--, in [0016]-[0018], and see Fig. 4, and, --training and use of a generative adversarial network adapted for generating a sCT image from a received CBCT image--, in [0024]; and, --the single generator is trained to convert the CBCT image appearance to CT image in a way that removes artefacts in original CBCT images and converts to the correct CT numbers while, at the same time, being trained based on some level of structure deformation. When the shape distribution or other feature distribution in CT images domain have large amount of differences compared to the original CBCT images domain--, [0036], and, --Radiotherapy system 100 may use a GAN to generate sCT images from a received CBCT image. The sCT image may represent an improved CBCT image with sharp-edge looking features that are akin to real CT images. Radiotherapy system 100 may thus produce sCT type of images for medical analysis in real time using lower quality CBCT images that are captured of a region of a subject.--, in [0038]-[0040], and, the radiotherapy processing computing system 110 may obtain image data 152 from the image data source 150 (e.g., CBCT images)….computing system 110 may instruct a CBCT device to obtain an image of a target region of a subject (e.g., a brain region). Computing system 110 may store the image data in storage device 116 with an associated indication of a time and target region captured by the CBCT image.--, in [0046]-[0047]; --0052] In an example, the image data 152 may include one or more MRI image (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric Mill, 4D cine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI),…etc.,--, in [0052]-[0053], and, -- a true CBCT image 602 is received and provided to multiple deformable offset layers 660A in a first path. The CBCT image 602 passes through the deformable offset layers 660A in an interleaved manner with convolution blocks in the convolution blocks 661A…..first generation result 612 is an sCT image produced with offset layers and second generation result 614 is an sCT image produced without offset layers. The result 612 that includes the sCT image produced with the offset layers is provided to the first discriminator model 630 for the CT domain while result 614 is not provided to the first discriminator model 630.
[0119] Referring back to FIG. 6A first generation results 612 (e.g., sCT image) may also be concurrently provided to the second generator model 608 together with the-second generation results 614 via third and fourth paths, respectively.--, in [0118]-0119]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662