Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability should not be negated by the way the invention was made.
Claims 1, 10-16, 21-22 and 25-28 are rejected under 35 U.S.C. 103 as being unpatentable over Kushida (US 2021/0224997 in view of Yim (US 2022/0301152).
As per claims 1 and 21-22, Kushida teaches, a method and a system for performing automated retinal segmentation for performing retinal segmentation (Kushida, ¶[0017] “FIG. 4 is an explanatory drawing for describing segmentation of a retina portion, a vitreous body portion and a choroid portion.” This represents performing retinal segmentation ), the method comprising: receiving an optical coherence tomography (OCT) image of a retina (Kushida, ¶[006] “ image into data for display, favorable contrast of a retina portion is obtained by discarding a certain amount of low-intensity-side data.” This represents receiving an optical coherence tomography (OCT) image of a retina); generating a layer element image using the OCT image and a first neural network (Kushida, ¶[0093] “a 3D scan is suitable in a case where it is desired to observe a layer structure or a layer thickness of the entire retina.” And ¶[0259] “[0259] In this case, for example, a convolutional neural network (CNN) can be used as a machine learning model for segmentation.” ), the layer element image identifying a set of retinal layer elements using a set of layer element indicators ( Kushida, ¶[0333] “a retinal layer boundary, a retinal layer boundary edge” This represents layer element image identifying a set of retinal layer elements using a set of layer element indicators by having the boundary and boundary edge to analyze, see also fig.10, showing different convolutional layers); the initial pathological element image visually identifying a set of retinal pathological elements using a set of pathological element indicators that assigns a different group of pixels to each retinal pathological element of the set of retinal pathological elements ( Kushida, ¶[0147] “Note that, as parameters that are set for convolutional layer groups included in the configuration 1001 illustrated in FIG. 10, it is possible to perform image quality improving processing of a certain accuracy by, for example, setting the kernel size of the filters to a width of three pixels and a height of three pixels, and the number of filters to 64. However, it is necessary to pay attention in this regard because if the settings of parameters with respect to layer groups and node groups constituting a neural network differ, in some cases the degrees to which a tendency trained based on training data is reproducible in the output data will differ. That is, in many cases, the appropriate parameters will differ according to the form at the time of implementation, and therefore parameters can be changed to preferable values as needed.” Different parameters are changed preferable values as needed); and refining the initial pathological element image using the layer element image to generate a refined pathological element image (Kushida, fig.10 image up top and refined pathological element image at the bottom using the layer element image), the refined pathological element image visually identifying the set of retinal pathological elements using the set of pathological element indicators (Kushida, ¶ [0210] “Therefore, in Modification 4, when integrating partial images obtained using a learned model, the image quality improving unit 322 modifies pixel values of connecting portions of regions that are observation targets based on pixel values of the surrounding pixels so that the image edges become inconspicuous. By this means, an image that is suitable for diagnosis in which a sense of incongruity caused by image edges is reduced can be generated.” The edges represents pathological element indicators), the set of pathological element indicators assigning an updated group of pixels to at least one retinal pathological element of the set of retinal pathological elements (Kushida, ¶[0210] “he image quality improving unit 322 modifies pixel values of connecting portions of regions that are observation targets based on pixel values of the surrounding pixels so that the image edges become inconspicuous.” This represents that the pixel values change).
Kushida doesn’t clearly teach but suggests generating an initial pathological element image using the OCT image and a second neural network.
However, Yim teaches generating an initial pathological element image using the OCT image and a second neural network (Yim, ¶ [0010] “The image of the eye tissue captured and processed by the first classification neural network(s) may be an optical coherence tomography (OCT) image, but this is not essential and the image of the eye tissue may be captured using other techniques. The image may, but need not be, be a three-dimensional image comprising a plurality of voxels.” And ¶ [0012] “In implementations the first and second classification neural network(s) may be trained to generate additional outputs characterizing referral decisions and additional diagnoses.” Discloses using more than one neural network to generate that initial image).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Kushida with those of Yim to be able to use a second neural network.
The motivation would have been to improve the redundancy and ¶ [0012] “Such outputs may be used when training the system and afterwards disregarded.” and for efficiency to be able to disregard other images.
As per claim 10, Kushida in view of Yim teaches, the method of claim 1, wherein a retinal layer element of the set of retinal layer elements is either a retinal layer or a boundary associated with the retinal layer (Kushida, ¶[0114] “In the case of a tomographic image 400 illustrated in FIG. 4, a boundary 401 between a vitreous body portion and a retina portion, and a boundary 402 between the retina portion and a choroid portion are detected. By detecting the boundaries 401 and 402 in the tomographic image 400, a region 403 of the retina portion between the boundary 401 and the boundary 402, a region 404 of the vitreous body portion that is on a shallow layer side from the boundary 401, and a region 405 of the choroid portion that is on a deep layer side from the boundary 402 can be identified.” This represents boundary associated with the retinal layer).
As per claims 11 and 27, Kushida in view of Yim teaches, the method of claim 10, wherein the retinal layer is selected from a group consisting of an internal limiting membrane (ILM) layer, an external limiting membrane (ELM) layer, an outer plexiform layer-Henle fiber layer (OPL-HFL), a retinal pigment epithelial (RPE) layer, a layer of RPE detachment, a Bruch's membrane (BM) layer, and an ellipsoid zone (EZ) (Kushida, ¶[0260] “a label image in which labels such as inner limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner segment-outer segment junction (ISOS), retinal pigment epithelium (RPE), Bruch's membrane (BM) and choroid have been given to regions can be used. Note that, as other regions, for example, a label image in which labels such as vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL),” this represents selecting from ILM and RPE).
As per claims 12 and 28, Kushida in view of Yim teaches, the method of claim 1, wherein the set of retinal pathological elements includes at least one of intraretinal fluid (IRF), subretinal fluid (SRF), fluid associated with pigment epithelial detachment (PED), hyperreflective material (HRM), subretinal hyperreflective material (SHRM), intraretinal hyperreflective material (IHRM), hyperreflective foci (HRF), a retinal fluid pocket, or a disruption (Kushida, fig.4 middle of the image shows a disruption).
As per claims 13 and 25, Kushida in view of Yim teaches, the method of claim 1, wherein each of the set of layer element indicators and the set of pathological element indicators includes at least one of a color indicator, a shape indicator, a pattern indicator, a shading indicator, a line, a curve, a marker, a label, a tag, or text (Kushida, ¶[0262] “Here, the term “rule-based region detection processing” refers to detection processing that utilizes, for example, known regularity such as the regularity of the shape of the retina.” This represents a shape indicator).
As per claim 14, Kushida in view of Yim teaches, the method of claim 1, wherein the first neural network comprises a first U-Net and the second neural network comprises a second U-Net (Kushida, ¶[0356] “In a U-Net type machine learning model, positional information (spatial information) that has been made ambiguous in a plurality of levels configured as an encoder is configured (for example, using a skip connection) so that the information can be used in levels of the same dimension (levels corresponding to each other) in a plurality of levels configured as a decoder.” a first U-Net of the neural network is being represented here ).
As per claim 15, Kushida in view of Yim teaches, the method of claim 1, wherein: the first neural network is trained using a first training dataset comprising a first plurality of training OCT images and a plurality of training layer element images (Kushida, ¶ [0352] Note that, the input data included in the training data may be a plurality of medical images of different sites of the subject and of different kinds.” This represents that the training data are images); and the second neural network is trained using a second training dataset comprising a second plurality of training OCT images and a plurality of training pathological element images (Yim, ¶ [0010] “The image of the eye tissue captured and processed by the first classification neural network(s) may be an optical coherence tomography (OCT) image, but this is not essential and the image of the eye tissue may be captured using other techniques. The image may, but need not be, be a three-dimensional image comprising a plurality of voxels.” And ¶ [0012] “In implementations the first and second classification neural network(s) may be trained to generate additional outputs characterizing referral decisions and additional diagnoses.” Discloses using more than one neural network to generate that initial image).
As per claim 16, Kushida in view of Yim teaches, the method of claim 15, wherein at least a portion of the first plurality of training OCT images is included in the second plurality of training OCT images (Kushida, ¶ [0381] “Next, the LSTM 2540 is illustrated in detail in FIG. 25B. A forget gate network FG, an input gate network IG and an output gate network OG are illustrated in FIG. 25B, and each of these networks is a sigmoid layer.” There would be some overlaps between the layers, therefore some images in the first and second layers).
As per claim 26, Kushida in view of Yim teaches, the method of claim 22, wherein the image input comprises an SD-OCT image (Kushida, ¶[0397] In addition, while a spectral domain OCT (SD-OCT) apparatus which uses the SLD as a light source is described as the OCT apparatus in the foregoing examples and modifications, the configuration of the OCT apparatus according to the present invention is not limited thereto.” This represents SD-OCT image).
Allowable Subject Matter
Claims 2-9 and 23-24 objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all the limitations of the base claim and any intervening claims. The subject matter of claim 2 “reassigning a portion of the group of pixels in the initial pathological element image based on whether an anatomical characterization of the first retinal pathological element as identified by the pathological element indicator is anatomically feasible, wherein the anatomical characterization of the first retinal pathological element includes at least one of a location, a size, a shape, a length, a width, a thickness, or a volume of the retinal pathological element.” was not found in the prior art. This helps by reducing false positives as also claim 4. AI and computer vision models look at pixels. Sometimes, a cluster of pixels might look exactly like a pathology (for example, a hemorrhage or an exudate) because of lighting, artifacts, or camera noise. However, by checking if that finding is "anatomically feasible," the software applies real-world biological rules to double-check its own work. If a computer claims there is a massive fluid pocket in an area of the retina where that specific layer is too thin to hold that volume, the system realizes it's an error and "reassigns" those pixels back to normal or to a different classification. Pure mathematical algorithms don't inherently know what a human eye looks like; they just know pixel contrasts. By incorporating characteristics like location, size, shape, length, width, thickness, or volume, that the method uses clinical anatomy as a boundary. This takes human logic and applies it automatically to analyzing the images.
Regarding claim 3 “updating a group of pixels in the initial pathological element image assigned to a retinal pathological element of the set of retinal pathological elements to form the updated group of pixels for the retinal pathological element in the refined pathological element image by constraining an allowable area for the retinal pathological element based on the layer element image, wherein the updated group of pixels includes fewer pixels than the group of pixels.” was not found in the prior art. It is important because the retina is organized into distinct structured layers (e.g., the nerve fiber layer, the photoreceptor layer, Bruch's membrane). Certain eye diseases or "pathological elements" (like drusen, exudates, microaneurysms, or fluid pockets) only occur within specific layers therefore important to segment them specifically as applicant at the pixel level.
As per claim 5, “wherein generating the layer element image comprises: generating, via the first neural network, a multi-channel map using the OCT image, wherein the multi-channel map comprises a plurality of segmented images in which each segmented image of the plurality of segmented images identifies a corresponding retinal layer of interest.” was also not found in the prior art. This subject matter is important because in medical practice, a major indicator of eye health is how thick a specific layer is. If the RNFL is thinning, it’s a sign of glaucoma. If the layer is thickening, there might be fluid leakage. By cleanly segmenting each layer into its own "channel," as applicant claims, a computer can instantly calculate the exact thickness or volume of that specific layer across the entire scan making this calculation useful for medical use. This is much more accurate than trying to measure the whole retina as a single block. The retina isn't just one blank sheet; it is made up of about 10 distinct, microscopic layers (like the ILM, RNFL, RPE, and photoreceptors). Different eye diseases attack different layers: Glaucoma damages the Nerve Fiber Layer (RNFL). Macular Degeneration (AMD) affects the deeper Retinal Pigment Epithelium (RPE) and photoreceptors. Diabetic Retinopathy causes swelling (edema) that disrupts fluid balance across multiple layers. And therefore, useful creating a “multi-channel map” makes this useful.
Claims 17-20 are allowed. This is the same subject matter as claim 5 but in independent form. “Wherein generating the layer element image comprises generating, via the first neural network, a multi-channel map using the OCT image, wherein the multi-channel map comprises a plurality of segmented images in which each segmented image of the plurality of segmented images identifies a corresponding retinal layer of interest.” And this subject matter was not found in the prior art. This subject matter is important because in medical practice, a major indicator of eye health is how thick a specific layer is. If the RNFL is thinning, it’s a sign of glaucoma. If the layer is thickening, there might be fluid leakage. By cleanly segmenting each layer into its own "channel," as applicant claims, a computer can instantly calculate the exact thickness or volume of that specific layer across the entire scan making this calculation useful for medical use. This is much more accurate than trying to measure the whole retina as a single block. The retina isn't just one blank sheet; it is made up of about 10 distinct, microscopic layers (like the ILM, RNFL, RPE, and photoreceptors). Different eye diseases attack different layers: Glaucoma damages the Nerve Fiber Layer (RNFL). Macular Degeneration (AMD) affects the deeper Retinal Pigment Epithelium (RPE) and photoreceptors. Diabetic Retinopathy causes swelling (edema) that disrupts fluid balance across multiple layers. And therefore, useful creating a “multi-channel map” makes this useful.
Prior art of interest of particular interest is:
Yang (US 2023/0326024), Abstract, “A method and system for evaluating geographic atrophy in a retina. A set of fundi autofluorescence (FAF) images of the retina is received at a machine learning system. A set of optical coherence tomography (OCT) images of the retina is received at the machine learning system. A lesion growth rate is predicted, via the machine learning system, for a geographic atrophy lesion in the retina using the set of FAF images and the set of OCT images.”
Nakagawa (US 2018/0240240), Abstract, “The image processing apparatus includes a boundary line extraction means that extracts a boundary line of a layer from an input image obtained by capturing an image of a target object composed of a plurality of layers. The boundary line extraction means is configured to first extract boundary lines at upper and lower ends of the target object, limit a search range using the extracted boundary lines at the upper and lower ends to extract another boundary line, limit the search range using an extraction result of the other boundary line to extract still another boundary line, and then sequentially repeat similar processes to extract subsequent boundary lines. In another aspect, the image processing apparatus includes a boundary line extraction means that extracts a boundary line of a layer from an input image obtained by capturing an image of a target object composed of a plurality of layers and a search range setting means that utilizes an already extracted boundary line extracted by the boundary line extraction means to dynamically set a search range for another boundary line. According to such an image processing apparatus and image processing method, boundary lines of layers can be extracted with a high degree of accuracy from a captured image of a target object composed of a plurality of layers.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SANTIAGO GARCIA whose telephone number is (571)270-5182. The examiner can normally be reached Monday-Friday 9:30am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SANTIAGO GARCIA/Primary Examiner, Art Unit 2673
/SG/
Prosecution Insights