DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Oath/Declaration
2. The receipt of Oath/Declaration is acknowledged.
Information Disclosure Statement
3. The information disclosure statements (IDS) submitted on 06/25/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Drawings
4. The drawing(s) filed on 06/29/2023 are accepted by the Examiner.
Claim Objections
5. Claim 3 is objected to because of the following informalities:
Claim 3, line 3, please change ***combing*** to ***combining***.
Appropriate correction is required.
Status of Claims
6. Claims 1-20 are pending in this application.
Claim Rejections - 35 USC § 103
7. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
8. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
9. The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
10. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
11. Claims 1-5, 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over in view of Fang et al. (US 2023/0245772 A1) in view of Bhuiyan et al. (US 2019/0014982), and further in view of Gribble et al. (US 2023/0018494), hereinafter ‘Fang’, ‘Bhuiyan’ and ‘Gribble’.
Regarding Claim 1:
Fang discloses an artificial intelligence (AI)-aided classification system for Alzheimer’s disease (AD) screening from a source dataset comprising retinal photographs, the system comprising a deep learning (DL) model created by a process comprising:
Fang discloses machine learning model(s) trained to process retinal fundus images acquired by an image acquisition system to classify retinal features contained in the images and to predict, based on the classified retinal features, whether the images are indicative of the presence or onset of Alzheimer's disease (Fang: Abstract; Fig. 1; Fig. 11; ¶¶[0090-0092]).
a) application of one or more data pre-processing methods and one or more on-the-fly data augmentation methods to normalize the retinal photographs including region of interest (ROI)
Fang discloses wherein a source database comprising retinal fundus images are used by the machine learning model (Fang: ¶[0062]).
Fang further teaches that when compiling the images in the dataset, that these images are selected based on image quality criterion such that images with insufficient composition, exposure/contrast, artifacts, and sharpness/focus are not included in the dataset. This constitutes one or more pre-processing methods so that the machine learning system is not impaired by not introducing artifacts and biasing features into the dataset (Fang: Figs. 3 and 4; ¶¶[0071-0073]).,
Fang expressly discloses determining “…importance scores at each pixel location across all patch scales were summarized and normalized to generate attention maps, where the intensity of each pixel represents its importance.” during the training process (Fang: Fig. 10 ¶[0092]). Note that Fang’s sliding window patch’s reads on the claimed region of interest (ROI).
Fang does not expressly teach data augmentation methods to normalize the retinal photographs including region of interest (ROI) cropping; and b) background subtraction with median filters,
Bhuiyan discloses data augmentation methods to normalize the retinal photographs including region of interest (ROI) cropping, and b) background subtraction with median filters,
Bhuiyan teaches “selecting vessel areas by cropping square shaped retinal regions from fundus images for downstream analysis (Bhuiyan: ¶[0026]; Fig. 14A), thereby normalizing image inputs for classification.
Bhuiyan further discloses estimating a background intensity image by smoothing the retinal image using a median filter and generating a shade corrected image by subtracting the estimated background from the original retinal image (Bhuiyan: ¶¶[0018; 0097], thereby performing illumination normalization through background subtraction using median filtering.
Fang in view of Bhuiyan are combinable because they are from the same field of endeavor of image processing (retinal image processing).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the retinal preprocessing technique of cropping, and background subtraction with median filters as taught by Bhuiyan, within the retinal image preparation stage of Fang. The suggestion/motivation for doing so is to improve the quality of retinal inputs to the deep learning model by to mitigating illumination artifacts. Therefore, it would have been obvious to combine Fang with Bhuiyan to obtain the invention as specified.
The proposed combination of Fang in view of Bhuiyan further disclose
c) removal of an illuminated rim and a stacking rim to create a multitude of cropped images to
Bhuiyan discloses identification and removal of central light reflex artifacts within retinal vessel interiors caused by illumination (Bhuiyan: ¶[0148]; Figs. 6B-6C), which correspond to illumination induced rim artifacts that degrade feature extraction. Further Bhuiyan as previously disclosed above, generates cropped retinal vessel image regions for subsequent processing (Bhuiyan: ¶[0026]).
Fang in view of Bhuiyan do not disclose generate a 6-channel input for training the DL model.
Gribble discloses generate a 6-channel input for training the DL model.
Gribble discloses regional segmentation of retinal images into anatomical regions including temporal regions including temporal rim, nasal rim, inferior rim, and superior rim regions (see ¶¶[0070; 0097; 0099]; Fig. 13). Such segmentation corresponds to removal of peripheral retinal rim regions when training the model.
Gribble expressly discloses “stacked images as CNN input” including “six different locations” obtaining retinal image mosaics and stacks of images from multiple retinal regions for CNN input (see ¶¶[0100-0101]). Also see the stacked image panel shown at Fig. 17C, left side, depicting 6 multiple cropped stacked retinal images for multi-channel input.
Fang, Bhuiyan & Gribble are combinable because they are in the same field of endeavor of image processing (retinal image processing). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose removal of an illuminated rim and a stacking rim to create a multitude of cropped images to generate a 6-channel input for training the DL model. The suggestion/motivation for doing so is to improve robustness and classification performance. Therefore, it would have been obvious to combine Fang, Bhuiyan & Gribble to obtain the invention as specified in claim 1.
Regarding Claim 2:
The proposed combination of Fang, Bhuiyan & Gribble further discloses the system according to claim 1, the DL model comprising a bilateral model, a hybrid model, and a unilateral model.
Gribble teaches an ensemble-based prediction architecture comprising multiple independent CNN based prediction models that receive distinct retinal image inputs and subject level metadata in parallel, with the outputs of the respective models combined to generate a final classification (see Fig. 9A; ¶¶[0088-0092]).
The plurality of independent input pathways within the ensemble correspond to model configurations trained using: (i) image inputs from one eye (unilateral), (ii) image inputs from both eyes (bilateral), and (iii) retinal image inputs in combination with subject level metadata (hybrid), thereby constituting unilateral, bilateral, and hybrid DL model configurations.
Fang, Bhuiyan & Gribble are combinable because they are in the same field of endeavor of image processing (retinal image processing). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose the DL model comprising a bilateral model, a hybrid model, and a unilateral model. The suggestion/motivation for doing so is to improve robustness and classification performance. Therefore, it would have been obvious to combine Fang, Bhuiyan & Gribble to obtain the invention as specified in claim 2.
Regarding Claim 3:
The proposed combination of Fang, Bhuiyan & Gribble further discloses the system according to claim 2, wherein:
the bilateral model is trained primarily on both eyes’ four images;
Gribble teaches parallel CNN prediction models receiving multiple retinal image inputs corresponding to distinct anatomical retinal locations (Gribble: Fig. 9A).
the hybrid model is trained primarily on both eyes’ four images combing demographic information; and
Gribble teaches combining retinal image inputs with subject level metadata within the ensemble prediction architecture such as ‘ethnicity/skin pigmentation’ (Gribble: Fig. 9B; ¶¶[0090-0091]).
the unilateral model is trained primarily on a single eye’s two images.
Gribble teaches configuring individual prediction models to receive retinal image inputs independently, enabling unilateral model training using images derived from a single eye. ‘one or more regions of the retina in one or both eyes of the patient’ (Gribble: Fig. 4 step 300; Also see Fig. 9A).
Fang, Bhuiyan & Gribble are combinable because they are in the same field of endeavor of image processing (retinal image processing). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the plurality of cropped an illumination normalized retinal image representations produced by Fang and Bhuiyan, as the stacked CNN input taught by Gribble by stacking the representations as separate feature planes, thereby forming a multi-channel (e.g., six channel) input tensor for DL training. The suggestion/motivation for doing so is to improve robustness and classification performance. Therefore, it would have been obvious to combine Fang, Bhuiyan & Gribble to obtain the invention as specified in claim 3.
Regarding Claim 4:
The proposed combination of Fang, Bhuiyan & Gribble further discloses the system according to claim 1, the source dataset comprising retinal photographs and demographic information derived from different centers, and the DL model comprising a feature fusion module to integrate features captured from the different centers.
Bhuiyan teaches identifying retinal anatomical regions, including the optic disc, for subsequent analysis (Bhuiyan: ¶0139], and ¶[0142]), and Optic Disc center detection (Bhuiyan: ¶¶[0130-0144]).
Gribble teaches combining retinal image inputs with subject level metadata within the ensemble prediction architecture such as ‘ethnicity/skin pigmentation’ (Gribble: Fig. 9B; ¶¶[0090-0091]).
Gribble further discloses performing pixel-wise prediction on hyperspectral retinal image data and generating a spatial probability output indicative of disease presence (Gribble; ¶[0093]; ¶[0099]; Fig. 14).
Fang, Bhuiyan & Gribble are combinable because they are in the same field of endeavor of image processing (retinal image processing). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose wherein the source dataset comprising retinal photographs and demographic information derived from different centers, and the DL model comprising a feature fusion module to integrate features captured from the different centers. The suggestion/motivation for doing so is to enhance classification accuracy through aggregated feature extraction across multiple retinal locations. Such modification would improve classification performance by leveraging complementary spatial information derived from multiple retinal regions, as expressly taught by Bhuiyan, while utilizing Gribble’s disclosed CNN input structure obtained at different anatomical locations. Therefore, it would have been obvious to combine Fang, Bhuiyan & Gribble to obtain the invention as specified in claim 4.
Regarding Claim 5:
The proposed combination of Fang, Bhuiyan & Gribble further discloses the system according to claim 1, the DL model comprising demographic information integrated with both eyes’ four images from one subject; and demographic information integrated by bilinear transformation.
Fang expressly stores and uses demographic information, including age, gender, and clinical data alongside retinal images in the UK Biobank dataset (Fang: ¶[0065]), and uses age and gender as critical variables in subject matching (Fang: matching “with the identical combination of gender and age” ¶[0074]; Figs. 6A and 6B). Additionally, Fang processes both left and right eye images simultaneously within the same pipeline (Fang:¶[0071]).
Gribble teaches stacking multiple retinal-region images as CNN input channels and organizing images for unilateral, bilateral, and hybrid model configurations that receive inputs from one eye, both eyes, and combinations with subject level metadata (e.g., ethnicity/skin pigmentation) which read on hybrid models trained on two image or four image inputs plus demographic information (Gribble: ¶¶[0090-0091]).
Fang, Bhuiyan & Gribble are combinable because they are in the same field of endeavor of image processing (retinal image processing). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to integrate demographic information with retinal image derived features in Fang and Bhuiyan’s DL classification model in view of Gribble’s teaching for stacking multiple cropped regions into a multi channel CNN input and configuring unilateral/bilateral/hybrid models using both-eyes and single-eye image sets plus demographic metadata. The suggestion/motivation for doing so is to improve disease prediction accuracy and provide complementary statistical predictors of AD risk when fused with image features in a common learned representation. Therefore, it would have been obvious to combine Fang, Bhuiyan & Gribble to obtain the invention as specified in claim 5.
Regarding Claims 15 and 16:
The proposed combination of Fang, Bhuiyan, & Gribble, explained in the rejection of claims 1-5, renders obvious the method of claims 15 and 16 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claims 1-5 are equally applicable to claims 15 and 16.
12. Claims 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over Fang, Bhuiyan & Gribble as applied to claim 1 above, and further in view of Tan et al. (“Efficientnet: Rethinking model scaling for convolutional neural networks” In International conference on machine learning (pp. 6105-6114). PMLR. (Year: 2019)), hereinafter ‘Tan’.
Regarding Claim 6:
The proposed combination of Fang, Bhuiyan & Gribble further discloses the system according to claim 1, the DL model comprising EfficientNet
Gribble teaches wherein the design of the algorithm can build on top of state of the art algorithms from the convolutional neural network (CNN) family (such as EfficientNet…) and modifying the architectures to accept hyperspectral images instead of color (RGB) images, adapting the layers to support the spatial-spectral requirements of the analysis, and changing the width, depth and length of the networks according to the capacity needed for detecting signal in multispectral and/or hyperspectral spectral retinal images (Gribble: ¶[0098]).
Fang, Bhuiyan & Gribble do not expressly disclose the DL model comprising EfficientNet-b2 as a backbone to extract features.
Tan discloses the DL model comprising EfficientNet-b2 as a backbone to extract features.
Tan teaches a compound scaling method that uniformly scales network width, depth and resolution with a set of fixed scaling coefficients (Tan: Page 2, Col. 2 (bottom)).
Tan further discloses in Table 2 performance results for EfficientNet models, including EfficientNet-B2.
Fang, Bhuiyan, Gribble & Tan are combinable because they are in the same field of endeavor of image processing (image processing using CNN modeling). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use an established CNN architecture such as EfficientNet-B2 as a backbone for medical analysis as disclosed by Tan. The suggestion/motivation for doing so is to improve disease prediction accuracy by choosing a model that balances model performance and computational resources. Therefore, it would have been obvious to combine Fang, Bhuiyan, Gribble & Tan to obtain the invention as specified in claim 6.
Regarding Claim 7:
The proposed combination of Fang, Bhuiyan, Gribble & Tan further discloses the system according to claim 6, the DL model comprising a domain adaptation technique to deal with dataset discrepancies.
Tan teaches domain adaptive techniques for handling dataset discrepancies, as shown in Table 5 (page 7). It is well known that in order to handle medical images from different sources, such as different cameras and hospitals, that domain adaption techniques such that taught by Tan are necessary in order to overcome discrepancies in the different datasets. See Table 6 of Tan for different datasets used for training the CNN.
Fang, Bhuiyan, Gribble & Tan are combinable because they are in the same field of endeavor of image processing (image processing using CNN modeling). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to the disclose a DL model comprising a domain adaptation technique to deal with dataset discrepancies as disclosed by Tan. The suggestion/motivation for doing so is to reduce performance degradation by using transfer learning and pretrained models which is a form of domain adaption. Therefore, it would have been obvious to combine Fang, Bhuiyan, Gribble & Tan to obtain the invention as specified in claim 7.
Regarding Claim 8:
The proposed combination of Fang, Bhuiyan & Gribble further disclose the system according to claim 1, the DL model created by a process comprising two stages.
Fang discloses the trained machine learning model is configured in a multiple-stage (multi-stage) pipeline architecture comprising multiple stages that are separately trained (Fang: ¶[0014]).
13. Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Fang, Bhuiyan & Gribble & Tan.
Regarding Claim 18:
The proposed combination of Fang, Bhuiyan, Gribble & Tan, explained in the rejection of claims 1-7, renders obvious the system of claim 18 because these steps occur in the operation of the proposed combination as discussed above. Thus, the arguments similar to that presented above for claims 1-7 are equally applicable to claim 18.
Allowable Subject Matter
14. Claims 9-14, 17, 19 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 9:
None of the prior art cited disclose or suggest the system according to claim 8, the two stages comprising: a first stage, trained with supervised learning on the source dataset with image-level annotations; and a second stage, introducing a domain adaptation method by estimating the pseudo labels for the retinal photographs from the source dataset using a domain-specific batch normalization technique.
Regarding Claims 10-14:
It follows that Claims 10-14 are then inherently allowable for depending on allowable base claim 9.
Regarding Claim 17:
None of the prior art cited disclose or suggest the method according to claim 16, comprising:
i)extracting features from the source domain using EfficientNet-b2 as a backbone;
j)applying a domain adaptation technique to deal with dataset discrepancies;
k)conducting a first stage, trained with supervised learning on the source dataset with image-level annotations wherein images from the source domain and from the target domain were fed into separate batch normalization layers;
l)conducting a second stage, introducing a domain adaptation method by estimating pseudo labels for the retinal photographs from the target domain using domain-specific batch normalization technique wherein images from the source domain and from the target domain, respectively, are fed into separate batch normalization layers;
m)identifying an imbalance of data between a first class with more data and a second class with less data;
n)applying an over sampling for the second class;
o)applying a training objective function utilizing both source domain images and target domain images;
p)generating heatmaps to show the significant locations which are related to the AD with a Gradient-weighted Class Activation method; and
q)providing both a pre-diagnosis image assessment and an AD binary classification by a cloud-based web application.
Regarding Claims 19-20:
It follows that Claims 19-20 are then inherently allowable for depending on allowable base claim 18.
Conclusion
15. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yu et al. (US 2021/0343006) relates to fundus image processing field, especially a preprocessing method for quantitative analysis of fundus image, and storage device. A preprocessing method for quantitative analysis of fundus image, wherein the step includes, acquiring a to-be-processed fundus image; performing optic disk positioning on the to-be-processed fundus image; performing macular fovea positioning on the to-be-processed image; and calculating a quantization parameter of a distance between a center of the macular fovea and a bitamporal edge of the optic disk. Through this method, the data obtained is converted from absolute representation to relative representation, and through normalization, a meaningful and comparable quantification is formed between people, between different people, and even between inspection results of different instruments. analyze data. It ensures that fundus images from different sources can form meaningful and comparable quantitative indicators.
16. Any inquiry concerning this communication or earlier communications from the examiner should be directed to NEIL R MCLEAN whose telephone number is (571)270-1679. The examiner can normally be reached Monday-Thursday, 6AM - 4PM, PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi M Sarpong can be reached at 571.270.3438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NEIL R MCLEAN/Primary Examiner, Art Unit 2681