Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 10, and 14 have been amended.
Claims 6, 11, and 18 have been cancelled.
Claims 1-5, 7-10, 12-17, and 19-20 are still pending for consideration.
Response to Arguments
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-5, 7-10, 12-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Manivannan et al. (US 20220084210 A1) in view of Selvaraju et al. NPL “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” and further in view of Vijaykeerthy et al. (US 20210012156 A1).
Regarding claim 1, Manivannan et al. teaches a method for evaluating geographic atrophy (see Abstract; “An automated segmentation and identification system/method for identifying geographic atrophy (GA) phenotypic patterns in fundus autofluorescence images”), the method comprising: receiving a set of retinal images (see para [0007]; “The ophthalmic diagnostic device is used to acquire an image of the fundus of the eye. Preferably, the image is a fundus autofluorescence (FAF) image since GA lesions are typically more easily discernable in such images”), training each model of a plurality of models to predict a set of geographic atrophy (GA) progression parameters for a geographic atrophy (GA) lesion using the set of retinal images (see para [0008]; “GA segmentation may further be based on a two-stage segmentation process, e.g., a hybrid process that combines a supervised classifier… with an active contour algorithm. ….This first stage classifier/segmentation ML model identifies initial GA regions (lesions) in the image, and the results are fed to the active counter algorithm for a second stage segmentation”, see also para [0060]; “a GA segmentation/classifier The supervised classifier is preferably a machine learning (ML) model, and may be implemented as a support vector machine (SVM) or a deep learning neural network, preferably a U-Net type convolutional neural network. In this case, the NN was trained using manually segmented images (e.g., images with GA regions segmented by human experts) as training outputs and corresponding non-segmented images as training inputs”, and para [0009]; “The identified GA regions are then submitted for analysis to identify their specific phenotype….. Either of these two phenotypes indicates a high progression rate GA region” Note: the two-stage hybrid (SVM/U-Net + active-contour) constitutes plural trained models operating on retina images)). However, Manivannan et al. does not teach and generating a visualization output for each model of the plurality of models, wherein the visualization output for a corresponding model of the plurality of models provides information about how the corresponding model uses the set of retinal images to predict the set of GA progression parameters; and modifying a model of the plurality of models to form a new model based on the visualization output generated for the model to improve a performance of the model.
In the same field of endeavor Selvaraju et al. teaches and generating a visualization output for each model of the plurality of models, wherein the visualization output for a corresponding model of the plurality of models provides information about how the corresponding model uses the set of retinal images to predict the set of GA progression parameters (see Abstract; “(Grad-CAM), uses the gradients of any target concept flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept”, see also page 5, right col. last para; “Thus, Grad-CAM is a strict generalization of CAM. This generalization allows us to generate visual explanations from CNN-based models that cascade convolutional layers with much more complex interactions, such as those for image”). Accordingly, it would have been obvious to one of ordinary skill in the art before the invention of the claimed invention to modify an automated segmentation and identification system/method for identifying geographic atrophy (GA) phenotypic patterns in fundus autofluorescence images of Manivannan et al. in view of the use of Grad-CAM: visual explanations from deep networks via gradient-based localization of Selvaraju et al. in order to create a high-resolution class-discriminative visualization (see Abstract). However, the combination of Manivannan et al. and Selvaraju et al. as a whole does not teach and modifying a model of the plurality of models to form a new model based on the visualization output generated for the model to improve a performance of the model.
In the same field of endeavor, Vijaykeerthy et al. teaches and modifying a model of the plurality of models to form a new model based on the visualization output (see para [0016]; “generating explanations include, for example, Class Activation Mapping (CAM) and Gradient-weighted Class Activation Mapping (Grad-CAM), which quantify the influence of each pixel in an image on the outputs of the model”, see also para [0028]; “the explanation is computed at the current parameter configuration and used to refine the model”, and para [0045]; “The model may then be retrained using the trainable explanation-guided loss, which guides the model to focus on the actual important regions”, see also para [0048]; “re-training comprises at least (i) generating machine explanations …..(ii) comparing the user explanations to the machine explanations, and (iii) adjusting a parameter configuration of the machine learning model based on said comparing”) generated for the model to improve a performance of the model (see para [0002]; “the trained ML model often focuses on the background of the object which adversely affects the generalization performance of the model” see also para [0016]; “provides a scheme that forces a ML model to focus on objects holistically instead of the background”). Accordingly, it would have been obvious to one of ordinary skill in the art before the invention of the claimed invention to modify an automated segmentation and identification system/method for identifying geographic atrophy (GA) phenotypic patterns in fundus autofluorescence images of Manivannan et al. in view of the use of Grad-CAM: visual explanations from deep networks via gradient-based localization of Selvaraju et al. and techniques for explanation guided learning of Vijaykeerthy et al. in order to improve generalization performance of the model (see Abstract).
Regarding claim 2, the rejection of claim 1 is incorporated herein.
Selvaraju in the combination further teach wherein the generating comprises: generating a gradient activation map for a corresponding retinal image in the set of retinal images for the corresponding model, wherein the gradient activation map indicates a set of regions in the corresponding retinal image that contributed to the set of GA progression parameters predicted by the corresponding model for the GA lesion (see Abstract; “Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’ in a classification network or a sequence of words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept”, see also page 2, right col., 3rd para; “our proposed method Gradient-weighted Class Activation Mapping (Grad-CAM), are highly class-discriminative (the ‘cat’ explanation exclusively highlights the ‘cat’ regions but not ‘dog’ regions in Fig. 1c, and vice versa in Fig. 1i)”).
Regarding claim 3, the rejection of claim 1 is incorporated herein.
Selvaraju et al. in the combination further teach wherein the plurality of models includes a deep learning model and further comprising: validating the deep learning model using the visualization output generated for the deep learning model (see page 3, left col. 2nd para; “We conduct human studies (Sec. 5) that show Guided Grad-CAM explanations are class-discriminative and not only help humans establish trust, but also help untrained users successfully discern a ‘stronger’ network from a ‘weaker’ one, even when both make identical predictions”, see also page 14, B; “validate our design choices for computing Grad-CAM visualizations”).
Regarding claim 4, the rejection of claim 1 is incorporated herein.
Manivannan et al. in the combination further teach wherein the plurality of models includes a first deep learning model and a second deep learning model further comprising (see page [0063]; “both SVM-based and DL-based classification provide good, initial GA delineation, or segmentation”, see also para [0080]; “The preferred segmentation process, however, is a two-step segmentation that combines GA classification (e.g., pixel-by-pixel) with active contour segmentation. The first of this two-step process may be a trained, machine model, such as a SVM or a (e.g., deep learning) neural network that segments/classifies/identifies GA regions within the image”):
Selvaraju et al. in the combination further teach performing a comparison of the visualization output generated for the first deep learning model with the visualization output generated for the second deep learning model (see page 3, 2nd para; “users successfully discern a ‘stronger’ network from a ‘weaker’ one, even when both make identical predictions”).
Regarding claim 5, the rejection of claim 4 is incorporated herein.
Selvaraju et al. in the combination further teach further comprising: selecting either the first deep learning model or the second deep learning model as a best model for predicting the set of GA progression parameters based on the comparison (see page 3, 2nd para; “users successfully discern a ‘stronger’ network from a ‘weaker’ one, even when both make identical predictions” Note; implies choosing the better model).
Regarding claim 7, the rejection of claim 1 is incorporated herein.
Manivannan et al. in the combination further teach wherein the set of GA progression parameters comprises at least one of a growth rate for the GA lesion or a baseline lesion area for the GA lesion (see para [0008]; “This first stage classifier/segmentation ML model identifies initial GA regions (lesions) in the image”, see also para [0078]; “GA region is identifying as any phenotype associated with a high progression rate”).
Regarding claim 8, the rejection of claim 1 is incorporated herein.
Manivannan et al. in the combination further teach wherein the set of retinal images comprises at least one of a set of fundus autofluorescence (FAF) images or a set of optical coherence tomography (OCT) images (see para [0049]; “the present invention is described as applied to widefield FAF images, but may equally be applied to other types of ophthalmic imaging modalities, such as OCT/OCTA that can generate images that provide visualization of GA”).
Regarding claim 9, the rejection of claim 8 is incorporated herein.
Manivannan et al. in the combination further teach wherein the set of fundus autofluorescence (FAF) images is a set of baseline FAF images and wherein the set of optical coherence tomography (OCT) images is a set of baseline OCT images (see para [0049]; “the present invention is described as applied to widefield FAF images, but may equally be applied to other types of ophthalmic imaging modalities, such as OCT/OCTA that can generate images that provide visualization of GA. For example, it may be applied to an en face OCT/OCTA image”).
Regarding claim 10, the scope of claim 10 is fully encompassed by the scope of claim 1, accordingly, the rejection of claim 1 is fully applicable here.
Regarding claim 12, the rejection of claim 10 is incorporated herein.
Manivannan et al. in the combination further teach wherein the set of GA progression parameters comprises at least one of a growth rate for the GA lesion or a baseline lesion area for the GA lesion (see para [0008]; “This first stage classifier/segmentation ML model identifies initial GA regions (lesions) in the image”, see also para [0078]; “GA region is identifying as any phenotype associated with a high progression rate”).
Regarding claim 13, the rejection of claim 10 is incorporated herein.
Manivannan et al. in the combination further teach wherein the set of retinal images comprises at least one of a set of fundus autofluorescence (FAF) images or a set of optical coherence tomography (OCT) images (see para [0049]; “the present invention is described as applied to widefield FAF images, but may equally be applied to other types of ophthalmic imaging modalities, such as OCT/OCTA that can generate images that provide visualization of GA”).
Regarding claim 14, the scope of claim 14 is fully encompassed by the scope of claim 1, accordingly, the rejection of claim 1 is fully applicable here (see also para [0122]; “a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs))”).
Regarding claim 15, the rejection of claim 14 is incorporated herein.
Selvaraju in the combination further teach wherein the generating comprises: generating a gradient activation map for a corresponding retinal image in the set of retinal images for the corresponding model, wherein the gradient activation map indicates a set of regions in the corresponding retinal image that contributed to the set of GA progression parameters predicted by the corresponding model for the GA lesion (see Abstract; “Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’ in a classification network or a sequence of words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept”, see also page 2, right col., 3rd para; “our proposed method Gradient-weighted Class Activation Mapping (Grad-CAM), are highly class-discriminative (the ‘cat’ explanation exclusively highlights the ‘cat’ regions but not ‘dog’ regions in Fig. 1c, and vice versa in Fig. 1i)”).
Regarding claim 16, the rejection of claim 14 is incorporated herein.
Selvaraju et al. in the combination further teach wherein the corresponding model is a corresponding deep learning model and wherein the processor is configured to execute the machine executable code to cause the processor to validate the corresponding deep learning model using the visualization output generated for the corresponding deep learning model (see page 3, left col. 2nd para; “We conduct human studies (Sec. 5) that show Guided Grad-CAM explanations are class-discriminative and not only help humans establish trust, but also help untrained users successfully discern a ‘stronger’ network from a ‘weaker’ one, even when both make identical predictions”, see also page 14, B; “validate our design choices for computing Grad-CAM visualizations”).
Regarding claim 17, the rejection of claim 14 is incorporated herein.
Manivannan et al. in the combination further teach wherein the plurality of models includes a first deep learning model and a second deep learning model (see page [0063]; “both SVM-based and DL-based classification provide good, initial GA delineation, or segmentation”, see also para [0080]; “The preferred segmentation process, however, is a two-step segmentation that combines GA classification (e.g., pixel-by-pixel) with active contour segmentation. The first of this two-step process may be a trained, machine model, such as a SVM or a (e.g., deep learning) neural network that segments/classifies/identifies GA regions within the image”):
Selvaraju et al. in the combination further teach and wherein the processor is configured to execute the machine executable code to cause the processor to: perform a comparison of the visualization output generated for the first deep learning model with the visualization output generated for the second deep learning model; and select either the first deep learning model or the second deep learning model as a best model for predicting the set of GA progression parameters based on the comparison. (see page 3, 2nd para; “users successfully discern a ‘stronger’ network from a ‘weaker’ one, even when both make identical predictions”).
Regarding claim 19, the rejection of claim 14 is incorporated herein.
Manivannan et al. in the combination further teach wherein the set of GA progression parameters comprises at least one of a growth rate for the GA lesion or a baseline lesion area for the GA lesion (see para [0008]; “This first stage classifier/segmentation ML model identifies initial GA regions (lesions) in the image”, see also para [0078]; “GA region is identifying as any phenotype associated with a high progression rate”).
Regarding claim 20, the rejection of claim 14 is incorporated herein.
Manivannan et al. in the combination further teach wherein the set of retinal images comprises at least one of a set of fundus autofluorescence (FAF) images or a set of optical coherence tomography (OCT) images (see para [0049]; “the present invention is described as applied to widefield FAF images, but may equally be applied to other types of ophthalmic imaging modalities, such as OCT/OCTA that can generate images that provide visualization of GA”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WINTA GEBRESLASSIE whose telephone number is (571)272-3475. The examiner can normally be reached Monday-Friday9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WINTA GEBRESLASSIE/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights