DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending in the application.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 6 ,11 and 16 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang et al. (Wang K, Xu C, Li G, Zhang Y, Zheng Y, Sun C. Combining convolutional neural networks and self-attention for fundus diseases identification. Scientific Reports. 2023 Jan 2;13(1):76. Hereafter Wang).
As per claim 1. Wang teaches the invention substantially as claimed including a method for training a medical image recognition model (Abstract), comprising:
obtaining a plurality of training images and a preset disease label corresponding to each training image of the plurality of training image (page 9 section “Dataset”: “The training set is a structured ophthalmology database that includes the ages of 3,500 patients, color fundus images of their left and right eyes, and diagnostic keywords from clinicians … As illustrated in Figure 5(a), these data categorize patients into eight categories: normal (N), DR (D), glaucoma (G), cataract (C), AMD (A), hypertension (H), Myopia (M), and other diseases/abnormalities (O)”; Fig. 5);
obtaining the medical image recognition model by training a medical image recognition network based on the plurality of training images and the preset disease label corresponding to each training image, the medical image recognition network comprising a feature extraction network, an attention mechanism module, and a classification network (See below Fig. 4, in which stages S0-S3 corresponds to a feature extraction model, stage S4 corresponds to an attention mechanism model, and “Sigmoid classifier” represents a classification model), and training the medical image recognition network comprising:
PNG
media_image1.png
464
977
media_image1.png
Greyscale
inputting each training image into the feature extraction network and obtaining a training feature map of each training image (See below Table 7, which shows dimensions of feature maps from each stage. The feature maps from stage 3 have dimensions 14*14*512);
PNG
media_image2.png
236
469
media_image2.png
Greyscale
determining a training weighted feature map of each training feature map by inputting each training feature map into the attention mechanism module (Fig. 4 stage S4; The feature maps from state 4 have dimensions 7*7*512; Abstract: “Therefore, given that the self-attention (SA) model with a global receptive field may have robust global-level feature modeling ability, we propose a multistage fundus image classification model MBSaNet which combines CNN and SA mechanism. The convolution block extracts the local information of the fundus image, and the SA module further captures the complex relationships between different spatial positions, thereby directly detecting one or more fundus diseases in retinal fundus image”);
obtaining a predicted disease label corresponding to each training image by invoking the classification network to classify each training image based on the training weighted feature map of each training feature map (Fig. 4 “Sigmoid classifier”; The classified target disease labels include eight categories: normal (N), DR (D), glaucoma (G), cataract (C), AMD (A), hypertension (H), Myopia (M), and other diseases/abnormalities (O));
calculating a loss value according to the predicted disease label and the preset disease label corresponding to each training image, adjusting the feature extraction network, the attention mechanism module, and the classification network according to the loss value, and obtaining a feature extraction model corresponding to the feature extraction network, an attention mechanism model corresponding to the attention mechanism module and a classification model corresponding to the classification network (See below bridging paragraph of pages 13-14. Eqn. (11) shows a loss function, in which yi is the ground truth label (i.e., preset disease label), and p(yi) is the probability that sample i is predicted to be positive (i.e., predicted disease label). The below paragraph also discloses that the training is performed in an iterative manner by using an optimization function, such as SGD or Adam. During the training process, the parameters of the feature extraction network, the attention mechanism module, and the classification network are adjusted, When the training converges, a feature extraction model corresponding to the feature extraction network, an attention mechanism model corresponding to the attention mechanism module and a classification model corresponding to the classification network are obtained.); and
PNG
media_image3.png
374
801
media_image3.png
Greyscale
PNG
media_image4.png
120
794
media_image4.png
Greyscale
constructing the medical image recognition model based on the feature extraction model, the attention mechanism model, and the classification model (Fig. 4).
As per claim 6, an independent claim, Wang teaches a method (Abstract) for recognizing a medical image, comprising:
obtaining a fundus image to be recognized (Fig. 3; Fig. 4);
obtaining a target disease label of the fundus image to be recognized by invoking a medical image recognition model that has been trained to recognize the fundus image to be recognized (Fig. 4 shows a medical image recognition model that is trained to recognize the fundus image. The obtained target disease labels include eight categories: normal (N), DR (D), glaucoma (G), cataract (C), AMD (A), hypertension (H), Myopia (M), and other diseases/abnormalities (O)), the medical image recognition model comprising a feature extraction model, an attention mechanism model, and a classification model (See below Fig. 4, in which stages S0-S3 corresponds to a feature extraction model, stage S4 corresponds to an attention mechanism model, and “Sigmoid classifier” represents a classification model), and obtaining the target disease label comprising:
PNG
media_image1.png
464
977
media_image1.png
Greyscale
obtaining a fundus feature map by invoking a feature extraction model to extract features from the fundus image to be recognized (See below Table 7 which shows dimensions of feature maps from each stage. The feature maps from stage 3 have dimensions 14*14*512);
PNG
media_image2.png
236
469
media_image2.png
Greyscale
inputting the fundus feature map into the attention mechanism model, and obtaining an attention-weighted feature map (Fig. 4 stage S4; The feature maps from state 4 have dimensions 7*7*512; Abstract “Therefore, given that the self-attention (SA) model with a global receptive field may have robust global-level feature modeling ability, we propose a multistage fundus image classification model MBSaNet which combines CNN and SA mechanism. The convolution block extracts the local information of the fundus image, and the SA module further captures the complex relationships between different spatial positions, thereby directly detecting one or more fundus diseases in retinal fundus image”); and
obtaining a target disease label of the fundus image to be recognized by invoking the classification model to classify the attention-weighted feature map (Fig. 4 “Sigmoid classifier”; The classified target disease labels include eight categories: normal (N), DR (D), glaucoma (G), cataract (C), AMD (A), hypertension (H), Myopia (M), and other diseases/abnormalities (O)).
Claim 11, an independent system claim, recites steps corresponding to method claim 1. Therefore the recited steps of 11 are mapped to Wang in the same manner as corresponding steps in claim 1. Wang additionally teaches system elements, such as at least one processors and a storage device storing at least one instruction (See below paragraph from page 3).
PNG
media_image5.png
135
801
media_image5.png
Greyscale
Claim 16, dependent upon claim 11, recites steps corresponding to method claim 6. Therefore the recited steps of 16 are mapped to Wang in the same manner as corresponding steps in claim 6.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-3, 7-8, 12-13 and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Wang K, Xu C, Li G, Zhang Y, Zheng Y, Sun C. Combining convolutional neural networks and self-attention for fundus diseases identification. Scientific Reports. 2023 Jan 2;13(1):76. Hereafter Wang), in view of Cai et al. (US 20250252711 A1, hereafter Cai).
As per claim 2, dependent upon claim 1, Wang does not teach the recited limitations.
Cai in an analogous field discloses a convolutional layer of a computer model that generates a dynamic convolutional filter based on the input feature map of the convolutional layer. The convolutional layer includes an attention model that generates a set of attention weights to dynamically adjust the convolutional filter applied by the model based on the input to the convolutional layer (Abstract). Specifically, Cai teaches the attention model comprises a perception layer (FIG. 6 #610; para. [0044] “In one embodiment, the channel squeeze and excitation layer 610 is implemented with a fully-connected layer, such as a multi-layer-perceptron (MLP), followed by a batch normalization layer (BN) and an activation layer, such as a rectified linear unit (ReLU) layer”). Cai further taches:
obtaining a plurality of training pooling values by performing a pooling processing on each training feature map (FIG. 6 “Input Feature Map x” and #600 “Channel Pooling Layer”);
inputting each training pooling value of the plurality of training pooling values into the training perception layer (FIG. 6 #610), and obtaining a training weight vector corresponding to each training feature map based on a weight of each neuron in the training perception layer for each training pooling value (FIG. 6 att_sn, att_cn, att_fn, att_wn being weight vectors; Note the channel squeeze and excitation layer 610 is implemented with a fully-connected layer in which every neuron in one layer is connected to every neuron in the adjacent layer. Para. [0045]); and
generating the training weighted feature map for each training feature map based on each training feature map and the corresponding training weight vector (FIG. 6 “Output Feature Map y”; para. [0046]).
It would have been obvious for a person with ordinary skill in the art before the effective filing date of the claimed invention to modify Wang to incorporate the teaching of Cai to include a training perception layer and generate training weighted feature map as claimed. The motivation for doing so is to determine a dynamic convolutional filter based on an input feature map in order to improve object detection accuracy (Cai para. [0016]).
As per claim 3, dependent upon claim 2, Wang in view of Cai further teaches:
performing a global average pooling processing on each training feature map and obtaining training global average pooling values (Cai FIG. 6 #600 “Channel Pooling Layer”; para. [0043] “The channel pooling layer 600 pools the values across the spatial dimensions of the input feature map to obtain representative values in the channel descriptor. In this example, the channel pooling layer 600 may perform global average pooling or global maximum pooling to determine a value for each channel of the input feature map that is representative of that channel for the feature map as a whole”);
inputting each of the training global average pooling values into the training perception layer (Cai FIG. 6 #610), and obtaining a training weight vector corresponding to each training feature map based on a weight of each neuron in the training perception layer for each training global average pooling value (Cai FIG. 6 att_sn, att_cn, att_fn, att_wn being weight vectors; Note the channel squeeze and excitation layer 610 is implemented with a fully-connected layer in which every neuron in one layer is connected to every neuron in the adjacent layer. Para. [0045]), and then obtain a training weighted feature map of each training feature map based on each training feature map and the corresponding training weight vector (Cai FIG. 6 “Output Feature Map y”; para. [0046]).
As per claim 7, dependent upon claim 6, Wang in view of Cai teaches the attention mechanism model comprises a perception layer (Cai FIG. 6 #610; para. [0044] “In one embodiment, the channel squeeze and excitation layer 610 is implemented with a fully-connected layer, such as a multi-layer-perceptron (MLP), followed by a batch normalization layer (BN) and an activation layer, such as a rectified linear unit (ReLU) layer”). Cai further teaches:
performing a pooling processing on the fundus feature map and obtaining pooling values (Cai FIG. 6 “Input Feature Map x” and #600 “Channel Pooling Layer”);
inputting each of the pooling values into the perception layer (Cai FIG. 6 #610), and obtaining a target weight vector of the fundus feature map based on a weight of each neuron in the perception layer for each pooling value (Cai FIG. 6 att_sn, att_cn, att_fn, att_wn being weight vectors; Note the channel squeeze and excitation layer 610 is implemented with a fully-connected layer in which every neuron in one layer is connected to every neuron in the adjacent layer. Para. [0045]); and
generating the attention-weighted feature map based on the fundus feature map and the target weight vector (Cai FIG. 6 “Output Feature Map y”; para. [0046]).
As per claim 8, dependent upon claim 7, Wang in view of Cai further teaches: performing a global average pooling on the fundus feature map and obtaining global average pooling values (Cai FIG. 6 #600 “Channel Pooling Layer”; para. [0043] “The channel pooling layer 600 pools the values across the spatial dimensions of the input feature map to obtain representative values in the channel descriptor. In this example, the channel pooling layer 600 may perform global average pooling or global maximum pooling to determine a value for each channel of the input feature map that is representative of that channel for the feature map as a whole”);
inputting each of the global average pooling values into the perception layer (Cai FIG. 6 #610), and obtaining the target weight vector based on a weight of each neuron in the perception layer for each global average pooling value (Cai FIG. 6 att_sn, att_cn, att_fn, att_wn being weight vectors; Note the channel squeeze and excitation layer 610 is implemented with a fully-connected layer in which every neuron in one layer is connected to every neuron in the adjacent layer. Para. [0045]); and
generating the attention-weighted feature map based on the fundus feature map and the target weight vector (Cai FIG. 6 “Output Feature Map y”; para. [0046]).
Claim 12, dependent upon claim 11, recites similar steps as in corresponding method claim 2. Therefore the recited steps of 12 are mapped to Wang in view of Cai in the same manner as corresponding steps in claim 2. Additionally the rationale and motivation for combining Wang and Cai is also applicable to claim 12.
Claim 13, dependent upon claim 12, recites similar steps as in corresponding method claim 3. Therefore the recited steps of 13 are mapped to Wang in view of Cai in the same manner as corresponding steps in claim 3.
Claim 17, dependent upon claim 16, recites similar steps as in corresponding method claim 7. Therefore the recited steps of 17 are mapped to Wang in view of Cai in the same manner as corresponding steps in claim 7.
Claim 18, dependent upon claim 17, recites similar steps as in corresponding method claim 8. Therefore the recited steps of 18 are mapped to Wang in view of Cai in the same manner as corresponding steps in claim 8.
Allowable Subject Matter
Claims 4-5, 9-10, 14-15 and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Additional reference He et al. (He X, Deng Y, Fang L, Peng Q. Multi-modal retinal image classification with modality-specific attention network. IEEE transactions on medical imaging. 2021 Feb 24;40(6):1591-602. Hereafter He) teaches a multi-modal fundus image classification model using modality-specific attention network (Abstract). The model includes feature extraction layers, multi-scale attention subnet, region-guided attention subnet and a classifier (FIG. 4-5; page 1594-1595 section B. Multi-Scale Attention Module for Fundus Image). He further teaches calculating a loss function based on ground truth label and predicted label (page 1596 section D. Classifier and Loss Function eqn. (14)).
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XUEMEI G CHEN whose telephone number is (571)270-3480. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XUEMEI G CHEN/Primary Examiner, Art Unit 2661
Prosecution Insights