DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Amendment
The preliminary amendment filed on March 15th, 2024 has been entered and acknowledged.
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record on file.
Information Disclosure Statement(s)
The Information disclosure statement(s) (IDS) filed on March 15th, 2024, May 31st, 2024 and September 17th, 2025 have been acknowledged and considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
1) “a classification model configured to classify…..,” as recited in claims 1, 8 and 9 and mentioned in claims 2-4;
The specification discloses sufficient structure, material and/or act for the recited classification model to perform the recited function as disclosed in [0007] wherein the classification model is constructed by a convolutional neural network and an attention branch network that visualizes and an interest region of the convolutional neural network.
furthermore, additional claim limitations evoke 112(f) are:
2) “a prediction unit configured to carry….” as recited in claim 1;
The specification does not disclose sufficient structure, material and/or acts for the recited prediction unit to perform the recited functions, the closest disclosure can be found in [0019] wherein the prediction unit is mentioned as by using the classification model, however, is disclosed as being associated to and not sufficiently disclosed to perform the recited functions.
3) “a learning unit configured to carry out….” as recited in claim 1 and mentioned in claims 2-4;
The specification does not disclose sufficient structure, material and/or act for the recited learning unit to perform the recited function. the closest disclosure can be found in [0047] wherein it discloses the prediction unit classified the presence or absence of the disease….by using a trained classification model however, it’s disclosed as being in association with the trained classification model being neural network, and this disclosure does not teach fully the functions recited in the claims.
4) “a feature extractor configured to generate….” as recited in claim 2;
The specification does not disclose sufficient structure, material and/or act for the recited feature extractor to perform the recited function, the disclosure mentions the feature extractor nominally without providing clear structure material and/or act for it to perform the recited function.
5) “a prediction step of carrying out a prediction….” as recited in claim 8 and claim 9;
The specification does not disclose sufficient structure, material and/or acts, for the recited prediction step to perform the recited functions, the disclosure mentions the prediction step nominally without giving it structure, material and/or act to perform the recited functions.
6) “a learning step of carrying out supervised learning….” as recited in claim 8 and claim 9;
The specification does not disclose sufficient structure, material and/or act for the recited learning step to perform the recited functions, the closest disclosure can be found in [0014-0015] wherein the learning step is disclosed to perform the recited functions however, does not provide which specific structure to perform this function nor a specific material and/or act such as algorithms, flowchart to perform the functions.
Because some of these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have some of these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-4 and 8-9 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Specifically, claims 1, 2, 8 and 9 recite the features of “a prediction unit configured to carry….” as recited in claim 1; “a learning unit configured to carry out….” as recited in claim 1 and mentioned in claims 2-4; “a feature extractor configured to generate….” as recited in claim 2; “a prediction step of carrying out a prediction….” as recited in claim 8 and claim 9; “a learning step of carrying out supervised learning….” as recited in claim 8 and claim 9, which, as discussed above in the 112f interpretation section, do not have sufficient support from the disclosure for these recited features to perform the recited functions hence, are lack of written descriptions. The dependent claims 3-4 are rejected based on dependency and doesn’t overcome the deficiencies of the claims 1, 2, 8 and 9.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-4 and 8-9 are rejected under 35 U.S.C. 112(b), Claim limitation of “a prediction unit configured to carry….” as recited in claim 1; “a learning unit configured to carry out….” as recited in claim 1 and mentioned in claims 2-4; “a feature extractor configured to generate….” as recited in claim 2; “a prediction step of carrying out a prediction….” as recited in claim 8 and claim 9; “a learning step of carrying out supervised learning….” as recited in claim 8 and claim 9 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. the features of “a prediction unit configured to carry….” as recited in claim 1; “a learning unit configured to carry out….” as recited in claim 1 and mentioned in claims 2-4; “a feature extractor configured to generate….” as recited in claim 2; “a prediction step of carrying out a prediction….” as recited in claim 8 and claim 9; “a learning step of carrying out supervised learning….” as recited in claim 8 and claim 9, which, as discussed above in the 112f interpretation section, do not have sufficient support from the disclosure for these recited features to perform the recited functions hence, are lack of written descriptions. The dependent claims 3-4 are rejected based on dependency and doesn’t overcome the deficiencies of the claims 1, 2, 8 and 9. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 6 and 8-9 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Yuxing Tang et. al. (“Attention-Guided Curriculum Learning for Weakly Supervised Classification and Localization of Thoracic Diseases on Chest Radiographs, Sept. 2018, Part of the book series: Lecture Notes in Computer Science, LNIP, Vol. 11046” hereinafter as “Tang”).
(as best understood based on the 112f interpretation above) Regarding claim 1, Tang discloses a medical image diagnostics assistance device for assisting diagnosis of a medical image (title and abstract and page 250, last par., discloses the current invention label image region to have presence or absence of certain disease), comprising: a classification model configured to classify at least presence or absence of a disease from the medical image (page 250, last par., discloses the processing to label the images to have presence or absence of certain disease using classification model of an attention-guided curriculum learning framework such as disclosed in page 251, 1st par.); a prediction unit configured to carry out prediction using the classification model (as illustrated in figure 2, the framework can use the classification model to carry out the prediction performed by the FC/fully connected layer [prediction model]); and a learning unit configured to carry out supervised learning of the classification model before the classification model is used by the prediction unit (section 2.2 discloses the training of the framework model uses weakly supervised object detection indicating a supervised learning of the classification model to train the network before using it for prediction in a more accurate and improved performance), wherein in the supervised learning carried out by the learning unit (page 252, last par., discloses the training or the supervised learning as discussed previously, is carried out in a baseline model in section 2.1 hence, indicating the use of learning unit to perform of the baseline model), a training medical image for which at least the presence or absence of the disease is previously known is used as supervised data (section 2.2 discloses the training data includes image data including information of regions with presence or absence of the disease or the corresponding labels present in the training images for training of the model), the classification model is constructed by a convolutional neural network and an attention branch network (section 2.3, 3rd par., discloses the network of the classification model includes a convolutional network and an attention guided seeding for the attention map [analogous to the attention branch network as claimed]) that visualizes an interest region of the convolutional neural network (the attention branch perform attention mapping iteratively, as disclosed in page 254, 1st 3 paragraphs, to visualize the region of interest for the convolutional neural network), and in a stage where the supervised learning of the classification model is carried out by the learning unit, the attention branch network is provided with preliminary information indicating a classification region which is a region required for classifying the presence or absence of the disease on the training medical image (page 252 discloses the training of the neural network includes using of the supervised learning method, as discussed previously, including using the attention branch to use the attention map to guide the convolutional block during the training with an objective function to optimize the multi-label classification, as disclosed in page 254, the attention map is analogous to the preliminary information as disclosed in which a region of classification region is determined for providing the labeling on the training image such as disclosed in how the features are mined in page 252, last par.).
Regarding claim 6, Tang discloses the medical image diagnostics assistance device according to claim 1, wherein any one of VGG16, ResNet50, and DenseNet121 is used as the convolutional neural network (“any one of” indicates a selection, therefore, only one of the options is the instant scope of the claim, the examiner selects “ResNet50” to be mapped to Tang’s page 255, 1st par., wherein the ResNet50 is used as a backbone of the CNN convolutional neural network).
(as best understood based on the 112f interpretation above) Regarding claim 8, Tang discloses a medical image diagnostics assistance method for assisting diagnosis of a medical image (title and abstract), comprising: a prediction step of carrying out prediction using the classification model configured to classify at least presence or absence of a disease from the medical image (as illustrated in figure 2, the framework can use the classification model to carry out the prediction performed by the FC/fully connected layer [prediction model]; page 250, last par., discloses the current invention label image region to have presence or absence of certain disease; page 250, last par., discloses the processing to label the images to have presence or absence of certain disease using classification model of an attention-guided curriculum learning framework such as disclosed in page 251, 1st par.); and a learning step of carrying out supervised learning of the classification model before the classification model is carried out (section 2.2 discloses the training of the framework model uses weakly supervised object detection indicating a supervised learning of the classification model to train the network before using it for prediction in a more accurate and improved performance), wherein in the supervised learning carried out by the learning step (page 252, last par., discloses the training or the supervised learning as discussed previously, is carried out in a baseline model in section 2.1 hence, indicating the use of learning unit to perform of the baseline model), a training medical image for which at least the presence or absence of the disease is previously known is used as supervised data (section 2.2 discloses the training data includes image data including information of regions with presence or absence of the disease or the corresponding labels present in the training images for training of the model), the classification model is constructed by a convolutional neural network and an attention branch network (section 2.3, 3rd par., discloses the network of the classification model includes a convolutional network and an attention guided seeding for the attention map [analogous to the attention branch network as claimed]) that visualizes an interest region of the convolutional neural network (the attention branch perform attention mapping iteratively, as disclosed in page 254, 1st 3 paragraphs, to visualize the region of interest for the convolutional neural network), and in the learning step, the attention branch network is provided with preliminary information indicating a classification region which is a region required for classifying the presence or absence of the disease on the training medical image (page 252 discloses the training of the neural network includes using of the supervised learning method, as discussed previously, including using the attention branch to use the attention map to guide the convolutional block during the training with an objective function to optimize the multi-label classification, as disclosed in page 254, the attention map is analogous to the preliminary information as disclosed in which a region of classification region is determined for providing the labeling on the training image such as disclosed in how the features are mined in page 252, last par.).
(as best understood based on the 112f interpretation above) Regarding claim 9, Tang discloses a non-transitory computer readable medium holding instructions that causes a computer to carry out steps comprising: a prediction step of carrying out prediction using the classification model configured to classify at least presence or absence of a disease from the medical image (as illustrated in figure 2, the framework can use the classification model to carry out the prediction performed by the FC/fully connected layer [prediction model]; page 250, last par., discloses the current invention label image region to have presence or absence of certain disease; page 250, last par., discloses the processing to label the images to have presence or absence of certain disease using classification model of an attention-guided curriculum learning framework such as disclosed in page 251, 1st par.); and a learning step of carrying out supervised learning of the classification model before the classification model is carried out (section 2.2 discloses the training of the framework model uses weakly supervised object detection indicating a supervised learning of the classification model to train the network before using it for prediction in a more accurate and improved performance), wherein in the supervised learning carried out by the learning step (page 252, last par., discloses the training or the supervised learning as discussed previously, is carried out in a baseline model in section 2.1 hence, indicating the use of learning unit to perform of the baseline model), a training medical image for which at least the presence or absence of the disease is previously known is used as supervised data (section 2.2 discloses the training data includes image data including information of regions with presence or absence of the disease or the corresponding labels present in the training images for training of the model), the classification model is constructed by a convolutional neural network and an attention branch network (section 2.3, 3rd par., discloses the network of the classification model includes a convolutional network and an attention guided seeding for the attention map [analogous to the attention branch network as claimed]) that visualizes an interest region of the convolutional neural network (the attention branch perform attention mapping iteratively, as disclosed in page 254, 1st 3 paragraphs, to visualize the region of interest for the convolutional neural network), and in the learning step, the attention branch network is provided with preliminary information indicating a classification region which is a region required for classifying the presence or absence of the disease on the training medical image (page 252 discloses the training of the neural network includes using of the supervised learning method, as discussed previously, including using the attention branch to use the attention map to guide the convolutional block during the training with an objective function to optimize the multi-label classification, as disclosed in page 254, the attention map is analogous to the preliminary information as disclosed in which a region of classification region is determined for providing the labeling on the training image such as disclosed in how the features are mined in page 252, last par.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Yuxing Tang et. al. (“Attention-Guided Curriculum Learning for Weakly Supervised Classification and Localization of Thoracic Diseases on Chest Radiographs, Sept. 2018, Part of the book series: Lecture Notes in Computer Science, LNIP, Vol. 11046” hereinafter as “Tang”) in view of Hiroshi Fukui et. al. (“Attention Branch Network: Learning of Attention Mechanism for Visual Explanation, 2019, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10705-10714” hereinafter as “Fukui”) further in view of Rahul Kumar (“CornerNet: Detecting Objects as Paired Keypoints, Mar. 2020, Medium.com, Rahul/cse/ccu” hereinafter as “Kumar”) and Pei-Chang Guo (“A Frobenius Norm Regularization Method for Convolutional Kernels to Avoid Unstable Gradient Problem, July 2019, Machine Learning, arXiv admin note: text overlap” hereinafter as “Guo”) and Haibao Wang et. al. (“Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”, July 2020, IEEE Transactions on Cognitive and Development Systems, Vol. 13, Issue 4” hereinafter as “Wang”).
(as best understood based on the 112f interpretation above) Regarding claim 2, Tang discloses the medical image diagnostics assistance device according to claim 1, wherein the attention branch network includes a feature extractor configured to generate a feature quantity map by extracting a feature quantity required for classifying the medical image (as discussed above in claim 1 of the attention branch as disclosed in Tang’s page 253, last par., of the attention branch which includes a feature channel [feature extractor], as disclosed in page 254, 1st par., to extract the features form the image to generate a heatmap [feature quantity map, based on BRI/broadest reasonable interpretation] for classifying the image as shown in figure 2), an attention branch configured to generate an attention map using class activation mapping (as disclosed in page 254, 1st par., the processing of the attention branch then generate an attention map using an activation map, such as disclosed in page 253, 1st par.), the attention map generated by the attention branch is reflected in the feature quantity map generated by the feature extractor (section 2.3, 1st 2 pars., discloses the heatmaps highlight the regions in the image for disease recognition which indicates that the heatmap of the attention map, as disclosed in page 253, last 2 paragraphs, have feature highlighted generated by the feature channel according to page 253, last par., and page 254, 1st par., and section 3, 2nd par.).
However, Tang does not explicitly disclose and a perception branch, in the stage where the supervised learning of the classification model is carried out by the learning unit, the perception branch outputs the feature quantity map weighted by the attention map, as a classification result of the training medical image, a loss function of the attention branch network is a sum of a learning error of the attention branch, a learning error of the perception branch, and a regularization term, the regularization term is Frobenius norm of a matrix obtained by a Hadamard product of the attention map and a weight map, and the weight map corresponds to the classification region.
In the same field of convolutional neural network processing (title and abstract, Fukui), Fukui discloses and a perception branch, in the stage where the supervised learning of the classification model is carried out by the learning unit (page 10706, 1st column, 2nd par., discloses the convolutional network further includes the attention branch and a perception branch where the supervised learning is carried out according to page 10706, 2nd col., 2nd par.), the perception branch outputs the feature quantity map weighted by the attention map (page 10706, 1st col., 2nd par., discloses the perception branch is to output the probabilities of class using the feature and the attention maps to the convolution layers indicating the branch output the weighted feature and the attention map [feature quantity map] using gradient-weighted class activation mapping according to page 10706, 2nd col., 1st par., by BRI, covers the scope of the claim), as a classification result of the training medical image, a loss function of the attention branch network is a sum of a learning error of the attention branch, a learning error of the perception branch (section 3.3, 1st par., discloses the training is based on a training loss function [loss function as claimed] which is based on sum of errors/losses at both branches of the attention branch and the perception branch such as shown in equation 3), and a regularization term (section 3.3 further discloses each branch loss is calculated based on a combination of softmax function and cross-entropy [together being analogous to the recited regularization term as claimed, since cross-entropy and softmax function are functions of a final activation function converting unformalized data to probability distribution indicating a regularization]).
Thus, it would have been obvious for a person of ordinary skill in the art before the effective filing date to modify Tang to have a convolutional neural network with an attention branch and a perception branch in the stage where the supervised learning of the classification model is carried out by the learning unit, the perception branch outputs the feature quantity map weighted by the attention map, as a classification result of the training medical image, a loss function of the attention branch network is a sum of a learning error of the attention branch, a learning error of the perception branch, and a regularization term as taught by Fukui to arrive at the claimed invention discussed above. Such a modification is the result of combing prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to perform convolutional image classification more effectively (abstract, Fukui).
However, Tang in view of Fukui does not explicitly disclose the regularization term is Frobenius norm of a matrix obtained by a Hadamard product of the attention map and a weight map, and the weight map corresponds to the classification region.
In the same field of convolutional neural network machine learning (title and abstract, Guo), Guo discloses the regularization term is Frobenius norm of a matrix (page 2, 2nd to the last par., discloses in convolutional network learning, a regularization term can be used to weight the neural network such as further discloses in page 7, last par., wherein the regularization term can be of a Frobenius norm method in this paper).
Thus, it would have been obvious for a person of ordinary skill in the art before the effective filing date to modify Tang in view of Fukui to have a convolutional neural network with an attention branch and a perception branch in the stage where the supervised learning of the classification model is carried out by the learning unit, the perception branch outputs the feature quantity map weighted by the attention map, as a classification result of the training medical image, a loss function of the attention branch network is a sum of a learning error of the attention branch, a learning error of the perception branch, and a regularization term, the regularization term is Frobenius norm of a matrix as taught by Guo to arrive at the claimed invention discussed above. Such a modification is the result of combing prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to perform convolutional neural network learning more effectively (abstract and page 7, last par., Guo).
However, Tang in view of Fukui and Guo does not explicitly disclose a matrix obtained by a Hadamard product of the attention map and a weight map, and the weight map corresponds to the classification region.
In the same field of Neural Networks Learning (title and abstract, Wang), Wang discloses a matrix obtained by a Hadamard product of the attention map and a weight map, and the weight map corresponds to the classification region (section II.B discloses the Frebinus norm by a matrix of a product of the Hadamard product of the attention feature map and a feature-weighted RF [weight map as claimed] corresponding to the weighted sum within the spatial extend of the GRF indicating the classification region).
Thus, it would have been obvious for a person of ordinary skill in the art before the effective filing date to modify Tang in view of Fukui and Guo to have a convolutional neural network with an attention branch and a perception branch in the stage where the supervised learning of the classification model is carried out by the learning unit, the perception branch outputs the feature quantity map weighted by the attention map, as a classification result of the training medical image, a loss function of the attention branch network is a sum of a learning error of the attention branch, a learning error of the perception branch, and a regularization term, the regularization term is Frobenius norm of a matrix obtained by a Hadamard product of the attention map and a weight map, and the weight map corresponds to the classification region as taught by Wang to arrive at the claimed invention discussed above. Such a modification is the result of combing prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to perform neural network feature learning more effectively (abstract, Wang).
Regarding claim 7, Tang in view of Fukui and Guo and Wang discloses the medical image diagnostics assistance device according to claim 2, wherein an output from the perception branch is visualized by applying Grad-CAM to the output from the perception branch (Fukui, page 10706, 2nd col., 1st par., discloses the perception branch with visual explanation of gradient-weighted class activation mapping of a Grad-Cam output from the perception branch). The motivation for combination of arts is the same as for claim 2 above.
Claims 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Yuxing Tang et. al. (“Attention-Guided Curriculum Learning for Weakly Supervised Classification and Localization of Thoracic Diseases on Chest Radiographs, Sept. 2018, Part of the book series: Lecture Notes in Computer Science, LNIP, Vol. 11046” hereinafter as “Tang”) in view of Hiroshi Fukui et. al. (“Attention Branch Network: Learning of Attention Mechanism for Visual Explanation, 2019, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10705-10714” hereinafter as “Fukui”) further in view of Rahul Kumar (“CornerNet: Detecting Objects as Paired Keypoints, Mar. 2020, Medium.com, Rahul/cse/ccu” hereinafter as “Kumar”) and Pei-Chang Guo (“A Frobenius Norm Regularization Method for Convolutional Kernels to Avoid Unstable Gradient Problem, July 2019, Machine Learning, arXiv admin note: text overlap” hereinafter as “Guo”) and Haibao Wang et. al. (“Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”, July 2020, IEEE Transactions on Cognitive and Development Systems, Vol. 13, Issue 4” hereinafter as “Wang”) and Sun Jing et. al. (foreign patent document “CN 102722891 B” hereinafter as “Jing”).
(as best understood based on the 112f interpretation above) Regarding claim 3, Tang in view of Fukui further in view of Guo and Wang discloses the medical image diagnostics assistance device according to claim 2 (such as discussed above in claim 2), wherein in the stage where the supervised learning of the classification model is carried out by the learning unit (Tang, page 252, last par., discloses the training or the supervised learning as discussed previously, is carried out in a baseline model in section 2.1 hence, indicating the use of learning unit to perform of the baseline model), a segmentation image of a first portion which is a portion of the classification region (Tang, page 250, 2nd to the last par., discloses the image is segmented for the neural network processing a portion of the segmented for labelling).
Tang in view of Fukui further in view of Guo and Wang does not explicitly disclose the attention branch network receives the weight map prepared by carrying out convex hull processing on a segmentation image of a first portion which is a portion of the classification region.
(mapping is based on page count of the translation part of the document) In the same field of image segmentation (“Technical field” section and “background technique” section, 1st par, Jing), Jing discloses the attention branch network receives the weight map prepared by carrying out convex hull processing on a segmentation image of a first portion which is a portion of the classification region (page 3 discloses the segmentation of the image for a portion being segmented uses the difference from the outside of the convex hull to reject the background noise and form an edge weight map, and by using the weighted convex hull to calculate the priori image of the image can remove the background for image segmentation).
Thus, it would have been obvious for a person of ordinary skill in the art before the effective filing date to modify Tang in view of Fukui further in view of Guo and Wang to have a stage where the supervised learning is carried out, wherein in the stage where the supervised learning of the classification model is carried out by the learning unit, the attention branch network receives the weight map prepared by carrying out convex hull processing on a segmentation image of a first portion which is a portion of the classification region as taught by Jing to arrive at the claimed invention discussed above. Such a modification is the result of combing prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to detect target in the image more effectively (“technical field” section, Jing).
(as best understood based on the 112f interpretation above) Regarding claim 4, Tang in view of Fukui further in view of Guo and Wang discloses the medical image diagnostics assistance device according to claim 2 (as discussed above in claim 1), wherein in the stage where the supervised learning of the classification model is carried out by the learning unit (Tang, page 252, last par., discloses the training or the supervised learning as discussed previously, is carried out in a baseline model in section 2.1 hence, indicating the use of learning unit to perform of the baseline model), the attention branch network receives the weight map (as discussed above in claim 1, as disclosed in Wang’s section II.B discloses the Frebinus norm by a matrix of a product of the Hadamard product of the attention feature map and a feature-weighted RF [weight map as claimed] corresponding to the weighted sum within the spatial extend of the GRF indicating the classification region)).
However, Tang in view of Fukui further in view of Guo and Wang does not explicitly disclose the weight map prepared by combining a segmentation image of a first portion which is a portion of the classification region and a segmentation image of a second portion which is another portion of the classification region.
(mapping is based on page count of the translation part of the document) In the same field of image segmentation (“Technical field” section and “background technique” section, 1st par, Jing), Jing discloses the weight map prepared by combining a segmentation image of a first portion which is a portion of the classification region and a segmentation image of a second portion which is another portion of the classification region (the weight map is obtained according to formula 2, based on calculated edged image of the edge feature and calculated color different between internal superpixel and the external superpixel indicating two different portions of the segmentation of the image, such as recited in the claim, moreover, the region of the internal superpixel and the external superpixel are of image classification regions according to page 2, 3rd par., as being combined to obtain the weight map).
Thus, it would have been obvious for a person of ordinary skill in the art before the effective filing date to modify Tang in view of Fukui further in view of Guo and Wang to have a stage where the supervised learning is carried out, the attention branch network receives the weight map prepared by combining a segmentation image of a first portion which is a portion of the classification region and a segmentation image of a second portion which is another portion of the classification region as taught by Jing to arrive at the claimed invention discussed above. Such a modification is the result of combing prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to detect target in the image more effectively (“technical field” section, Jing).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Yuxing Tang et. al. (“Attention-Guided Curriculum Learning for Weakly Supervised Classification and Localization of Thoracic Diseases on Chest Radiographs, Sept. 2018, Part of the book series: Lecture Notes in Computer Science, LNIP, Vol. 11046” hereinafter as “Tang”) in view of Hiroshi Fukui et. al. (“Attention Branch Network: Learning of Attention Mechanism for Visual Explanation, 2019, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10705-10714” hereinafter as “Fukui”) further in view of Rahul Kumar (“CornerNet: Detecting Objects as Paired Keypoints, Mar. 2020, Medium.com, Rahul/cse/ccu” hereinafter as “Kumar”) and Pei-Chang Guo (“A Frobenius Norm Regularization Method for Convolutional Kernels to Avoid Unstable Gradient Problem, July 2019, Machine Learning, arXiv admin note: text overlap” hereinafter as “Guo”) and Haibao Wang et. al. (“Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”, July 2020, IEEE Transactions on Cognitive and Development Systems, Vol. 13, Issue 4” hereinafter as “Wang”) and Sun Jing et. al. (foreign patent document “CN 102722891 B” hereinafter as “Jing”) and Zhu Weifang et. al. (foreign patent document “CN 112819798 A” hereinafter as “Weifang”).
Regarding claim 5, Tang in view of Fukui further in view of Guo and Wang and Jing discloses the medical image diagnostics assistance device according to claim 3 (such as discussed above in claim 3), wherein the segmentation image of the first portion and/or a combination of the segmentation image of the first portion and a segmentation image of a second portion which is another portion of the classification region (“and/or” indicates a selection, therefore, only one of the options is the instant scope of the claim, the examiner selects “the segmentation image of the first portion and a segmentation image of a second portion….” which is disclosed in Jing’s the weight map is obtained according to formula 2, based on calculated edged image of the edge feature and calculated color different between internal superpixel and the external superpixel indicating two different portions of the segmentation of the image, such as recited in the claim, moreover, the region of the internal superpixel and the external superpixel are of image classification regions according to page 2, 3rd par., as being combined to obtain the weight map).
However, Tang in view of Fukui further in view of Guo and Wang and Jing does not explicitly disclose portion of the classification region are/is generated by using U-Net.
(mapping is based on the page counts of the translation part of the document) In the same field of medical image segmentation (“technical field” section, Weifang), Weifang discloses portion of the classification region are/is generated by using U-Net (page 3, last par., discloses the image segmentation of the portion of the image is generated by a U-net).
Thus, it would have been obvious for a person of ordinary skill in the art before the effective filing date to modify Tang in view of Fukui further in view of Guo and Wang and Jing to have a system that perform image segmentation to obtain different portions of the segmentation of the image generated by a U-net as taught by Weifang to arrive at the claimed invention discussed above. Such a modification is the result of combing prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to perform image segmentation effectively (“technical field” section, Weifang).
Pertinent Prior Art(s)
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Jhimli Mitra et. al. “US 2020/0297219 A1” discloses convolutional neural network for finding anomalies in medial image (abstract) using discriminator with an adversarial loss ([0027]) with images with training images with anomalies labeled ([0029]) generated by GAN with image reconstruction ([0030]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHUONG HAU CAI whose telephone number is (571)272-9424. The examiner can normally be reached M-F 8:30 am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHUONG HAU CAI/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673