DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Preliminary amendments have been made, Claims 1-15 are amended. Claims 16-20 are new. Claims 1-20 are pending.
Priority
This application is a continuation of International Application No. PCT/CN2022/101015, filed on June 24, 2022, which claims priority to Chinese Patent Application No. 202110745619.5, filed on June 30, 2021.
Information Disclosure Statement
The IDS filed 08/05/24, 11/08/24, and 12/19/24 are considered.
Claim Objection
Claim 8 recites in lines 3-5 a “to-be-trained model image”. This seems to be an erroneous addition of the word “image” to the “to-be-trained model”. Examiner has removed the word image during interpretation in the prior art rejection
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 8-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In particular, in is unclear how a “to-be-trained” model (untrained essentially) is capable of performing all the steps listed. Furthermore, how is the model “to-be-trained” when there is no training step? If the “updating” step of claim 8 is the training, examiner recommends amending the language to language that communicates that the image classification model is a trained version of the to-be-trained model such as “updating, based on the target loss until a model training condition is met, a model parameter of the to-be-trained model to obtain an image classification model, wherein the image classification model is the to-be-trained model trained”. The remaining claims are rejected for their dependency on a rejected claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
35 U.S.C. 101 requires that a claimed invention must fall within one of the four eligible categories of invention (i.e. process, machine, manufacture, or composition of matter) and must not be directed to subject matter encompassing a judicially recognized exception as interpreted by the courts. MPEP 2106. Three categories of subject matter are found to be judicially recognized exceptions to 35 U.S.C. § 101 (i.e. patent ineligible) (1) laws of nature, (2) physical phenomena, and (3) abstract ideas. MPEP 2106(II). To be patent-eligible, a claim directed to a judicial exception must as whole be integrated into a practical application or directed to significantly more than the exception itself (MPEP 2106). Hence, the claim must describe a process or product that applies the exception in a meaningful way, such that it is more than a drafting effort designed to monopolize the exception.
Claims 1-7 and 15-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without integration into a practical application or recitation of significantly more. In the analysis below, the method of claim 1 is considered representative of independent claims 1 and 15 since all of the independent claims recite very similar steps despite being directed to different statutory matter. Furthermore, each of independent claims 1, 8, and 15 are directed to one of the four statutory categories of eligible subject matter; thus, the claims pass Step 1 of the Subject Matter Eligibility Test (See flowchart in MPEP 2106). Any differing content in claims 8 and 15 will be directly addressed during the analysis.
Step 2A, Prong 1 Analysis
The independent claims are directed to generating, based on the first feature, a first classification result, wherein the first classification result is for determining a first category of the reference image;
generating, based on the third feature, a second classification result; and
generating, based on the first classification result and the second classification result, a third classification result,
wherein the third classification result is for determining a second category of the to-be-classified image.
Each of the above steps can be performed mentally. An individual can mentally look at features and determine classification of the features to determine the category of the images. As such, the description in independent claims 1 and 15 is an abstract idea – namely, a mental process. Accordingly, the analysis under prong one of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Additional elements
The independent claims further recites obtaining a first feature of a reference image and a second feature of a to-be-classified image;
generating, based on the first feature and the second feature, a third feature. Independent claim 15 includes the additional element of a memory and processors
Step 2A, prong 2 analysis
The above-identified additional elements do not integrate the judicial exception into a practical application.
The steps of generating features can be defined as data gathering steps of obtaining features from images. Such data gathering steps amount to insignificant pre-solution activity which does not integrate the abstract idea into a practical application (MPEP 2106.05(g)).
Each of the additional elements (memory and processors) amounts to merely using a computer as a tool to perform the claimed mental process. Implementing an abstract idea on a computer does not integrate a judicial exception into a practical application (See MPEP 2106.05(f)).
Moreover, the additional elements of the claims do not recite an improvement in the functioning of a computer or other technology or technical field, the claimed steps are not performed using a particular machine, the claimed steps do not effect a transformation, and the claims do not apply the judicial exception in any meaningful way beyond generically linking the use of the judicial exception to a particular technological environment (See MPEP 2106.04(d)). Therefore, the analysis under prong two of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Step 2B
Finally, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The steps of generating features do not amount to more than pre-solution and data gathering. Thus, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea).
Each of the additional elements (memory and processors) are generic computer features which perform generic computer functions that are well-understood, routine, and conventional and do not amount to more than implementing the abstract idea with a computerized system. Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea).
Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation, and mere implementation on a generic computer does not add significantly more to the claims. Accordingly, the analysis under step 2B of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
For all of the foregoing reasons, independent claims 1, 10, and 18 do not recite eligible subject matter under 35 USC 101.
Dependent claims 2-7 are dependent on independent claim 1 and therefore include all of the limitations of claim 1. Therefore, claims 1-7 recite the same abstract idea of a mental process which can be performed in the mind.
Claim 2 recites performing addition processing on the first classification result and the second classification result to obtain the third classification result. An individual can mentally add together their mental classification results. Thus, the feature of claim 2 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 3 recites performing first addition processing on the first classification result and the second classification result to obtain a fourth classification result; performing second addition processing on the first classification result and a model parameter of an image classification model to obtain a fifth classification result; performing multiplication processing on the fifth classification result and a preset weight parameter to obtain a sixth classification result; and performing subtraction processing on the fourth classification result and the sixth classification result to obtain the third classification result. An individual can mentally perform all the mathematical operations on the classification results. Thus, the feature of claim 3 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 4 recites calculating, based on the first feature, a first probability that the reference image belongs to the first category and the second category to obtain the first classification result. An individual can mentally calculate a probability that the reference image belongs to the first category and the second category to obtain the first classification result. Thus, the feature of claim 4 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 5 recites performing feature fusion processing on the first feature and the second feature to obtain the third feature. This further describes the steps of data gathering, which do not integrate into a practical application or constitute as significantly more. Thus, the feature of claim 5 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 6 recites wherein the feature fusion processing comprises at least one of addition processing, multiplication processing, subtraction processing, concatenation processing, or concatenation convolution processing. This further describes the steps of data gathering, which do not integrate into a practical application or constitute as significantly more. Thus, the feature of claim 6 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 7 recites performing first feature extraction processing on the reference image to obtain the first feature and performing second feature extraction processing on the to-be-classified image to obtain the second feature. This further describes the steps of data gathering, which do not integrate into a practical application or constitute as significantly more. Thus, the feature of claim 7 is directed to the mental process. Accordingly, the claim does not recite any additional limitations that integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Dependent claims 16-20 are dependent on independent claim 15 and therefore include all of the limitations of claim 15. Therefore, claims 16-20 recite the same abstract idea of a mental process which can be performed in the mind.
The content of claim 16 is similar to the content of claim 2, hence the 101 analysis for claim 16 is the same analysis done for claim 2 dictating that claim 16 is not eligible under 35 USC 101.
The content of claim 17 is similar to the content of claim 3, hence the 101 analysis for claim 17 is the same analysis done for claim 3 dictating that claim 17 is not eligible under 35 USC 101.
The content of claim 18 is similar to the content of claim 4, hence the 101 analysis for claim 18 is the same analysis done for claim 4 dictating that claim 18 is not eligible under 35 USC 101.
The content of claim 19 is similar to the content of claim 5, hence the 101 analysis for claim 19 is the same analysis done for claim 5 dictating that claim 19 is not eligible under 35 USC 101.
The content of claim 20 is similar to the content of claim 6, hence the 101 analysis for claim 20 is the same analysis done for claim 6 dictating that claim 20 is not eligible under 35 USC 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-7, 15-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over JIE et al. (US 20210390695 A1 Hereinafter “JIE”) in view of SHORT et al. (US 20200302157 A1 Hereinafter “SHORT”).
Regarding claim 1, JIE teaches a method, comprising:
obtaining a first feature of a reference image and a second feature of a to-be-classified image (Fig. 2, [0043]: “In step S204, a first image feature of the target image and a second image feature of the reference image may be determined. For example, the first image feature of the target image and the second image feature of the reference image may be determined in the same manner or in a different manner”);
generating, based on the first feature and the second feature, a third feature (Fig. 2, [0047]: “In step S206, the first image feature and the second image feature may be fused to determine a to-be-classified image feature”);
generating, based on the first feature, a first classification result, wherein the first classification result is for determining a first category of the reference image (Fig. 4, [0087-88]: “In step S408, a first image processing result and a second image processing result of the target image may be determined according to the first target image feature and the second target image feature. In certain embodiments, because the first neural network and the second neural network are different networks obtained by using the same training method, the first image processing result and the second image processing result are in the same type. For example, the first image processing result and the second image processing result may be at least one of an image classification result, an image segmentation result, and a target detection result. This depends on specific manners and training sets used by a person skilled in the art for training neural networks”. The image classification result can be obtained for the reference image (which acts as the first image), “Similarly, the first target the reference image feature and the second target the reference image feature of the reference image may be obtained through steps similar to steps S402 to S410”);
generating, based on the third feature, a second classification result (Fig. 5, [0123]: “The classification result generating unit 540 may be configured to determine, by using the to-be-classified image feature generated by the fusion unit 530, a probability that the target image belongs to a preset category”. To determine the probability that the target image is in a category it need to be classified, additionally a “classification result generating unit” is generating the output, presuming that a classification is output); and
JIE does not expressly disclose using the classification result from the first features and the third features to generate a third classification for the to-be-classified image.
However, SHORT teaches using a combination of classifications to improve the accuracy of a single classification (Fig. 1, [0030]: “In an exemplary embodiment, the method 100 can include a step 130 of combining outputs (e.g., 155, 165) of the classifying 120 into a single output confidence score 140 by using a weighted fusion of the allocated confidence factors, as described in detail herein”. By combining JIE and SHORT in this way, it would result in the first classification and the fused feature classification being combined to generate a classification for the to-be-classified image).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify JIE’s image classification method to include SHORT’s combination of multiple classifications for classifying a to-be-classified image because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify JIE to include SHORT is expressly provided by SHORT, stating that combining outputs can lead to improved confidence over independent machine learning decisions ([0032]: “As described herein, combining outputs associated with all data representation source modes, along with measured or assigned quality, can improve confidence over independent machine learning decisions”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify JIE’s image classification method to include SHORT’s combination of multiple classifications for classifying a to-be-classified image with the motivation of improving classification confidence. The person of ordinary skill in the art would have recognized the benefit of improved classification confidence.
Regarding claim 2, the combination of JIE and SHORT teaches the method of claim 1, in addition, SHORT further teaches wherein generating the third classification result comprises performing addition processing on the first classification result and the second classification result to obtain the third classification result ([0030]: “In an exemplary embodiment, the method 100 can include a step 130 of combining outputs (e.g., 155, 165) of the classifying 120 into a single output confidence score 140 by using a weighted fusion of the allocated confidence factors, as described in detail herein”. The fusion is the addition processing).
The rationale for this combination is similar to the rationale of the combination of claim 1 due to similar methods of combination (the fusion being part of the original classification combination) and benefits.
Regarding claim 4, the combination of JIE and SHORT teaches the method of claim 1, in addition, JIE further teaches wherein generating the first classification result comprises calculating, based on the first feature, a first probability that the reference image belongs to the first category and the second category to obtain the first classification result (Fig. 4, [0087-88]: “In step S408, a first image processing result and a second image processing result of the target image may be determined according to the first target image feature and the second target image feature. In certain embodiments, because the first neural network and the second neural network are different networks obtained by using the same training method, the first image processing result and the second image processing result are in the same type. For example, the first image processing result and the second image processing result may be at least one of an image classification result, an image segmentation result, and a target detection result. This depends on specific manners and training sets used by a person skilled in the art for training neural networks”. The image classification result can be obtained for the reference image (which acts as the first image), “Similarly, the first target the reference image feature and the second target the reference image feature of the reference image may be obtained through steps similar to steps S402 to S410”. These classification results are to determine the probability of the image falling within one of the categories “Taking a classification process for a medical image as an example, preset categories may include a health category and a disease category” [0057], “In some embodiments, probabilities that the target image and the reference image belong to the preset category may be determined according to confidence scores for a plurality of dimensions outputted by the first generally-connected network” [0059]), and wherein generating the second classification result comprises calculating, based on the third feature, a second probability that the to-be- classified image belongs to the first category and the second category to obtain the second classification result (Fig. 5, [0123]: “The classification result generating unit 540 may be configured to determine, by using the to-be-classified image feature generated by the fusion unit 530, a probability that the target image belongs to a preset category”. To determine the probability that the target image is in a category it need to be classified, additionally a “classification result generating unit” is generating the output, presuming that a classification is output. These classification results are to determine the probability of the image falling within one of the categories “Taking a classification process for a medical image as an example, preset categories may include a health category and a disease category” [0057], “In some embodiments, probabilities that the target image and the reference image belong to the preset category may be determined according to confidence scores for a plurality of dimensions outputted by the first generally-connected network” [0059]).
Regarding claim 5, the combination of JIE and SHORT teaches the method of claim 1, in addition, JIE further teaches wherein generating the third feature comprises performing feature fusion processing on the first feature and the second feature to obtain the third feature (Fig. 2, [0047]: “In step S206, the first image feature and the second image feature may be fused to determine a to-be-classified image feature”).
Regarding claim 6, the combination of JIE and SHORT teaches the method of claim 5, in addition, JIE further teaches wherein the feature fusion processing comprises at least one of addition processing, multiplication processing, subtraction processing, concatenation processing, or concatenation convolution processing (Fig. 2, [0048]: “In some embodiments, the first image feature and the second image feature may be stitched, to determine the to-be-classified image feature”. The feature stitching is the addition processing, the “or” limitation only requires one of the listed limitations be met).
Regarding claim 7, the combination of JIE and SHORT teaches the method of claim 1, in addition, JIE further teaches wherein obtaining the a first feature and the second feature comprises:
performing first feature extraction processing on the reference image to obtain the first feature (Fig. 2, [0043]: “In step S204, a first image feature of the target image and a second image feature of the reference image may be determined. For example, the first image feature of the target image and the second image feature of the reference image may be determined in the same manner or in a different manner”. This determination is extraction of the features “For example, multi-viewing-angle scans of the breast on both left and right sides may be simultaneously received based on a network structure used for comparing breasts on both left and right sides, feature extraction is individually performed on each scan of a breast on each side, then feature fusion is performed, and a feature obtained after fusion is simultaneously used for predicting whether the breasts on both left and right sides are positive in breast cancer.” [0069]); and
performing second feature extraction processing on the to-be-classified image to obtain the second feature (Fig. 2, [0043]: “In step S204, a first image feature of the target image and a second image feature of the reference image may be determined. For example, the first image feature of the target image and the second image feature of the reference image may be determined in the same manner or in a different manner”. This determination is extraction of the features “For example, multi-viewing-angle scans of the breast on both left and right sides may be simultaneously received based on a network structure used for comparing breasts on both left and right sides, feature extraction is individually performed on each scan of a breast on each side, then feature fusion is performed, and a feature obtained after fusion is simultaneously used for predicting whether the breasts on both left and right sides are positive in breast cancer.” [0069]).
Regarding claim 15, the content of claim 15 is similar to the content of claim 1, with the additional teachings of a memory and processor. JIE also discloses this information ([0138]: “A hardware unit may be implemented using processing circuitry and/or memory. Each unit can be implemented using one or more processors (or processors and memory)”). Therefore, claim 15 is rejected for the same reasons of obviousness as claim 1, along with the additional teachings above.
Regarding claim 16, the content of claim 16 is similar to the content of claim 2, therefore it is rejected for the same reasons of obviousness as claim 2.
Regarding claim 18, the content of claim 18 is similar to the content of claim 4, therefore it is rejected for the same reasons of obviousness as claim 4.
Regarding claim 19, the content of claim 19 is similar to the content of claim 5, therefore it is rejected for the same reasons of obviousness as claim 5.
Regarding claim 20, the content of claim 20 is similar to the content of claim 6, therefore it is rejected for the same reasons of obviousness as claim 6.
Claims 8-14 are rejected under 35 U.S.C. 103 as being unpatentable over JIE et al. (US 20210390695 A1 Hereinafter “JIE”) in view of SHORT et al. (US 20200302157 A1 Hereinafter “SHORT”) in further view of Wu (US 20210192747 A1 Hereinafter “Wu”).
Regarding claim 8, JIE teaches the method, comprising:
obtaining a reference image and a to-be-classified image (Fig. 2, [0036]: “As shown in FIG. 2, in step S202, a target image and at least one reference image about the target image may be received”);
inputting the reference image and the to-be-classified image to a to-be-trained model image ([0044]: “That is to say, the target image and the reference image may be processed by using the parameter shared neural network”);
obtaining, using the to-be-trained model, a first feature of the reference image and a second feature of the to-be-classified ([0044]: “In some embodiments, convolution processing may be performed, by using a first neural network including at least one convolutional layer, on the target image to obtain the first image feature. Further, convolution processing may be performed, by using the first neural network, on the reference image to obtain the second image feature”);
generating, based on the first feature and the second feature and using the to-be-trained model, a third feature ([0069]: “For example, multi-viewing-angle scans of the breast on both left and right sides may be simultaneously received based on a network structure used for comparing breasts on both left and right sides, feature extraction is individually performed on each scan of a breast on each side, then feature fusion is performed, and a feature obtained after fusion is simultaneously used for predicting whether the breasts on both left and right sides are positive in breast cancer”. Paragraphs 66-70 further describe the network being used for fusion);
generating, based on the first feature and using the to- be-trained model, a first classification result (Fig. 4, [0087-88]: “In step S408, a first image processing result and a second image processing result of the target image may be determined according to the first target image feature and the second target image feature. In certain embodiments, because the first neural network and the second neural network are different networks obtained by using the same training method, the first image processing result and the second image processing result are in the same type. For example, the first image processing result and the second image processing result may be at least one of an image classification result, an image segmentation result, and a target detection result. This depends on specific manners and training sets used by a person skilled in the art for training neural networks”. The image classification result can be obtained for the reference image (which acts as the first image), “Similarly, the first target the reference image feature and the second target the reference image feature of the reference image may be obtained through steps similar to steps S402 to S410”);
generating, based on the third feature and using the to-be-trained model, a second classification result (Fig. 5, [0123]: “The classification result generating unit 540 may be configured to determine, by using the to-be-classified image feature generated by the fusion unit 530, a probability that the target image belongs to a preset category”. To determine the probability that the target image is in a category it need to be classified, additionally a “classification result generating unit” is generating the output, presuming that a classification is output); and
determining, based on the first classification result, a first predicted category of the reference image (Fig. 4, [0087-88]: “In step S408, a first image processing result and a second image processing result of the target image may be determined according to the first target image feature and the second target image feature. In certain embodiments, because the first neural network and the second neural network are different networks obtained by using the same training method, the first image processing result and the second image processing result are in the same type. For example, the first image processing result and the second image processing result may be at least one of an image classification result, an image segmentation result, and a target detection result. This depends on specific manners and training sets used by a person skilled in the art for training neural networks”. The image classification result can be obtained for the reference image (which acts as the first image), “Similarly, the first target the reference image feature and the second target the reference image feature of the reference image may be obtained through steps similar to steps S402 to S410”. These classification results are to determine the probability of the image falling within one of the categories “Taking a classification process for a medical image as an example, preset categories may include a health category and a disease category” [0057], “In some embodiments, probabilities that the target image and the reference image belong to the preset category may be determined according to confidence scores for a plurality of dimensions outputted by the first generally-connected network” [0059]);
determining, based on the (Fig. 5, [0123]: “The classification result generating unit 540 may be configured to determine, by using the to-be-classified image feature generated by the fusion unit 530, a probability that the target image belongs to a preset category”. To determine the probability that the target image is in a category it need to be classified, additionally a “classification result generating unit” is generating the output, presuming that a classification is output. These classification results are to determine the probability of the image falling within one of the categories “Taking a classification process for a medical image as an example, preset categories may include a health category and a disease category” [0057], “In some embodiments, probabilities that the target image and the reference image belong to the preset category may be determined according to confidence scores for a plurality of dimensions outputted by the first generally-connected network” [0059]);
obtaining, based on ([0073]: “To train the first neural network and the first generally-connected network, parameters of the first neural network and the first generally-connected network may be adjusted, to minimize a loss between the probability that the first training image belongs to the preset category and an actual category to which the first training image belongs”. For the loss to be calculated for the first training image, there is a preset category the image how been classified to correspond to), a target loss, wherein the target loss ([0072]: “With reference to the method shown in FIG. 2, convolution processing may be performed, by using the first neural network, on the first training image and the first reference training image respectively to obtain a first training image feature and a second training image feature”. This section describes using training data similar to how the target image and reference target image are used, the target image and reference target image could replace the first training image and the first reference training image because they are analogous in method and training purpose. They use these image to find a loss between the First training image (target image for training) and actual category for the image (“To train the first neural network and the first generally-connected network, parameters of the first neural network and the first generally-connected network may be adjusted, to minimize a loss between the probability that the first training image belongs to the preset category and an actual category to which the first training image belongs”[0073]); and
updating, based on the target loss until a model training condition is met, a model parameter of the to-be-trained model to obtain an image classification model ([0073]: “To train the first neural network and the first generally-connected network, parameters of the first neural network and the first generally-connected network may be adjusted, to minimize a loss between the probability that the first training image belongs to the preset category and an actual category to which the first training image belongs”).
JIE does not expressly disclose using the classification result from the first features and the third features to generate a third classification for the to-be-classified image.
However, SHORT teaches using a combination of classifications to improve the accuracy of a single classification (Fig. 1, [0030]: “In an exemplary embodiment, the method 100 can include a step 130 of combining outputs (e.g., 155, 165) of the classifying 120 into a single output confidence score 140 by using a weighted fusion of the allocated confidence factors, as described in detail herein”. By combining JIE and SHORT in this way, it would result in the first classification and the fused feature classification being combined to generate a classification for the to-be-classified image).
At the time the invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify JIE’s image classification method to include SHORT’s combination of multiple classifications for classifying a to-be-classified image because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify JIE to include SHORT is expressly provided by SHORT, stating that combining outputs can lead to improved confidence over independent machine learning decisions ([0032]: “As described herein, combining outputs associated with all data representation source modes, along with measured or assigned quality, can improve confidence over independent machine learning decisions”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify JIE’s image classification method to include SHORT’s combination of multiple classifications for classifying a to-be-classified image with the motivation of improving classification confidence. The person of ordinary skill in the art would have recognized the benefit of improved classification confidence.
The combination of JIE and SHORT does not expressly disclose using the classification loss from each of the combined classifications individually to generate a total loss, which is used to update the parameters of neural networks.
However, Wu teaches using the classification loss from each of the combined classifications individually to generate a total loss, which is used to update the parameters of neural networks ([0063]: “At block 106, a total loss is obtained according to the portrait classification loss, the background classification loss and the fusion loss correspondingly; parameters of the portrait branch network and the background branch network are adjusted according to the total loss”).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify the combination of JIE and SHORT’s classification loss function to include Wu’s combination of multiple classification losses because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, Wu’s combination of multiple classification losses permits accurate training of networks by taking into account the losses of multiple classification operations. This known benefit in Wu is applicable to the combination of JIE and SHORT’s classification loss function as they both share characteristics and capabilities, namely, they are directed to fusion of features and classifications for images to determine accurate classification of images. Therefore, it would have been recognized that modifying the combination of JIE and SHORT’s classification loss function to include Wu’s combination of multiple classification losses would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Wu’s combination of multiple classification losses in fusion of features and classifications for images to determine accurate classification of images and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 9, the combination of JIE, SHORT, and Wu teaches the method of claim 8, in addition, JIE further teaches wherein obtaining the target loss comprises:
obtaining, based on the second reality category and the second predicted category a second sub-loss, wherein the second sub-loss indicates the second difference ([0072]: “With reference to the method shown in FIG. 2, convolution processing may be performed, by using the first neural network, on the first training image and the first reference training image respectively to obtain a first training image feature and a second training image feature”. This section describes using training data similar to how the target image and reference target image are used, the target image and reference target image could replace the first training image and the first reference training image because they are analogous in method and training purpose. They use these image to find a loss between the First training image (target image for training) and actual category for the image (“To train the first neural network and the first generally-connected network, parameters of the first neural network and the first generally-connected network may be adjusted, to minimize a loss between the probability that the first training image belongs to the preset category and an actual category to which the first training image belongs”[0073]. The loss acts as the sub-loss); and
Wu further teaches a first classification loss that’s acts as a sub-loss and performing an addition processing (fusion) to generate the target loss ([0063]: “At block 106, a total loss is obtained according to the portrait classification loss, the background classification loss and the fusion loss correspondingly; parameters of the portrait branch network and the background branch network are adjusted according to the total loss”).
The rationale for this combination is similar to the rationale of the combination of claim 8 for Wu due to similar methods of combination (their method of calculating loss requiring usage of the first classification and second in an addition processing manner) and benefits.
Regarding claim 10, the combination of JIE, SHORT, and Wu teaches the method of claim 9, in addition, SHORT further teaches further comprising performing, using the to-be-trained model, second addition processing on the first classification result and the second classification result to obtain the third classification result ([0030]: “In an exemplary embodiment, the method 100 can include a step 130 of combining outputs (e.g., 155, 165) of the classifying 120 into a single output confidence score 140 by using a weighted fusion of the allocated confidence factors, as described in detail herein”. The fusion is the addition processing and JIE’s model would be used in the current combination)
The rationale for this combination is similar to the rationale of the combination of claim 8 for SHORT due to similar methods of combination (the fusion being part of the original classification combination) and benefits.
Regarding claim 11, the combination of JIE, SHORT, and Wu teaches the method of claim 8, in addition, JIE further teaches further comprising:
calculating, based on the first feature and using the to-be-trained model, a first probability that the reference image belongs to the first predicted category and the second predicted category to obtain the first classification result (Fig. 4, [0087-88]: “In step S408, a first image processing result and a second image processing result of the target image may be determined according to the first target image feature and the second target image feature. In certain embodiments, because the first neural network and the second neural network are different networks obtained by using the same training method, the first image processing result and the second image processing result are in the same type. For example, the first image processing result and the second image processing result may be at least one of an image classification result, an image segmentation result, and a target detection result. This depends on specific manners and training sets used by a person skilled in the art for training neural networks”. The image classification result can be obtained for the reference image (which acts as the first image), “Similarly, the first target the reference image feature and the second target the reference image feature of the reference image may be obtained through steps similar to steps S402 to S410”. These classification results are to determine the probability of the image falling within one of the categories “Taking a classification process for a medical image as an example, preset categories may include a health category and a disease category” [0057], “In some embodiments, probabilities that the target image and the reference image belong to the preset category may be determined according to confidence scores for a plurality of dimensions outputted by the first generally-connected network” [0059]); and
calculating, based on the third feature and using the to-be-trained model, a second probability that the to-be-classified image belongs to the first predicted category and the second predicted category to obtain the second classification result (Fig. 5, [0123]: “The classification result generating unit 540 may be configured to determine, by using the to-be-classified image feature generated by the fusion unit 530, a probability that the target image belongs to a preset category”. To determine the probability that the target image is in a category it need to be classified, additionally a “classification result generating unit” is generating the output, presuming that a classification is output. These classification results are to determine the probability of the image falling within one of the categories “Taking a classification process for a medical image as an example, preset categories may include a health category and a disease category” [0057], “In some embodiments, probabilities that the target image and the reference image belong to the preset category may be determined according to confidence scores for a plurality of dimensions outputted by the first generally-connected network” [0059]).
Regarding claim 12, the combination of JIE, SHORT, and Wu teaches the method of claim 8, in addition, JIE further teaches further comprising performing, using the to-be-trained model, feature fusion processing on the first feature and the second feature to obtain the third feature (Fig. 2, [0047]: “In step S206, the first image feature and the second image feature may be fused to determine a to-be-classified image feature”).
Regarding claim 13, the combination of JIE, SHORT, and Wu teaches the method of claim 12, in addition, JIE further teaches wherein the feature fusion processing comprises at least one of addition processing, multiplication processing, subtraction processing, concatenation processing, or concatenation convolution processing (Fig. 2, [0048]: “In some embodiments, the first image feature and the second image feature may be stitched, to determine the to-be-classified image feature”. The feature stitching is the addition processing, the “or” limitation requires only one of the listed items be met).
Regarding claim 14, the combination of JIE, SHORT, and Wu teaches the method of claim 8, in addition, JIE further teaches further comprising:
performing first feature extraction processing on the reference image to obtain the first feature (Fig. 2, [0043]: “In step S204, a first image feature of the target image and a second image feature of the reference image may be determined. For example, the first image feature of the target image and the second image feature of the reference image may be determined in the same manner or in a different manner”. This determination is extraction of the features “For example, multi-viewing-angle scans of the breast on both left and right sides may be simultaneously received based on a network structure used for comparing breasts on both left and right sides, feature extraction is individually performed on each scan of a breast on each side, then feature fusion is performed, and a feature obtained after fusion is simultaneously used for predicting whether the breasts on both left and right sides are positive in breast cancer.” [0069]); and
performing second feature extraction processing on the to-be-classified image to obtain the second feature (Fig. 2, [0043]: “In step S204, a first image feature of the target image and a second image feature of the reference image may be determined. For example, the first image feature of the target image and the second image feature of the reference image may be determined in the same manner or in a different manner”. This determination is extraction of the features “For example, multi-viewing-angle scans of the breast on both left and right sides may be simultaneously received based on a network structure used for comparing breasts on both left and right sides, feature extraction is individually performed on each scan of a breast on each side, then feature fusion is performed, and a feature obtained after fusion is simultaneously used for predicting whether the breasts on both left and right sides are positive in breast cancer.” [0069]).
Allowable Subject Matter
Claims 3 and 17 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 101 set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Alzamzmi et al. (US 20190320974 A1) teaches feature fusion and decision fusion for data classification
Dal Mutto et al. (US 20200372625 A1) teaches combining classifications
Hu et al. (US 20210365741 A1) teaches 2 networks, one for global features and one for local features, fuse features and classify fused features
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698