DETAILED ACTION
This communication is in response to the application filed 4/18/23 in which claims 1-19 were presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to one or more abstract ideas without significantly more.
Claim 1
An image detection method performed by a computer device, the method comprising:
iteratively training a plurality of neural network models until the plurality of neural network models converge, to obtain a plurality of trained neural network models, each iteration of training comprising:
for each sample image in a plurality of sample images corresponding to the iteration:
separately inputting the sample image into the plurality of neural network models, to obtain a fuzzy probability value set of the sample image, the fuzzy probability value set comprising fuzzy probability values outputted from each of the plurality of neural network models, and
calculating, based on the fuzzy probability value set and preset label information of the sample image, a loss parameter of the sample image,
selecting, based on a distribution of loss parameters of the plurality of sample images, target sample images from the plurality of sample images, and
updating, based on the target sample images, the plurality of neural network models; and
performing fuzzy detection on an image to be detected by at least one trained neural network model of the plurality of trained neural network models, to obtain a fuzzy detection result.
Step 1: YES. Claim 1 is directed to a method which is a statutory category.
Step 2A Prong 1: YES. The limitation “obtain a fuzzy probability value set…,” under a broadest reasonable interpretation, is a mathematical concept. See Spec. ¶ 43. Similarly, the limitation “calculating…” is a mathematical concept. See Spec. ¶¶ 43, 104, 109, 113. The limitation “selecting, based on a distribution of loss parameters of the plurality, target sample images…” is a process that may be performed by evaluation, reasoning, and judgment and, therefore, falls under the Mental Concepts category of abstract ideas.
Step 2A Prong 2/Step 2B: NO. The additional elements fail to integrate the judicial exception into a practical application or provide an inventive concept.
The limitation “iteratively training a plurality of neural network models until the plurality of neural network models converge, to obtain a plurality of trained neural network models, each iteration of training comprising” describes neural network training at a high level of generalization in terms of generic computer components and, therefore, is mere instruction to apply the exception.
The limitation “for each sample image in a plurality of sample images corresponding to the iteration: separately inputting the sample image into the plurality of neural network models” recites a data inputting step and, therefore, constitutes insignificant extra-solution activity.
The limitation “updating, based on the target sample images, the plurality of neural network models” is mere instruction to apply the exception using generic computer components.
The limitation “performing fuzzy detection on an image to be detected by at least one trained neural network model of the plurality of trained neural network models, to obtain a fuzzy detection result” is mere instruction to apply the exception using generic computer components.
Claim 10 is an apparatus claim corresponding to claim 1 and, therefore, is similarly analyzed. Claim 10 additionally recites “at least one processor; and at least one non-volatile memory having stored thereon a plurality of neural network models and a computer program, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform operations of.” Recitation of generic computer components at a high level of generalization constitutes mere instruction to “apply” the judicial exception.
Claim 19 is media claim corresponding to claim 1 and, therefore, is similarly analyzed. Claim 19 additionally recites a “non-transitory computer-readable storage medium storing a plurality of instructions thereon, the instructions being executable by at least one processor to perform operations of an image detection method comprising.” Recitation of generic computer components at a high level of generalization constitutes mere instruction to “apply” the judicial exception.
Accordingly, claims 1, 10, and 19 are ineligible.
Claim 2
The method according to claim 1, wherein the calculating of the loss parameter of the sample image comprises:
calculating first cross entropies between the preset label information and each fuzzy probability value in the fuzzy probability value set of the sample image;
summing the calculated first cross entropies to obtain a first sub-loss parameter of the sample image; and
determining the loss parameter corresponding to the sample image based on the first sub-loss parameter of the sample image.
Step 2A Prong 1: YES. Each of the limitations “calculating…,” “summing…,” “determining…” recite a mathematical calculation and, therefore, fall under the Mathematical Concepts grouping of abstract ideas.
Accordingly, claim 2 is ineligible.
Claim 11 is an apparatus claim corresponding to claim 2 and, therefore, is similarly ineligible.
Claim 3
The method according to claim 2, wherein the calculating of the loss parameter of the sample image further comprises:
calculating relative entropies between each pair of fuzzy probability values in the fuzzy probability value set of the sample image; and
summing the relative entropies to obtain a second sub-loss parameter corresponding to the sample image,
wherein the loss parameter of the sample image is determined as a weighted summation of at least the first sub-loss parameter and the second sub-loss parameter.
Step 2A Prong 1: YES. Each of the limitations recites a mathematical calculation and, therefore, falls under the Mathematical Concepts grouping of abstract ideas.
Accordingly, claim 3 is ineligible.
Claim 12 is an apparatus claim corresponding to claim 3 and, therefore, is similarly ineligible.
Claim 4
The method according to claim 2, wherein the calculating of the loss parameter of the sample image further comprises:
acquiring probability distribution information of preset label information of the plurality of sample images;
generating a corresponding feature vector based on the probability distribution information;
calculating second cross entropies between the feature vector and the fuzzy probability value set corresponding to the sample image; and
summing the calculated second cross entropies to obtain a third sub-loss parameter of the sample image,
wherein the loss parameter of the sample image is determined as a weighted summation of at least the first sub-loss parameter and the third sub-loss parameter.
Step 2A Prong 1: YES. The limitation “generating a corresponding feature vector based on the probability distribution information” may be performed mentally or with the aid of pen and paper and, therefore, falls under the Mental Processes grouping of abstract ideas. See Spec. ¶ 63.
The limitations “calculating…” and “summing…” recite mathematical calculations and, therefore, fall under the Mathematical Concepts grouping of abstract ideas. Similarly, the wherein clause recites determining the loss parameter as a weighted summation and, therefore, also falls under the Mathematical Concepts grouping of abstract ideas.
Step 2A Prong 2/Step 2B: NO. The additional elements of the claim fail to integrate the judicial exception into a practical application or provide an inventive concept.
The limitation “acquiring probability distribution information of preset label information of the plurality of sample images,” is a data inputting step and, therefore, constitutes insignificant extra-solution activity. Receiving information is well understood routine and conventional. See MPEP 2106.05(d) (receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362).
Accordingly, claim 4 is ineligible.
Claim 13 is an apparatus claim corresponding to claim 4 and, therefore, is similarly ineligible.
Claim 5
The method according to claim 3, wherein the calculating of the loss parameter of the sample image further comprises:
acquiring probability distribution information of preset label information of the plurality of sample images;
generating a feature vector based on the probability distribution information;
calculating second cross entropies between the feature vector and the fuzzy probability value set corresponding to the sample image; and
summing the calculated second cross entropies to obtain a third sub-loss parameter of the sample image,
wherein the loss parameter of the sample image is determined as a weighted summation of at least the first sub-loss parameter, the second sub-loss parameter, and the third sub-loss parameter.
Step 2A Prong 1: YES. The limitation “generating a corresponding feature vector based on the probability distribution information” may be performed mentally or with the aid of pen and paper and, therefore, falls under the Mental Processes grouping of abstract ideas. See Spec. ¶ 63.
The limitations “calculating…” and “summing…” recite mathematical calculations and, therefore, fall under the Mathematical Concepts grouping of abstract ideas. Similarly, the wherein clause recites determining the loss parameter as a weighted summation and, therefore, also falls under the Mathematical Concepts grouping of abstract ideas.
Step 2A Prong 2/Step 2B: NO. The additional elements of the claim fail to integrate the judicial exception into a practical application or provide an inventive concept.
The limitation “acquiring probability distribution information of preset label information of the plurality of sample images,” is a data inputting step and, therefore, constitutes insignificant extra-solution activity. Receiving information is well understood routine and conventional. See MPEP 2106.05(d) (receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362).
Accordingly, claim 5 is ineligible.
Claim 14 is an apparatus claim corresponding to claim 5 and, therefore, is similarly ineligible.
Claim 6
The method according to claim 1, wherein the selecting of the target sample images from the plurality of sample images comprises:
acquiring a current number of iterations of training on the plurality of neural network models;
calculating a target number based on the current number of iterations; and
selecting the target number of sample images in an order of loss parameters from small to large to obtain the target sample images.
Step 2A Prong 1: YES. The limitation “calculating a target number based on the current number of iterations” recites a mathematical calculation and, therefore, falls under the Mathematical Concepts grouping of abstract ideas. The limitation “selecting the target number of sample images in an order of loss parameters from small to large to obtain the target sample images” is a process that may be performed manually by human reasoning and, therefore, falls under the Mental Processes grouping of abstract ideas.
Step 2A Prong 2/Step 2B: NO. The additional elements of the claim fail to integrate the judicial exception into a practical application or provide an inventive concept.
The limitation “acquiring a current number of iterations of training on the plurality of neural network models” is a data inputting step and, therefore, constitutes insignificant extra-solution activity. Receiving information is well understood routine and conventional. See MPEP 2106.05(d) (receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362).
Accordingly, claim 6 is ineligible.
Claim 15 is an apparatus claim corresponding to claim 6 and, therefore, is similarly ineligible.
Claim 7
The method according to claim 6, wherein the calculating of the target number comprises:
[1] acquiring a preset screening rate which is used to control the screening of the plurality of sample images;
[2] calculating a proportion of the target sample images in the plurality of sample images based on the screening rate and the current number of iterations; and
[3] calculating the target number of the target sample images based on the proportion and a number of the plurality of sample images.
Step 2A Prong 1: YES. Limitations [2] and [3] each recite performing a calculation and, therefore, fall under the Mathematical Concepts grouping of abstract ideas.
Step 2A Prong 2/Step 2B: NO. The additional elements of the claim fail to integrate the judicial exception into a practical application or provide an inventive concept.
Limitation [1] recites a data inputting step and, therefore, constitutes insignificant extra-solution activity. The type of data does not cause the data inputting to integrate the exception into a practical application. Receiving information is well understood routine and conventional. See MPEP 2106.05(d) (receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362).
Accordingly, claim 7 is ineligible.
Claim 16 is an apparatus claim corresponding to claim 7 and, therefore, is similarly ineligible.
Claim 8
The method according to claim 1, wherein the performance of the fuzzy detection comprises:
[1] separately performing fuzzy detection on the image to be detected by each of the plurality of trained neural network models to thereby obtain, for each of the trained plurality of neural network models, a fuzzy probability value, and
[2] obtaining an average value of the obtained fuzzy probability values as a fuzzy probability corresponding to the image to be detected.
Step 2A Prong 1: YES. Limitation [2] recites obtaining an average value of the fuzzy probability values. Under a broadest reasonable interpretation, obtaining an average of values is a mathematical calculation and, therefore, the limitation falls under the Mathematical Concepts grouping of abstract ideas.
Step 2A Prong 2/Step 2B: NO. Limitation [1] recites performing fuzzy detection on an image by each of the trained neural networks to obtain a fuzzy probability value. The limitation recites generic computer components at a high level of abstraction and, therefore, constitutes mere instructions to “apply” the judicial exception.
Accordingly, claim 8 is ineligible.
Claim 17 is an apparatus claim corresponding to claim 8 and, therefore, is similarly ineligible.
Claim 9
The method according to claim 1, wherein the performance of the fuzzy detection comprises:
[1] acquiring prediction accuracy rates of each of the plurality of trained neural network models,
[2] ranking the prediction accuracy rates, and
[3] performing the fuzzy detection by a neural network model with a highest prediction accuracy rate according to the ranking.
Step 2A Prong 1: YES. Limitation [2] describes ranking the prediction accuracy rates, a step that may be performed mentally and, therefore, falls under the Mental Processes grouping of abstract ideas.
Step 2A Prong 2/Step 2B: NO. The additional elements of the claim fail to integrate the judicial exception into a practical application.
Limitation [1] recites a data gathering step. Data gathering constitutes insignificant extra-solution activity. The type/source of data does not cause data gathering to practically integrate the exception. Receiving information is well understood routine and conventional. See MPEP 2106.05(d) (receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362).
Limitation [3] recites performing detection by a neural network. The recitation of generic computer components at a high level of generality is a mere instruction to “apply” the exception.
Accordingly, claim 9 is ineligible.
Claim 18 is an apparatus claim corresponding to claim 9 and, therefore, is similarly ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1, 10, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Huszar (US 2018/0240031 A1; published Aug. 23, 2018) in view of Manna, Ankur, et al. "A fuzzy rank-based ensemble of CNN models for classification of cervical cytology." Scientific Reports 11.1 (2021): 14538 (“Manna”) and Foody, Giles M. "Cross-entropy for the evaluation of the accuracy of a fuzzy land cover classification with fuzzy ground data." ISPRS Journal of Photogrammetry and remote sensing 50.5 (1995): 2-12 (“Foody”).
Regarding claim 1, Huszar discloses [a]n image detection method performed by a computer device, the method comprising:
iteratively training a plurality of neural network models until the plurality of neural network models converge, to obtain a plurality of trained neural network models, each iteration of training comprising: (Huszar Abstract (“Systems and methods provide a deep neural network trained via active learning. An example method includes generating, from a set of labeled objects, a plurality of differing training sets, assigning each of the plurality of training sets to a respective deep neural network in a committee of networks, and initializing each of the deep neural networks in the committee by training the deep neural network using the respective assigned training set. The method further includes iteratively training the deep neural networks in the committee until convergence and using one of the deep neural networks to make predictions for unlabeled objects.”))
for each sample image in a plurality of sample images corresponding to the iteration:
separately inputting the sample image into the plurality of neural network models, to obtain a fuzzy probability value set of the sample image, the fuzzy probability value set comprising fuzzy probability values outputted from each of the plurality of neural network models, and (Huszar ¶ 10 (“In one aspect, a method includes initializing committee members in a committee, each committee member being a deep neural network trained on a different set of labeled objects, i.e., labeled training data. The method also includes providing an unlabeled object as input to each of the committee members and obtaining a prediction from each committee member. The prediction can be a classification, a score, etc.”)).
Huszar does not expressly disclose obtain a fuzzy probability value set of the sample image, the fuzzy probability value set comprising fuzzy probability values outputted from each of the plurality of neural network models (but see Manna Introduction, pg. 2, paragraphs 1-2 (“Ensemble learning is one such alternative where decision scores from multiple classifiers are fused to predict the final class label of an input sample. An ensemble model is aimed to capture the salient features of all its constituent models thus performing better than the individual base classifiers. Such models are robust since ensembling diminishes the dispersion or spread of the predictions made by the base models. The variance in the prediction errors of the base classifiers gets reduced in the ensemble model by the addition of some bias to the competing base learners.
In the present work, we formulate a fusion strategy that uses the decision scores obtained by three base Convolutional Neural Network (CNN) classifiers, namely, Inception v3 by Szegedy et al.4, Xception by5 and DenseNet-169 by Huang et al.6 (pre-trained on the ImageNet dataset7) to form the ensemble. We use a fuzzy ranking-based approach, where the probability scores are subjected to two non-linear functions, an exponentially decaying function, and the tanh function, to assign the ranks to the class probabilities predicted by a base learner. The ranks assigned by the two non-linear functions are multiplied. The same process is repeated for each base learner, and the rank products from each classifier are added to get the final ranks. We use two different functions of different concavities so that they can generate complementary results. Fusion entails consolidating the multiple ranks associated with an identity and determining a new rank that would aid in establishing the final decision. The main motive of using two ranks is to consider the closeness to and deviation from the expected result corresponding to the primary classification result. Lesser deviation corresponds to a lower value of the product and a better result. So, the class having the lowest value of this sum of products of ranks is deemed as the predicted class of the ensemble model. Here, the two non-linear functions have opposite concavity in the range [0, 1] and hence a higher confidence score results in a larger value of rank in one function and a smaller value in the other, and our aim to minimize this product. If the confidence score of a prediction is high, then this sum of products yields a lower value than if the confidence score is low which are explained in detail later.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huszar to incorporate the teachings of Manna to use a fusion strategy to combine fuzzy classification probabilities of base classifiers in an ensemble system, at least because doing so would take into account the confidence of predictions. See Manna pg. 2 (“Ensemble learning is a strategy that considers decisions obtained from more than one model for making the final decision. Some simple fusion schemes have been explored in literature like Sarwar et al.16 who used an average probability-based ensemble and Xue et al.17 who used a majority voting based ensemble technique. However, such simplistic ensemble models do not take into account the confidence of predictions and use pre-determined or fixed weights associated with the base learners. Keeping this in mind, in this research, we propose a novel ensemble technique which fuses the decision scores from three base CNN based classifiers, namely Inception v34, Xception5 and DenseNet-1696 while taking into account the confidence in predictions of the base learners.”)).
Huszar further discloses:
calculating, based on the fuzzy probability value set and preset label information of the sample image, a loss parameter of the sample image, (Huszar ¶ 10 (“The method includes determining whether the various predictions satisfy a diversity metric. Satisfying the diversity metric means that the predictions represent a data object for which the parameters under the posterior disagree about the outcome the most. In some implementations the diversity metric is a Bayesian Active Learning by Disagreement (BALD) score.”), ¶ 28 (“The modules in the active learning system 100 also include a label evaluator 140. After the committee members in the classifier committee 150 have been initialized, the label evaluator 140 is configured to receive the output of the various committee members in the classifier committee 150 for a specific unlabeled object, e.g., from unlabeled objects 120. For example, after initialization, the system 100 may provide a large number of unlabeled objects 120 to the committee members in the classifier committee 150. Each committee member provides an output, e.g., a predicted classification, for each unlabeled object. The label evaluator 140 may evaluate the diversity of the predictions to determine whether the predictions for the unlabeled object satisfy a diversity metric. The diversity metric measures how much variance exists in the predictions. In some implementations, any unlabeled objects that meet some threshold satisfy the diversity metric. In some implementations, some quantity of unlabeled objects having the highest diversity satisfy the diversity metric. In some implementations, the diversity metric may represent the predictions for which the parameters under the posterior disagree about the outcome the most. In some implementations, the label evaluator 140 may use a Bayesian Active Learning by Disagreement (BALD) criteria as the diversity metric. The BALD criteria is described by Houlsby et al. in “Bayesian Active Learning for Classification and Preference Learning,” (2011), available at https://pdfs.semanticscholar.org/7486/e148260329785fb347ac6725bd4123d8dad6.pdf. The BALD criterion aims at maximizing the mutual information between the newly acquired labelled example and the parameters of the neural network. This mutual information can be equivalently computed in terms of the average Kullback-Leibler divergence between the probabilistic predictions made by each member of a committee and the average prediction.”))
selecting, based on a distribution of loss parameters of the plurality of sample images, target sample images from the plurality of sample images, and (Huszar ¶ 10 (“An unlabeled data object that satisfies the diversity metric is an informative object. The method may include identifying several informative objects.”), ¶ 29 (“The label evaluator 140 may identify any unlabeled objects that satisfy the diversity metric as informative objects 115. Identification can be accomplished in any manner, such as setting a flag or attribute for the unlabeled object, saving the unlabeled object or an identifier for the unlabeled object in a data store, etc.”))
updating, based on the target sample images, the plurality of neural network models; and (Huszar ¶ 10 (“The method includes re-training the committee members with the newly labeled data objects. The method may include repeating the identification of informative objects, labeling of informative objects, and re-training the committee members until the committee members reach convergence. In other words, eventually the committee members may agree enough that very few, if any, unlabeled data objects result in predictions that satisfy the diversity metric. Any one of the trained committee members may then be used in labeling additional data objects.”))
performing fuzzy detection on an image to be detected by at least one trained neural network model of the plurality of trained neural network models, to obtain a fuzzy detection result (Huszar ¶ 9 (“Eventually the committee members reach a consensus and the trained model can be provided for use in classifying unlabeled objects.”), ¶ 10 (“In other words, eventually the committee members may agree enough that very few, if any, unlabeled data objects result in predictions that satisfy the diversity metric. Any one of the trained committee members may then be used in labeling additional data objects.”)).
Huszar does not expressly disclose performing fuzzy detection on an image to obtain a fuzzy detection result (but see Foody Abstract (“Fuzzy classifications have been used to represent land cover when pixels may have multiple and partial class membership. A fuzzy classification can be derived by softening the output of a conventional “hard” classification. Thus, for example, the probabilities of class membership may be derived from a conventional probability-based classification and mapped to represent the land cover of a site. The accuracy of the representation provided by a fuzzy classification is, however, difficult to evaluate. Conventional measures of classification accuracy cannot be used since they are appropriate only for “hard” classifications. The accuracy of a classification may, however, be indicated by the way in which the probability of class membership is partitioned between the classes and this may be expressed by entropy measures. Here cross-entropy is proposed as a means of evaluating the accuracy of a fuzzy classification, by illustrating how closely a fuzzy classification represents land cover when multiple and partial class membership is a feature of both the remotely sensed and ground data sets. Cross-entropy is calculated from the probability distributions of class membership derived from the remotely sensed and ground data sets. The use of cross-entropy as an indicator of classification accuracy was investigated with reference to land cover classifications of two contrasting test sites. The results show that cross-entropy may be used to indicate the accuracy of the representation of land cover when the classification of the remotely sensed data and ground data are both fuzzy.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Huszar to incorporate the teachings of Foody to perform fuzzy classification of an image, at least because doing so “would convey considerably more information on the class membership properties of each pixel.” Foody pg. 3.
Claim 10 is an apparatus claim corresponding to claim 1 and, therefore, is similarly rejected. Huszar further discloses [a]n image detection apparatus, comprising:
at least one processor; and (Huszar ¶ 42 (“Processing device 402 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device 402 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processing device 402 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 402 is configured to execute instructions 426 (e.g., instructions for an application ranking system) for performing the operations and steps discussed herein.”))
at least one non-volatile memory having stored thereon a plurality of neural network models and a computer program, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform operations of: (Huszar ¶ 44 (“The data storage device 418 may include a computer-readable storage medium 428 on which is stored one or more sets of instructions 426 (e.g., instructions for the application ranking system) embodying any one or more of the methodologies or functions described herein. The instructions 426 may also reside, completely or at least partially, within the main memory 404 and/or within the processing device 402 during execution thereof by the computing device 400, the main memory 404 and the processing device 402 also constituting computer-readable media. The instructions may further be transmitted or received over a network 420 via the network interface device 408.”)).
Claim 19 is a computer-readable memory claim corresponding to claim 1 and, therefore, is similarly rejected. Huszar further discloses [a] non-transitory computer-readable storage medium storing a plurality of instructions thereon, the instructions being executable by at least one processor to perform operations of an image detection method comprising: (Huszar ¶ 45 (“While the computer-readable storage medium 428 is shown in an example implementation to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media and magnetic media. The term “computer-readable storage medium” does not include transitory signals.”)).
Claims 2, 3, 11, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Huszar, Manna, and Foody as applied to claims 1 and 10 above, and further in view of Hall (WO 2021056043 A1; published Apr. 1, 2021).
Regarding claim 2, Huszar, in view of Manna and Foody, discloses the invention of claim 1 as discussed above. Huszar does not expressly disclose wherein the calculating of the loss parameter of the sample image comprises: calculating first cross entropies between the preset label information and each fuzzy probability value in the fuzzy probability value set of the sample image; summing the calculated first cross entropies to obtain a first sub-loss parameter of the sample image; and determining the loss parameter corresponding to the sample image based on the first sub-loss parameter of the sample image (but see Hall ¶ 17 (“First, the set of Teacher model(s) are trained on the dataset of interest. The Teacher models can be of any neural network or model architecture, and can even be completely different architectures from each other or the Student model. They can either share the same dataset exactly, or have disjoint or overlapping subsets of the original dataset. Once the Teacher models are trained, the Student is trained using a distillation loss function to mimic the outputs of the Teacher models. The distillation process begins by first applying the Teacher model to a dataset that is made available to both the Teacher and Student models, known as the ‘transfer dataset’. The transfer dataset can be hold-out, blind dataset drawn from the original dataset, or could be the original dataset itself. Furthermore, the transfer dataset does not have to be completely labelled, i.e. with some portion of the data not associated with a known outcome. This removal of the labelling restriction allows for the dataset to be artificially increased in size. Then the Student model is applied to the transfer dataset. The output probabilities (soft labels) of the Teacher model are compared with the output probabilities of the Student model via a divergence measure function, such as Kullback-Leibler (KL)-Divergence, or ‘relative entropy’ function, computed from the distributions. A divergence measure is an accepted mathematical method for measuring the “distance” between two probability distributions. The divergence measure is then summed together with a standard cross-entropy classification loss function, so that the loss function is effectively minimizing both the classification loss, improving model performance, and also the divergence of the Student model from the Teacher model, simultaneously. Typically, the soft label matching loss (the divergence component of the new loss) and the hard label classification loss (the original component of the loss) are weighted with respect to each other (introducing an extra tuneable parameter to the training process) to control the contribution of each of the two terms in the new loss function.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huszar to incorporate the teachings of Hall to calculate and sum the relative and cross entropy losses, at least because doing so would effectively minimize both the classification loss, improving model performance, and also the divergence of the untrained model, simultaneously.
Claim 11 is an apparatus claim corresponding to claim 2 and, therefore, is similarly rejected.
Regarding claim 3, Huszar, in view of Manna, Foody, and Hall, discloses the invention of claim 2 as discussed above. Huszar does not expressly disclose:
wherein the calculating of the loss parameter of the sample image further comprises:
calculating relative entropies between each pair of fuzzy probability values in the fuzzy probability value set of the sample image; and summing the relative entropies to obtain a second sub-loss parameter corresponding to the sample image, wherein the loss parameter of the sample image is determined as a weighted summation of at least the first sub-loss parameter and the second sub-loss parameter (but see Hall ¶ 17 (“First, the set of Teacher model(s) are trained on the dataset of interest. The Teacher models can be of any neural network or model architecture, and can even be completely different architectures from each other or the Student model. They can either share the same dataset exactly, or have disjoint or overlapping subsets of the original dataset. Once the Teacher models are trained, the Student is trained using a distillation loss function to mimic the outputs of the Teacher models. The distillation process begins by first applying the Teacher model to a dataset that is made available to both the Teacher and Student models, known as the ‘transfer dataset’. The transfer dataset can be hold-out, blind dataset drawn from the original dataset, or could be the original dataset itself. Furthermore, the transfer dataset does not have to be completely labelled, i.e. with some portion of the data not associated with a known outcome. This removal of the labelling restriction allows for the dataset to be artificially increased in size. Then the Student model is applied to the transfer dataset. The output probabilities (soft labels) of the Teacher model are compared with the output probabilities of the Student model via a divergence measure function, such as Kullback-Leibler (KL)-Divergence, or ‘relative entropy’ function, computed from the distributions. A divergence measure is an accepted mathematical method for measuring the “distance” between two probability distributions. The divergence measure is then summed together with a standard cross-entropy classification loss function, so that the loss function is effectively minimizing both the classification loss, improving model performance, and also the divergence of the Student model from the Teacher model, simultaneously. Typically, the soft label matching loss (the divergence component of the new loss) and the hard label classification loss (the original component of the loss) are weighted with respect to each other (introducing an extra tuneable parameter to the training process) to control the contribution of each of the two terms in the new loss function.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huszar to incorporate the teachings of Hall to calculate and sum the relative and cross entropy losses, at least because doing so would effectively minimize both the classification loss, improving model performance, and also the divergence of the untrained model, simultaneously.
Claim 12 is an apparatus claim corresponding to claim 3 and, therefore, is similarly rejected.
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Huszar, Manna, and Foody as applied to claims 1 and 10 above, and further in view of Devarakonda, Aditya, Maxim Naumov, and Michael Garland. "Adabatch: Adaptive batch sizes for training deep neural networks." arXiv preprint arXiv:1712.02029 (2017) (“Devarakonda”) and Jiang, Angela H., et al. "Accelerating deep learning by focusing on the biggest losers, 2019." URL https://arxiv. org/abs/1910.00762 (2019) (“Jiang”).
Regarding claim 6, Huszar, in view of Manna and Foody, discloses the invention of claim 1 as discussed above. Huszar does not expressly disclose:
wherein the selecting of the target sample images from the plurality of sample images comprises: acquiring a current number of iterations of training on the plurality of neural network models; calculating a target number based on the current number of iterations; and (but see Devarakonda page 1 (“Our approach to resolving this trade-off between small and large batch sizes is to adaptively increase the batch size during training. We begin with a small batch size r, chosen to encourage rapid convergence in early epochs, and then progressively increase the batch size between selected epochs as training proceeds. For the experiments reported in this paper, we double the batch size at specific intervals and simultaneously adapt the learning rate α so that the ratio α/r remains constant. Our adaptive batch size technique has several advantages. It delivers the accuracy of training with small batch sizes, while improving performance by increasing the amount of work available per processor in later epochs. Furthermore, the large batches used in later epochs expose sufficient parallelism to create the opportunity for distributing work across many processors, where those are available. Our approach can also be combined with other existing techniques for constructing learning rate decay schedules to increase batch sizes even further.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huszar to incorporate the teachings of Devarakonda to adaptively increase the batch size during training, at least because doing so would increase the accuracy of training with small batch sizes, while improving performance by increasing the amount of work available per processor in later training epochs.
Huszar and Devarakonda do not expressly disclose selecting the target number of sample images in an order of loss parameters from small to large to obtain the target sample images (but see Jiang page 1 (“Motivated by the hinge loss (Rosasco et al., 2004), which provides zero loss whenever an example is correctly predicted by sufficient margin, this paper introduces Selective Backprop (SB), a simple and effective sampling technique for prioritizing high-loss training examples throughout training. We suspect, and confirm experimentally, that examples with low loss correspond to gradients with small norm and thus contribute little to the gradient update. Thus, Selective Backprop uses the loss calculated during the forward pass as a computationally cheap proxy for the gradient norm, enabling us to decide whether to apply an update without having to actually compute the gradient.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huszar to incorporate the teachings of Jiang to prioritize high-loss samples during each training iteration, at least because doing so would reduce computationally expensive back-propagation with samples that contribute little to the gradient update.
Claim 15 is an apparatus claim corresponding to claim 6 and, therefore, is similarly rejected.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Huszar, Manna, and Foody as applied to claims 1 and 10 above, and further in view of Bonakdar (US 2022/0138932 A1; published May 5, 2022).
Regarding claim 8, Huszar, in view of Manna and Foody, discloses the invention of claim 1 as discussed above. Huszar does not expressly disclose:
wherein the performance of the fuzzy detection comprises: separately performing fuzzy detection on the image to be detected by each of the plurality of trained neural network models to thereby obtain, for each of the trained plurality of neural network models, a fuzzy probability value, and obtaining an average value of the obtained fuzzy probability values as a fuzzy probability corresponding to the image to be detected (but see Bonakdar ¶ 128 (“FIG. 5 is a flowchart outlining an example operation of liver detection and predetermined amount of anatomical structure determination logic of an AI pipeline in accordance with one illustrative embodiment. As shown in FIG. 5, the liver detection operation of the AI pipeline starts by receiving the input volume (step 510) and divides the input volume into a plurality of overlapping sections of a predetermined number of slices for each section (step 520). The slices for each section are input to a trained ML/DL computer model that estimates the axial scores for the first and last slices in each section (step 530). The axial scores for the first and last slices are used to extrapolate to scores for the most inferior slice in the volume (MISV) and the most superior slice in the volume (MSSV) for the input volume (step 540). This results in a plurality of estimates for the axial scores for the MISV and MS SV which are then combined through a function of the individual estimates to thereby generate an estimate of the axial scores for the MISV and MSSV of the input volume, e.g., a weighted mean of the like (step 550). Based on the estimate of the axial score for the MISV and MSSV, the axial scores are compared to criteria for determining whether or not a predetermined amount of an anatomical structure of interest, e.g., the liver, is present in the input volume (step 560).”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huszar to incorporate the teachings of Bonakdar to combine the fuzzy probability values generated by the ensemble model with a weighted mean, at least because doing so would “provide a more efficient and correct analysis of medical image data to detect lesions in an imaged anatomical structure, e.g., the liver or other organs.” Bonakdar ¶ 43.
Claim 17 is an apparatus claim corresponding to claim 8 and, therefore, is similarly rejected.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Huszar, Manna, and Foody as applied to claims 1 and 10 above, and further in view of He (US 2020/0313612 A1; published Oct. 1, 2020).
Regarding claim 9, Huszar, in view of Manna and Foody, discloses the invention of claim 1 as discussed above. Huszar does not expressly disclose wherein the performance of the fuzzy detection comprises: acquiring prediction accuracy rates of each of the plurality of trained neural network models, ranking the prediction accuracy rates, and performing the fuzzy detection by a neural network model with a highest prediction accuracy rate according to the ranking (but see He ¶ 83 (“Step 2: the FLIR A310 thermal imager obtains 720 original thermal images with defects and extracts 720 images from PCA, ICA and NMF respectively. There is a total of 2880 thermal images. These defective silicon photovoltaic cell data are imported into LeNet-5, VGG-16 and GoogLeNet convolutional neural networks for defect classification and identification. 80% of the data is used as the train data set and 20% of the data is used as the test data set. Therefore, the number of training data sets is 2304 and the number of test data sets is 576. After establishing the data, we fine-tune the data obtained by using the LeNet-5, VGG-16, and GoogLeNet models. The training results in FIG. 5 show that all three classification algorithms achieved high accuracy values. The GoogLeNet model has the best classification accuracy and loss function. At the 30.sup.th repeated operation, the GoogLeNet model shows more than 90% accuracy. The GoogleNet model achieves the maximum 100% classification accuracy in the 77.sup.th repeated operation during training, and the corresponding loss function value is 0.002. From the 77.sup.th repeated operation and onwards, the GoogLeNet model basically maintains 100% classification accuracy. In the 21.sup.st repeated operation in the training, the VGG-16 model achieves the maximum accuracy value, the final classification accuracy rate is fixed at 94.67%. The LeNet-5 model has the highest accuracy rate of 89.65%. Therefore, it can be proved that GoogLeNet has the best defect classification effect.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Huszar to incorporate the teachings of He to select the model with the highest accuracy rate, at least because doing so would ensure the final classification is accurate.
Claim 18 is an apparatus claim corresponding to claim 9 and, therefore, is similarly rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Melville, Prem, and Raymond J. Mooney. "Diverse ensembles for active learning." Proceedings of the twenty-first international conference on Machine learning. 2004.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAHID KHAN whose telephone number is (571)270-0419. The examiner can normally be reached M-F, 9-5 est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571)272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAHID K KHAN/Primary Examiner, Art Unit 2146
Prosecution Insights