DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendment filed on 9/25/2025. Claims 1, 8, 9, 14, 15, and 20 have been amended. Claims 4, 12, and 18 are cancelled. Claims 1-3, 5-11, 13-17, and 19-20 are pending and have been examined.
Claim Rejections - 35 USC § 112
The rejections under 35 USC § 112 to Claims 1-11, 13-17, and 19-20 are WITHDRAWN in view of Applicant’s amendments to Claims 1, 8, 9, 14, 15, and 20.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-3, 5-11, 13-17, and 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Step 1:
The claim recites a method, which is one of the four statutory categories of patentable subject matter.
Step 2A prong 1:
The claim recites an abstract idea. Specifically, the limitation determining which of the first or second mode of data requires fewer resources to label or annotate… comprises determining if labeling or annotating one mode of data requires greater or less monetary cost, greater or less computational resources, greater or less human labor, or greater or less time than the other mode of data amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation evaluating a performance of a classifier for the mode of data requiring fewer resources to label or annotate amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation selecting a first subset of the second set of pairs of data, wherein the subset selected are those for which the classifier for the mode of data requiring fewer resources to label or annotate outputs a result indicating a presence of a characteristic amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation selecting a second subset of the second set of pairs of data, wherein the subset selected are those for which the classifier for the mode of data requiring fewer resources to label or annotate outputs a result indicating an absence of the characteristic amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation estimating a performance of the classifier for the mode of data requiring greater resources to label or annotate by comparing the output of the classifier for the mode requiring greater resources to label or annotate to the output of the classifier for the mode requiring fewer resources to label or annotate amounts to a mental process as it can be performed in a human mind.
Step2A prong 2:
The additional element of obtaining a first set of pairs of data… from a first study does not integrate the abstract idea into practical application because obtaining data from a first study is considered an insignificant extra solution activity of “mere data gathering” MPEP 2106.05(g).
The additional element of obtaining a second set of pairs of data… from a second study does not integrate the abstract idea into practical application because obtaining data from a first study is considered an insignificant extra solution activity of “mere data gathering” MPEP 2106.05(g).
The additional element of operating the classifier for the mode of data requiring fewer resources to label or annotate using the data element of the second set of pairs of data corresponding to the mode of data requiring fewer resources to label or annotate as inputs is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
The additional element of operating a classifier for the mode of data requiring greater resources to label or annotate using the selected first and second subsets of the second set of pairs of data as inputs is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
Step 2B:
The additional element of obtaining a first set of pairs of data… from a first study does not amount to significantly more because the additional element is an insignificant extra solution activity and further is a well understood routine and conventional activity. See MPEP 2106.05(d)(II)(i), (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)).
The additional element of obtaining a second set of pairs of data… from a second study does not amount to significantly more because the additional element is an insignificant extra solution activity and further is a well understood routine and conventional activity. See MPEP 2106.05(d)(II)(i), (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)).
The additional element of operating the classifier for the mode of data requiring fewer resources to label or annotate using the data element of the second set of pairs of data corresponding to the mode of data requiring fewer resources to label or annotate as inputs is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
The additional element of operating a classifier for the mode of data requiring greater resources to label or annotate using the selected first and second subsets of the second set of pairs of data as inputs is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
Therefore, the claim is ineligible.
Regarding Claim 2:
Claim 2 incorporates the rejection of Claim 1. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 1. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 3:
Claim 3 incorporates the rejection of Claim 2. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 1. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 5:
Claim 5 which incorporates the rejection of Claim 1, recites a further abstract idea estimating the performance of the classifier for the mode of data requiring greater resources to label or annotate by comparing the output of the classifier for the mode requiring greater resources to the output of the classifier for the mode requiring fewer resources to label or annotate further comprises performing a direct estimation of a conditional probability distribution or utilizing a form of Bayes rule which is a mathematical concept. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 6:
Claim 6 incorporates the rejection of Claim 1. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 1. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 7:
Claim 7 incorporates the rejection of Claim 1. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 1. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 8:
Claim 8 incorporates the rejection of Claim 1.
Step 2A prong 1:
The claim recites an additional abstract idea. Specifically, the limitation labeling a plurality of data elements of the mode of data requiring fewer resources to annotate or label the label indicating the presence or absence of the characteristic amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation selecting a plurality of outputs of the operated classifier for the mode of data requiring fewer resources to label or annotate indicating the presence of the characteristic amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation selecting a plurality of outputs of the operated classifier for the mode of data requiring fewer resources to label or annotate indicating the absence of the characteristic amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation performing a review… to produce a set of correctly labeled data of the mode requiring fewer resources amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation based on the correctly labeled data, evaluating the performance of the classifier for the mode requiring fewer resources in terms of a Positive Predictive Value and a Negative Predictive Value for that classifier amounts to a mental process as it can be performed in a human mind.
Step 2A prong 2:
The additional element of operating the classifier for the mode of data requiring fewer resources to label or annotate using the data element of the first set of pairs of data corresponding to the mode of data requiring fewer resources to label or annotate as inputs is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
Step 2B:
The additional element of operating the classifier for the mode of data requiring fewer resources to label or annotate using the data element of the first set of pairs of data corresponding to the mode of data requiring fewer resources to label or annotate as inputs is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
Therefore, the claim is ineligible.
Regarding Claim 9:
Step 1:
The claim recites a system, which is one of the four statutory categories of patentable subject matter.
Step 2A prong 1:
The claim recites an abstract idea. Specifically, the limitation determine if labeling or annotating one of either the first or second modes of data requires fewer resources than the other mode of data in pairs… comprises determining if labeling or annotating one mode of data requires greater or less monetary cost, greater or less computational resources, greater or less human labor, or greater or less time than the other mode of data amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation evaluate a performance of a classifier for the mode of data requiring fewer resources amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation estimate a performance of the classifier for the mode of data requiring greater resources based on the output of the classifier for the mode requiring greater resources compared to the output of the classifier for the mode requiring fewer resources amounts to a mental process as it can be performed in a human mind.
Step2A prong 2:
The additional element of using memory is a generic computer component amounting to mere instructions to apply the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(f).
The additional element of using electronic processors is a generic computer component amounting to mere instructions to apply the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(f).
The additional element of obtain a first set of pairs of data… are obtained from a first study does not integrate the abstract idea into practical application because obtaining data from a first study is considered an insignificant extra solution activity of “mere data gathering” MPEP 2106.05(g).
The additional element of obtain a second set of pairs of data… from a second study does not integrate the abstract idea into practical application because obtaining data from a first study is considered an insignificant extra solution activity of “mere data gathering” MPEP 2106.05(g).
The additional element of operate the classifier for the mode of data requiring fewer resources… the classifier outputs a result indicating a presence of a characteristic and a result indicating an absence of the characteristic is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
The additional element of operate a classifier for the mode of data requiring greater resources using the selected subset of pairs of data is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
Step 2B:
The additional element of using memory is a generic computer component amounting to mere instructions to apply the abstract idea, therefore does not amount to significantly more MPEP 2106.05(f).
The additional element of using electronic processors is a generic computer component amounting to mere instructions to apply the abstract idea, therefore does not amount to significantly more MPEP 2106.05(f).
The additional element of obtain a first set of pairs of data… are obtained from a first study does not amount to significantly more because the additional element is an insignificant extra solution activity and further is a well understood routine and conventional activity. See MPEP 2106.05(d)(II)(i), (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)).
The additional element of obtain a second set of pairs of data… from a second study does not amount to significantly more because the additional element is an insignificant extra solution activity and further is a well understood routine and conventional activity. See MPEP 2106.05(d)(II)(i), (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)).
The additional element of operate the classifier for the mode of data requiring fewer resources… the classifier outputs a result indicating a presence of a characteristic and a result indicating an absence of the characteristic is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
The additional element of operate a classifier for the mode of data requiring greater resources using the selected subset of pairs of data is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
Therefore, the claim is ineligible.
Regarding Claim 10:
Claim 10 incorporates the rejection of Claim 9. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 9. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 11:
Claim 11 incorporates the rejection of Claim 10. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 9. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 13:
Claim 13 which incorporates the rejection of Claim 9, recites a further abstract idea estimating the performance of the classifier for the more mode of data requiring greater resources based on the output of the classifier for the more mode requiring greater resources compared to the output of the classifier for the mode requiring fewer resources further comprises performing a direct estimation of conditional probability distributions or utilizing a form of Bayes rule which is a mathematical concept. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 14:
Claim 14 incorporates the rejection of Claim 9.
Step 2A prong 1:
The claim recites an additional abstract idea. Specifically, the limitation labeling a set of the mode requiring fewer resources of the data pair as indicating the presence or absence of a characteristic amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation reviewing the first subset of data by a human to produce a set of correctly labeled data of the mode requiring fewer resources amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation based on the correctly labeled data by the human, evaluating the performance of the classifier for the mode requiring fewer resources in terms of a Positive Predictive Value and a Negative Predictive Value for that classifier amounts to a mental process as it can be performed in a human mind.
Step 2A prong 2:
The additional element of operating the classifier for the mode requiring fewer resources to select a first subset of data indicating the presence and the absence of the characteristic is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
Step 2B:
The additional element of operating the classifier for the mode requiring fewer resources to select a first subset of data indicating the presence and the absence of the characteristic is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
Therefore, the claim is ineligible.
Regarding Claim 15:
Step 1:
The claim recites a non-transitory computer readable medium, which is one of the four statutory categories of patentable subject matter.
Step 2A prong 1:
The claim recites an abstract idea. Specifically, the limitation determining which of the first or second mode of data requires fewer resources to label or annotate… comprises determining if labeling or annotating one mode of data requires greater or less monetary cost, greater or less computational resources, greater or less human labor, or greater or less time than the other mode of data amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation evaluate a performance of a classifier for the mode of data requiring fewer resources amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation estimate a performance of the classifier for the mode of data requiring greater resources based on the output of the classifier for the mode requiring greater resources compared to the output of the classifier for the mode requiring fewer resources amounts to a mental process as it can be performed in a human mind.
Step2A prong 2:
The additional element of using electronic processors is a generic computer component amounting to mere instructions to apply the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(f).
The additional element of obtain a first set of pairs of data… are obtained from a first study does not integrate the abstract idea into practical application because obtaining data from a first study is considered an insignificant extra solution activity of “mere data gathering” MPEP 2106.05(g).
The additional element of obtain a second set of pairs of data… from a second study does not integrate the abstract idea into practical application because obtaining data from a first study is considered an insignificant extra solution activity of “mere data gathering” MPEP 2106.05(g).
The additional element of operate the classifier for the mode of data requiring fewer resources… the classifier outputs a result indicating a presence of a characteristic and a result indicating an absence of the characteristic is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
The additional element of operate a classifier for the mode of data requiring greater resources using the selected subset of pairs of data is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
Step 2B:
The additional element of using electronic processors is a generic computer component amounting to mere instructions to apply the abstract idea, therefore does not amount to significantly more MPEP 2106.05(f).
The additional element of obtain a first set of pairs of data… are obtained from a first study does not amount to significantly more because the additional element is an insignificant extra solution activity and further is a well understood routine and conventional activity. See MPEP 2106.05(d)(II)(i), (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)).
The additional element of obtain a second set of pairs of data… from a second study does not amount to significantly more because the additional element is an insignificant extra solution activity and further is a well understood routine and conventional activity. See MPEP 2106.05(d)(II)(i), (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network)).
The additional element of operate the classifier for the mode of data requiring fewer resources… the classifier outputs a result indicating a presence of a characteristic and a result indicating an absence of the characteristic is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
The additional element of operate a classifier for the mode of data requiring greater resources using the selected subset of pairs of data is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
Therefore, the claim is ineligible.
Regarding Claim 16:
Claim 16 incorporates the rejection of Claim 15. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 15. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 17:
Claim 17 incorporates the rejection of Claim 16. This claim further recites a description of the abstract idea of the first and second mode of data in the abstract ideas of Claim 15. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 19:
Claim 19 which incorporates the rejection of Claim 15, recites a further abstract idea estimating the performance of the classifier for the more mode of data requiring greater resources based on the output of the classifier for the more mode requiring greater resources compared to the output of the classifier for the mode requiring fewer resources further comprises performing a direct estimation of conditional probability distributions or utilizing a form of Bayes rule which is a mathematical concept. The claim does not recite any additional elements that integrate the abstract idea into practical application or amount to significantly more. The claim is ineligible.
Regarding Claim 20:
Claim 20 incorporates the rejection of Claim 15.
Step 2A prong 1:
The claim recites an additional abstract idea. Specifically, the limitation labeling a set of the mode requiring fewer resources of the data pair as indicating the presence or absence of a characteristic amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation reviewing the first subset of data by a human to produce a set of correctly labeled data of the mode requiring fewer resources amounts to a mental process as it can be performed in a human mind.
The claim recites an additional abstract idea. Specifically, the limitation based on the correctly labeled data by the human, evaluating the performance of the classifier for the mode requiring fewer resources in terms of a Positive Predictive Value and a Negative Predictive Value for that classifier amounts to a mental process as it can be performed in a human mind.
Step 2A prong 2:
The additional element of operating the classifier for the mode requiring fewer resources to select a first subset of data indicating the presence and the absence of the characteristic is generally linked to the abstract idea, therefore does not integrate the abstract idea into practical application MPEP 2106.05(h).
Step 2B:
The additional element of operating the classifier for the mode requiring fewer resources to select a first subset of data indicating the presence and the absence of the characteristic is generally linked to the abstract idea, therefore does not amount to significantly more MPEP 2106.05(h).
Therefore, the claim is ineligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 5, 7, 9-11, 13, 15-17, and 19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Shin et al. “Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation”, hereinafter “Shin”.
Regarding Claim 1, Shin teaches:
A method, comprising:
obtaining a first set of pairs of data, wherein each pair of data in the first set comprises a first data element and a second data element, the first data element being a first mode of data and the second data element being a second mode of data, wherein the first and the second data elements of the first set are obtained from a first study (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 11, col. 2, ¶2, “we treat each x-ray image and report as a sample”, first study is a mini-batch, p.4, col. 1, ¶2, “each mini-batch consists of a balanced number of samples per disease case and a random selection of normal case samples”);
determining which of the first or second mode of data requires fewer resources to label or annotate, wherein determining which of the first or second mode of data requires fewer resources to label or annotate comprises determining if labeling or annotating one mode of data requires greater or less monetary cost, greater or less computational resources, greater or less human labor, or greater or less time than the other mode of data (First mode of data is images, second mode of data is reports, reports are annotated before being inputted into model therefore are determined to require less resources to annotate, first mode of data is guaranteed to require less computational resources because second mode of data requires first mode of data to be already annotated as preprocessing step before annotating second mode of data, p. 3, col. 1, ¶2, “MeSH terms for each radiology report… are annotated… We use these to train our model”, images require more resources because they use the overall recurrent neural cascade model for annotation, p. 1, col. 1, Abstract, “we present a deep learning model to efficiently detect a disease from an image and annotate its contexts”);
evaluating a performance of a classifier for the mode of data requiring fewer resources to label or annotate (Classifier is overall recurrent neural cascade model comprising CNN and RNN, Figure 5 shows report annotations being input into recurrent neural cascade model, Table 5 evaluates performance metrics of the recurrent neural cascade model);
obtaining a second set of pairs of data, wherein each pair of data in the second set comprises a first data element and a second data element, the first data element being of the first mode of data and the second data element being of the second mode of data, wherein the first and the second data elements of the second set are obtained from a second study (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 11, col. 2, ¶2, “we treat each x-ray image and report as a sample”, Second study is a minibatch different than the first minibatch, p.4, col. 1, ¶2, “each mini-batch consists of a balanced number of samples per disease case and a random selection of normal case samples”);
operating the classifier for the mode of data requiring fewer resources to label or annotate using the data element of the second set of pairs of data corresponding to the mode of data requiring fewer resources to label or annotate as inputs (Classifier is operated for reports by inputting reports and receiving output, p. 7, Figure 5 shows report annotations input into recurrent neural cascade model resulting in classification of images);
selecting a first subset of the second set of pairs of data, wherein the subset selected are those for which the classifier for the mode of data requiring fewer resources to label or annotate outputs a result indicating a presence of a characteristic and selecting a second subset of the second set of pairs of data, wherein the subset selected are those for which the classifier for the mode of data requiring fewer resources to label or annotate outputs a result indicating an absence of the characteristic (All the data input into the classifier will result in indicating a presence or an absence of characteristics based on input report annotations, subsets are selected when data is input into model, p. 7, col. 2, Figure 5 shows images and text data input into model will result in indicating absence or presence of certain characteristics, p. 8, Figure 6, true annotation);
operating a classifier for the mode of data requiring greater resources to label or annotate using the selected first and second subsets of the second set of pairs of data as inputs (Image data requires greater resources than annotating reports because annotating the images requires annotating reports as a preprocessing step, p. 3, col. 2, ¶3, “We use… 17 unique disease annotation patterns… to label the images and train CNNs”, p. 7, col. 2, Figure 5 showing images input into a CNN classifier); and
estimating a performance of the classifier for the mode of data requiring greater resources to label or annotate by comparing the output of the classifier for the mode requiring greater resources to label or annotate to the output of the classifier for the mode requiring fewer resources to label or annotate (The overall recurrent neural cascade model classifier compares output from its RNN operated with input report annotations with output of its CNN operated with input images, comparison is done by inputting CNN output into RNN in the overall recurrent neural cascade model, p. 5, col. 2, ¶3, “We then use the CNN prediction of the input image as the first word as the input to the RNN”, p. 7, col. 2, Figure 5 shows both CNN and RNN are part of recurrent neural cascade model classifier, p. 8, col. 1, Table 5 shows performance metrics estimation after comparison).
Regarding Claim 2, Shin teaches the method of Claim 1 as referenced above. Shin further teaches:
wherein the first mode of data is an image, and the second mode of data is a text description indicating the presence or absence of the characteristic in the image (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 8, Figure 6 showing images and text description extracted from report indicating presence or absence of characteristics).
Regarding Claim 3, Shin teaches the method of Claim 2 as referenced above. Shin further teaches:
wherein the image is an x-ray or scan of a portion of a person's body, and the characteristic is a tumor or nodule in the image (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, “MeSH terms for each radiology report… are annotated… We use these to train our model”, p. 3, Table 1 shows MeSH terms including nodule as a characteristic).
Regarding Claim 5, Shin teaches the method of Claim 1 as referenced above. Shin further teaches:
wherein estimating the performance of the classifier for the mode of data requiring greater resources to label or annotate by comparing the output of the classifier for the mode requiring greater resources to the output of the classifier for the mode requiring fewer resources to label or annotate further comprises performing a direct estimation of a conditional probability distribution or utilizing a form of Bayes rule (Performance is estimated through training by comparing CNN output to RNN, “We set the initial state of RNNs as the CNN image embedding (CNN(I)), and the first annotation word as the initial input”, p. 5, col. 1, Equation 11 shows a direct estimation of conditional probability distribution as loss function)
Regarding Claim 7, Shin teaches the method of Claim 1 as referenced above. Shin further teaches:
wherein the first mode of data is an image, and the second mode of data is a caption for the image (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 1, Figure 1 shows x-ray image and captions from report).
Regarding Claim 9, Shin teaches:
A system, comprising:
a set of computer-executable instructions stored in a memory and one or more electronic processors configured to execute the set of computer-executable instructions, wherein when executed, the instructions cause the one or more electronic processors (The machine learning architecture is trained and tested, demonstrating that Shin performs their method on a computer, in which processor, memory, and storage devices are inherent, p. 5, col. 2, ¶4, “We evaluate the annotation generation on the BLEU [47] score averaged over all of the images and their annotations in the training, validation, and test set”, p. 8 col. 2, ¶3, “We thank NVIDIA for the K40 GPU donation”) to
obtain a first set of pairs of data, wherein each pair comprises a first data element of a first mode of data and a second data element of a second mode of data, wherein both the first and the second modes of data are obtained from a first study (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 11, col. 2, ¶2, “we treat each x-ray image and report as a sample”, first study is a mini-batch, p.4, col. 1, ¶2, “each mini-batch consists of a balanced number of samples per disease case and a random selection of normal case samples”);
determine if labeling or annotating one of either the first or second modes of data requires fewer resources than the other mode of data in the pairs, wherein determining if labeling or annotating one of either the first or second modes of data requires fewer resources than the other mode of data in the pairs comprises determining if labeling or annotating one mode of data requires greater or less monetary cost, greater or less computational resources, greater or less human labor, or greater or less time than the other modes of data (First mode of data is images, second mode of data is reports, reports are annotated before being inputted into model therefore are determined to require less resources to annotate, first mode of data is guaranteed to require less computational resources because second mode of data requires first mode of data to be already annotated as preprocessing step before annotating second mode of data, p. 3, col. 1, ¶2, “MeSH terms for each radiology report… are annotated… We use these to train our model”, images require more resources because they use the recurrent neural cascade model for annotation, p. 1, col. 1, Abstract, “we present a deep learning model to efficiently detect a disease from an image and annotate its contexts”);
evaluate a performance of a classifier for the mode of data requiring fewer resources (Classifier is overall recurrent neural cascade model comprising CNN and RNN, Figure 5 shows report annotations being input into Recurrent Neural Cascade Model, Table 5 evaluates performance metrics of the Recurrent Neural Cascade Model);
obtain a second set of the pairs of data, wherein each pair comprises a first data element of the first mode of data and a second data element of the second mode of data, wherein both the first and the second modes of data in the second set of pairs of data are obtained from a second study (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 11, col. 2, ¶2, “we treat each x-ray image and report as a sample”, Second study is a minibatch different than the first minibatch, p.4, col. 1, ¶2, “each mini-batch consists of a balanced number of samples per disease case and a random selection of normal case samples”);
operate the classifier for the mode of data requiring fewer resources(p. 7, Figure 5 shows report annotations input into recurrent neural cascade model resulting in classification of images) and selecting a subset of pairs of data from the second set of pairs of data, wherein the subset selected represents inputs to the classifier for which the classifier outputs a result indicating a presence of a characteristic and a result indicating an absence of the characteristic (All the data input into the classifier includes subsets of data that indicate a presence and an absence of a characteristic, subsets are selected when data is input into model, p. 7, col. 2, Figure 5 shows images and text data input into model will result in indicating absence or presence of certain characteristics, p. 8, Figure 6, true annotation);
operate a classifier for the mode of data requiring greater resources using the selected subset of pairs of data (Image data requires greater resources than annotating reports because annotating the images requires annotating reports as a preprocessing step, p. 3, col. 2, ¶3, “We use… 17 unique disease annotation patterns… to label the images and train CNNs”, p. 7, col .2, Figure 5, showing images input into a CNN classifier); and
estimate a performance of the classifier for the mode of data requiring greater resources based on the output of the classifier for the mode requiring greater resources compared to the output of the classifier for the mode requiring fewer resources (The overall recurrent neural cascade model classifier compares output from its RNN operated with input report annotations with output of its CNN operated with input images, comparison is done by inputting CNN output into RNN in the overall recurrent neural cascade model, p. 5, col. 2, ¶3, “We then use the CNN prediction of the input image as the first word as the input to the RNN”, p. 7, col. 2, Figure 5 shows both CNN and RNN are part of recurrent neural cascade model classifier, p. 8, col. 1, Table 5 shows performance metrics estimation after comparison).
Regarding Claim 10, the rejection of Claim 9 is incorporated and further, the claim is rejected for the same reasons as set forth in Claim 2.
Regarding Claim 11, the rejection of Claim 10 is incorporated and further, the claim is rejected for the same reasons as set forth in Claim 3.
Regarding Claim 13, the rejection of Claim 9 is incorporated and further, the claim is rejected for the same reasons as set forth in Claim 5.
Regarding Claim 15, Shin teaches:
A non-transitory computer readable medium containing a set of computer- executable instructions, wherein when the set of instructions are executed by one or more electronic processors, the instructions cause the processors (The machine learning architecture is trained and tested, demonstrating that Shin performs their method on a computer, in which processor, memory, and storage devices are inherent, p. 5, col. 2, ¶4, “We evaluate the annotation generation on the BLEU score averaged over all of the images and their annotations in the training, validation, and test set”, p. 8 col. 2, ¶3, “We thank NVIDIA for the K40 GPU donation”) to:
obtain a first set of pairs of data, wherein each pair comprises a first data element of a first mode of data and a second data element of a second mode of data, wherein both the first and the second modes of data are obtained from a first study (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 11, col. 2, ¶2, “we treat each x-ray image and report as a sample”, first study is a mini-batch, p.4, col. 1, ¶2, “where each mini-batch consists of a balanced number of samples per disease case and a random selection of normal case samples”);
determine if labeling or annotating one of either the first or second modes of data requires fewer resources than the other mode of data in the pairs, wherein determining which of the first or second mode of data requires fewer resources to label or annotate comprises determining if labeling or annotating one mode of data requires greater or less monetary cost, greater or less computational resources, greater or less human labor, or greater or less time than the other mode of data (First mode of data is images, second mode of data is reports, reports are annotated before being inputted into model therefore are determined to require less resources to annotate, first mode of data is guaranteed to require less computational resources because second mode of data requires first mode of data to be already annotated as preprocessing step before annotating second mode of data, p. 3, col. 1, ¶2, “MeSH terms for each radiology report… are annotated… We use these to train our model”, images require more resources because they use the recurrent neural cascade model for annotation, p. 1, col. 1, Abstract, “we present a deep learning model to efficiently detect a disease from an image and annotate its contexts”);
evaluate a performance of a classifier for the mode of data requiring fewer resources (Classifier is overall recurrent neural cascade model comprising CNN and RNN, Figure 5 shows report annotations being input into recurrent neural cascade model, Table 5 evaluates performance metrics of the recurrent neural cascade model);
obtain a second set of the pairs of data, wherein each pair comprises a first data element of the first mode of data and a second data element of the second mode of data, wherein both the first and the second modes of data in the second set of pairs of data are obtained from a second study (p. 2, col. 2, ¶5, “We use a publicly available radiology dataset of chest x-rays and report”, p. 11, col. 2, ¶2, “we treat each x-ray image and report as a sample”, Second study is a minibatch different than the first minibatch, p.4, col. 1, ¶2, “where each mini-batch consists of a balanced number of samples per disease case and a random selection of normal case samples”);
operate the classifier for the mode of data requiring fewer resources(p. 7, Figure 5 shows report annotations input into recurrent neural cascade model resulting in classification of images) and selecting a subset of pairs of data from the second set of pairs of data, wherein the subset selected represents inputs to the classifier for which the classifier outputs a result indicating a presence of a characteristic and a result indicating an absence of the characteristic (All the data input into the classifier includes subsets of data that indicate a presence and an absence of a characteristic, subsets are selected when data is input into model, p. 7, col. 2, Figure 5 shows images and text data input into model will result in indicating absence or presence of certain characteristics, p. 8, Figure 6, true annotation);
operate a classifier for the mode of data requiring greater resources using the selected subset of pairs of data (Image data requires greater resources than annotating reports because annotating the images requires annotating reports as a preprocessing step, p. 3, col. 2, ¶3, “We use… 17 unique disease annotation patterns… to label the images and train CNNs”, p. 7, col .2, Figure 5, showing images input into a CNN classifier); and
estimate a performance of the classifier for the mode of data requiring greater resources based on the output of the classifier for the mode requiring greater resources compared to the output of the classifier for the mode requiring fewer resources (The overall recurrent neural cascade model classifier compares output from its RNN operated with input report annotations with output of its CNN operated with input images, comparison is done by inputting CNN output into RNN in the overall recurrent neural cascade model, p. 5, col. 2, ¶3, “We then use the CNN prediction of the input image as the first word as the input to the RNN”, p. 7, col. 2, Figure 5 shows both CNN and RNN are part of recurrent neural cascade model classifier, p. 8, col. 1, Table 5 shows performance metrics estimation after comparison).
Regarding Claim 16, the rejection of Claim 15 is incorporated and further, the claim is rejected for the same reasons as set forth in Claim 2.
Regarding Claim 17, the rejection of Claim 16 is incorporated and further, the claim is rejected for the same reasons as set forth in Claim 3.
Regarding Claim 19, the rejection of Claim 15 is incorporated and further, the claim is rejected for the same reasons as set forth in Claim 5.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 6 is rejected under 35 U.S.C. 103 as being obvious over Shin, in view of Tariq et al. “Lung Disease Classification using Deep Convolutional Neural Network”, hereinafter Tariq.
Regarding Claim 6, Shin teaches the method of Claim 1 as referenced above. Shin teaches:
the second mode of data is a text description… (p. 2, col. 1, ¶2, “A publicly available radiology dataset is exploited which contains… reports”)
Shin does not expressly teach:
wherein the first mode of data is an audio track
However, Tariq teaches:
wherein the first mode of data is an audio track (p. 1, col. 1, Abstract, “we extracted spectrogram features and labels of the annotated lung sound samples and used them as an input to our 2D Convolutional Neural Network (CNN) model”, p. 4, col. 1, ¶1, “lungs sound dataset”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the audio data of Tariq inputted into a CNN with the CNN of the recurrent neural cascade model in Shin. The motivation to do so would be to accurately classify lung disease diagnosis (p. 4, col. 2, ¶2, “we developed… for high-performance classification in lung disease diagnosis”).
Claims 8, 14, and 20 are rejected under 35 U.S.C. 103 as being obvious over Shin, in view of Ouannes et al. “Understanding precision and recall”, hereinafter Ouannes.
Regarding Claim 8, Shin teaches the method of Claim 1 as referenced above. Shin further teaches:
wherein evaluating the performance of the classifier for the mode of data requiring fewer resources to label or annotate further comprises:
labeling a plurality of data elements of the mode of data requiring fewer resources to annotate or label, the label indicating the presence or absence of the characteristic (Shin, p. 3, col. 1, paragraph 2, “MeSH terms for each radiology report… are annotated”, p. 1, Figure 1 sh