Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/10/2026 has been entered.
Response to Amendment
Applicant’s Amendment and remarks dated 3/10/2026 have been considered. Claims 1-4, 7-15, and 18-22 are pending.
Response to Arguments
On page 8 of Applicant’s 3/10/2026 Amendment and remarks, Applicant asserts that the original claims and paras. 0023 and 0048-0051 of the instant specification provide sufficient written description support for the claim amendments.
The examiner agrees that together with at least para. 0019, the portions of the disclosure identified by Applicant provide sufficient written description support for the amendments to the independent claims.
On pages 10-11 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, and with respect to Step 2A, Prong 1, Applicant argues that the “to obtain a first classification value indicative of a first classification prediction for the audio sample” and “to obtain a second classification value indicative of a second classification prediction for the audio sample” limitations are not mental processes.
The examiner respectfully disagrees. Applicant’s arguments with respect to the first and “second neural network sub-model” limitations are addressed under Step 2A, Prong 2 and Step 2B. As explained in the rejection, the output of such neural network sub-models merely automates a mental process that can be performed, for example, by a doctor that is listening to audio of a patient’s cough, or by a doctor who can view images of the log-mel spectrogram data and make a prediction about whether the patient has a disease or not.
The examiner further notes that as explained by MPEP 2106.04(a)(2) II.C, this is also a judicial exception under the “managing personal behavior or relationships or interactions between people” sub-grouping, where the MPEP explains that a “mental process that a neurologist should follow when testing a patient for nervous system malfunctions” is considered to be an example of “managing personal behavior”, and in the instant case, a doctor can follow a procedure by listening to an audio sample of a patient’s cough in order to predict if the patient has COVID-19)
On page 11 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, and with respect to Step 2A, Prong 1, Applicant argues:
PNG
media_image1.png
252
650
media_image1.png
Greyscale
The examiner respectfully disagrees. First, it may be “difficult” or “complex” to analyze “spectral and waveform signatures of cough sounds”, it is still possible for a human to analyze such data mentally and make a prediction about whether a disease is present. While a computer (with a neural network) may make this process more efficient, the use of a computer or generic neural network does not mean that this is not a mental process. Second, Applicant has not provided any evidence that a person, such as a physician, could not make such a finding based on such data. The examiner notes that the 12/4/2025 memo re: “Subject Matter Eligibility Declarations” explains that Rule 132 declarations could be filed to provide such an evidentiary submission.
On page 11 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, and with respect to Step 2A, Prong 2, Applicant argues:
PNG
media_image2.png
214
644
media_image2.png
Greyscale
The examiner respectfully disagrees. As the examiner correctly pointed out on page 8 of the previous action, the claims do not recite “spectral waveforms or signatures of cough sounds.” The term “signature” is not used in the claims. Regardless, even though Applicant has now amended the independent claims to specifically recite an “audio sample of a cough”, the claims still do not relate to an improvement to disease detection technology.
MPEP 2106.04(d)(1) explains: “Conversely, if the specification explicitly sets forth an improvement but in a conclusory manner (i.e., a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art), the examiner should not determine the claim improves technology. Second, if the specification sets forth an improvement in technology, the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement.” Here, in view of the instant specification, one of ordinary skill would not recognize an inventive contribution to have been made or to be reflected by the claims. The claims, at a high-level, merely use 2 neural network sub-models, trained with generic training data and particular loss functions, to come up with first and second predictions, and such first and second predictions are “combined” to obtain an ensemble result that is used to predict the presence of a disease. No improvements are made to neural network technology or any other technical field. The improvements, if any, relate to the mental processes of analyzing data to come up with prediction results, and then using such prediction results to indicate the presence of a disease.
On pages 11-12 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, and with respect to Step 2A, Prong 2, Applicant argues:
PNG
media_image3.png
196
658
media_image3.png
Greyscale
PNG
media_image4.png
170
662
media_image4.png
Greyscale
The examiner respectfully disagrees. The claims, at a high-level, merely use 2 neural network sub-models, trained with generic training data and particular loss functions, to come up with first and second predictions, and such first and second predictions are “combined” to obtain an ensemble result that is used to predict the presence of a disease. No improvements are made to neural network technology or any other technical field. The improvements, if any, relate to the mental processes of analyzing data to come up with prediction results, and then using such prediction results to indicate the presence of a disease.
On page 12 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, and with respect to Step 2A, Prong 1, Applicant argues that the office action does not assess the limitations of claim 1 “as a whole.”
PNG
media_image5.png
328
644
media_image5.png
Greyscale
The examiner respectfully disagrees. As a “whole”, the claim does not relate to any improvements to “audio processing technology” or “disease detection.” The “audio processing technology” steps are merely data gathering steps using known, generic techniques for processing audio steps. As a whole, the claims relate to determining an indication of a disease from cough audio data, which is both a mental process and an example of “managing personal behavior”, which are each types of judicial exceptions.
On page 12 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, Applicant argues that independent claim 12 should be subject matter eligible for the same reasons argued with respect to claim 1.
The examiner respectfully disagrees for the same reasons explained with respect to claim 1.
On page 12 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 101, Applicant argues that the claims disclose “improvement in the functioning of a computer, or an improvement to other technology or technical field.”
The examiner respectfully disagrees. There is no improvement to any functioning of a computer or any neural network technologies in the claims. Nor is there any improvement to any other technical field. As discussed above, there is no improvement to the technical field of “disease detection” and Applicant has not identified any other alleged technical field that has been improved.
On pages 13-14 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the rejections under 35 U.S.C. 103, Applicant argues that the GRUBELE reference does not teach
The recited “remove audio segments from the plurality of audio segments where silence is detected limitation.”
PNG
media_image6.png
112
644
media_image6.png
Greyscale
PNG
media_image7.png
280
654
media_image7.png
Greyscale
The examiner respectfully disagrees with Applicant’s interpretation of GRUBELE and the office action. As Applicant notes, GRUBELE expressly teaches that “Silence detection 322 may be used to automatically detect audio segments of relative silence which can be used for later automated noise removal based on current background noise conditions.” (col. 12, lines 62-65). On page 37 of the 7/2/2025 office action, it explains that “the QIAO-COYLE-PALMA-GRUEBELE combination now applies the silence detection techniques of GRUEBELE to remove silent segments from the audio sample segments of COYLE.” (emphasis added). Therefore, where silence is detected, automated noise removal is used to remove the silenced portions of the text. Applicant’s argument that “The Office attempts to argue that detecting segments of silence in audio data and removing the non-silent segments is equivalent to removal of the silent segments themselves” mischaracterizes the office action, as it has also been the examiner’s position that GRUBELE specifically teaches removing silent portions from audio.
On page 14 of Applicant’s 3/10/2026 Amendment and remarks, with respect to the dependent claims, Applicant argues that such dependent claims should be allowed for the same reasons argued with respect to claim 1.
The examiner respectfully disagrees for the same reasons explained with respect to claim 1.
Claim Objections
Claims 11 and 22 are objected to because of the following informalities:
In claim 11, lines 5-6, “into a first class indicative of a presence of a disease or a second class indicative of no presence of the disease” should read “into [[a]]the first class indicative of [[a]]the presence of [[a]]the disease or [[a]]the second class indicative of no presence of the disease”
In claim 22, lines 4-5, “into a first class indicative of a presence of a disease or a second class indicative of no presence of the disease” should read “into [[a]]the first class indicative of [[a]]the presence of [[a]]the disease or [[a]]the second class indicative of no presence of the disease”
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4, 7-15, and 18-22 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Step 1 of the Alice/Mayo framework, Claims 1-4 and 7-11 are directed to a system (a machine) and Claims 12-15 and 18-22 are directed to a method (a process), which each fall within one of the four statutory categories of inventions.
Regarding Claim 1
Step 2A, prong 1 (Is the claim directed to a law of nature, a natural phenomenon or an abstract idea).
Claim 1 recites the following mental processes, that in each case under the broadest reasonable interpretation, covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper but for the recitation of generic computer components (e.g., “processor”, “memory”).
to obtain a first classification value indicative of a first classification prediction for the audio sample (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human, such as a trained doctor, can listen to an audio sample and predict whether the audio sample pertains to a first classification prediction, such as the audio sample being associated with a COVID-19 patient; the examiner further notes that as explained by MPEP 2106.04(a)(2) II.C, this is also a judicial exception under the “managing personal behavior or relationships or interactions between people” sub-grouping, where the MPEP explains that a “mental process that a neurologist should follow when testing a patient for nervous system malfunctions” is considered to be an example of “managing personal behavior”, and in the instant case, a doctor can follow a procedure by listening to an audio sample of a patient’s cough in order to predict if the patient has COVID-19)
to obtain a second classification value indicative of a second classification prediction for the audio sample (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human, such as a trained doctor, can listen to an audio sample and predict whether the audio sample pertains to a second classification prediction, such as the audio sample NOT being associated with a COVID-19 patient; the examiner further notes that as explained by MPEP 2106.04(a)(2) II.C, this is also a judicial exception under the “managing personal behavior or relationships or interactions between people” sub-grouping, where the MPEP explains that a “mental process that a neurologist should follow when testing a patient for nervous system malfunctions” is considered to be an example of “managing personal behavior”, and in the instant case, a doctor can follow a procedure by listening to an audio sample of a patient’s cough in order to predict if the patient has COVID-19))
combine the first classification value and the second classification value to obtain an ensembled classification result indicative of an ensembled classification prediction for the audio sample of the cough, the ensembled classification prediction being indicative of a first class indicative of the presence of the disease or a second class indicative of no presence of the disease. (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human can mentally combine two values (e.g., by averaging, or weighting the values) to determine the probability of a first or second class being indicative, such as the presence of COVID-19)
Step 2A, prong 2 (Does the claim recite additional elements that integrate the judicial exception into a practical application?).
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements (e.g., “processor”, “memory”) which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “A system, comprising: a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to” limitations, such limitations are recited at a high-level of generality and amount to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional elements of processor and memory. These additional elements are recited at a high-level of generality and amount to no more than mere instructions to apply the exception using generic computer components (processor and memory). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “segment an audio sample of a cough into a plurality of audio segments of the audio sample by application of a silence detection process”; “downsample the plurality of audio segments”; “remove audio segments from the plurality of audio segments where silence is detected”; and “transform each audio segment of a plurality of segments of the audio sample to a spectral log-mel scale to obtain a set of log-mel spectrogram data” limitations, such additional elements of data gathering are recited at a high level of generality and amounts to extra-solution activity of receiving and pre-processing data, i.e. pre-solution activity of gathering data for use in the claimed process (see MPEP 2106.05(g)). Moreover, such limitations are recited at a high-level of generality and amount to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result (e.g., the process of segmenting audio into silent periods for removal, using common audio processing techniques like downsampling and transforming to a different scale, without specific details or inventive concepts with respect to such audio processing). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “transform each audio segment of a plurality of segments of the audio sample to a spectral log-mel scale to obtain a set of log-mel spectrogram data” limitation,
Regarding the “apply a first neural network sub-model to a set of log-mel spectrogram data of an audio sample” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a neural network sub-model. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a neural network sub-model, which is claimed only at a high-level such that it is a generic computing automation). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “wherein the first neural network sub-model is trained to identify a presence of a disease by optimizing cross-entropy loss” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (use of a particular loss function for training a neural network model). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Regarding the “apply a second neural network sub-model to the set of log-mel spectrogram data of the audio sample” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of a neural network sub-model. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (a neural network sub-model, which is claimed only at a high-level such that it is a generic computing automation). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “wherein the second neural network sub-model is trained to identify the presence of the disease by optimizing focal loss” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (use of a particular loss function for training a neural network model). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Accordingly, at Step 2A, prong 2, after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Step 2B (Does the claim recite additional elements that amount to significantly more than the judicial exception?)
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements (e.g., “processor”, “memory”) are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “A system, comprising: a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to” limitations, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “segment an audio sample of a cough into a plurality of audio segments of the audio sample by application of a silence detection process”; “downsample the plurality of audio segments”; “remove audio segments from the plurality of audio segments where silence is detected”; and “transform each audio segment of a plurality of segments of the audio sample to a spectral log-mel scale to obtain a set of log-mel spectrogram data” limitations, as discussed above, the additional element of data gathering and pre-processing are recited at a high level of generality and amount to extra-solution activity of receiving and pre-processing data, i.e. pre-solution activity of gathering data for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory"). Moreover, such limitations are recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitations attempt to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “apply a first neural network sub-model to a set of log-mel spectrogram data of an audio sample” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “wherein the first neural network sub-model is trained to identify a presence of a disease by optimizing cross-entropy loss” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding the “apply a second neural network sub-model to the set of log-mel spectrogram data of the audio sample” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “wherein the second neural network sub-model is trained to identify the presence of the disease by optimizing focal loss” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Accordingly, at Step 2B, after considering all claim elements individually and as an ordered combination, it is determined that the claims do not amount to significantly more than the judicial exception.
Regarding Claim 2
Step 2A, Prong 1
wherein the cross-entropy loss function is given by:
PNG
media_image8.png
34
280
media_image8.png
Greyscale
wherein y є { 0, 1} corresponds to a respective classification label and p corresponds to the first classification value. (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human can do this calculation using pencil and paper; this is also a mathematical calculation, which is another type of abstract idea)
Step 2A, Prong 2
This limitation of claim 2 further amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (use of a particular loss function for training a neural network model). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Step 2B
This limitation of claim 2 further amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding Claim 3
Step 2A, Prong 1
wherein the focal loss function is given by:
PNG
media_image9.png
52
296
media_image9.png
Greyscale
wherein α and y are each a modulation hyperparameter. (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human can do this calculation using pencil and paper; this is also a mathematical calculation, which is another type of abstract idea)
Step 2A, Prong 2
This limitation of claim 3 further amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (use of a particular loss function for training a neural network model). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Step 2B
This limitation of claim 3 further amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding Claim 4
Step 2A, Prong 2
Regarding the “wherein the set of log-mel spectrogram data includes a plurality of spectrogram segments of the set of log-mel spectrogram data, each spectrogram segment of the plurality of spectrogram segments corresponding to a respective audio segment of a plurality of audio segments of the audio sample” limitation, this limitation merely describes the type of data being processed and analyzed, and therefore such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (log-mel spectrogram data). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Step 2B
Regarding the “wherein the set of log-mel spectrogram data includes a plurality of spectrogram segments of the set of log-mel spectrogram data, each spectrogram segment of the plurality of spectrogram segments corresponding to a respective audio segment of a plurality of audio segments of the audio sample” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding Claim 7
Step 2A, Prong 1
combine the first classification value and the second classification value by averaging the first classification value and the second classification value into the ensembled classification result. (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human can average the first and second classification values using pencil and paper; the examiner further notes that this is a mathematical calculation, which is another type of abstract idea)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 8
Step 2A, Prong 2
Regarding the “wherein the set of training data includes a plurality of audio samples, wherein each audio sample of the plurality of audio samples includes a classification label” limitation, this limitation merely describes the data being processed (labeled data), and therefore such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (supervised training using labeled data). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Step 2B
Regarding the “wherein the set of training data includes a plurality of audio samples, wherein each audio sample of the plurality of audio samples includes a classification label” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding Claim 9
Step 2A, Prong 1
augment the set of training data by mixing, at random, a pair of inputs of the set of training data with a pair of corresponding outputs of the set of training data. (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human can augment a data set by mixing up the associated inputs/outputs to create negative data samples)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 10
Step 2A, Prong 1
upsample the set of training data from more than one dataset. (under the broadest reasonable interpretation, this limitation can be performed mentally (or with physical aids, such as pencil and paper), for example, a human can upsample mathematically using a pencil and paper with respect to numerical audio samples from the audio from more than one dataset; the examiner further notes that this is a mathematical calculation, which is another type of abstract idea)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 11
Step 2A, Prong 2
Regarding the “train the first neural network sub-model and the second neural network sub-model on a set of training data to classify the audio sample into a first class indicative of a presence of a disease or a second class indicative of no presence of the disease” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of training neural networks at a high level using certain data. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (neural network training). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “wherein the set of training data includes a plurality of training samples, each training sample of the plurality of training samples being classified into the first class or the second class” limitation, this limitation merely describes the data being processed (labeled data), and therefore such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (supervised training using labeled data). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not integrate a judicial exception into a practical application.
Step 2B
Regarding the “train the first neural network sub-model and the second neural network sub-model on a set of training data to classify the audio sample into a first class indicative of a presence of a disease or a second class indicative of no presence of the disease” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “wherein the set of training data includes a plurality of training samples, each training sample of the plurality of training samples being classified into the first class or the second class” limitation, such limitation amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use as explained above, which does not amount to significantly more than the judicial exception. MPEP 2106.05(h).
Regarding Claims 12-15 and 18-22
Claim 12 recites a method that corresponds to the system of claim 1, and is therefore rejected for the same reasons explained above with respect to claim 1.
Claims 13-15 and 18-22 depend from claim 12 and recite a method that corresponds to the systems of claims 2-5 and 7-11, respectively, and are therefore rejected for the same reasons explained with respect to claim 1 and claims 2-5 and 7-11, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 7, 10-12, 15, 18, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Qiao, Zhi, et al. "FLANNEL (focal loss based neural network ensemble) for COVID-19 detection." Journal of the American Medical Informatics Association 28.3 (Oct. 30, 2020): pp. 444-452, hereinafter referenced as QIAO, in view of US 20070276278 A1, hereinafter referenced as COYLE, and further in view of US 20220138931 A1, hereinafter referenced as PALMA, and further in view of US 10489103 B1, hereinafter referenced as GRUEBELE.
Regarding Claim 1
QIAO teaches:
A system, comprising: (QIAO, p. 445, Intro section, left column: “we propose a FLANNEL (Focal Loss bAsed Neural Network EnsembLe) model. FLANNEL utilizes an ensemble structure, using 5 state-of-the-art convolutional neural network (CNN) classifiers as based models. The neural weight module combines each base model using a learnt weighted ensemble to get the final classification results. ... our focus is to make accurate prediction on COVID-19 cases without hindering the performance of other classes.”)
a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to: (QIAO, p. 445, “Ensemble models” section, left+right columns: “Instead of training a complex single neural network with a large number of layers and parameters, decomposing the architecture into smaller and simpler individual base models has been shown to be more accurate and require less time and memory for training”;
QIAO, p. 448, “Implementation Details” section, left column: “All the base models and FLANNEL are implemented in PyTorch and trained on 3 NVIDIA Tesla P100 GPUs (NVIDIA, Santa Clara, CA) over 200 epochs.”)
Examiner’s Note: The FLANNEL model is trained using Tesla P100 GPUs (corresponding to recited “processor”), where one of ordinary skill would understand that the instructions executed by the processor are stored at least temporarily in memory)
apply a first neural network sub-model to ... obtain a first classification value indicative of a first classification prediction, wherein the first neural network sub-model is trained to identify a presence of a disease by optimizing a cross-entropy loss; (QIAO, p. 447, “Stage 1: Base learner training” section, left column: “Owing to the limited amount of training data of x-ray images, we use the listed pretrained models from the ImageNet Large Scale Visual Recognition Challenge (http://www.image-net.org/challenges/LSVRC/) and fine-tune each model with respect to the COVID-19 identification task.”;
QIAO, p. 448, “Results: Baseline models for performance comparison” section, left column: “To verify the advantages of FLANNEL for the imbalanced datasets, we compare it to a version of FLANNEL that replaces the Focal Loss with multiclass standard cross-entropy loss (denoted as FLANNEL_w/o_focal)”
QIAO, p. 447, “Results” section, right column: “All of the models were fine-tuned using their default parameter settings and by using the Adam optimizer.”
QIAO, p. 448, “Evaluation strategy” section, left column: “In order to overall verify the prediction accuracy, we first measure the overall accuracy of the model in distinguishing the 4 classes (COVID-19 viral pneumonia, non–COVID-19 viral pneumonia, bacterial pneumonia, and normal images). The main intention of the study is the detection of COVID-19 among kinds of respiratory-related x-ray images.”;
Examiner’s Note: The version of FLANNEL trained without focal loss, and instead using a cross-entropy loss optimized using an Adam optimizer, corresponds to the recited “first neural network sub-model”, which predicts whether COVID-19 (corresponding to recited “presence of a disease”) is detected from the input data)
apply a second neural network sub-model to obtain a second classification value indicative of a second classification prediction, wherein the second neural network sub-model is trained to identify the presence of the disease by optimizing focal loss (QIAO, p. 447, “Stage 2: Ensemble model learning” section, right column:
PNG
media_image10.png
305
558
media_image10.png
Greyscale
QIAO, p. 447, “Results” section, right column: “All of the models were fine-tuned using their default parameter settings and by using the Adam optimizer.”
QIAO, p. 448, “Results: Baseline models for performance comparison” section, left column: “To verify the advantages of FLANNEL for the imbalanced datasets, we compare it to a version of FLANNEL that replaces the Focal Loss with multiclass standard cross-entropy loss (denoted as FLANNEL_w/o_focal)”
QIAO, p. 448, “Evaluation strategy” section, left column: “In order to overall verify the prediction accuracy, we first measure the overall accuracy of the model in distinguishing the 4 classes (COVID-19 viral pneumonia, non–COVID-19 viral pneumonia, bacterial pneumonia, and normal images). The main intention of the study is the detection of COVID-19 among kinds of respiratory-related x-ray images.”;
Examiner’s Note: The version of FLANNEL trained with focal loss corresponds to the recited “second neural network sub-model”, which predicts whether COVID-19 is detected from the input data and optimizes Focal Loss using an Adam optimizer)
However, QIAO fails to explicitly teach:
segment an audio sample of a cough into a plurality of audio segments of the audio sample by application of a silence detection process;
downsample the plurality of audio segments;
remove audio segments from the plurality of audio segments where silence is detected;
transform each audio segment of a plurality of segments of the audio sample to a spectral log-mel scale to obtain a set of log-mel spectrogram data.
the set of log-mel spectrogram data of the audio sample to ... for the audio sample;
the set of log-mel spectrogram data of the audio sample to ... for the audio sample;
combine the first classification value and the second classification value to obtain an ensembled classification result indicative of an ensembled classification prediction for the audio sample of the cough, the ensembled classification prediction being indicative of a first class indicative of the presence of a disease or a second class indicative of no presence of the disease.
However, in a related field of endeavor (patient monitoring with respect to coughing, see para. 0002), COYLE teaches:
transform each audio segment of a plurality of segments of the audio sample to a spectral log-mel scale to obtain a set of log-mel spectrogram data (COYLE, para. 0097: “Audio signals (from, for example, a throat microphone) are processed with a speech recognition front-end to determine if an audio event contains voiced or unvoiced speech. ... The pitch value is computed by measuring the peak-to-peak power present in the Cepsturm or Mel Frequency Cepstral Coefficients (MFCCs).”;
COYLE, para. 0110: “For these further tests, the characteristics of a speech audio signal are considered to be stationary over time increments of approximately 10 msec., and the pitch of the audio signal is therefore analyzed over such segments of such time duration.”;
COYLE, para. 0117: “In the final step of cepsturm determination, the log mel spectrum is transformed back to time resulting in the mel frequency cepsturm coefficients (MFCC). The cepstral representation of the speech spectrum provides a representation of the local spectral properties of the signal for the given frame analysis”;
Examiner’s Note: COYLE teaches collecting audio samples from patients and calculating “log mel spectrum” (corresponding to recited “log-mel spectrogram data”) for use in cough analysis with respect to 10 ms segments, where such representation is of the “local spectral properties” of the signal; the QIAO-COYLE combination now modifies the models of QIAO to utilize log-mel spectrum data as of COYLE, where each spectrogram is of a 10 msec time segment as in COYLE)
the set of log-mel spectrogram data of the audio sample to ... for the audio sample; (COYLE, para. 0097: “Audio signals (from, for example, a throat microphone) are processed with a speech recognition front-end to determine if an audio event contains voiced or unvoiced speech. ... The pitch value is computed by measuring the peak-to-peak power present in the Cepsturm or Mel Frequency Cepstral Coefficients (MFCCs).”;
COYLE, para. 0111: “Generally, MFCCs are based on the known variation of the human ear's critical bandwidths so that these coefficients are expressed in a mel-frequency scale, which is linear at frequencies less than 1000 Hz and logarithmic at frequencies above 1000 Hz.”;
COYLE, para. 0117: “In the final step of cepstrum determination, the log mel spectrum is transformed back to time resulting in the mel frequency cepsturm coefficients (MFCC). The cepstral representation of the speech spectrum provides a representation of the local spectral properties of the signal for the given frame analysis”;
Examiner’s Note: COYLE teaches collecting audio samples from patients and calculating “log mel spectrum” (corresponding to recited “log-mel spectrogram data”) for use in cough analysis; the QIAO-COYLE combination now modifies the first model of QIAO (that uses cross-entropy loss) to utilize the log-mel spectrum data of COYLE, in addition to the x-ray imaging data of QIAO)
the set of log-mel spectrogram data of the audio sample to ... for the audio sample; (COYLE, para. 0097: “Audio signals (from, for example, a throat microphone) are processed with a speech recognition front-end to determine if an audio event contains voiced or unvoiced speech. ... The pitch value is computed by measuring the peak-to-peak power present in the Cepsturm or Mel Frequency Cepstral Coefficients (MFCCs).”;
COYLE, para. 0111: “Generally, MFCCs are based on the known variation of the human ear's critical bandwidths so that these coefficients are expressed in a mel-frequency scale, which is linear at frequencies less than 1000 Hz and logarithmic at frequencies above 1000 Hz.”;
COYLE, para. 0117: “In the final step of cepsturm determination, the log mel spectrum is transformed back to time resulting in the mel frequency cepsturm coefficients (MFCC). The cepstral representation of the speech spectrum provides a representation of the local spectral properties of the signal for the given frame analysis”;
Examiner’s Note: COYLE teaches collecting audio samples from patients and calculating “log mel spectrum” (corresponding to recited “log-mel spectrogram data”) for use in cough analysis; the QIAO-COYLE combination now modifies the second model of QIAO (that uses focal loss) to utilize the log-mel spectrum data of COYLE, in addition to the x-ray imaging data of QIAO)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of QIAO with respect to using log-mel spectrum data. As disclosed by COYLE, one of ordinary skill would have been motivated to do so in order to utilize “portable and easy-to-use monitoring methods and systems that provide objective and quantitative data on cough.” (para. 0004). One of ordinary skill would further understand the benefit of augmenting the x-ray images of QIAO with the audio data of COYLE in order to improve the COVID-19 classification system of QIAO.
However, QIAO and COYLE fail to explicitly teach:
segment an audio sample of a cough into a plurality of audio segments of the audio sample by application of a silence detection process;
downsample the plurality of audio segments;
remove audio segments from the plurality of audio segments where silence is detected;
combine the first classification value and the second classification value to obtain an ensembled classification result indicative of an ensembled classification prediction for the audio sample of the cough, the ensembled classification prediction being indicative of a first class indicative of the presence of the disease or a second class indicative of no presence of the disease.
However, in a related field of endeavor (identification of anatomical features of interest in the healthcare field, see para. 0001), PALMA teaches:
combine the first classification value and the second classification value to obtain an ensembled classification result indicative of an ensembled classification prediction for the audio sample of the cough, the ensembled classification prediction being indicative of a first class indicative of the presence of the disease or a second class indicative of no presence of the disease.
(PALMA, para. 0015: “a first machine learning model of the ensemble is trained with a first loss function that penalizes errors in false negative classifications of lesions, and a second machine learning model of the ensemble is trained with a second loss function, different from the first machine learning model, that penalizes errors in false positive classifications of lesions. By having multiple machine learning models implementing different loss functions, each machine learning model may compensate for the weaknesses of the other machine learning models with regard to sensitivity and specificity.”
PALMA, para. 0109: “A third loss function of the ensemble of ML/DL computer models as a whole compares the results of the decoders of the ML/DL computer model 136 to each other and forces them to be consistent with each other. The lesion prediction results of the first and second ML/DL computer models 134, 136 are combined to generate a final lesion prediction for the ensemble, while the other ML/DL computer model 132 that generates a prediction of a liver mask provides an output representing the liver and its contour.”
Examiner’s Note: the QIAO-COYLE-PALMA combination now modifies the COVID-19 detection classifier of QIAO to operate on the log-mel spectrum data of COYLE (with respect to audio cough signals as disclosed by paras. 0051-0052 of COYLE) and further to combine the outputs of the FLANNEL (cross-entropy) and FLANNEL (focal loss) models of QIAO in the ensemble method of PALMA, which combines the predictions from the 2 FLANNEL models of QIAO to create a final prediction of whether COVID-19 is present or not)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions. As disclosed by PALMA, one of ordinary skill would have been motivated to do so because by “having an ensemble of differently trained machine learning computer models, the machine learning computer models may compensate for the loss/error function focus and/or potential weaknesses in training of other computer models of the ensemble and thereby improve the overall lesion detection.” (para. 0013).
However, QIAO, COYLE, and PALMA fail to explicitly teach:
segment an audio sample of a cough into a plurality of audio segments of the audio sample by application of a silence detection process; (GRUEBELE teaches
downsample the plurality of audio segments;
remove audio segments from the plurality of audio segments where silence is detected;
However, in a related field of endeavor (collecting and processing audio data, including via wearable electronic medical devices, see col. 1, lines 6-15), GRUEBELE teaches:
segment an audio sample of a cough into a plurality of audio segments of the audio sample by application of a silence detection process; (GRUEBELE, col. 11, lines 49-53: “Turning back to FIG. 11, audio noise removal 302 consists of first calculating a noise profile through frequency domain analysis of a segment of “noise” audio, and then real-time frequency domain removal of detected noise profiles in an audio stream.”;
GRUEBELE, col. 12, lines 32-34: “The audio stream is segmented into Opus or other audio file blobs which are then stored in distributed local & cloud database 26.”;
GRUEBELE, col. 12, lines 62-65: “Silence detection 322 may be used to automatically detect audio segments of relative silence which can be used for later automated noise removal based on current background noise conditions”
Examiner’s Note: the QIAO-COYLE-PALMA-GRUEBELE combination now applies the silence detection techniques of GRUEBELE to the audio sample segments of coughs as in COYLE, where paras. 0051-0052 of COYLE discloses processing of microphone data of cough events)
downsample the plurality of audio segments; (GRUEBELE, col. 11, lines 30-37: “It is then volume normalized 290, SINC 32 down-sampled to F/2 292 and SINC 32 up-sampled to F 294. Subtract 296 is then used to generate an array of possible bit errors which has most non-error energy removed. This then passes through pseudo-code 298 and pseudo-code 300, is resampled to 24 Khz 301 which produces the final error-corrected 24 Khz audio stream.”;
Examiner’s Note: the QIAO-COYLE-PALMA-GRUEBELE combination now applies the audio downsampling techniques of GRUEBELE to the audio sample segments of COYLE)
remove audio segments from the plurality of audio segments where silence is detected; (GRUEBELE, col. 12, lines 62-65: “Silence detection 322 may be used to automatically detect audio segments of relative silence which can be used for later automated noise removal based on current background noise conditions”
Examiner’s Note: the QIAO-COYLE-PALMA-GRUEBELE combination now applies the silence detection techniques of GRUEBELE to remove silent segments from the audio sample segments of coughs of COYLE to remove the segments of “relative silence” that are directed using silence detection 322)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE. As disclosed by GRUEBELE, accuracy is improved by removing noise before processing. (col. 12, lines 65-67). One of ordinary skill would understand that segmenting based on silence periods will remove silent audio samples from the training set, which will be beneficial so that the models of QIAO are only trained on non-silent data).
Regarding Claim 4
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. However, QIAO fails to explicitly teach:
wherein the set of log-mel spectrogram data includes a plurality of spectrogram segments of the set of log-mel spectrogram data, each spectrogram segment of the plurality of spectrogram segments corresponding to a respective audio segment of a plurality of audio segments of the audio sample.
However, in a related field of endeavor (patient monitoring with respect to coughing, see para. 0002), COYLE teaches:
wherein the set of log-mel spectrogram data includes a plurality of spectrogram segments of the set of log-mel spectrogram data, each spectrogram segment of the plurality of spectrogram segments corresponding to a respective audio segment of a plurality of audio segments of the audio sample. (COYLE, para. 0097: “Audio signals (from, for example, a throat microphone) are processed with a speech recognition front-end to determine if an audio event contains voiced or unvoiced speech. ... The pitch value is computed by measuring the peak-to-peak power present in the Cepsturm or Mel Frequency Cepstral Coefficients (MFCCs).”;
COYLE, para. 0110: “For these further tests, the characteristics of a speech audio signal are considered to be stationary over time increments of approximately 10 msec., and the pitch of the audio signal is therefore analyzed over such segments of such time duration.”;
COYLE, para. 0117: “In the final step of cepsturm determination, the log mel spectrum is transformed back to time resulting in the mel frequency cepsturm coefficients (MFCC). The cepstral representation of the speech spectrum provides a representation of the local spectral properties of the signal for the given frame analysis”;
Examiner’s Note: COYLE teaches collecting audio samples from patients and calculating “log mel spectrum” (corresponding to recited “log-mel spectrogram data”) for use in cough analysis with respect to 10 ms segments; the QIAO-COYLE-PALMA-GRUEBELE combination now modifies the models of QIAO to utilize log-mel spectrum data as of COYLE, where each spectrogram is of a 10 msec time segment as in COYLE)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE. As disclosed by COYLE, one of ordinary skill would have been motivated to do so in order to utilize “portable and easy-to-use monitoring methods and systems that provide objective and quantitative data on cough.” (para. 0004). One of ordinary skill would further understand the benefit of augmenting the x-ray images of QIAO with the audio data of COYLE in order to improve the COVID-19 classification system of QIAO.
Regarding Claim 7
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. However, QIAO and COYLE fail to explicitly teach:
combine the first classification value and the second classification value by averaging the first classification value and the second classification value into the ensembled classification result.
However, in a related field of endeavor (identification of anatomical features of interest in the healthcare field, see para. 0001), PALMA teaches:
combine the first classification value and the second classification value by averaging the first classification value and the second classification value into the ensembled classification result. (PALMA, para. 0015: “a first machine learning model of the ensemble is trained with a first loss function that penalizes errors in false negative classifications of lesions, and a second machine learning model of the ensemble is trained with a second loss function, different from the first machine learning model, that penalizes errors in false positive classifications of lesions. By having multiple machine learning models implementing different loss functions, each machine learning model may compensate for the weaknesses of the other machine learning models with regard to sensitivity and specificity.”
PALMA, para. 0071: “The detection output from the two ML/DL models is averaged to produce a final lesion detection.”
Examiner’s Note: the QIAO-COYLE-PALMA-GRUEBELE combination now modifies the COVID-19 detection classifier of QIAO to operate on the log-mel spectrum data of COYLE and further to average the outputs of the FLANNEL (cross-entropy) and FLANNEL (focal loss) models of QIAO in the ensemble method of PALMA, which combines the predictions from the 2 FLANNEL models of QIAO (by averaging) to create a final prediction of whether COVID-19 is present or not)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE. As disclosed by PALMA, one of ordinary skill would have been motivated to do so because by “having an ensemble of differently trained machine learning computer models, the machine learning computer models may compensate for the loss/error function focus and/or potential weaknesses in training of other computer models of the ensemble and thereby improve the overall lesion detection.” (para. 0013).
Regarding Claim 10
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. However, QIAO, COYLE, and PALMA fail to explicitly teach:
upsample the set of training data from more than one dataset.
However, in a related field of endeavor (collecting and processing audio data, including via wearable electronic medical devices, see col. 1, lines 6-15), GRUEBELE teaches:
upsample the set of training data from more than one dataset. (GRUEBELE, col. 11, lines 30-37: “It is then volume normalized 290, SINC 32 down-sampled to F/2 292 and SINC 32 up-sampled to F 294. Subtract 296 is then used to generate an array of possible bit errors which has most non-error energy removed. This then passes through pseudo-code 298 and pseudo-code 300, is resampled to 24 Khz 301 which produces the final error-corrected 24 Khz audio stream.”;
Examiner’s Note: the QIAO-COYLE-PALMA-GRUEBELE combination now trains the models of QIAO to use audio cough data as in COYLE, where such training data is upsampled from more than one dataset (as in QIAO) using the upsampling techniques of GRUEBELE)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling and upsampling techniques of GRUEBELE. As disclosed by GRUEBELE, accuracy is improved by removing noise before processing. (col. 12, lines 65-67). One of ordinary skill would understand that segmenting based on silence periods will remove silent audio samples from the training set, which will be beneficial so that the models of QIAO are only trained on non-silent data).
Regarding Claim 11
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. QIAO further teaches:
train the first neural network sub-model and the second neural network sub-model on a set of training data to classify the audio sample into a first class indicative of a presence of a disease or a second class indicative of no presence of the disease; (QIAO, p. 445, “Study design” section, right column: “We combined 2 publicly available datasets to generate the experimental data: (1) COVID Chest X-ray dataset (https://github.com/ieee8023/covid-chestxray-dataset) and (2) Kaggle Chest X-ray images (pneumonia) dataset (https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia). The choice of these 2 datasets for our experiment was guided by the fact that both are open source and fully accessible to the research community and the general public.”;
QIAO, p. 447, “Stage 1: Base learning training” section, left column: “Owing to the limited amount of training data of x-ray images, we use the listed pretrained models from the ImageNet Large Scale Visual Recognition Challenge (http://www.image-net.org/challenges/LSVRC/) and fine-tune each model with respect to the COVID-19 identification task.”)
wherein the set of training data includes a plurality of training samples, each training sample of the plurality of training samples being classified into the first class or the second class. (QIAO, p. 448, Box 1: The FLANNEL Algorithm:
PNG
media_image11.png
572
572
media_image11.png
Greyscale
Examiner’s Note: In Algorithm 1, the “Labels” correspond to the recited classification of each sample into a first or second class (COVID-19 yes/no))
However, QIAO fails to explicitly teach:
the audio sample
However, in a related field of endeavor (patient monitoring with respect to coughing, see para. 0002), COYLE teaches:
the audio sample (COYLE, para. 0097: “Audio signals (from, for example, a throat microphone) are processed with a speech recognition front-end to determine if an audio event contains voiced or unvoiced speech. ... The pitch value is computed by measuring the peak-to-peak power present in the Cepsturm or Mel Frequency Cepstral Coefficients (MFCCs).”)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE. As disclosed by COYLE, one of ordinary skill would have been motivated to do so in order to utilize “portable and easy-to-use monitoring methods and systems that provide objective and quantitative data on cough.” (para. 0004). One of ordinary skill would further understand the benefit of augmenting the x-ray images of QIAO with the audio data of COYLE in order to improve the COVID-19 classification system of QIAO.
Claim 12 recites a method that corresponds to the system of claim 1, and is therefore rejected for the same reasons explained above with respect to claim 1.
Claim 15 depends from claim 12 and recites a method that corresponds to the system of claim 4, and is therefore rejected for the same reasons explained with respect to claims 4 and 12.
Claim 18 depends from claim 12 and recites a method that corresponds to the system of claim 7, and is therefore rejected for the same reasons explained with respect to claims 7 and 12.
Claim 21 depends from claim 12 and recites a method that corresponds to the system of claim 10, and is therefore rejected for the same reasons explained with respect to claims 10 and 12.
Claim 22 depends from claim 12 and recites a method that corresponds to the system of claim 11, and is therefore rejected for the same reasons explained with respect to claims 11 and 12.
Claims 2-3 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over QIAO in view of COYLE and PALMA and GRUEBELE and further in view of Lin, Tsung-Yi, et al. "Focal Loss for Dense Object Detection." IEEE Transactions on Pattern Analysis and Machine Intelligence 42.2 (2018): pp. 318-327, hereinafter referenced as LIN.
Regarding Claim 2
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. However, QIAO, COYLE, PALMA, and GRUEBELE do not distinctly disclose:
wherein the cross-entropy loss function is given by:
PNG
media_image8.png
34
280
media_image8.png
Greyscale
wherein y є { 0, 1} corresponds to a respective classification label and p corresponds to the first classification value.
The examiner notes that QIAO does teach a specific cross-entropy loss (see p. 447, “Stage 2: Ensemble model learning” section, left column), but such cross-entropy loss equation of QIAO does not precisely match the formula claimed in claim 2.
However, in a related field of endeavor (loss functions that deal with class imbalance, see p. 318, section 1), LIN teaches:
wherein the cross-entropy loss function is given by:
PNG
media_image8.png
34
280
media_image8.png
Greyscale
wherein y є { 0, 1} corresponds to a respective classification label and p corresponds to the first classification value. (LIN, p. 320, section 3:
PNG
media_image12.png
424
520
media_image12.png
Greyscale
Examiner’s Note: Claim 2 recites that y is either 0 or 1. For y = 1, the claimed equation
CE = -ylog(p) – (l – y) log (1-p) becomes -log(p), which matches the focal loss equation in LIN for y=1, when equation (2) is substituted into equation (1). For y = 0, the claimed equation
CE = -ylog(p) – (l – y) log (1-p) becomes –log (1 – p), which matches the focal loss equation in LIN for y=0; the QIAO-COYLE-PALMA-GRUEBELE-LIN combination now modifies the cross-entropy loss function of QIAO to utilize the specific formulas for cross-entropy loss as recited in LIN)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE, and further with the specific cross-entropy loss function of LIN. As disclosed by LIN, one of ordinary skill would have been motivated to do so because LIN teaches a specific cross-entropy loss for binary classification (QIAO has more than 1 classification), so for a yes/no determination on COVID-19, one of ordinary skill would use a binary cross-entropy loss as in LIN. (LIN, p. 320, section 3).
Regarding Claim 3
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. However, QIAO, COYLE, PALMA, and GRUEBELE do not distinctly disclose:
wherein the focal loss function is given by:
PNG
media_image9.png
52
296
media_image9.png
Greyscale
wherein α and y are each a modulation hyperparameter.
The examiner notes that QIAO does teach a specific focal loss (see p. 447, “Stage 2: Ensemble model learning” section, right column), but such focal loss equation of QIAO does not precisely match the formula claimed in claim 3.
However, in a related field of endeavor (loss functions that deal with class imbalance, see p. 318, section 1), LIN teaches:
wherein the focal loss function is given by:
PNG
media_image9.png
52
296
media_image9.png
Greyscale
wherein α and y are each a modulation hyperparameter. (LIN, p. 320, section 3.2:
PNG
media_image13.png
618
524
media_image13.png
Greyscale
Examiner’s Note: For y = 1, the claimed equation FL = -α log(p) (1-p)y becomes -α log(p) (1-p), and in LIN, when equation (2) is substituted into equation (5), FL = - αt (1 – pt) log (pt)y becomes
FL = - αt (1 – p) log (p), which matches the claimed equation for y = 1. When y is not equal to 1, the claimed equation – (1 – α) log (1-p) py matches equation (5) of LIN when equation (2) is substituted, where FL = - αt (1 – pt)y log (pt)y becomes - αt (p)y log (1 - p)y, and if α2 = (1 - αt), becomes = - (1 - α2) (p)y log (1 - p)y which matches the claimed equation for y not equal to 1; the QIAO-COYLE-PALMA-GRUEBELE-LIN combination now modifies the focal loss function of QIAO to utilize the specific formulas for focal loss as recited in LIN)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE, and further with the specific focal loss function of LIN. As disclosed by LIN, one of ordinary skill would have been motivated to do so because LIN teaches that a focal loss, with a modulating factor (α) and focusing parameter (y) improves performance with respect to large class imbalances. (LIN, p. 320, section 3.2).
Claim 13 depends from claim 12 and recites a method that corresponds to the system of claim 2, and is therefore rejected for the same reasons explained with respect to claims 2 and 12.
Claim 14 depends from claim 12 and recites a method that corresponds to the system of claim 3, and is therefore rejected for the same reasons explained with respect to claims 3 and 12.
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over QIAO in view of COYLE and PALMA and GRUEBELE and further in view of US 20210142782 A1, hereinafter referenced as WOLF.
Regarding Claim 8
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. However, QIAO fails to explicitly teach:
wherein the set of training data includes a plurality of audio samples, wherein each audio sample of the plurality of audio samples includes a classification label.
However, in a related field of endeavor (patient monitoring with respect to coughing, see para. 0002), COYLE teaches:
wherein the set of training data includes a plurality of audio samples (COYLE, para. 0124: “ Generally, threshold values can be determined for a given population from a training data set of cough data obtained from the target population. Specifically, the threshold values are adjusted so that the methods best recognize actual coughs in the training data set.”)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE. As disclosed by COYLE, one of ordinary skill would have been motivated to do so in order to utilize “portable and easy-to-use monitoring methods and systems that provide objective and quantitative data on cough.” (para. 0004). One of ordinary skill would further understand the benefit of augmenting the x-ray images of QIAO with the audio data of COYLE in order to improve the COVID-19 classification system of QIAO.
However, QIAO, COYLE, PALMA, and GRUEBELE fail to explicitly teach:
wherein each audio sample of the plurality of audio samples includes a classification label.
However, in a related field of endeavor (modeling based on audio samples, see para. 0005), WOLF teaches:
wherein each audio sample of the plurality of audio samples includes a classification label. (WOLF, para. 0034: “For example, the CTC 616 can label or classify outputs from the plurality of convolutional blocks 112 and convolutional layers 113 and provide the labels or classifications back the neural network to update or train weights of the encoder 110 such that subsequent and/or future audio samples 108 provided to the encoder can be used to train the neural network based in part on the labels or classifications from previous audio samples 108 and properties of the subsequent and/or future audio samples 108.”; (EN): the QIAO-COYLE-PALMA-GREUBELE-WOLF combination now trains the models of QIAO to use audio cough data as in COYLE, where such training data has classification labels as in WOLF)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE, and further with the audio training data having classification labels as in WOLF. One of ordinary skill would understand that training using labeled data (e.g., supervised learning) can improve the accuracy of a machine learning model.
Claim 19 depends from claim 12 and recites a method that corresponds to the system of claim 8, and is therefore rejected for the same reasons explained with respect to claims 8 and 12.
Claims 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over QIAO in view of COYLE and PALMA and GRUEBELE and further in view of US 20210043186 A1, hereinafter referenced as NAGANO.
Regarding Claim 9
QIAO, COYLE, PALMA, and GRUEBELE disclose the system of claim 1. However, QIAO, COYLE, PALMA, and GRUEBELE fail to explicitly teach:
augment the set of training data by mixing, at random, a pair of inputs of the set of training data with a pair of corresponding outputs of the set of training data.
However, in a related field of endeavor (augmenting audio data, see para. 0001), NAGANO teaches:
augment the set of training data by mixing, at random, a pair of inputs of the set of training data with a pair of corresponding outputs of the set of training data. (NAGANO, para. 0056: “Referring back to FIG. 1, the data augmentation module 130 is configured to output the partially prolonged copy as augmented speech data to the adaptation data store 142. ... When used in combination during training process of the domain adaptation, the resultant augmented speech data may be used randomly as a mixture with the original data and/or other speech data, may be used sequentially with the original data and/or other speech data, or may be used alternately with the original speech data and/or other speech data.”; (EN): the QIAO-COYLE-PALMA-GRUEBELE-NAGANO combination now trains the models of QIAO to use audio cough data as in COYLE, where such training data is augmented by randomly mixing original data with output data as in NAGANO)
Before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the FLANNEL models of QIAO which detect COVID-19 from data, to utilize the teachings of COYLE with respect to using log-mel spectrum data, and further with the teachings of PALMA pertaining to using an ensemble method to combine the predictions of multiple machine learning models having different loss functions, and further with the silence detection and downsampling techniques of GRUEBELE, and further with the data augmentation techniques of NAGANO. As disclosed by NAGANO, one of ordinary skill would have been motivated to do so in order to increase the “amount of a training dataset” so that there is more data to train models with. (para. 0002).
Claim 20 depends from claim 12 and recites a method that corresponds to the system of claim 9, and is therefore rejected for the same reasons explained with respect to claims 9 and 12.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bansal, Vipin, et al. "Cough Classification for COVID-19 based on audio mfcc features using Convolutional Neural Networks." 2020 IEEE international conference on computing, power and communication technologies (GUCON). IEEE, 2020. “Cough classification is a well-researched topic by use of various audio feature extraction techniques. These are for cough detection, cough classification and disease diagnosis based on a cough. Barata et al. [9] proposed mel spectrogram visual imagery as an input to Convolutional Neural Network for cough detection on 43 participants with 6737 cough, 3985 laughter, 3695 throat clearing, 731 speech and 443 forced expiration audio signals. The evaluation was done using KNN, random forest and CNN architectures. CNN outperformed the other approaches.” (p. 605, section III.A).
Dhekane, Shariva, et al. "Modern Transfer Learning-Based Preliminary Diagnosis of COVID-19 Using Forced Cough Recordings with Mel-Frequency Cepstral Coefficients." Applied Information Processing Systems: Proceedings of ICCET 2021. Singapore: Springer Singapore, July 21, 2021. 137-146. Discloses using AI techniques to detect COVID-19 through cough sounds. (p. 138, section 1). “The coughing sound recorded while data collection is resampled to 16 kHz. Librosa library was used for audio processing. From the resampled audio, the leading and trailing silence was removed. Mel-Frequency Cepstral Coefficients (MFCCs) [6] were extracted and stored in the form of Mel spectrogram images from this silence-removed audio.” (p. 139, section 2.2).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL C. LEE/Examiner, Art Unit 2128
Prosecution Insights