DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the response to this office action, the Examiner respectfully requests that support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line numbers in the specification and/or drawing figure(s). This will assist the Examiner in prosecuting this application.
Response to Amendment
2. The Amendment filed May 11, 2026 has been entered. Claims 1, 4-8, 11-15, and 18-20 has been amended. Claims 1-20 are pending in the application.
Claim Rejections - 35 USC § 102
3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
4. Claims 1, 8, and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Khanzada U.S. Patent Application Publication 20220037022.
Regarding claim 1, Khanzada teaches a computer-based diagnostic device (Some aspects include a computer implemented method, including: obtaining, with one or more processors, a set of data comprising a plurality of patient records, wherein: each patient record includes a plurality of parameters and corresponding values for a patient; the plurality of parameters and corresponding values for a patient comprises an audio file of patient's vocal noises, such as cough, breathing, or speech; and the set of data also includes a diagnostic indicator indicating whether or not the patient has been diagnosed with COVID-19, par [0005], see Khanzada), comprising:
a memory (par [0007], see Khanzada); and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors (Some aspects include a system, including: one or more processors; and memory storing instructions that when executed by the processors cause the processors to effectuate operations of the above-mentioned process, par [0007], see Khanzada) to:
receive, from a client device (That said, the present techniques have use on other platforms, e.g., public kiosks, desktop computers, servers receiving similar data from a remote client device, and the like, par [0014], see Khanzada), source audio data captured from a patient (the plurality of parameters and corresponding values for a patient comprises an audio file of patient's vocal noises, such as cough, breathing, or speech, par [0005]; In some cases, different forms of audio input, e.g., coughing, reading a specified phrase, reciting of a syllable (e.g., asking the user to say “aaaaaaaah” or “eeeeeee” for 5 seconds), and deep breathing, may each constitute a different channel of input. The audio input may be captured with a microphone of the user's smartphone, par [0024] see Khanzada);
apply a plurality of spectral feature extraction algorithms to the source audio data (In some embodiments, temporal data analysis may be used. By using data from the same patient using a user-interface to record data multiple times throughout days and weeks, the algorithm is expected to be able to infer the user's stage in the COVID-19 disease and predict development of the disease and outcomes. In some cases, even after COVID-19 recovery, patients' ear, nose, throat, and lung tissue are still affected, along with the presence of antibodies, par [0025], see Khanzada) to generate a plurality of two-dimensional numerical arrays (In some embodiments, the second extracted feature may be the mel-frequency spectrogram (corresponds to two-dimensional numerical array), another audio feature. Though MFCCs are derived from the spectrogram, the spectrogram encodes raw power information without any transformations. Spectrograms may be extracted using the librosa package with the same parameters as for the MFCCs and interpolated to size (par [0043], see Khanzada)); the mel-spectrograms may be extracted using the librosa package for the same parameters used to extract MFCCs. Each mel-spectrogram color image may be reshaped to the size of (224,224,3), the original input size of the ResNet-50 convolutional neural network, (par [0045], see Khanzada); Various algorithms may be used to decide where a split occurs, including entropy, Gini impurity, Chi-Square, Information Gain, or Reduction in Variance. Decision trees are often helpful to rapidly identify the most significant variables among a large number of variables, as well as identify relationships between two or more variables. Additionally, decision trees may handle both numerical and non-numerical data, (par [0062], see Khanzada), each two-dimensional numerical array comprising values representing a distinct spectral characteristic of the source audio data (In some embodiments, the way in which audio features are extracted from voice audio files may affect model performance. There are expected to be several useful features to train the network on, such as mel-frequency cepstral coefficients and mel-frequency spectrograms (corresponds to two-dimensional numerical array), both being audio features . In some embodiments, heterogeneous classifiers may be used, one of them being trained on mel-spectrograms and the other being trained on MFCCs, par [0044], see Khanzada. It is noted the spectrogram "encodes raw power information" distinct from the MFCC transformation (par [0044]), with the mel-spectrogram reshaped into a multi-dimensional array (e.g., 224x224x3 (par [0045]), or 64x64x1(par [0056])) for input to a CNN. "the second extracted feature may be the melfrequency spectrogram ... " and "Each mel-spectrogram color image may be reshaped to the size of (224,224,3) (par [0045], see Khanzada); and
provide the plurality of two-dimensional numerical arrays to a neural network model trained (In some embodiments, the model is a multi-branch ensemble learning architecture based on a ResNet-50 3D convolutional neural network that is pre-trained on ImageNet dataset and stripped of the top layer (e.g., classification layer). The input for the CNN may be a mel-spectrogram color image of size (224 pixels, 224 pixels, 3 RGB layers, or larger or smaller in any of these dimensions) and the output of the CNN may be passed to both a global average pooling layer and a global maximum pooling layer in two separate and parallel links. Each of these layers may be followed by batch normalization and dropout layers before concatenated together in a single dense (e.g., non-linear, like layers with a sigmoid or hyperbolic tangent activation function) layer to make the first branch, par [0048], see Khanzada) to generate one or more diagnoses of the patient based at least in part on the plurality of two-dimensional numerical arrays (In some embodiments, the extracted high-level features at the high end of the three branches may be fused together before being passed to a sequential feed-forward neural network (SFFN) that is followed by a softmax layer for a multi-label classification task. The three labels are as follows for some embodiments: negative COVID-19 (healthy), negative COVID-19 (Symptomatic) and positive COVID-19. Other embodiments may include more, like negative with low confidence, negative with high confidence, positive with low confidence, positive with high confidence, and indeterminate (par [0051], see Khanzada); Various algorithms may be used to decide where a split occurs, including entropy, Gini impurity, Chi-Square, Information Gain, or Reduction in Variance. Decision trees are often helpful to rapidly identify the most significant variables among a large number of variables, as well as identify relationships between two or more variables. Additionally, decision trees may handle both numerical and non-numerical data, (par [0062], see Khanzada).
Khanzada thus teaches all the claimed limitations.
Regarding claim 8, this claim merely reflects a non-transitory computer-readable medium comprising computer readable instructions executable by one or more processors to perform the Claim 1 and is therefore rejected for the same reasons. It is noted that Khanzada teaches system memory 1020 may include a non-transitory computer readable storage medium that may have program instructions stored thereon that are executable by a computer processor (e.g., one or more of processors 1010a-1010n) to cause the subject matter and the functional operations described herein (see Fig. 3, par [0081], see Khanzada).
Regarding claim 15, this claim merely reflects the method to the apparatus claim of Claim 1 and is therefore rejected for the same reasons.
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
8. Claims 2, 3, 9-10, and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Khanzada U.S. Patent Application Publication 20220037022 in view of Alqudah et al. "Deep learning models for detecting respiratory pathologies from raw lung auscultation sounds", Soft Computing (2022), 26 September 2022, pages 13405-13429 (hereinafter, “Alqudah”), previously cited).
Regarding claim 2, Khanzada teaches the computer-based diagnostic device of claim 1. However Khanzada U.S. Patent Application Publication 20220037022does not explicitly disclose wherein the one or more diagnoses include COPD, healthy, URTI, Bronchiectasis, Pneumonia, and Bronchiolitis.
Alqudah teaches deep learning models for detecting respiratory pathologies from raw lung auscultation sounds (see Title) in which in this paper, the data used incorporated two different datasets, both datasets are consisting of stethoscope lung sounds classified with different respiratory diseases. Table 1 provides a detailed overview of the used dataset and the four datasets from their merging. Each dataset will
be discussed in detail in the next two sections (page 13412 left column, last paragraph – right column first paragraph, see Alqudah). Table 1 teaches normal, asthma, bronchiectasis, bronchiolitis, COPD, LRTI, pneumonia, and URTI (see Table 1, page 13410). See also Confusion Matrix in Fig. 4 (B) in page 13413 see Alqudah.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the deep learning models for detecting respiratory pathologies from raw lung auscultation sounds taught by Alqudah with the computer-based diagnostic device of Khanzada such that to obtain wherein the one or more diagnoses include COPD, healthy, URTI, Bronchiectasis, Pneumonia, and Bronchiolitis in order to improve the diagnosis performance of many diseases especially respiratory diseases as suggested by Alqudah in Abstract.
Regarding claim 3, Khanzada teaches the computer-based diagnostic device of claim . However, Khanzada does not explicitly disclose wherein the audio data is captured by a digital stethoscope.
Alqudah teaches deep learning models for detecting respiratory pathologies from raw lung auscultation sounds (see Title) in which initially, we intend to test the proposed method on more datasets. Then, we plan to develop an embedded system that
integrate the developed CNN-LSTM model into the system with a digital stethoscope (see page 13427, left column, last-paragraph, see Alqudah).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the deep learning models for detecting respiratory pathologies from raw lung auscultation sounds taught by Alqudah with the computer-based diagnostic device of Khanzada such that to obtain wherein the audio data is captured by a digital stethoscope in order to improve the diagnosis performance of many diseases especially respiratory diseases as suggested by Alqudah in Abstract.
Regarding claim 9, this claim has similar limitations as Claim 2 and is therefore rejected for the same reasons.
Regarding claim 10, this claim has similar limitations as Claim 3 and is therefore rejected for the same reasons.
Regarding claim 16, this claim has similar limitations as Claim 2 and is therefore rejected for the same reasons.
Regarding claim 17, this claim has similar limitations as Claim 3 and is therefore rejected for the same reasons.
9. Claims 4, 5, 7, 11, 12, 14, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Khanzada U.S. Patent Application Publication 20220037022 in view of Xing U.S. Patent Application Publication 20180276540 (previously cited), and further in view of Paraskevopoulos et al. U.S. Patent Application Publication 20200335086 (hereinafter, “Paraskevopoulos”).
Regarding claim 4, Khanzada teaches the computer-based diagnostic device of claim 1. Khanzada further teaches machine learning algorithms can potentially be a powerful tool for a preliminary indication of a person's COVID-19 status. Some embodiments implement such models to accurately infer COVID-19 infection from smartphone-acquired voice sounds and images (par [0014], see Khanzada); in some embodiments, the algorithms may be configured to detect and distinguish various diseases, including other coronaviruses such as the flu, common cold, SARS, and COVID- 20, along with respiratory conditions such as whooping cough and asthma (par [0036], see Khanzada). In some embodiments, the way in which audio features are extracted from voice audio files may affect model performance. There are expected to be several useful features to train the network on, such as mel-frequency cepstral coefficients and mel-frequency spectrograms, both being audio features (par [0044], see Khanzada).
However, Khanzada does not explicitly disclose wherein the plurality of spectral feature extraction algorithms includes one or more of a chromagram algorithm, .
Xing teaches modeling of the latent embedding of music using deep neural network (see Title) in which in some embodiments, other time-frequency analysis may be used as known in the art. For example, mel-Frequency Analysis, and/or mel-frequency cepstrum (MFC) or mel-frequency cepstral coefficients analysis (MFCC) (see par [0041], see Xing). Perceptual scale of pitches such as melody spectrogram (MEL-spectrogram) 416 and mel-frequency cepstrum consisting of various Mel-frequency cepstral coefficients (MFCC) 432 may be generated by taking a discrete cosine transformation of the mel logarithmic powers (see par [0050], see Xing). A two channel waveform 440 may be generated as illustrated in FIG. 4C. Additionally, or alternatively, a spectrogram in the linear scale 444 as illustrated in FIG. 4D, a spectrogram on the log scale 448 as illustrated in FIG. 4E, a melodic range linear analysis 452 as illustrated in FIG. 4F, and or a melodic range analysis on the linear scale as illustrated in FIG. 4G may be generated in certain embodiments (see par [0051], see Xing). Other acoustic signal representations such as spectral contrast and music harmonics such as tonal centroid features (tonnetz); all the acoustic signal representations, e.g. chromagram, tempogram, mel-spectrogram, (see par [0052], see Xing).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modeling of the latent embedding of music using deep neural network taught by Xing with the computer-based diagnostic device of Khanzada such that to obtain wherein the plurality of spectral feature extraction algorithms includes one or more of a chromagram algorithm, a spectral contrast algorithm, and a tonal centroid algorithm in order to perform backward propagation to artificially construct an audio hyper-image as suggested by Xing in paragraph [0062].
However, Khanzada in view of Xing does not explicitly disclose wherein the plurality of spectral feature extraction algorithms includes a mel-scaled spectrogram algorithm.
Paraskevopoulos teaches speech data augmentation (see Title) in which feature Extraction and Classification: The data augmentation methods have been evaluated in terms of the classification performance of a CNN. In particular, we have chosen the VGG19 architecture [10], which results in state-of-the-art performance on IEMOCAP. The network takes as input mel-scaled spectrograms, that are extracted from fix-sized segments of 3 seconds, after breaking each spoken utterance (par [0042], see Paraskevopoulos).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the speech data augmentation taught by Paraskevopoulos with the computer-based diagnostic device of Khanzada in view of Xing such that to obtain wherein the plurality of spectral feature extraction algorithms includes a mel-scaled spectrogram algorithm in order to improve classification performance as compared to traditional speech data augmentation method as suggested by Paraskevopoulos in Abstract.
Regarding claim 5, Khanzada in view of Xing in view of Paraskevopoulos teaches the computer-based diagnostic device of claim 4. Khanzada in view of Xing in view of Paraskevopoulos, as modified teaches wherein the plurality of spectral feature extraction algorithms (Khanzada further teaches machine learning algorithms can potentially be a powerful tool for a preliminary indication of a person's COVID-19 status. Some embodiments implement such models to accurately infer COVID-19 infection from smartphone-acquired voice sounds and images (par [0014], see Khanzada); in some embodiments, the algorithms may be configured to detect and distinguish various diseases, including other coronaviruses such as the flu, common cold, SARS, and COVID- 20, along with respiratory conditions such as whooping cough and asthma (par [0036], see Khanzada)) include each of a mel-frequency cepstral coefficient algorithm (In some embodiments, the way in which audio features are extracted from voice audio files may affect model performance. There are expected to be several useful features to train the network on, such as mel-frequency cepstral coefficients and mel-frequency spectrograms, both being audio features (par [0043], see Khanzada)), a chromagram algorithm, a spectral contrast algorithm, and atonal centroid algorithm (Other acoustic signal representations such as spectral contrast and music harmonics such as tonal centroid features (tonnetz); all the acoustic signal representations, e.g. chromagram, tempogram, mel-spectrogram, (see par [0052], see Xing)), and a mel-scaled spectrogram algorithm (In particular, we have chosen the VGG19 architecture [10], which results in state-of-the-art performance on IEMOCAP. The network takes as input mel-scaled spectrograms, that are extracted from fix-sized segments of 3 seconds, after breaking each spoken utterance (par [0042], see Paraskevopoulos)). The motivation is in order to improve classification performance as compared to traditional speech data augmentation method as suggested by Paraskevopoulos in Abstract.
Regarding claim 7, Khanzada in view of Xing in view of Paraskevopoulos teaches the computer-based diagnostic device of claim 4. Khanzada in view of Xing in view of Paraskevopoulos, as modified teaches wherein the two-dimensional numerical arrays (i.e., spectrogram images) are separately provided to the trained neural network model as distinct data inputs (i.e., for each branch) (In some embodiments, the network architecture may use several heterogeneous classifiers and fuse together the extracted high-level features from spectrogram images, using ResNet-50 CNN (convolutional neural networks), and from MFCCs using a deep neural network. The network architecture, number of hidden layers for each branch, and number of units per each layer are hypermaters that may be determined using a grid-search. The model may be trained using categorical cross entropy loss, a stochastic gradient descent optimizer with a learning rate of 1e-2 and 2500 decay steps (par [0052], see Khanzada)). The motivation is in order to improve classification performance as compared to traditional speech data augmentation method as suggested by Paraskevopoulos in Abstract.
Regarding claim 11, this claim has similar limitations as Claim 4 and is therefore rejected for the same reasons.
Regarding claim 12, this claim has similar limitations as Claim 5 and is therefore rejected for the same reasons.
Regarding claim 14, this claim has similar limitations as Claim 7 and is therefore rejected for the same reasons.
Regarding claim 18, this claim has similar limitations as Claim 4 and is therefore rejected for the same reasons.
Regarding claim 19, this claim merely reflects the method to the apparatus claim of Claim 5 and is therefore rejected for the same reasons.
10. Claims 6, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Khanzada U.S. Patent Application Publication 20220037022 in view of Xing U.S. Patent Application Publication 20180276540 (previously cited) in view of Paraskevopoulos et al. U.S. Patent Application Publication 20200335086 (hereinafter, “Paraskevopoulos”), and further in view of Anushiravani et al. U.S. Patent Application Publication 20200388287 (hereinafter, “Anushiravani”, previously cited).
Regarding claim 6, Khanzada in view of Xing in view of Paraskevopoulos teaches the computer-based diagnostic device of claim 4. Khanzada in view of Xing in view of Paraskevopoulos, as modified teaches wherein the values of each of the two-dimensional numerical arrays (In some embodiments, the model is a multi-branch ensemble learning architecture based on a ResNet-50 3D convolutional neural network that is pre-trained on ImageNet dataset and stripped of the top layer (e.g., classification layer). The input for the CNN may be a mel- spectrogram color image of size (224 pixels (i.e., values), 224 pixels, 3 RGB layers, or larger or smaller in any of these dimensions) and the output of the CNN may be passed to both a global average pooling layer and a global maximum pooling layer in two separate and parallel links (par [0048], see Khanzada)).
However, Khanzada in view of Xing in view of Paraskevopoulos does not explicitly disclose represent one or more of respiratory oscillations, pitch content, amplitude of breathing noises, peaks and valleys in the source audio data, and chord sequences in the source audio data.
Anushiravani teaches intelligent health monitoring (see Title) in which referring to FIG. 26, a time-frequency representation of an audio signal such as a spectrogram is shown. It is possible to detect multiple sound events (i.e., source audio data) within the same analysis window such as speech, television sound, cough, kitchen noise, etc. (par [0169], see Anushiravani) represent one or more of respiratory oscillations (see patterns of frame size 302, 303, Fig. 3A, par [0088]; abnormal breathing pattern, par [0135], see Anushiravani), pitch content (wheezing, par [0147], see Anushiravani), amplitude of breathing noises (see amplitude of frame size 302, Fig. 3A, par [0088], see Anushiravani), peaks and valleys in the audio data (see peaks and valleys frame size 302, Fig. 3A, par [0088], see Anushiravani), and chord sequences in the source audio data (see sequences 303 of frame size 302, Fig. 3A, par [0088], see Anushiravani).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the intelligent health monitoring taught by Anushiravani with the computer-based diagnostic device of Khanzada in view of Xing in view of Paraskevopoulos such that to obtain to represent one or more of respiratory oscillations, pitch content, amplitude of breathing noises, peaks and valleys in the source audio data, and chord sequences in the source audio data for purpose of improving a disease state for a user, as suggested by Anushiravani in paragraph [0113].
Regarding claim 13, this claim has similar limitations as Claim 6 and is therefore rejected for the same reasons.
Regarding claim 20, this claim has similar limitations as Claim 6 and is therefore rejected for the same reasons.
Response to Arguments
11. Applicant's arguments with respect to claims 1-20 have been considered but are moot in view of the new grounds of rejection.
Conclusion
12. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhang et al. “Pulmonary disease detection and classification in patient respiratory audio files using long short-term memory neural networks”. Frontiers in Medicine, Nov. 2023, 11 pages. Publication disclosing instant application by the inventors of the instant application.
Applicant's amendment necessitated the new grounds of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CON P TRAN whose telephone number is (571) 272-7532. The examiner can normally be reached M-F (08:30 AM- 05:00 PM) ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VIVIAN C. CHIN can be reached at 571-272-7848. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.P.T/Examiner, Art Unit 2695
/VIVIAN C CHIN/Supervisory Patent Examiner, Art Unit 2695
Prosecution Insights