DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The Information Disclosure Statements (IDS) filed 10/13/2023, 02/12/2024, and 03/18/2025 have been considered by the Examiner except where lined through because the reference was not provided.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 214. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 1 and 15 are objected to because of the following informalities: "for each segment" should read "for each of the plurality of segments" for claim language consistency, "an associated a confidence score" should read "an associated confidence score", "the labels and associated confidence scores" should read "the labels and the associated confidence scores". Appropriate correction is required.
Claim 8 is objected to because of the following informalities: "a processers" should read "a processor", "for each segment" should read "for each of the plurality of segments" for claim language consistency, "an associated a confidence score" should read "an associated confidence score", "the labels and associated confidence scores" should read "the labels and the associated confidence scores". Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-7, 13, and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding Claim 1, the claim recites “storing the labels and associated confidence scores in storage” and later recites “retrieving a subset of the plurality of segments from the storage”. However, the plurality of segments is never stored in storage, only the labels and associated confidence scores are stored in the storage. Therefore, it is unclear how “a subset of the plurality of segments” can be retrieved from the storage if the plurality of segments is never stored in storage. For the purposes of examination, “retrieving a subset of the plurality of segments from the storage” is herein interpreted to be “retrieving a subset of the plurality of segments”. Due to the aforementioned reason, claim 1 is rendered indefinite. Claims 2-7 are rejected due to their dependence on claim 1.
Regarding Claims 5, 13, and 20, the claims recite “confidence scores associated with the plurality of segments”. It is unclear if “confidence scores associated with the plurality of segments” are the same as “associated confidence scores” recited in independent claims 1, 8, and 15, from which claims 5, 13, and 20 are respectively dependent. Therefore, claims 5, 13, and 20 are rendered indefinite. For the purposes of examination, “confidence scores associated with the plurality of segments” are herein interpreted to refer the “associated confidence scores” recited in the respective independent claims. Examiner suggests amending the claim to read “the confidence scores associated with the plurality of segments”.
Regarding Claim 6, the claim recites “the microphone is attached to a stethoscope”. There is insufficient antecedent basis for “the microphone”. Furthermore, claim 6 is a method claim and it is unclear how the recitation of structural components further limits the method of claim 1. Therefore, claim 6 is rendered indefinite. For the purposes of examination, “the microphone is attached to a stethoscope” is herein interpreted to be “the audio data is received from a stethoscope”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 4-9, 12-16, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dockendorf et al. (US Patent Pub. No. 20230293137) hereinafter Dockendorf in view of Shriberg et al. (US Patent No. 10748644) hereinafter Shriberg.
Regarding Claim 1, Dockendorf discloses a method for training an audio-based machine learning model with active learning (the mobile device can have a display screen 204 on which a minimally processed waveform and frequency distribution 206 can be viewed and can be transmitted 206 for further analysis on supplemental compute resources 207 [0028]; Examples include artificial intelligence (AI), neural networks, regression, Markov models, Gaussian mixture models, support vector machines, random forest, and other machine learning (ML) algorithms and methods for the diagnosis, guidance, detection, screening, monitoring, risk analysis, and prognostication, which we henceforth refer to as artificial intelligence or machine learning models. [0031]; fig 1), the method comprising:
receiving audio data corresponding to biological sounds produced by a body of a patient (placing a digital stethoscope on a patient's chest so the body sounds from the heart are transduced and transferred to the point of processing 401 [0036]; In one embodiment, shown in FIG. 6, sound is input 601 [0043]; fig 4);
segmenting the audio data into a plurality of segments (In a further step, the body sounds can be preprocessed and filtered 402 to, for example, enhance the signal-to-noise ratio, alter the sampling rate, change scales, normalize, and/or otherwise prepare the body sounds for processing by the AI. [0036]; Segmentation and localization of the exam data stream using the accelerometer 120 and/or sound data can enable visuals and interactions. [0058]; processed for disease processes of interest 602 [0043]; figs 4 & 6; Examiner notes that segmentation is part of the preprocessing steps 402/602);
executing an audio-based machine learning model on the plurality of segments (The AI can proceed to process 403 the body sound data [0036]; fig 4; element 603 in fig 6), wherein the audio-based machine learning model is configured to output, for each segment, a label of a medical condition and an associated a confidence score (The AI can proceed to process 403 the body sound data to determine a likelihood of one or more pathologies or regress physiologic parameters. For each pathology, a threshold can be used to determine whether further processing is needed depending on the likelihood, probability, or severity 404 of the attribute detected 505, as shown in FIG. 5, associated and the relevant pathology. [0036]; Automatic segmentation of the periodic signals used in conjunction with the aggregated attribution information provides a discrete list of rankable instances. This can be preferred for lengthy signals and affords the ability to, per period, assign confidence values [0042]; The AI's predicted likelihood of disease (step 603) [0043]; Adding classification to the segmentation and localization provides the ability to visually label each site as appropriate with detected conditions. [0058]; figs 4 & 6);
storing the labels and associated confidence scores in storage (The processing results can be stored 408 for later review, archival purposes, or to prevent reprocessing of the data. [0036]; fig 4);
and training the audio-based machine learning model via active learning (AI decision attribution can build trust with the user 405, 406, result in effective use of time, enable the user to make accuracy decisions using prior knowledge, and automatically annotate data for future AI training. [0041]; fig 4), wherein the training includes:
retrieving a subset of the plurality of segments (The processing results can be stored 408 for later review [0036]; The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; fig 4);
receiving annotations from a human annotator, wherein the annotations are associated with medical conditions (AI decision attribution can build trust with the user 405, 406, result in effective use of time, enable the user to make accuracy decisions using prior knowledge, and automatically annotate data for future AI training [0041]; The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. This pre-demarcation can reduce the time required to create labeled datasets and enables AI- and human-in-the-loop iteration for labeling the dataset where the human identifies examples, edge cases, and errors and the AI can label the bulk of the data from what it has learned so far. [0060]; fig 4);
and training the audio-based machine learning model based on the annotations to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sounds (Here, AI attribution methodologies are additionally applied for the development of a novel disease process isolation and noise reduction algorithm and workflow (FIG. 6). While illustrated with attribution methods, these novel workflows can also be built upon back calculations of attention, disambiguated latent variables, secondary models specifically trained for explanatory AI, or other forward calculations of input data importance. [0041]; In one embodiment, after acquisition 401, 601 and processing of the data 403, 405, 601-604 as shown in FIGS. 4 and 6, the user is presented with the detected pathologies or conditions. [0042]; The AI's predicted likelihood of disease (step 603) and the integrated gradients method of attribution (step 505, in FIG. 5 and step 604 in FIG. 6) of this likelihood to the source sub-bands are used to weight the filter bank components like a common audio equalizer being continuously adjusted to amplify sounds related to the disease process. [0043]; figs 4 & 6).
Dockendorf fails to disclose training the audio-based machine learning model via active learning, wherein the training includes:
training the audio-based machine learning model based on the annotations until convergence to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sounds.
However, Shriberg teaches training an active learning model until convergence (An active learning model, by contrast, may employ a human to converge more quickly. The active learner may ask targeted questions to the human in order to do this. For example, a machine learning algorithm may be employed on a large amount of unlabeled audio samples. (col. 76, lines 55-59)).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the method of Dockendorf such that training the audio-based machine learning model via active learning, wherein the training includes: training the audio-based machine learning model based on the annotations until convergence to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sound, as taught by Shriberg, because training an active learning model until convergence minimizes loss and produces a stable model. The use of a known technique to improve similar devices (methods or products) in the same way is likely to be obvious. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 415-421, USPQ2d 1385, 1395 – 97 (2007) (see MPEP § 2143, C.).
Regarding Claim 4, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 1. Dockendorf further discloses wherein the plurality of segments are each associated with heartbeats (The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; The system and the digital stethoscope device used therewith are not limited to use with a single organ or system; rather, body sounds 201 can be detected nearly everywhere on a human body 301, though, many optimal locations can be utilized for detecting specific sounds. For example, sounds generated by the heart could be obtained by making observations near the organ [0029]; FIG. 11 illustrates an embodiment of the segmentation of the acceleration signals 1101 that uses a detector bank 1102 that outputs time-varying probability signals for sensing a heartbeat 1103 or relocation 1104, non-target body sounds 1105 (along with any number of other random body sound detectors 1106, e.g., cough that can be analyzed by other processing systems 1108. [0057]), the predicted medical condition includes a heart murmur (The AI can estimate 1305 the PAP from the sounds and other data collected. Additionally, the AI can directly estimate the need for a change in medication 1305… Estimating blood pressure, pulmonary artery pressure and varying intensity of heart valve murmurs will allow the user to optimize the use of medications [0071]; fig 13).
Regarding Claim 5, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 1. Dockendorf further discloses the subset of the plurality of segments are selected for retrieval based upon confidence scores associated with the plurality of segments being under a threshold (In one embodiment, if the probability does not exceed the threshold for detection, no attribution is computed and instead the low probability can be reported to the user 407. The processing results can be stored 408 for later review, archival purposes, or to prevent reprocessing of the data. [0036]; after acquisition 401, 601 and processing of the data 403, 405, 601-604 as shown in FIGS. 4 and 6, the user is presented with the detected pathologies or conditions... For each condition under inspection, the original and/or transformed representations of the input signal are shown with regions of interest highlighted for further scrutiny by a user or expert (FIG. 5). [0042]).
Regarding Claim 6, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 1. Dockendorf further discloses the audio data is received from a stethoscope (acquiring body sounds 201 can be initiated by placing a digital stethoscope on a patient's chest so the body sounds from the heart are transduced and transferred to the point of processing 401 [0036]; fig 4).
Regarding Claim 7, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 1. Dockendorf further discloses the audio data is a continuous stream of audio data (Segmentation and localization of the exam data stream using the accelerometer 120 and/or sound data can enable visuals and interactions. For example, playback of a data stream containing multiple recording sites can be started. In one embodiment, a user can listen to the continuous sound stream [0058]).
Regarding Claim 8, Dockendorf discloses a system comprising:
a processor (a cellular phone 1402 [0071]; figs 14 & 15; Examiner notes a cellular phone has a processor);
and a non-transitory memory coupled to the processor comprising instructions executable by the processor (a mobile application 1302 on a cellular phone 1402 [0071]; The mobile application can also show the PCG data 1410 that was obtained, when sufficient data is collected 1304, the data can be sent to a supplemental compute resource 207, such as, for example, a remote computer system for analysis and AI processing. [0071]; figs 14 & 15; Examiner notes a cellular phone has a non-transitory memory coupled to a processor), the processor operable when executing the instructions to:
receive audio data generated from a microphone and corresponding to biological sounds produced by a body of a patient (mobile application 1302 on a cellular phone 1402 that can be utilized by a user to obtain data with a digital stethoscope 100 [0071]; fig 15);
segment the audio data into a plurality of segments (In a further step, the body sounds can be preprocessed and filtered 402 to, for example, enhance the signal-to-noise ratio, alter the sampling rate, change scales, normalize, and/or otherwise prepare the body sounds for processing by the AI. [0036]; Segmentation and localization of the exam data stream using the accelerometer 120 and/or sound data can enable visuals and interactions. [0058]; processed for disease processes of interest 602 [0043]; figs 4 & 6; Examiner notes that segmentation is part of the preprocessing steps 402/602);
execute an audio-based machine learning model on the plurality of segments (The AI can proceed to process 403 the body sound data [0036]; fig 4; element 603 in fig 6), wherein the audio-based machine learning model is configured to output, for each segment, a label of a medical condition and an associated a confidence score (The AI can proceed to process 403 the body sound data to determine a likelihood of one or more pathologies or regress physiologic parameters. For each pathology, a threshold can be used to determine whether further processing is needed depending on the likelihood, probability, or severity 404 of the attribute detected 505, as shown in FIG. 5, associated and the relevant pathology. [0036]; Automatic segmentation of the periodic signals used in conjunction with the aggregated attribution information provides a discrete list of rankable instances. This can be preferred for lengthy signals and affords the ability to, per period, assign confidence values [0042]; The AI's predicted likelihood of disease (step 603) [0043]; Adding classification to the segmentation and localization provides the ability to visually label each site as appropriate with detected conditions. [0058]; figs 4 & 6);
store the labels and associated confidence scores (The processing results can be stored 408 for later review, archival purposes, or to prevent reprocessing of the data. [0036]; fig 4);
and train the audio-based machine learning model via active learning, wherein the training includes (AI decision attribution can build trust with the user 405, 406, result in effective use of time, enable the user to make accuracy decisions using prior knowledge, and automatically annotate data for future AI training. [0041]; fig 4):
retrieving a subset of the plurality of segments (The processing results can be stored 408 for later review [0036]; The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; fig 4);
receiving annotations from a human annotator, wherein the annotations are associated with medical conditions (AI decision attribution can build trust with the user 405, 406, result in effective use of time, enable the user to make accuracy decisions using prior knowledge, and automatically annotate data for future AI training [0041]; The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. This pre-demarcation can reduce the time required to create labeled datasets and enables AI- and human-in-the-loop iteration for labeling the dataset where the human identifies examples, edge cases, and errors and the AI can label the bulk of the data from what it has learned so far. [0060]);
and training the audio-based machine learning model based on the annotations to yield a trained sound-based machine learning model configured to output a predicted medical condition (Here, AI attribution methodologies are additionally applied for the development of a novel disease process isolation and noise reduction algorithm and workflow (FIG. 6). While illustrated with attribution methods, these novel workflows can also be built upon back calculations of attention, disambiguated latent variables, secondary models specifically trained for explanatory AI, or other forward calculations of input data importance. [0041]; In one embodiment, after acquisition 401, 601 and processing of the data 403, 405, 601-604 as shown in FIGS. 4 and 6, the user is presented with the detected pathologies or conditions. [0042]; The AI's predicted likelihood of disease (step 603) and the integrated gradients method of attribution (step 505, in FIG. 5 and step 604 in FIG. 6) of this likelihood to the source sub-bands are used to weight the filter bank components like a common audio equalizer being continuously adjusted to amplify sounds related to the disease process. [0043]; figs 4 & 6).
Dockendorf fails to disclose training the audio-based machine learning model via active learning, wherein the training includes:
training the audio-based machine learning model based on the annotations until convergence to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sounds.
However, Shriberg teaches training an active learning model until convergence (An active learning model, by contrast, may employ a human to converge more quickly. The active learner may ask targeted questions to the human in order to do this. For example, a machine learning algorithm may be employed on a large amount of unlabeled audio samples. (col. 76, lines 55-59)).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the system of Dockendorf such that training the audio-based machine learning model via active learning, wherein the training includes: training the audio-based machine learning model based on the annotations until convergence to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sound, as taught by Shriberg, because training an active learning model until convergence minimizes loss and produces a stable model. The use of a known technique to improve similar devices (methods or products) in the same way is likely to be obvious. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 415-421, USPQ2d 1385, 1395 – 97 (2007) (see MPEP § 2143, C.).
Regarding Claim 9, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 8. Dockendorf further discloses the audio data is a continuous stream of audio data (Segmentation and localization of the exam data stream using the accelerometer 120 and/or sound data can enable visuals and interactions. For example, playback of a data stream containing multiple recording sites can be started. In one embodiment, a user can listen to the continuous sound stream [0058]).
Regarding Claim 12, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 8. Dockendorf further discloses wherein the plurality of segments are each associated with heartbeats (The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; The system and the digital stethoscope device used therewith are not limited to use with a single organ or system; rather, body sounds 201 can be detected nearly everywhere on a human body 301, though, many optimal locations can be utilized for detecting specific sounds. For example, sounds generated by the heart could be obtained by making observations near the organ [0029]; FIG. 11 illustrates an embodiment of the segmentation of the acceleration signals 1101 that uses a detector bank 1102 that outputs time-varying probability signals for sensing a heartbeat 1103 or relocation 1104, non-target body sounds 1105 (along with any number of other random body sound detectors 1106, e.g., cough that can be analyzed by other processing systems 1108. [0057]), the predicted medical condition includes a murmur (The AI can estimate 1305 the PAP from the sounds and other data collected. Additionally, the AI can directly estimate the need for a change in medication 1305… Estimating blood pressure, pulmonary artery pressure and varying intensity of heart valve murmurs will allow the user to optimize the use of medications [0071]; fig 13).
Regarding Claim 13, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 8. Dockendorf further discloses the subset of the plurality of segments are selected for retrieval based upon confidence scores associated with the plurality of segments being under a threshold (In one embodiment, if the probability does not exceed the threshold for detection, no attribution is computed and instead the low probability can be reported to the user 407. The processing results can be stored 408 for later review, archival purposes, or to prevent reprocessing of the data. [0036]; after acquisition 401, 601 and processing of the data 403, 405, 601-604 as shown in FIGS. 4 and 6, the user is presented with the detected pathologies or conditions... For each condition under inspection, the original and/or transformed representations of the input signal are shown with regions of interest highlighted for further scrutiny by a user or expert (FIG. 5). [0042]).
Regarding Claim 14, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 8. Dockendorf discloses the microphone is attached to a stethoscope (mobile application 1302 on a cellular phone 1402 that can be utilized by a user to obtain data with a digital stethoscope 100 [0071]; fig 15).
Regarding Claim 15, Dockendorf discloses a computer-readable non-transitory storage medium embodying software that is operable (a mobile application 1302 on a cellular phone 1402 [0071]; The mobile application can also show the PCG data 1410 that was obtained, when sufficient data is collected 1304, the data can be sent to a supplemental compute resource 207, such as, for example, a remote computer system for analysis and AI processing. [0071]; figs 14 & 15; Examiner notes a cellular phone a computer-readable non-transitory storage medium embodying software), when executed, to:
receive audio data generated from a microphone and corresponding to biological sounds produced by a body of a patient (placing a digital stethoscope on a patient's chest so the body sounds from the heart are transduced and transferred to the point of processing 401 [0036]; In one embodiment, shown in FIG. 6, sound is input 601 [0043]; fig 4);
segment the audio data into a plurality of segments (In a further step, the body sounds can be preprocessed and filtered 402 to, for example, enhance the signal-to-noise ratio, alter the sampling rate, change scales, normalize, and/or otherwise prepare the body sounds for processing by the AI. [0036]; Segmentation and localization of the exam data stream using the accelerometer 120 and/or sound data can enable visuals and interactions. [0058]; processed for disease processes of interest 602 [0043]; figs 4 & 6; Examiner notes that segmentation is part of the preprocessing steps 402/602),
execute an audio-based machine learning model on the plurality of segments (The AI can proceed to process 403 the body sound data [0036]; fig 4; element 603 in fig 6), wherein the audio-based machine learning model is configured to output, for each segment, a label of a medical condition and an associated a confidence score (The AI can proceed to process 403 the body sound data to determine a likelihood of one or more pathologies or regress physiologic parameters. For each pathology, a threshold can be used to determine whether further processing is needed depending on the likelihood, probability, or severity 404 of the attribute detected 505, as shown in FIG. 5, associated and the relevant pathology. [0036]; Automatic segmentation of the periodic signals used in conjunction with the aggregated attribution information provides a discrete list of rankable instances. This can be preferred for lengthy signals and affords the ability to, per period, assign confidence values [0042]; The AI's predicted likelihood of disease (step 603) [0043]; Adding classification to the segmentation and localization provides the ability to visually label each site as appropriate with detected conditions. [0058]; figs 4 & 6);
store the labels and associated confidence scores (The processing results can be stored 408 for later review, archival purposes, or to prevent reprocessing of the data. [0036]; fig 4);
and train the audio-based machine learning model via active learning (AI decision attribution can build trust with the user 405, 406, result in effective use of time, enable the user to make accuracy decisions using prior knowledge, and automatically annotate data for future AI training. [0041]; fig 4), wherein the training includes:
retrieving a subset of the plurality of segments (The processing results can be stored 408 for later review [0036]; The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; fig 4);
receiving annotations from a human annotator, wherein the annotations are associated with medical conditions (AI decision attribution can build trust with the user 405, 406, result in effective use of time, enable the user to make accuracy decisions using prior knowledge, and automatically annotate data for future AI training [0041]; The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. This pre-demarcation can reduce the time required to create labeled datasets and enables AI- and human-in-the-loop iteration for labeling the dataset where the human identifies examples, edge cases, and errors and the AI can label the bulk of the data from what it has learned so far. [0060]; fig 4);
and training the audio-based machine learning model based on the annotations to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sounds (Here, AI attribution methodologies are additionally applied for the development of a novel disease process isolation and noise reduction algorithm and workflow (FIG. 6). While illustrated with attribution methods, these novel workflows can also be built upon back calculations of attention, disambiguated latent variables, secondary models specifically trained for explanatory AI, or other forward calculations of input data importance. [0041]; In one embodiment, after acquisition 401, 601 and processing of the data 403, 405, 601-604 as shown in FIGS. 4 and 6, the user is presented with the detected pathologies or conditions. [0042]; The AI's predicted likelihood of disease (step 603) and the integrated gradients method of attribution (step 505, in FIG. 5 and step 604 in FIG. 6) of this likelihood to the source sub-bands are used to weight the filter bank components like a common audio equalizer being continuously adjusted to amplify sounds related to the disease process. [0043]; figs 4 & 6).
Dockendorf fails to disclose training the audio-based machine learning model via active learning, wherein the training includes:
training the audio-based machine learning model based on the annotations until convergence to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sounds.
However, Shriberg teaches training an active learning model until convergence (An active learning model, by contrast, may employ a human to converge more quickly. The active learner may ask targeted questions to the human in order to do this. For example, a machine learning algorithm may be employed on a large amount of unlabeled audio samples. (col. 76, lines 55-59)).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the storage medium of Dockendorf such that training the audio-based machine learning model via active learning, wherein the training includes: training the audio-based machine learning model based on the annotations until convergence to yield a trained sound-based machine learning model configured to output a predicted medical condition associated with input biological sound, as taught by Shriberg, because training an active learning model until convergence minimizes loss and produces a stable model. The use of a known technique to improve similar devices (methods or products) in the same way is likely to be obvious. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 415-421, USPQ2d 1385, 1395 – 97 (2007) (see MPEP § 2143, C.).
Regarding Claim 16, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 15. Dockendorf further discloses the audio data is a continuous stream of audio data (Segmentation and localization of the exam data stream using the accelerometer 120 and/or sound data can enable visuals and interactions. For example, playback of a data stream containing multiple recording sites can be started. In one embodiment, a user can listen to the continuous sound stream [0058]).
Regarding Claim 19, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 15. Dockendorf further discloses wherein the plurality of segments are each associated with heartbeats (The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; The system and the digital stethoscope device used therewith are not limited to use with a single organ or system; rather, body sounds 201 can be detected nearly everywhere on a human body 301, though, many optimal locations can be utilized for detecting specific sounds. For example, sounds generated by the heart could be obtained by making observations near the organ [0029]; FIG. 11 illustrates an embodiment of the segmentation of the acceleration signals 1101 that uses a detector bank 1102 that outputs time-varying probability signals for sensing a heartbeat 1103 or relocation 1104, non-target body sounds 1105 (along with any number of other random body sound detectors 1106, e.g., cough that can be analyzed by other processing systems 1108. [0057]), the predicted medical condition includes a heart murmur (The AI can estimate 1305 the PAP from the sounds and other data collected. Additionally, the AI can directly estimate the need for a change in medication 1305… Estimating blood pressure, pulmonary artery pressure and varying intensity of heart valve murmurs will allow the user to optimize the use of medications [0071]; fig 13).
Regarding Claim 20, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 15. Dockendorf further discloses the subset of the plurality of segments are selected for retrieval based upon confidence scores associated with the plurality of segments being under a threshold (In one embodiment, if the probability does not exceed the threshold for detection, no attribution is computed and instead the low probability can be reported to the user 407. The processing results can be stored 408 for later review, archival purposes, or to prevent reprocessing of the data. [0036]; after acquisition 401, 601 and processing of the data 403, 405, 601-604 as shown in FIGS. 4 and 6, the user is presented with the detected pathologies or conditions... For each condition under inspection, the original and/or transformed representations of the input signal are shown with regions of interest highlighted for further scrutiny by a user or expert (FIG. 5). [0042]).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dockendorf (US Patent Pub. No. 20230293137) in view of Shriberg (US Patent No. 10748644) as applied to claim 1 above, and further in view of Kirkpatrick et al. (US Patent Pub. No. 20240252139) hereinafter Kirkpatrick.
Regarding Claim 2, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 1. Dockendorf discloses each of the plurality of segments is associated with a respective audio event corresponding to an audio-based biomarker (FIG. 11 illustrates an embodiment of the segmentation of the acceleration signals 1101 that uses a detector bank 1102 that outputs time-varying probability signals for sensing a heartbeat 1103 or relocation 1104, non-target body sounds 1105 (along with any number of other random body sound detectors 1106, e.g., cough that can be analyzed by other processing systems 1108. Additional detectors processing other simultaneously acquired signals (for example, sound) can also be used to generate various other probabilities. [0057]; An AI model can be applied to the data sequence to find, classify, and separate the signal components due to each event. The AI can produce a signal for each component that is separated. The classifier labels event onsets and enables aggregation and display of exam statistics. The classifier can trigger further AI processing of the sounds, for example, of a cough, which have shown to contain important health information in conditions like pneumonia or COVID-19. The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]).
Dockendorf in view of Shriberg fails to teach the segmenting is performed via a pre- trained audio neural network (PANN).
However, Kirkpatrick teaches using a pre-trained audio neural network (PANN) to distinguish between audio events corresponding to respiratory sounds and non-respiratory sounds (the present embodiments are directed to a method of training a machine learning algorithm to identify lung sounds including: obtaining heart sound recordings corresponding to known lung conditions and corresponding to normal healthy lung function; obtaining non-lung sound recordings; providing the lung sound recordings and non-lung sound recordings to an audio classifier model; and using the audio classifier model to train the machine learning algorithm to distinguish between the lung sound recordings and non-lung sound recordings, wherein the heart sound recordings are obtained from recordings performed by the electronic stethoscope of claim 1, by alternative electronic stethoscopes, by alternative microphones, and by smartphone microphones, wherein non-lung sound recordings may include sounds such as speech, vehicle traffic, dogs barking, children crying, noise, and music, wherein the audio classifier model comprises a pre-trained audio neural network (PANN) [0028]).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the method of Dockendorf in view of Shriberg such that the segmenting is performed via a pre- trained audio neural network (PANN) instead of a detector bank. The simple substitution of one known element for another is likely to be obvious when predictable results are achieved. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 415-421, USPQ2d 1385, 1395 – 97 (2007) (see MPEP § 2143, B.).
Claim(s) 10 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dockendorf (US Patent Pub. No. 20230293137) in view of Shriberg (US Patent No. 10748644) as applied to claim 9 and 16 above, and further in view of Kirkpatrick (US Patent Pub. No. 20240252139).
Regarding Claim 10, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 9. Dockendorf discloses each of the plurality of segments is associated with a respective audio event corresponding to an audio-based biomarker (FIG. 11 illustrates an embodiment of the segmentation of the acceleration signals 1101 that uses a detector bank 1102 that outputs time-varying probability signals for sensing a heartbeat 1103 or relocation 1104, non-target body sounds 1105 (along with any number of other random body sound detectors 1106, e.g., cough that can be analyzed by other processing systems 1108. Additional detectors processing other simultaneously acquired signals (for example, sound) can also be used to generate various other probabilities. [0057]; An AI model can be applied to the data sequence to find, classify, and separate the signal components due to each event. The AI can produce a signal for each component that is separated. The classifier labels event onsets and enables aggregation and display of exam statistics. The classifier can trigger further AI processing of the sounds, for example, of a cough, which have shown to contain important health information in conditions like pneumonia or COVID-19. The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]).
Dockendorf in view of Shriberg fails to teach the segmenting of the audio data is performed via a pre- trained audio neural network (PANN).
However, Kirkpatrick teaches using a pre-trained audio neural network (PANN) to distinguish between audio events corresponding to respiratory sounds and non-respiratory sounds (the present embodiments are directed to a method of training a machine learning algorithm to identify lung sounds including: obtaining heart sound recordings corresponding to known lung conditions and corresponding to normal healthy lung function; obtaining non-lung sound recordings; providing the lung sound recordings and non-lung sound recordings to an audio classifier model; and using the audio classifier model to train the machine learning algorithm to distinguish between the lung sound recordings and non-lung sound recordings, wherein the heart sound recordings are obtained from recordings performed by the electronic stethoscope of claim 1, by alternative electronic stethoscopes, by alternative microphones, and by smartphone microphones, wherein non-lung sound recordings may include sounds such as speech, vehicle traffic, dogs barking, children crying, noise, and music, wherein the audio classifier model comprises a pre-trained audio neural network (PANN) [0028]).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the system of Dockendorf in view of Shriberg such that the segmenting of the audio data is performed via a pre- trained audio neural network (PANN) instead of a detector bank. The simple substitution of one known element for another is likely to be obvious when predictable results are achieved. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 415-421, USPQ2d 1385, 1395 – 97 (2007) (see MPEP § 2143, B.).
Regarding Claim 17, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 16. Dockendorf discloses each of the plurality of segments is associated with a respective audio event corresponding to an audio-based biomarker (FIG. 11 illustrates an embodiment of the segmentation of the acceleration signals 1101 that uses a detector bank 1102 that outputs time-varying probability signals for sensing a heartbeat 1103 or relocation 1104, non-target body sounds 1105 (along with any number of other random body sound detectors 1106, e.g., cough that can be analyzed by other processing systems 1108. Additional detectors processing other simultaneously acquired signals (for example, sound) can also be used to generate various other probabilities. [0057]; An AI model can be applied to the data sequence to find, classify, and separate the signal components due to each event. The AI can produce a signal for each component that is separated. The classifier labels event onsets and enables aggregation and display of exam statistics. The classifier can trigger further AI processing of the sounds, for example, of a cough, which have shown to contain important health information in conditions like pneumonia or COVID-19. The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]).
Dockendorf in view of Shriberg fails to teach the segmenting of the audio data is performed via a pre- trained audio neural network (PANN).
However, Kirkpatrick teaches using a pre-trained audio neural network (PANN) to distinguish between audio events corresponding to respiratory sounds and non-respiratory sounds (the present embodiments are directed to a method of training a machine learning algorithm to identify lung sounds including: obtaining heart sound recordings corresponding to known lung conditions and corresponding to normal healthy lung function; obtaining non-lung sound recordings; providing the lung sound recordings and non-lung sound recordings to an audio classifier model; and using the audio classifier model to train the machine learning algorithm to distinguish between the lung sound recordings and non-lung sound recordings, wherein the heart sound recordings are obtained from recordings performed by the electronic stethoscope of claim 1, by alternative electronic stethoscopes, by alternative microphones, and by smartphone microphones, wherein non-lung sound recordings may include sounds such as speech, vehicle traffic, dogs barking, children crying, noise, and music, wherein the audio classifier model comprises a pre-trained audio neural network (PANN) [0028]).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the storage medium of Dockendorf in view of Shriberg such that the segmenting of the audio data is performed via a pre- trained audio neural network (PANN) instead of a detector bank. The simple substitution of one known element for another is likely to be obvious when predictable results are achieved. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 415-421, USPQ2d 1385, 1395 – 97 (2007) (see MPEP § 2143, B.).
Claim(s) 3, 11, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Dockendorf (US Patent Pub. No. 20230293137) in view of Shriberg (US Patent No. 10748644) as applied to claims 1, 8, and 15 above, and further in view of Emokpae et al. (US Patent Pub. No. 20230008860) hereinafter Emokpae.
Regarding Claim 3, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 1. Dockendorf further discloses wherein the plurality of segments are each associated with breathing (The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; The system and the digital stethoscope device used therewith are not limited to use with a single organ or system; rather, body sounds 201 can be detected nearly everywhere on a human body 301, though, many optimal locations can be utilized for detecting specific sounds...if breathing or, specifically, the bronchi of the lungs were of interest, observations can be made at sites shown with the triangles 304, 305 in FIG. 3. [0029]), the predicted medical condition is respiratory in nature (Embodiments of the subject invention pertain to devices and methods for providing guidance to a user for obtaining sounds from the body of a patient, collecting the sounds obtained, and analyzing the sounds to determine a physical condition of the patient. [0003]; auditory signals are indicative of various conditions…if breathing or, specifically, the bronchi of the lungs were of interest, observations can be made at sites shown with the triangles 304, 305 in FIG. 3. [0029]).
Dockendorf in view of Shriberg fails to teach the predicted medical condition includes asthma.
However, Emokpae teaches audio data associated with breathing can be analyzed to detect asthma (Therefore, acoustic sensors are primarily used in tracking respiratory related symptoms. Cough and breathing sounds are analyzed to detect pulmonary related disease, namely, asthma, pneumonia, and lung inflammation. [0016]).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the method of Dockendorf in view of Shriberg such that wherein the plurality of segments are each associated with breathing, the predicted medical condition includes asthma, as taught by Emokpae, because the plurality of segments of Dockendorf in view of Shriberg are each associated with breathing and Emokpae teaches detecting asthma from audio data associated with breathing.
Regarding Claim 11, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 8. Dockendorf further discloses wherein the plurality of segments are each associated with breathing (The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; The system and the digital stethoscope device used therewith are not limited to use with a single organ or system; rather, body sounds 201 can be detected nearly everywhere on a human body 301, though, many optimal locations can be utilized for detecting specific sounds...if breathing or, specifically, the bronchi of the lungs were of interest, observations can be made at sites shown with the triangles 304, 305 in FIG. 3. [0029]), the predicted medical condition is respiratory in nature (Embodiments of the subject invention pertain to devices and methods for providing guidance to a user for obtaining sounds from the body of a patient, collecting the sounds obtained, and analyzing the sounds to determine a physical condition of the patient. [0003]; auditory signals are indicative of various conditions…if breathing or, specifically, the bronchi of the lungs were of interest, observations can be made at sites shown with the triangles 304, 305 in FIG. 3. [0029]).
Dockendorf in view of Shriberg fails to teach the predicted medical condition includes asthma.
However, Emokpae teaches audio data associated with breathing can be analyzed to detect asthma (Therefore, acoustic sensors are primarily used in tracking respiratory related symptoms. Cough and breathing sounds are analyzed to detect pulmonary related disease, namely, asthma, pneumonia, and lung inflammation. [0016]).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the system of Dockendorf in view of Shriberg such that wherein the plurality of segments are each associated with breathing, the predicted medical condition includes asthma, as taught by Emokpae, because the plurality of segments of Dockendorf in view of Shriberg are each associated with breathing and Emokpae teaches detecting asthma from audio data associated with breathing.
Regarding Claim 18, Dockendorf in view of Shriberg teaches the invention as discussed above in claim 15. Dockendorf further discloses wherein the plurality of segments are each associated with breathing (The segmentation of accelerometer and/or sound data into each discrete event and/or repeating events can be labeled for training AI models. [0060]; The system and the digital stethoscope device used therewith are not limited to use with a single organ or system; rather, body sounds 201 can be detected nearly everywhere on a human body 301, though, many optimal locations can be utilized for detecting specific sounds...if breathing or, specifically, the bronchi of the lungs were of interest, observations can be made at sites shown with the triangles 304, 305 in FIG. 3. [0029]), the predicted medical condition is respiratory in nature (Embodiments of the subject invention pertain to devices and methods for providing guidance to a user for obtaining sounds from the body of a patient, collecting the sounds obtained, and analyzing the sounds to determine a physical condition of the patient. [0003]; auditory signals are indicative of various conditions…if breathing or, specifically, the bronchi of the lungs were of interest, observations can be made at sites shown with the triangles 304, 305 in FIG. 3. [0029]).
Dockendorf in view of Shriberg fails to teach the predicted medical condition includes asthma.
However, Emokpae teaches audio data associated with breathing can be analyzed to detect asthma (Therefore, acoustic sensors are primarily used in tracking respiratory related symptoms. Cough and breathing sounds are analyzed to detect pulmonary related disease, namely, asthma, pneumonia, and lung inflammation. [0016]).
It would have been obvious to one having ordinary skill in the art at the time of the effective filing date to have modified the storage medium of Dockendorf in view of Shriberg such that wherein the plurality of segments are each associated with breathing, the predicted medical condition includes asthma, as taught by Emokpae, because the plurality of segments of Dockendorf in view of Shriberg are each associated with breathing and Emokpae teaches detecting asthma from audio data associated with breathing.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANKI M BAVA whose telephone number is (571)272-0416. The examiner can normally be reached Monday-Friday 9:00-6:00 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Sims can be reached at 571-272-7540. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANKI M BAVA/Examiner, Art Unit 3791
/ETSUB D BERHANU/Primary Examiner, Art Unit 3791
Prosecution Insights