DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
Applicant is reminded of the continuing obligation under 37 CFR 1.56, to timely apprise the Office of any information which is material to patentability of the claims under consideration in this application.
Specification
The disclosure is objected to because it contains an embedded hyperlink and/or other form of browser-executable code. Applicant is required to delete the embedded hyperlink and/or other form of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant' s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Objections
Claim 58 is objected to because of the following informalities: the recitation of “sound record.” should instead read --sound records.--. Further, the claim uses the term IMAGENET, which is a trade name or mark used in commerce. It should be (1a) in ALL CAPS or (1b) accompanied by the appropriate symbol (™, SM, or ®) wherever it appears, and be (2) accompanied by the generic terminology. See MPEP 608.01(v).
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “computing device” in claims 61, 65, and 70.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof (e.g. for “computing device,” a phone or computer as shown in Fig. 1).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 76 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 76, there is no support for a computer-readable storage medium storing instructions for execution by a processor.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 57-76 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 57, 75, and 76, the recitations of “the first sound record being separate from sound records in the first number of respiratory sound records” is unclear at least because there is no previous recitation of “the first number of respiratory sound records.” Further, it is self-referential to receive a first sound record that is separate from first sound records. It may be that Applicant intends to say that the first sound record is separate from the second number of respiratory sound records.
Further regarding claims 57, 75, and 76, the recitations of “the respiratory sounds acquired by auscultation” in the “applying” steps are unclear. Are the sounds from the second number of respiratory sound records, or the first sound record?
Regarding claims 66 and 67, the recitation of “a graphical representation of the first sound” is unclear because this already appears to be recited in claim 57. In claim 67, it is unclear to which graphical representation reference is being made.
Claims 58-74 are rejected because they depend on rejected claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 57, 61, 62, 65, 66, 68-71, and 73-76 are rejected under 35 U.S.C. 103 as being unpatentable over International Application Publication WO 2014/107798 (“Alshaer”), in view of US Patent Application Publication 2020/0093455 (“Wang”), US Patent Application Publication 2021/0361227 (“Chou”), and US Patent 5,218,969 (“Bredesen”).
Regarding claim 57, Alshaer teaches a computer-implemented method (method using a device; Abstract) comprising: …; receiving a second number of respiratory sound records, each of the respiratory sound records representing respiratory sounds acquired by auscultation, and each of the respiratory sound records having … one or more known sound classes (¶¶s 00187, 00202, 00235, etc.); …; receiving a first sound record representing the respiratory sounds of a subject (“sound and/or airflow produced by the subject while breathing”; ¶ 0011) acquired by auscultation, the first sound record being separate from sound records in the [second] number of respiratory sound records (“transducer responsive to sound and/or airflow”; ¶ 0011); transforming the received first sound record into a time-frequency domain graphical representation (Figure 25 and ¶ 00202); applying the time-frequency domain graphical representation to [a] neural network (“comparative process”; ¶ 00207 - also see ¶ 00249, artificial neural network) to determine a sound class for the respiratory sounds acquired by auscultation (stable or collapsible airway; ¶ 00207); and inferring a respiratory condition of the subject (“CSA (or normal) vs. OSA”; ¶ 00207) based at least on the sound class determined by the neural network (¶¶s 00207 and 00249).
Alshaer does not appear to explicitly teach pre-training initial convolutional layers of a neural network using supervised learning based on a first number of images unrelated to sound records; each of the respiratory sound records having a label selected from labels representing one or more known sound classes (although it does describe using a known data set for classifier training); and using a transfer-learning approach, training the neural network using the second number of respiratory sound records and the known sound classes, wherein the first number of images is larger than the second number of respiratory sound records.
Wang teaches using a deep convolutional neural network to classify medical images (¶ 0045). For the classification, a pre-trained ImageNet model (unrelated to sound records) is adapted, via transfer learning, to a more complete model (because knowledge obtained from ImageNet pre-training is used to train the more complete model as shown in Fig. 4) via the addition of a transition layer, a global pooling layer, a prediction layer, and a loss layer (¶ 0045). Pre-training via supervised learning occurred to obtain the ImageNet models, since the ImageNet data is labeled. The more complete model of Wang is considered a weakly-supervised multi-label image classification model (¶¶s 0044, 0045, Abstract, etc.), trained on labeled images (Abstract, Fig. 4, left frame, ¶ 0053).
Chou teaches processing spectrograms (¶¶s 0024, 0036, etc.) using a convolutional neural network (¶¶s 0024, 0048, 0071, etc.). There is a pre-training step based on unlabeled non-patient sound records, and then a supervised training step using labeled patient sound records (¶ 0041, the pre-training reducing the number of labeled patient sound records required).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to make the images of Alshaer correspond to a “known” data set by incorporating labels, as in Wang, since this is a recognized way of knowing what is in a data set, without requiring any further processing and classification, and for the purpose of enabling weakly-supervised classification as contemplated by Wang (¶ 0045). It would have been obvious to use these images in a transfer learning approach to train a convolutional neural network (Wang: Fig. 4, ¶ 0053), including pre-training initial layers using supervised learning based on images unrelated to sound records (i.e., ImageNet), as in Wang (Fig. 4), for the purpose of being able to build a high precision diagnosis system (Wang: ¶ 0004), while reducing the number of labeled patient sound recordings required to train the model (Chou: ¶ 0041).
Although in one sense Alshaer can be considered as teaching processing of auscultation data, it does not mention stethoscope-based auscultation.
Bredesen teaches using a stethoscope for auscultation (Abstract).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the methods of Alshaer-Wang-Chou with a stethoscope, as an alternative device for performing diagnosis based on breath sounds (Bredesen: Abstract, lung sounds), with predictable results (Alshaer: Abstract; Bredesen: Abstract), and for the purpose of implementing modern machine learning based classification with older diagnostic devices (Alshaer: ¶ 00249; Wang: ¶ 0003; Chou: ¶ 0041).
Regarding claims 61 and 62, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches displaying, on a computing device, a plurality of sound capture points associated with the subject and a plurality of numbers corresponding to the plurality of sound capture points, wherein respiratory sounds of the subject are acquired by auscultation at one of the plurality of sound capture points, wherein the plurality of numbers indicate a preferred order of auscultation at the plurality of sound capture points (Bredesen teaches displaying a plurality of sound capture points and related numerals on a display of a computing device (Figs. 6 and 10 and related description) and prompting an individual to follow a specific auscultation order with respect thereto (claims 5, 6, 10, and 11 - also see col. 14, lines 40-50 and col. 35, lines 43-54). It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a display to indicate a preferred auscultation order in the combination as in Bredesen, for the purpose of enabling more automated diagnosis (Bredesen: col. 1, lines 15-48)).
Regarding claim 65, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches presenting information about the inferred respiratory condition through a user interface of a computing device (Alshaer: ¶ 0092).
Regarding claim 66, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 65, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein the information presented through the user interface comprises a graphical representation of the first sound record during a period of time (Alshaer: Fig. 1, plot 130 - also see Figs. 7A, 14A, 23, 25, etc.).
Regarding claim 68, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 65, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein the information about the inferred respiratory condition presented through the user interface comprises information about management of the respiratory condition (Alshaer: ¶¶s 00102, 00114, 00119, etc., receiving treatment or recommendations).
Regarding claim 69, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches receiving multiple sound records taken at different sound capture points on the subject (Bredesen teaches displaying a plurality of sound capture points and related numerals on a display of a computing device (Figs. 6 and 10 and related description) and prompting an individual to follow a specific auscultation order with respect thereto (claims 5, 6, 10, and 11 - also see col. 14, lines 40-50 and col. 35, lines 43-54). It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to capture sound from different points according to a preferred auscultation order in the combination as in Bredesen, for the purpose of enabling more automated diagnosis (Bredesen: col. 1, lines 15-48) as well as comprehensively evaluating the patient (Bredesen: Abstract)).
Regarding claim 70, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 69, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein the sound capture points are determined algorithmically based on the respiratory condition (Bredesen: Fig. 4A, col. 8, line 55 to col. 9, line 13, showing the physician the next location to probe - also see col. 14, lines 40-50, etc.), and are presented on a user interface of a computing device (Bredesen teaches displaying a plurality of sound capture points and related numerals on a user interface of a mobile device (Figs. 1, 6, and 10 and related descriptions) - also see claims 5, 6, 10, and 11, col. 35, lines 43-54, etc. It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a display to indicate a preferred auscultation order in the combination as in Bredesen, for the purpose of enabling more automated diagnosis (Bredesen: col. 1, lines 15-48)).
Regarding claim 71, Alshaer-Wang-Chou-Bredesen teaches all the features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches receiving multiple sound records taken at a particular sound capture point on the subject (Alshaer: Fig. 1 and Abstract, recording sound data over time in a face mask which is arranged at one sound capture point (on the face)).
Regarding claim 73, Alshaer-Wang-Chou-Bredesen teaches all the features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein the first sound record has degraded quality (Alshaer: “turbulent signal component”; ¶ 0089).
Regarding claim 74, Alshaer-Wang-Chou-Bredesen teaches all the features with respect to claim 73, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein the degraded quality is based on noise (Alshaer: “ airflow on microphone”; ¶ 0089) or improper auscultation or a combination of them.
Regarding claim 75, Alshaer teaches [a] system comprising: at least one processor (¶ 0023, claim 13); and one or more memories coupled to the at least one processor and storing programming instructions for execution by the at least one processor (¶ 0023, claim 13) to perform operations comprising: …; receiving a second number of respiratory sound records, each of the respiratory sound records representing respiratory sounds acquired by auscultation, and each of the respiratory sound records having … one or more known sound classes (¶¶s 00187, 00202, 00235, etc.); …; receiving a first sound record representing the respiratory sounds of a subject (“sound and/or airflow produced by the subject while breathing”; ¶ 0011) acquired by auscultation, the first sound record being separate from sound records in the [second] number of respiratory sound records (“transducer responsive to sound and/or airflow”; ¶ 0011); transforming the received first sound record into a time-frequency domain graphical representation (Figure 25 and ¶ 00202); applying the time-frequency domain graphical representation to [a] neural network (“comparative process”; ¶ 00207 - also see ¶ 00249, artificial neural network) to determine a sound class for the respiratory sounds acquired by auscultation (stable or collapsible airway; ¶ 00207); and inferring a respiratory condition of the subject (“CSA (or normal) vs. OSA”; ¶ 00207) based at least on the sound class determined by the neural network (¶¶s 00207 and 00249).
Alshaer does not appear to explicitly teach pre-training initial convolutional layers of a neural network using supervised learning based on a first number of images unrelated to sound records; each of the respiratory sound records having a label selected from labels representing one or more known sound classes (although it does describe using a known data set for classifier training); and using a transfer-learning approach, training the neural network using the second number of respiratory sound records and the known sound classes, wherein the first number of images is larger than the second number of respiratory sound records.
Wang teaches using a deep convolutional neural network to classify medical images (¶ 0045). For the classification, a pre-trained ImageNet model (unrelated to sound records) is adapted, via transfer learning, to a more complete model (because knowledge obtained from ImageNet pre-training is used to train the more complete model as shown in Fig. 4) via the addition of a transition layer, a global pooling layer, a prediction layer, and a loss layer (¶ 0045). Pre-training via supervised learning occurred to obtain the ImageNet models, since the ImageNet data is labeled. The more complete model of Wang is considered a weakly-supervised multi-label image classification model (¶¶s 0044, 0045, Abstract, etc.), trained on labeled images (Abstract, Fig. 4, left frame, ¶ 0053).
Chou teaches processing spectrograms (¶¶s 0024, 0036, etc.) using a convolutional neural network (¶¶s 0024, 0048, 0071, etc.). There is a pre-training step based on unlabeled non-patient sound records, and then a supervised training step using labeled patient sound records (¶ 0041, the pre-training reducing the number of labeled patient sound records required).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to make the images of Alshaer correspond to a “known” data set by incorporating labels, as in Wang, since this is a recognized way of knowing what is in a data set, without requiring any further processing and classification, and for the purpose of enabling weakly-supervised classification as contemplated by Wang (¶ 0045). It would have been obvious to use these images in a transfer learning approach to train a convolutional neural network (Wang: Fig. 4, ¶ 0053), including pre-training initial layers using supervised learning based on images unrelated to sound records (i.e., ImageNet), as in Wang (Fig. 4), for the purpose of being able to build a high precision diagnosis system (Wang: ¶ 0004), while reducing the number of labeled patient sound recordings required to train the model (Chou: ¶ 0041).
Although in one sense Alshaer can be considered as teaching processing of auscultation data, it does not mention stethoscope-based auscultation.
Bredesen teaches using a stethoscope for auscultation (Abstract).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the methods of Alshaer-Wang-Chou with a stethoscope, as an alternative device for performing diagnosis based on breath sounds (Bredesen: Abstract, lung sounds), with predictable results (Alshaer: Abstract; Bredesen: Abstract), and for the purpose of implementing modern machine learning based classification with older diagnostic devices (Alshaer: ¶ 00249; Wang: ¶ 0003; Chou: ¶ 0041).
Regarding claim 76, Alshaer teaches [a] non-transitory computer-readable storage medium storing programming instructions for execution by at least one processor, that when executed by the at least one processor, cause a computer to perform operations (¶ 0023, claim 13) comprising: …; receiving a second number of respiratory sound records, each of the respiratory sound records representing respiratory sounds acquired by auscultation, and each of the respiratory sound records having … one or more known sound classes (¶¶s 00187, 00202, 00235, etc.); …; receiving a first sound record representing the respiratory sounds of a subject (“sound and/or airflow produced by the subject while breathing”; ¶ 0011) acquired by auscultation, the first sound record being separate from sound records in the [second] number of respiratory sound records (“transducer responsive to sound and/or airflow”; ¶ 0011); transforming the received first sound record into a time-frequency domain graphical representation (Figure 25 and ¶ 00202); applying the time-frequency domain graphical representation to [a] neural network (“comparative process”; ¶ 00207 - also see ¶ 00249, artificial neural network) to determine a sound class for the respiratory sounds acquired by auscultation (stable or collapsible airway; ¶ 00207); and inferring a respiratory condition of the subject (“CSA (or normal) vs. OSA”; ¶ 00207) based at least on the sound class determined by the neural network (¶¶s 00207 and 00249).
Alshaer does not appear to explicitly teach pre-training initial convolutional layers of a neural network using supervised learning based on a first number of images unrelated to sound records; each of the respiratory sound records having a label selected from labels representing one or more known sound classes (although it does describe using a known data set for classifier training); and using a transfer-learning approach, training the neural network using the second number of respiratory sound records and the known sound classes, wherein the first number of images is larger than the second number of respiratory sound records.
Wang teaches using a deep convolutional neural network to classify medical images (¶ 0045). For the classification, a pre-trained ImageNet model (unrelated to sound records) is adapted, via transfer learning, to a more complete model (because knowledge obtained from ImageNet pre-training is used to train the more complete model as shown in Fig. 4) via the addition of a transition layer, a global pooling layer, a prediction layer, and a loss layer (¶ 0045). Pre-training via supervised learning occurred to obtain the ImageNet models, since the ImageNet data is labeled. The more complete model of Wang is considered a weakly-supervised multi-label image classification model (¶¶s 0044, 0045, Abstract, etc.), trained on labeled images (Abstract, Fig. 4, left frame, ¶ 0053).
Chou teaches processing spectrograms (¶¶s 0024, 0036, etc.) using a convolutional neural network (¶¶s 0024, 0048, 0071, etc.). There is a pre-training step based on unlabeled non-patient sound records, and then a supervised training step using labeled patient sound records (¶ 0041, the pre-training reducing the number of labeled patient sound records required).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to make the images of Alshaer correspond to a “known” data set by incorporating labels, as in Wang, since this is a recognized way of knowing what is in a data set, without requiring any further processing and classification, and for the purpose of enabling weakly-supervised classification as contemplated by Wang (¶ 0045). It would have been obvious to use these images in a transfer learning approach to train a convolutional neural network (Wang: Fig. 4, ¶ 0053), including pre-training initial layers using supervised learning based on images unrelated to sound records (i.e., ImageNet), as in Wang (Fig. 4), for the purpose of being able to build a high precision diagnosis system (Wang: ¶ 0004), while reducing the number of labeled patient sound recordings required to train the model (Chou: ¶ 0041).
Although in one sense Alshaer can be considered as teaching processing of auscultation data, it does not mention stethoscope-based auscultation.
Bredesen teaches using a stethoscope for auscultation (Abstract).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the methods of Alshaer-Wang-Chou with a stethoscope, as an alternative device for performing diagnosis based on breath sounds (Bredesen: Abstract, lung sounds), with predictable results (Alshaer: Abstract; Bredesen: Abstract), and for the purpose of implementing modern machine learning based classification with older diagnostic devices (Alshaer: ¶ 00249; Wang: ¶ 0003; Chou: ¶ 0041).
Claims 58, 60, and 63 are rejected under 35 U.S.C. 103 as being unpatentable over Alshaer-Wang-Chou-Bredesen in view of US Patent Application Publication 2011/0035215 (“Sompolinsky”).
Regarding claim 58, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein the pre-training is performed using an ImageNet data set (Wang: Fig. 4, ¶ 0045), but does not appear to explicitly teach wherein training the neural network using the second number of respiratory sound records comprises training the neural network using color Mel spectrogram images corresponding to the second number of respiratory sound record.
Sompolinsky teaches a machine-based method comprising receiving a sound record (“speech signal”, ¶ 0017) representing respiratory sounds of a subject (¶ 0017), and by machine
transforming the received sound record into a time-frequency domain graphical representation (¶ 0052). Sompolinsky additionally teaches that the time-frequency domain graphical representation comprises a color (“colorbar”, “color of each spike”) Mel (“Mel scale”; ¶ 0041, Figure 8) spectrogram (¶ 0041, Figure 8).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to extend the classification capabilities of the system to color Mel spectrogram images by performing training and classification using these kinds of images (time-frequency domain graphical representations), as in Sompolinsky, since Alshaer describes employing the time-frequency domain in many different ways (Alshaer: “various pitch contour characteristics in the time, frequency and time/frequency domain”, “bands ratio”, “fundamental frequency plots”, “apnea-hypopnea index”, “leveraging characteristics of FFT and mel-frequency cepstrum coefficients (MFCC)”; ¶¶s 00202, 00124, 0057, 00250, etc.) and color can serve as an indicator, as Sompolinsky recognizes (¶ 0041), and for the purpose of classification based on more data (Sompolinsky: ¶ 0041, etc.).
Regarding claim 60, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein training the neural network comprises first training a convolutional neural network (CNN) using … spectrogram images (based on Wang and Chou, as above), and subsequently training a recurrent neural network (RNN) based on a long short-term memory model (LSTM) (Chou teaches using and/or training different models that work together, including an RNN based on LSTM and a CCN (Fig. 3, ¶ 0077, a sound model using a CNN; ¶¶s 0033 and 0048, an attention or aggregation model using an RNN based on LSTM; ¶ 0037, a diagnostic model using an RNN, etc. - also see Figs. 2, 4-6, etc.). It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use these particular models as the simple substation of one suitable neural network model for another with predictable results (machine-learning based classification - Chou: ¶¶s 0024, 0048)).
Alshaer-Wang-Chou-Bredesen is not explicit that the first training is based on RGB Mel-spectrogram images.
Sompolinsky teaches a machine-based method comprising receiving a sound record (“speech signal”, ¶ 0017) representing respiratory sounds of a subject (¶ 0017), and by machine transforming the received sound record into a time-frequency domain graphical representation (¶ 0052). Sompolinsky additionally teaches that the time-frequency domain graphical representation comprises a color (“colorbar”, “color of each spike”) Mel (“Mel scale”; ¶ 0041, Figure 8) spectrogram (¶ 0041, Figure 8).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to extend the classification capabilities of the system to color Mel spectrogram images by performing training and classification using these kinds of images (time-frequency domain graphical representations), as in Sompolinsky, since Alshaer describes employing the time-frequency domain in many different ways (Alshaer: “various pitch contour characteristics in the time, frequency and time/frequency domain”, “bands ratio”, “fundamental frequency plots”, “apnea-hypopnea index”, “leveraging characteristics of FFT and mel-frequency cepstrum coefficients (MFCC)”; ¶¶s 00202, 00124, 0057, 00250, etc.) and color can serve as an indicator, as Sompolinsky recognizes (¶ 0041), and for the purpose of classification based on more data (Sompolinsky: ¶ 0041, etc.).
Regarding claim 63, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen does not appear to explicitly teach wherein the time-frequency domain graphical representation comprises a color Mel spectrogram.
Sompolinsky teaches a machine-based method comprising receiving a sound record (“speech signal”, ¶ 0017) representing respiratory sounds of a subject (¶ 0017), and by machine
transforming the received sound record into a time-frequency domain graphical representation (¶ 0052). Sompolinsky additionally teaches that the time-frequency domain graphical representation comprises a color (“colorbar”, “color of each spike”) Mel (“Mel scale”; ¶ 0041, Figure 8) spectrogram (¶ 0041, Figure 8). Sompolinsky is considered to be analogous to the claimed invention because it is in the same field, namely a machine-based method for sound classification (Abstract).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to extend the classification capabilities of the system to color Mel spectrogram images by performing training and classification using these kinds of images (time-frequency domain graphical representations), as in Sompolinsky, since Alshaer describes employing the time-frequency domain in many different ways (Alshaer: “various pitch contour characteristics in the time, frequency and time/frequency domain”, “bands ratio”, “fundamental frequency plots”, “apnea-hypopnea index”, “leveraging characteristics of FFT and mel-frequency cepstrum coefficients (MFCC)”; ¶¶s 00202, 00124, 0057, 00250, etc.) and color can serve as an indicator, as Sompolinsky recognizes (¶ 0041), and for the purpose of classification based on more data (Sompolinsky: ¶ 0041, etc.).
Claim 59 is rejected under 35 U.S.C. 103 as being unpatentable over Alshaer-Wang-Chou-Bredesen-Sompolinsky in view of US Patent Application Publication 2017/0371856 (“Can”).
Regarding claim 58, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 58, as outlined above. Alshaer-Wang-Chou-Bredesen further teaches wherein the color Mel spectrogram images comprise RGB Mel spectrogram images (Chou: ¶ 0036), but does not appear to explicitly tech them being in .PNG format.
Can teaches storing data visualizations in PNG format (¶ 0058).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use a .PNG format in the combination as in Can, as the simple substitution of one file format for another with predictable results (training a classifier based on image data).
Claim 64 is rejected under 35 U.S.C. 103 as being unpatentable over Alshaer-Wang-Chou-Bredesen in view of US Patent Application Publication 2005/0015009 (“Mourad”).
Regarding claim 64, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 57, as outlined above. Alshaer-Wang-Chou-Bredesen does not appear to explicitly teach using an expert system for inferring the respiratory condition of the subject based at least on the sound class determined by the neural network.
Mourad teaches using an expert system to assist with medical diagnostic decisions (¶ 0101).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use an expert system in the inferring step of the combination, as in Mourad, for the purpose of enhancing accuracy of diagnosis (Mourad: ¶ 0101), and as the simple substitution of one inferring method for another with predictable results (inferring the diagnosis/respiratory condition).
Claim 67 is rejected under 35 U.S.C. 103 as being unpatentable over Alshaer-Wang-Chou-Bredesen in view of US Patent Application Publication 2002/0183642 (“Murphy”).
Regarding claim 67, Alshaer-Wang-Chou-Bredesen teaches all features with respect to claim 66, as outlined above. Alshaer-Wang-Chou-Bredesen does not appear to explicitly teach wherein the graphical representation of the first sound record is color coded according to sound class (although Chou does talk about color-coded spectrograms).
Murphy teaches color coding sound data to distinguish e.g. wheezes and crackles (¶ 0106, Fig. 16).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to color code the graphical representation of the first sound record of the combination, as in Murphy, for the purpose of being able to show classifications of the sound record (Murphy: ¶ 0106, Fig. 16).
Claim 72 is rejected under 35 U.S.C. 103 as being unpatentable over Alshaer-Wang-Chou-Bredesen in view of US Patent Application Publication 2011/0066042 (“Pandia”).
Regarding claim 72, Alshaer-Wang-Chou-Bredesen teaches all the features with respect to claim 71, as outlined above. Alshaer-Wang-Chou-Bredesen does not appear to explicitly teach performing a principal component analysis on the multiple sound records.
Pandia teaches performing principal component analysis to extra useful data from a combination of sensor signals (¶ 0259).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate principal component analysis into the combination, as in Pandia, for the purpose of obtaining data on inter-axis cross-talk (Pandia: ¶ 0259).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREY SHOSTAK whose telephone number is (408) 918-7617. The examiner can normally be reached Monday - Friday 7 am - 3 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Robertson can be reached on (571) 272-5001. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREY SHOSTAK/Primary Examiner, Art Unit 3791