DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims Pending
Claims 1-20 are currently under examination.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claim 8: The claim limitation “at least one processing device configured to…” “… predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input” has been interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because it uses a generic placeholder “processing device” coupled with functional language “configured to…” “… predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input” without reciting sufficient structure to achieve the function. Furthermore, the generic placeholder is not preceded by a structural modifier that has a known structural meaning before the phrase “processing device”
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
A review of the specification shows that the following appears to be the corresponding structure described in the specification for the 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph limitation:
A processor and machine learning algorithm, or equivalents thereof, as described on Par. 36, 65, and 68 of the disclosure filed on 02/05/2024, which lacks sufficient detail in regards to the structure of the algorithm that performs the indicated function, and will be interpreted as a generic algorithm capable of the indicated function.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1 and 15 recite the limitation “predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input” (Claim 1) and “predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input”, which lacks sufficient detail within the applicant’s specification as to the structure of the machine learning algorithm that performs the indicated function of “predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate” (Claim 1) and “predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate” (Claim 15). The applicant’s specification does state “the ML model can be a binary classification model. The classification model can be trained to determine the final output between two calculated RRs. To train the ML model, the electronic device 101 (or the server 106 or other device) can access a dataset that includes multiple training samples. In some embodiments, each training sample includes a motion-based respiratory signal, an rPPG-based respiratory signal, and a label indicating whether a motion-based RR or an rPPG-based RR is closer to a ground truth RR for that training sample” (Par. 68 of applicant’s spec.) and “ML classifier can be trained using any suitable set of features. In some embodiments, the features can include SNR, number of peaks, and skewness. The electronic device 101, server 106, or other device updates one or more parameters or weights of the ML model based on a comparison of the label and the prediction. In some cases, a class weight of 9 to 1 for rPPG-derived RR and motion-derived RR can be applied to the decision tree to resolve any class imbalance issues in the feature set” (Par. 69 of applicant’s spec.). However, this lacks sufficient detail in regards to any specific layers, biases, or weights used for the model itself, the specific manner in which the machine learning algorithm is trained to make the indicated prediction, or the manner in which the prediction is performed. As such, the claims are rejected.
Claim 8 recites the limitation “at least one processing device configured to…” “… predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input”, which lacks sufficient detail within the applicant’s specification as to the structure of the algorithm in the processing device and machine learning algorithm that performs the indicated function of “predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model”. The applicant’s specification does state “the ML model can be a binary classification model. The classification model can be trained to determine the final output between two calculated RRs. To train the ML model, the electronic device 101 (or the server 106 or other device) can access a dataset that includes multiple training samples. In some embodiments, each training sample includes a motion-based respiratory signal, an rPPG-based respiratory signal, and a label indicating whether a motion-based RR or an rPPG-based RR is closer to a ground truth RR for that training sample” (Par. 68 of applicant’s spec.) and “ML classifier can be trained using any suitable set of features. In some embodiments, the features can include SNR, number of peaks, and skewness. The electronic device 101, server 106, or other device updates one or more parameters or weights of the ML model based on a comparison of the label and the prediction. In some cases, a class weight of 9 to 1 for rPPG-derived RR and motion-derived RR can be applied to the decision tree to resolve any class imbalance issues in the feature set” (Par. 69 of applicant’s spec.). However, this lacks sufficient detail in regards to any specific layers, biases, or weights used for the model itself, the specific manner in which the machine learning algorithm is trained to make the indicated prediction, or the manner in which the prediction is performed. As such, the claim is rejected.
Claims 5, 12, and 19 recite the limitation “providing the features as input to the machine learning model which predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claim 5) and “provide the features as input to the machine learning model which predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claims 12 and 19), where the applicant specification lacks sufficient detail in regards to structure of the machine learning algorithm that performs the indicated functionality of “predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claim 5) and “predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claims 12 and 19). The corresponding structure provided within the applicant’s specification states a machine learning algorithm “the ML model can be a binary classification model. The classification model can be trained to determine the final output between two calculated RRs…” (Par. 68 of applicant’s spec.) and “ML classifier can be trained using any suitable set of features. In some embodiments, the features can include SNR, number of peaks, and skewness…” (Par. 69 of applicant’s spec.). However, this lacks sufficient detail in regards to the actual weights, biases, or layers used by the applicant for the machine learning algorithm, or the specific manner in which the machine learning algorithm is trained to make the indicated prediction. As such, the claim is rejected.
Claims 2-7, 9-14, and 16-20 are dependent on claims 1, 8, and 15, respectively, and as such are also rejected.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 15 recite the limitation “predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input” (Claim 1) and “predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input”, which fails to effectively define the metes and bounds of the claim as it is unclear as to manner in which the machine learning algorithm performs the indicated function of “predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate” (Claim 1) and “predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate” (Claim 15). The corresponding structure provided within the applicant’s specification states a machine learning algorithm “the ML model can be a binary classification model. The classification model can be trained to determine the final output between two calculated RRs…” (Par. 68 of applicant’s spec.) and “ML classifier can be trained using any suitable set of features. In some embodiments, the features can include SNR, number of peaks, and skewness…” (Par. 69 of applicant’s spec.), which fails to effectively define the metes and bounds of the claim as it is unclear as to the actual weights, biases, or layers used by the applicant for the machine learning algorithm, or the specific manner in which the machine learning algorithm is trained to make the indicated prediction. For examination purposes, this will be interpreted as any generic algorithm on a generic processor capable of the indicated function.
Claims 5, 12, and 19 recite the limitation “providing the features as input to the machine learning model which predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claim 5) and “provide the features as input to the machine learning model which predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claims 12 and 19), which fails to effectively define the metes and bounds of the claim as it is unclear as to manner in which the machine learning algorithm performs the indicated function of predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claim 5) and “predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR” (Claims 12 and 19). The corresponding structure provided within the applicant’s specification states a machine learning algorithm “the ML model can be a binary classification model. The classification model can be trained to determine the final output between two calculated RRs…” (Par. 68 of applicant’s spec.) and “ML classifier can be trained using any suitable set of features. In some embodiments, the features can include SNR, number of peaks, and skewness…” (Par. 69 of applicant’s spec.), which fails to effectively define the metes and bounds of the claim as it is unclear as to the actual weights, biases, or layers used by the applicant for the machine learning algorithm, or the specific manner in which the machine learning algorithm is trained to make the indicated prediction. For examination purposes, this will be interpreted as any generic algorithm on a generic processor capable of the indicated function.
Claim 8 limitation “at least one processing device configured to…” “… predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input”, invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The corresponding structure provided within the applicant’s specification states a machine learning algorithm “the ML model can be a binary classification model. The classification model can be trained to determine the final output between two calculated RRs…” (Par. 68 of applicant’s spec.) and “ML classifier can be trained using any suitable set of features. In some embodiments, the features can include SNR, number of peaks, and skewness…” (Par. 69 of applicant’s spec.), which fails to effectively define the metes and bounds of the claim as it is unclear as to the actual weights, biases, or layers used by the applicant for the machine learning algorithm, or the specific manner in which the machine learning algorithm is trained to make the indicated prediction. For examination purposes, this will be interpreted as any generic algorithm on a generic processor capable of the indicated function. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claims 2-7, 9-14, and 16-20 are dependent on claims 1, 8, and 15, respectively, and as such are also rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed towards a judicial exception without significantly more. These claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception or that are sufficient to amount to significantly more than the judicial exception.
Step 1 of the subject matter eligibility test
Claim 1, 8, and 15 are directed towards a method, device, and system, respectively, which each describes one of the four statutory categories of patentable subject matter.
Step 2A of the subject matter eligibility test
Prong 1: Claim 1 recites the abstract idea of a mental process as follows: “capturing a video of a person’s face”, “determining a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face”, “determining a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face”, “predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a…” “…model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input”, and “presenting one of the motion-based RR or the rPPG-based RR based on the prediction”.
Prong 1: Claims 8 and 15 recite the abstract idea of a mental process as follows: “capture a video of a person’s face”, “determine a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face”, “determine a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face”, “predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a…” “…model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input”, and “present one of the motion-based RR or the rPPG-based RR based on the prediction”.
The capturing a video of a person’s face, determining a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face, determining a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face, predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input, and presenting one of the motion-based RR or the rPPG-based RR based on the prediction can be practically performed by the human mind, with the aid of a pen and paper, but for performance on a generic processor, in a computer environment, or merely using the computer as a tool to perform the steps.
A person of ordinary skill in the art could reasonably capture a video of a user’s face based on receiving a video with a user’s face with a generic computer. A person of ordinary skill in the art could reasonably determine a motion-based respiratory rate (RR) and a motion-based respiratory signal based on receiving a video of a person’s face. A person of ordinary skill in the art could reasonably determine a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on receiving a video of a person’s face with a generic computer. A person of ordinary skill in the art could reasonably predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate mentally, or with a generic computer, based on receiving a piece of paper with a motion-based respiratory signal and the rPPG-based respiratory signal. A person of ordinary skill in the art could reasonably verbally present one of the motion-based RR or the rPPG-based RR based on receiving a prediction on a piece of paper.
There is currently nothing to suggest an undue level of complexity in the determining, predicting, capturing, or presenting steps. Therefore, a person would be able to practically be able to perform the determining and predicting steps mentally or with the aid of pen and paper.
Prong Two: Claims 1, 8, and 15 do not recite additional elements that integrate the mental process into a practical application. Therefore, the claims are “directed to” the mental process. The additional elements merely:
Recite the words “apply it” or an equivalent with the judicial exception, or include instructions to implement the abstract idea on a computer, or merely use the computer as a tool to perform the abstract idea (e.g., a processor with instructions, a computational device) and
Add insignificant extra-solution activity (the pre-solution activity of: using generic data-gathering components (e.g. a camera)
For claims 1, 8, and 15. The additional elements merely serve to gather data to be used by the abstract idea. The camera is merely used as a pre-solution step of necessary data gathering to be used by the abstract idea. There is no practical application because the abstract idea is not applied, relied on, or used in a meaningful way. The processing that is performed remains in the abstract realm, i.e. the gathered data is not used for a treatment or meaningful purpose. Additionally, there is no overall improvement to existing technology present. The mental process merely functions on generic computer elements that do not change the functionality of the device itself. Therefore, the additional elements, alone or in combination, do not integrate the abstract idea into a practical application.
Step 2B of the subject matter eligibility test for Claims 1, 8, and 15.
Per the Berkheimer requirement, the additional elements are well-understood, routine, and conventional. For example,
A camera and computational device as disclosed by Gargeya (US Pub. No. 20190191988) hereinafter Gargeya, “As shown in FIG. 9, the computer system 1101, which is a form of a data processing system, includes a bus 1102 that is coupled to a microprocessor 1103 and a ROM 1107 and volatile RAM 1105 and a non-volatile memory 1106…” “…displays (e.g., cathode ray tube (CRT) or liquid crystal display (LCD)), video cameras, and other devices that are well known in the art.” (Par. 57) and Snellenberg (US Pub. No. 20190021649) hereinafter Snellenberg “visible-light cameras 31 are used, along with other cameras that capture images at other wavelengths. In some embodiments, to reduce cost and complexity visible light cameras 31 may be commercially available cameras with visible-light image sensors such as those found in cellular phones. Other cameras may use image sensors calibrated to capture visible light, near-infrared light, infrared light, or some combination thereof.” (Par. 33).
A processor with instructions as disclosed by Kim (US Pub. No. 20170172431) hereinafter Kim “Examples of hardware components include processors, controllers, sensors, generators, drivers, and any other electronic components known to one of ordinary skill in the art. In one example, the hardware components are implemented by one or more processors or computers. A processor or computer is implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices known to one of ordinary skill in the art that is capable of responding to and executing instructions in a defined manner to achieve a desired result.” (Par. 131) and Bonomi (US Pub. No. 20180368737) hereinafter Bonomi “The processor is a hardware device for executing software, particularly that stored in memory. The processor can be any custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the computer 16, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, or generally any device for executing software instructions.” (Par. 33),
are all well-understood, routine, and conventional.
Claims 2-7, 9-14, and 16-20 do not include additional elements, alone or in combination that are sufficient to amount to significantly more than the judicial exception (i.e., an inventive concept) as all of the elements are directed to the further describing of the abstract idea, pre-solution activities, and computer implementation.
The dependent claims merely further define the abstract idea and are, therefore, directed to an abstract idea for similar reasons: they merely further describe the abstract idea:
identify face landmarks on the person’s face based on the video, wherein the face landmarks are on the person’s forehead and the person’s nose (Claims 2, 9, 16) (Examiner's Note: A person of ordinary skill in the art could reasonably receive a video with a generic computer and identify landmarks),
generate a motion signal based on vertical location changes of the face landmarks in the video (Claims 2, 9, 16) (Examiner's Note: A person of ordinary skill in the art could reasonably generate a motion signal based on having a video of a person’s face),
extract the motion-based respiratory signal using spectral analysis based on the motion signal (Claims 2, 9, 16) (Examiner's Note: A person of ordinary skill in the art could reasonably perform spectral analysis based on having a motion signal with a generic computer),
removing artifacts from the motion signal using a kurtosis-based motion artifacts detection technique (Claims 3, 10, 17) (Examiner's Note: A person of ordinary skill in the art could reasonably remove artifacts from a motion signal with a kurtosis based technique with a generic computer),
smoothing the motion signal using a filter (Claims 3, 10, 17) (Examiner's Note: A person of ordinary skill in the art could reasonably smooth a motion signal with a filter with a generic computer),
identifying regions of interest on the person’s face (Claims 4, 11, 18),
extracting an rPPG signal for each region of interest based on the video (Claims 4, 11, 18) (Claims 4, 11, 18) (Examiner's Note: A person of ordinary skill in the art could reasonably extract a signal for regions of interest on a video based on receiving a video with a generic computer),
extracting an inter-beat interval (IBI) signal based on a weighted combination of the rPPG signals corresponding to the regions of interest (Claims 4, 11, 18),
extracting the rPPG-based respiratory signal based on the IBI signal (Claims 4, 11, 18),
accessing a training dataset comprising multiple training samples, each training sample including a motion-based respiratory signal, an rPPG-based respiratory signal, and a label indicating whether a motion-based RR or an rPPG-based RR is closer to a ground truth RR for that training sample (claims 5, 12, and 19) (Examiner's Note: A person of ordinary skill in the art could reasonably access a training dataset based on receiving a dataset with multiple signals with a generic computer)
extracting features of the motion-based respiratory signal and the rPPG-based respiratory signal (Claims 5, 12, 19) (Examiner's Note: A person of ordinary skill in the art could reasonably provide extract features from respiratory signals with a generic computer based on receiving respiratory signals),
providing the features as input to the machine learning model which predicts whether the motion-based RR or the rPPG-based RR is more likely to be closer to the ground truth RR (Claims 5, 12, 19) (Examiner's Note: A person of ordinary skill in the art could reasonably provide an input to a machine learning model with a generic computer based on receiving features of respiratory signals),
updating parameters of the machine learning model based on a comparison of the label and the prediction (Claims 5, 12, 19) (Examiner's Note: A person of ordinary skill in the art could reasonably update a machine learning model with a generic computer based on receiving an output),
wherein the features include one or more of: a signal noise ratio, a number of peaks, and a skewness (Claims 6, 13, 20).
Further describe the pre-solution activity (or structure used for such activity):
the camera is coupled to a mobile device, a computer, or a television (Claims 7 and 14)
Per the Berkheimer requirement, the additional elements are well-understood, routine, and conventional. For example,
A camera coupled to a mobile device, computer, or television as disclosed by Gargeya and Snellenberg above,
are all well-understood, routine, and conventional.
Taken alone or in combination, the additional elements do not integrate the judicial exception into a practical application at least because the abstract idea is not applied, relied on, or used in a meaningful way. The additional elements do not add anything significantly more than the abstract idea. The collective functions of the additional elements merely provide computer/electronic implementation and processing, data gathering, and no additional elements beyond those of the abstract idea. There is no indication that the combination of elements improves the functioning of a mobile device, output device, improves technology other than the technical field of the claimed invention, etc. Therefore, the claims are rejected as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The claims are generally directed towards a method that comprises capturing a video of a face of a user, determining motion-based and remote photoplethysmography-based respiratory rates, predicting which respiratory rate is more accurate, and presenting one of the respiratory rates.
Claim(s) 1-2, 7-9, and 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Muehlsteff (US Pub. No. 20140275832) hereinafter Jens, and further in view of Xu (US Pub. No. 20120289850) hereinafter Xu, Frank (US Pub. No. 20200397306) hereinafter Frank, and Mccann (US Pub. No. 20200163586) hereinafter Mccann.
Regarding claim 1, Jens discloses A method comprising:
capturing a video of a person’s face using a camera (Par. 106, “The acquired video contains at least one part of the body showing breathing motion (e.g., the chest and/or belly) and at least one part of skin area. The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection, while the other is from the PPG signal”) (Fig. 8 (imaging unit – 20) (skin portion -24 includes the face)):
determining a motion-based respiratory signal based on the video (Par. 105, “a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”) (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection”);
determining an rPPG-based respiratory signal based on the video of the person’s face (Par. 103, “Another way to derive the respiratory signal remotely is by processing the photoplethysmography (PPG) signals calculated from the video. It is generally known in the art that a respiratory signal can be extracted from PPG signals, and that PPGs signal can be derived remotely by measuring the change in the skin area (called remote PPG or R-PPG)…”) (Par. 105, “The device 10e particularly comprises an imaging unit 20, in particular a camera representing the first detection unit and the second detection unit, for acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12” (portion 24 includes the face)) (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection, while the other is from the PPG signal”) (Par. 106 (PPG signal)).
Jens fails to explicitly disclose determining a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face.
However, Jens does disclose determining a motion-based respiratory signal based on the video (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection”) (Par. 105, “a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”), calculating respiratory rate (Par. 105, “the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used) and a weighting scheme”), and respiratory signals from two locations (Par. 105, “acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12 and a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”).
Xu teaches determining a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face (Claim 1, “capturing thermal images of a subject of interest intended to be monitored for respiration using a thermal image camera set to a temperature of a facial region of said subject, each thermal image comprising a plurality of pixels each corresponding to a surface temperature of said facial region across a thermal wavelength band; analyzing said thermal images to determine a facial feature associated with respiration; tracking said facial feature over time to identify a pattern of respiration; determining a respiration rate from said respiration pattern”).
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the method of Jens with that of Xu to include determining a motion-based respiratory rate (RR) of Xu and a motion-based respiratory signal based on the video of the person’s face through the combination of references as it would have yielded the predictable result of explicitly providing a respiratory rate output based on the signal and allow performance of both measurements from solely the face of the user.
Modified Jens highly suggests but fails to explicitly disclose determining a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face (Examiner's Note: Jen fails to explicitly disclose a separate determination step for determining a photoplethysmography (rPPG)-based RR).
Jens does disclose determining an rPPG-based respiratory signal based on the video of the person’s face (Par. 103, “Another way to derive the respiratory signal remotely is by processing the photoplethysmography (PPG) signals calculated from the video. It is generally known in the art that a respiratory signal can be extracted from PPG signals, and that PPGs signal can be derived remotely by measuring the change in the skin area (called remote PPG or R-PPG)…”) (Par. 105, “The device 10e particularly comprises an imaging unit 20, in particular a camera representing the first detection unit and the second detection unit, for acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12” (portion 24 includes the face)) and calculating respiratory rate (Par. 105, “the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used) and a weighting scheme”).
Frank teaches determining a remote photoplethysmography (rPPG)-based RR based on the video of the person’s face (Par. 61, “non-contact camera that captures images of the skin, where a computer extracts the PPG signal from the images using an imaging photoplethysmography (iPPG) technique. Other names known in the art for iPPG include: remote photoplethysmography…”) (Par. 183, “the computer 340 calculates, based on the current set of images, a current heart rate and/or a current respiration rate of the user. For example, the computer 340 may utilize one or more techniques described herein or known in the art for calculating heart rate and/or respiration from iPPG signals”) (Par. 182, “computer 340 may utilize the images 333 to detect additional physiological signals and/or conditions”) (Par. 114 (image of face)).
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the method of Jens and Xu with that of Frank to include determining a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face of Jens through the combination of references as it would have yielded the predictable result of providing additional respiration information regarding the user (Frank (Par. 183)).
Modified Jens fails to explicitly disclose predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input; and presenting one of the motion-based RR or the rPPG-based RR based on the prediction.
However, Jens does disclose predicting whether the motion-based signal or the rPPG-based signal is more likely to be accurate using a model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input (Par. 105, “processing unit 36e is configured to weight the first respiratory signal 80 by use of a first quality index, to weight the second respiratory signal 82 by use of a second quality index and to combine the weighted respiratory signal 84 and the weighted motion signal 86 to obtain a weighted combined respiratory signal 88. By the extraction unit 38 the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained.”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used..” “… In one embodiment, a simple combination method is to select the signal with the better quality.”); and
presenting one of the motion-based RR or the rPPG-based RR based on the prediction (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used..” “… In one embodiment, a simple combination method is to select the signal with the better quality.”) (Par. 105, “processing unit 36e is configured to weight the first respiratory signal 80 by use of a first quality index, to weight the second respiratory signal 82 by use of a second quality index and to combine the weighted respiratory signal 84 and the weighted motion signal 86 to obtain a weighted combined respiratory signal 88. By the extraction unit 38 the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained.”).
Mccann teaches predicting whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input (Claim 5, “wherein the processor outputs either: (a) a first (PPG-derived) estimate of the rate of respiration calculated by determining the frequency of a respiratory cycle induced variation in the output of the photoplethysmograph; or (b) a second (movement-derived) estimate of the rate of respiration calculated by determining the frequency of a respiratory cycle induced variation in the output of the one or more movement sensors, typically in dependence on (i) whether the output of the photoplethysmograph meets one or more quality criteria and/or (ii) whether the first estimate meets one or more accuracy criteria; and also (iii) whether the output of the movement sensors meets one or more quality criteria and/or (iv) whether the second estimate meets one or more accuracy criteria.”) (Par. 45 (machine learning algorithms)); and
presenting one of the motion-based RR or the rPPG-based RR based on the prediction (Claim 5, (output as indicated above)).
Jens, Xu, Frank, and Mccann are considered to be analogous art to the claimed invention as they are involved with respiratory measurements.
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the method of Jens, Xu, and Frank with that of Mccann to include predicting whether the motion-based RR of Xu or the rPPG-based RR of Frank is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal of Jens and Xu and the rPPG-based respiratory signal of Jens as input; and presenting one of the motion-based RR of Xu or the rPPG-based RR of Frank based on the prediction through the combination of references as differing output methods are known in the art (Jens (Par. 109)) and it would have yielded the predictable result of outputting a more accurate result.
Regarding claim 8, Jens discloses An electronic device (Fig. 8, device - 10e) comprising:
a camera configured to capture a video of a person’s face (Par. 106, “The acquired video contains at least one part of the body showing breathing motion (e.g., the chest and/or belly) and at least one part of skin area. The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection, while the other is from the PPG signal”) (Fig. 8 (imaging unit – 20) (skin portion -24 includes the face)); and
at least one processing device (Par. 104, “FIG. 8 shows a further embodiment of the device 10e for obtaining respiratory information of the subject according to the present invention in a more accurate and reliable way. The device 10e is similar to the device 10b shown in FIG. 4 and like elements are numbered with like reference numerals”)(Par. 105 (analysis unit 30e, processing unit -36e, extraction unit – 38)) (Par. 82, “The analysis unit 30, the processing unit 36 and the extracting unit 38 can be implemented by separate elements (e.g. processors or software functions), but can also be represented and implemented by a common processing apparatus.”) (Par. 114-120 (instructions and computational device implementation)) configured to:
determine a motion-based respiratory signal based on the video (Par. 105, “a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”) (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection”);
determine an rPPG-based respiratory signal based on the video of the person’s face (Par. 103, “Another way to derive the respiratory signal remotely is by processing the photoplethysmography (PPG) signals calculated from the video. It is generally known in the art that a respiratory signal can be extracted from PPG signals, and that PPGs signal can be derived remotely by measuring the change in the skin area (called remote PPG or R-PPG)…”) (Par. 105, “The device 10e particularly comprises an imaging unit 20, in particular a camera representing the first detection unit and the second detection unit, for acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12” (portion 24 includes the face)) (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection, while the other is from the PPG signal”) (Par. 106 (PPG signal)).
Jens fails to explicitly disclose determine a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face.
However, Jens does disclose determine a motion-based respiratory signal based on the video (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection”) (Par. 105, “a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”), calculating respiratory rate (Par. 105, “the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used) and a weighting scheme”), and respiratory signals from two locations (Par. 105, “acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12 and a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”).
Xu teaches determine a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face (Claim 1, “capturing thermal images of a subject of interest intended to be monitored for respiration using a thermal image camera set to a temperature of a facial region of said subject, each thermal image comprising a plurality of pixels each corresponding to a surface temperature of said facial region across a thermal wavelength band; analyzing said thermal images to determine a facial feature associated with respiration; tracking said facial feature over time to identify a pattern of respiration; determining a respiration rate from said respiration pattern”).
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the device of Jens with that of Xu to include determine a motion-based respiratory rate (RR) of Xu and a motion-based respiratory signal based on the video of the person’s face through the combination of references as it would have yielded the predictable result of explicitly providing a respiratory rate output based on the signal and allow performance of both measurements from solely the face of the user.
Modified Jens highly suggests but fails to explicitly disclose determine a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face (Examiner's Note: Jen fails to explicitly disclose a separate determination step for determining a photoplethysmography (rPPG)-based RR).
Jens does disclose determine an rPPG-based respiratory signal based on the video of the person’s face (Par. 103, “Another way to derive the respiratory signal remotely is by processing the photoplethysmography (PPG) signals calculated from the video. It is generally known in the art that a respiratory signal can be extracted from PPG signals, and that PPGs signal can be derived remotely by measuring the change in the skin area (called remote PPG or R-PPG)…”) (Par. 105, “The device 10e particularly comprises an imaging unit 20, in particular a camera representing the first detection unit and the second detection unit, for acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12” (portion 24 includes the face)) and calculating respiratory rate (Par. 105, “the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used) and a weighting scheme”).
Frank teaches determine a remote photoplethysmography (rPPG)-based RR based on the video of the person’s face (Par. 61, “non-contact camera that captures images of the skin, where a computer extracts the PPG signal from the images using an imaging photoplethysmography (iPPG) technique. Other names known in the art for iPPG include: remote photoplethysmography…”) (Par. 183, “the computer 340 calculates, based on the current set of images, a current heart rate and/or a current respiration rate of the user. For example, the computer 340 may utilize one or more techniques described herein or known in the art for calculating heart rate and/or respiration from iPPG signals”) (Par. 182, “computer 340 may utilize the images 333 to detect additional physiological signals and/or conditions”) (Par. 114 (image of face)).
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the device of Jens and Xu with that of Frank to explicitly include determine a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face of Jens through the combination of references as it would have yielded the predictable result of providing additional respiration information regarding the user (Frank (Par. 183)).
Modified Jens fails to explicitly disclose predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input; and
present one of the motion-based RR or the rPPG-based RR based on the prediction.
However, Jens does disclose predict whether the motion-based signal or the rPPG-based signal is more likely to be accurate using a model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input (Par. 105, “processing unit 36e is configured to weight the first respiratory signal 80 by use of a first quality index, to weight the second respiratory signal 82 by use of a second quality index and to combine the weighted respiratory signal 84 and the weighted motion signal 86 to obtain a weighted combined respiratory signal 88. By the extraction unit 38 the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained.”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used..” “… In one embodiment, a simple combination method is to select the signal with the better quality.”); and
present one of the motion-based RR or the rPPG-based RR based on the prediction (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used..” “… In one embodiment, a simple combination method is to select the signal with the better quality.”) (Par. 105, “processing unit 36e is configured to weight the first respiratory signal 80 by use of a first quality index, to weight the second respiratory signal 82 by use of a second quality index and to combine the weighted respiratory signal 84 and the weighted motion signal 86 to obtain a weighted combined respiratory signal 88. By the extraction unit 38 the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained.”).
Mccann teaches predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input (Claim 5, “wherein the processor outputs either: (a) a first (PPG-derived) estimate of the rate of respiration calculated by determining the frequency of a respiratory cycle induced variation in the output of the photoplethysmograph; or (b) a second (movement-derived) estimate of the rate of respiration calculated by determining the frequency of a respiratory cycle induced variation in the output of the one or more movement sensors, typically in dependence on (i) whether the output of the photoplethysmograph meets one or more quality criteria and/or (ii) whether the first estimate meets one or more accuracy criteria; and also (iii) whether the output of the movement sensors meets one or more quality criteria and/or (iv) whether the second estimate meets one or more accuracy criteria.”) (Par. 45 (machine learning algorithms)); and
present one of the motion-based RR or the rPPG-based RR based on the prediction (Claim 5, (output as indicated above)).
Jens, Xu, Frank, and Mccann are considered to be analogous art to the claimed invention as they are involved with respiratory measurements.
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the device of Jens, Xu, and Frank with that of Mccann to include predict whether the motion-based RR of Xu or the rPPG-based RR of Frank is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal of Jens and Xu and the rPPG-based respiratory signal of Jens as input; and present one of the motion-based RR of Xu or the rPPG-based RR of Frank based on the prediction through the combination of references as differing output methods are known in the art (Jens (Par. 109)) and it would have yielded the predictable result of outputting a more accurate result.
Regarding claim 15, Jens discloses A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to (Par. 104, “FIG. 8 shows a further embodiment of the device 10e for obtaining respiratory information of the subject according to the present invention in a more accurate and reliable way. The device 10e is similar to the device 10b shown in FIG. 4 and like elements are numbered with like reference numerals”)(Par. 105 (analysis unit 30e, processing unit -36e, extraction unit – 38)) (Par. 82, “The analysis unit 30, the processing unit 36 and the extracting unit 38 can be implemented by separate elements (e.g. processors or software functions), but can also be represented and implemented by a common processing apparatus.”) (Par. 114-120 (instructions and computational device implementation)):
capture a video of a person’s face using a camera (Par. 106, “The acquired video contains at least one part of the body showing breathing motion (e.g., the chest and/or belly) and at least one part of skin area. The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection, while the other is from the PPG signal”) (Fig. 8 (imaging unit – 20) (skin portion -24 includes the face));
determine a motion-based respiratory signal based on the video (Par. 105, “a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”) (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection”);
determine an rPPG-based respiratory signal based on the video of the person’s face (Par. 103, “Another way to derive the respiratory signal remotely is by processing the photoplethysmography (PPG) signals calculated from the video. It is generally known in the art that a respiratory signal can be extracted from PPG signals, and that PPGs signal can be derived remotely by measuring the change in the skin area (called remote PPG or R-PPG)…”) (Par. 105, “The device 10e particularly comprises an imaging unit 20, in particular a camera representing the first detection unit and the second detection unit, for acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12” (portion 24 includes the face)) (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection, while the other is from the PPG signal”) (Par. 106 (PPG signal)).
Jens fails to explicitly disclose determine a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face.
However, Jens does disclose determine a motion-based respiratory signal based on the video (Par. 106, “The acquired video is analysed to derive the respiratory signal in two ways: one is based on breathing motion detection”) (Par. 105, “a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”), calculating respiratory rate (Par. 105, “the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used) and a weighting scheme”), and respiratory signals from two locations (Par. 105, “acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12 and a second set of image data 26 detected from a body portion 28 of the subject 12 allowing the extraction of a second respiratory signal 82 related to respiratory information 48 of the subject 12”).
Xu teaches determine a motion-based respiratory rate (RR) and a motion-based respiratory signal based on the video of the person’s face (Claim 1, “capturing thermal images of a subject of interest intended to be monitored for respiration using a thermal image camera set to a temperature of a facial region of said subject, each thermal image comprising a plurality of pixels each corresponding to a surface temperature of said facial region across a thermal wavelength band; analyzing said thermal images to determine a facial feature associated with respiration; tracking said facial feature over time to identify a pattern of respiration; determining a respiration rate from said respiration pattern”).
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the system of Jens with that of Xu to include determine a motion-based respiratory rate (RR) of Xu and a motion-based respiratory signal based on the video of the person’s face through the combination of references as it would have yielded the predictable result of explicitly providing a respiratory rate output based on the signal and allow performance of both measurements from solely the face of the user.
Modified Jens highly suggests but fails to explicitly disclose determine a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face (Examiner's Note: Jen fails to explicitly disclose a separate determination step for determining a photoplethysmography (rPPG)-based RR).
Jens does disclose determine an rPPG-based respiratory signal based on the video of the person’s face (Par. 103, “Another way to derive the respiratory signal remotely is by processing the photoplethysmography (PPG) signals calculated from the video. It is generally known in the art that a respiratory signal can be extracted from PPG signals, and that PPGs signal can be derived remotely by measuring the change in the skin area (called remote PPG or R-PPG)…”) (Par. 105, “The device 10e particularly comprises an imaging unit 20, in particular a camera representing the first detection unit and the second detection unit, for acquiring a first set of image data 22 detected from a skin portion 24 of the subject 12 allowing the extraction of a first respiratory signal 80 related to a respiratory information of the subject 12” (portion 24 includes the face)) and calculating respiratory rate (Par. 105, “the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used) and a weighting scheme”).
Frank teaches determine a remote photoplethysmography (rPPG)-based RR based on the video of the person’s face (Par. 61, “non-contact camera that captures images of the skin, where a computer extracts the PPG signal from the images using an imaging photoplethysmography (iPPG) technique. Other names known in the art for iPPG include: remote photoplethysmography…”) (Par. 183, “the computer 340 calculates, based on the current set of images, a current heart rate and/or a current respiration rate of the user. For example, the computer 340 may utilize one or more techniques described herein or known in the art for calculating heart rate and/or respiration from iPPG signals”) (Par. 182, “computer 340 may utilize the images 333 to detect additional physiological signals and/or conditions”) (Par. 114 (image of face)).
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the system of Jens and Xu with that of Frank to explicitly include determine a remote photoplethysmography (rPPG)-based RR and an rPPG-based respiratory signal based on the video of the person’s face of Jens through the combination of references as it would have yielded the predictable result of providing additional respiration information regarding the user (Frank (Par. 183)).
Modified Jens fails to explicitly disclose predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input; and
present one of the motion-based RR or the rPPG-based RR based on the prediction.
However, Jens does disclose predict whether the motion-based signal or the rPPG-based signal is more likely to be accurate using a model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input (Par. 105, “processing unit 36e is configured to weight the first respiratory signal 80 by use of a first quality index, to weight the second respiratory signal 82 by use of a second quality index and to combine the weighted respiratory signal 84 and the weighted motion signal 86 to obtain a weighted combined respiratory signal 88. By the extraction unit 38 the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained.”) (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used..” “… In one embodiment, a simple combination method is to select the signal with the better quality.”); and
present one of the motion-based RR or the rPPG-based RR based on the prediction (Par. 109, “The two respiratory signals are combined, based on the quality indexes or not, to derive the output respiratory signal (and an overall quality index). The combination can be done by various methods such as logic (e.g., the one with the best quality is used..” “… In one embodiment, a simple combination method is to select the signal with the better quality.”) (Par. 105, “processing unit 36e is configured to weight the first respiratory signal 80 by use of a first quality index, to weight the second respiratory signal 82 by use of a second quality index and to combine the weighted respiratory signal 84 and the weighted motion signal 86 to obtain a weighted combined respiratory signal 88. By the extraction unit 38 the final respiratory signal 90, e.g. the respiratory rate, of the patient is obtained.”).
Mccann teaches predict whether the motion-based RR or the rPPG-based RR is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal and the rPPG-based respiratory signal as input (Claim 5, “wherein the processor outputs either: (a) a first (PPG-derived) estimate of the rate of respiration calculated by determining the frequency of a respiratory cycle induced variation in the output of the photoplethysmograph; or (b) a second (movement-derived) estimate of the rate of respiration calculated by determining the frequency of a respiratory cycle induced variation in the output of the one or more movement sensors, typically in dependence on (i) whether the output of the photoplethysmograph meets one or more quality criteria and/or (ii) whether the first estimate meets one or more accuracy criteria; and also (iii) whether the output of the movement sensors meets one or more quality criteria and/or (iv) whether the second estimate meets one or more accuracy criteria.”) (Par. 45 (machine learning algorithms)); and
present one of the motion-based RR or the rPPG-based RR based on the prediction (Claim 5, (output as indicated above)).
Jens, Xu, Frank, and Mccann are considered to be analogous art to the claimed invention as they are involved with respiratory measurements.
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the system of Jens, Xu, and Frank with that of Mccann to include predict whether the motion-based RR of Xu or the rPPG-based RR of Frank is more likely to be accurate using a trained machine learning model that receives the motion-based respiratory signal of Jens and Xu and the rPPG-based respiratory signal of Jens as input; and present one of the motion-based RR of Xu or the rPPG-based RR of Frank based on the prediction through the combination of references as differing output methods are known in the art (Jens (Par. 109)) and it would have yielded the predictable result of outputting a more accurate result.
Regarding claim 2, modified Jens fails to explicitly disclose the limitations of the claim.
However, Xu further teaches wherein determining the motion-based RR and the motion-based respiratory signal based on the video of the person’s face (as indicated in claim 1 above) comprises:
identifying face landmarks on the person’s face based on the video (Xu (Par. 31, “"locational relationship" refers to information about the location of a subject's extremities of the head and face which are used in accordance herewith to facilitate a determination of a location of the subject's facial features associated with respiration…” “…locational relationships which can be stored in a database as vectors and angles are shown in FIG. 3 are used to isolate the mouth 301 and nostrils, collectively at 302. Locational relationships are used herein to facilitate a determination of the location of the facial feature associated with respiration such that a respiration pattern can be determined.”) (Claim 1 (facial features))(Par. 38, Fig. 4, step 404)), wherein the face landmarks are on the person’s forehead and the person’s nose (Xu (Par. 31, Fig. 3 (face landmarks shown)));
generating a motion signal based on vertical location changes of the face landmarks in the video (Xu (Fig. 4, step 406 (tracking position changes))); and
extracting the motion-based respiratory signal based on the motion signal using spectral analysis (Xu (Fig. 4, step 406,408 (RGB plots of respiration pattern))).
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the method of Jens, Xu, Frank, and Mccann with that of Xu to include wherein determining the motion-based RR and the motion-based respiratory signal based on the video of the person’s face comprises: identifying face landmarks on the person’s face based on the video, wherein the face landmarks are on the person’s forehead and the person’s nose; generating a motion signal based on vertical location changes of the face landmarks in the video; and extracting the motion-based respiratory signal based on the motion signal using spectral analysis for the reasoning as indicated in claim 1 above.
Regarding claims 9 and 16, modified Jens discloses the method of claim 2 above, which comprises the device and system of claims 9 and 16. As the claims are similar, claims 9 and 16 are rejected in the same manner as claim 2.
Regarding claim 7, modified Jens further discloses wherein the camera is coupled to a mobile device, a computer (Jens (Par. 105, “device 10e particularly comprises an imaging unit 20, in particular a camera representing the first detection unit and the second detection unit”)), or a television.
Regarding claims 14, modified Jens discloses the method of claim 7 above, which comprises the device of claim 14. As the claims are similar, claim 14 is rejected in the same manner as claim 7.
Claim(s) 3, 10, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jens in view of Xu, Frank, and Mccann as applied to claims 2, 9, and 16 above, and further in view of Yu (US Pub. No. 20180132744) hereinafter Yu.
Jens, Xu, Frank, and Mccann teach the method of claim 2 above.
Regarding claim 3, modified Jens fails to explicitly disclose the limitations of the claim.
Modified Jens does teach determining the motion-based RR and the motion-based respiratory signal based on the video of the person’s face (as indicated in claim 1 above).
However, Yu teaches removing artifacts from the signal using a kurtosis-based artifacts detection technique (Par. 76, “algorithm A module 510 may determine a distribution type of the information, calculate a kurtosis of the information by using a built-in A algorithm, and generate a kurtosis calculation result…” “... If the kurtosis calculation result is smaller than the threshold value, it may indicate that the information includes noise(s), and the noise(s) may be further recognized and removed. Then the information may be transmitted to the algorithm C module 530”); and
smoothing the signal using a filter (Par. 76, “… pre-process the information to eliminate obvious noise(s) from the information through filtering”).
Jens, Xu, Frank, Mccann, and Yu are considered to be analogous art to the claimed invention as they are involved with physiological data.
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the method of Jens, Xu, Frank, and Mccann with that of Yu to include wherein determining the motion-based RR and the motion-based respiratory signal based on the video of the person’s face of Jens and Xu further comprises: removing artifacts from the motion signal of Jens and Xu using a kurtosis-based motion artifacts detection technique; and smoothing the motion signal of Jens and Xu using a filter through the combination of references as it would have yielded the predictable result of improving signal quality.
Regarding claims 10 and 17, modified Jens discloses the method of claim 3 above, which comprises the device and system of claims 10 and 17. As the claims are similar, claims 10 and 17 are rejected in the same manner as claim 3.
Claim(s) 4, 11, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jens in view of Xu, Frank, and Mccann as applied to claims 1, 8, and 15 above, and further in view of Zhao (US Pub. No. 20240081705) hereinafter Zhao.
Jens, Xu, Frank, and Mccann teach the method of claim 1 above.
Regarding claim 4, modified Jens fails to explicitly disclose the limitations of the claim.
Modified Jens does disclose determining the rPPG-based RR and the rPPG-based respiratory signal based on the video of the person’s face (as indicated in claim 1 above).
However, Zhao teaches wherein determining the rPPG-based RR and the rPPG-based respiratory signal based on the video of the person’s face comprises:
identifying regions of interest on the person’s face (Par. 69, “First, the face ROI tracking unit obtains the capture frame rate of the video capture device and the resolution of each image…” “… face feature point tracking is performed based on the frontal_face_detector of Python dlib (that is: extract 68 feature points of the face, including: eyebrows, eyes, nose, mouth and contour of face, etc.)”) (Par. 70, “The rPPG raw data extraction unit. First, the RGB image in the ROI region of face is subjected to YCbCr space conversion, and three channel thresholds Y, Cb and Cr are established:”);
extracting an rPPG signal for each region of interest based on the video (Par. 70, “The rPPG raw data extraction unit. First, the RGB image in the ROI region of face is subjected to YCbCr space conversion, and three channel thresholds Y, Cb and Cr are established:”) (Par. 71 (pixels));
extracting an inter-beat interval (IBI) signal based on a weighted combination of the rPPG signals corresponding to the regions of interest (Fig. 3, Par. 74 (waveform reconstruction unit)) (Par. 75, “Then, on the basis of waveform peak-seeking and valley-seeking, the respiration and heartbeat frequencies are initially estimated by calculating the average heartbeat interval (IBI), which are recorded as f.sub.br0 and f.sub.hr0, respectively.”); and
extracting the rPPG-based respiratory signal based on the IBI signal (Fig. 3, Par. 74 (waveform reconstruction unit)) (Par. 75, “Then, on the basis of waveform peak-seeking and valley-seeking, the respiration and heartbeat frequencies are initially estimated by calculating the average heartbeat interval (IBI), which are recorded as f.sub.br0 and f.sub.hr0, respectively.”).
Jens, Xu, Frank, Mccann, and Zhao are considered to be analogous art to the claimed invention as they are involved with physiological data.
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the method of Jens, Xu, Frank, and Mccann with that of Zhao to include wherein determining the rPPG-based RR and the rPPG-based respiratory signal based on the video of the person’s face of Jens comprises: identifying regions of interest on the person’s face; extracting an rPPG signal for each region of interest based on the video; extracting an inter-beat interval (IBI) signal based on a weighted combination of the rPPG signals corresponding to the regions of interest; and extracting the rPPG-based respiratory signal based on the IBI signal through the combination of references as it would have yielded the predictable result of improving signal quality (Zhao (Par. 43-44)), which is a known issue for R-PPG measurements (Jens (Par. 103)).
Regarding claims 11 and 18, modified Jens discloses the method of claim 4 above, which comprises the device and system of claims 11 and 18. As the claims are similar, claims 9 and 16 are rejected in the same manner as claim 4.
Claim(s) 5-6, 12-13, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jens in view of Xu, Frank, and Mccann as applied to claims 1, 8, and 15 above, and further in view of Aronovich (US Pub. No. 20240105334) hereinafter Aronovich.
Jens, Xu, Frank, and Mccann teach the method of claim 1 above.
Regarding claim 5, modified Jens fails to explicitly disclose the limitations of the claim.
However, Jens does disclose features extracted from signals (Jens (Par. 108, “…The quality index can be calculated based on the respiratory signal itself, or information extracted from the video or other context information, for example, signal to noise ratio, shape of respiratory signal versus expected physiological pattern, motion artefact, and so on”)).
However, Aronovich teaches wherein the machine learning model is a binary classifier model trained by (Par. 136, 178 (binary classifier model)):
accessing a training dataset comprising multiple training samples, each training sample including a signal, a second signal, and a label indicating whether a value is closer to a ground truth for that training sample (Fig. 3, step 302) (Par. 168, “at 302, the processor accesses data obtained from multiple sensors monitoring the subject and/or the sample individual is obtained, for example, as described with reference to 104 of FIG. 1.”) (Fig. 3, step 312-314) (Par. 173 (ground truth label for parameters)) (Par. 112, “Physiological features represent characteristics of the body of the subject. Examples of physiological parameters, and examples of sensors from which data used to extract the features as physiological parameters include: location of the subject obtained from a location sensor, motion of the subject obtained from a motion sensor…” “…respiration rate obtained from a respiration rate sensor, electro dermal activity obtained from an electrode sensor, activity data obtained from an activity sensor, and sleep data obtained from a sleep sensor.”) (Par. 111, “the features include one or more of: physiological, behavioral and environmental parameters. Other features that do not necessarily fall into the categories of physiological, behavioral, and environmental, may be used. The parameters are fed into the machine learning model, as described herein.”) (Par. 175, “316, the machine learning model is trained on the training dataset(s). The prediction of the headache for the subject is obtained as an outcome of the machine learning model in response to an input of physiological parameters obtained from data outputted by sensors monitoring the subject, for example, as described with reference to FIG. 1”) (Par. 98, “A record includes one or more of: sample physiological, behavioral, and environmental parameters, of a sample individual, and a ground truth label indicative of a state of a headache of the sample individual.”); and
for each training sample (Fig. 3):
extracting features of the respiratory signals (Fig. 3, step 304) (Par. 169, “At 304, the processor extracts features from the data. The features include parameters, which are fed into the ML model. For example, as described with reference to 106 of FIG. 1.”) (Par. 112, “…respiration rate obtained from a respiration rate sensor, electro dermal activity obtained from an electrode sensor, activity data obtained from an activity sensor, and sleep data obtained from a sleep sensor.”) (Par. 111, “the features include one or more of: physiological, behavioral and environmental parameters. Other features that do not necessarily fall into the categories of physiological, behavioral, and environmental, may be used. The parameters are fed into the machine learning model, as described herein.”);
providing the features as input to the machine learning model which predicts whether the value is more likely to be closer to the ground truth value (Par. 175, “At 316, the machine learning model is trained on the training dataset(s). The prediction of the headache for the subject is obtained as an outcome of the machine learning model in response to an input of physiological parameters obtained from data outputted by sensors monitoring the subject, for example, as described with reference to FIG. 1.”) (Par. 112, “…respiration rate obtained from a respiration rate sensor, electro dermal activity obtained from an electrode sensor, activity data obtained from an activity sensor, and sleep data obtained from a sleep sensor.”) (Par. 111, “the features include one or more of: physiological, behavioral and environmental parameters. Other features that do not necessarily fall into the categories of physiological, behavioral, and environmental, may be used. The parameters are fed into the machine learning model, as described herein.”) (Par. 173, (ground truth label)); and
updating parameters of the machine learning model based on a comparison of the label and the prediction (Fig. 3, step 318, Par. 180, “At 318, the ML model may be updated with one or more new records, for example, as described with reference to 118 of FIG. 1”).
Jens, Xu, Frank, Mccann, and Aronovich are considered to be analogous art to the claimed invention as they are involved with physiological data.
Therefore, it would have been obvious to a person of ordinary skill in the art to modify the method of Jens, Xu, Frank, and Mccann with that of Aronovich to include wherein the machine learning model is a binary classifier model trained by: accessing a training dataset comprising multiple training samples, each training sample including a motion-based respiratory signal of Jens and Xu, an rPPG-based respiratory signal of Jens, and a label indicating whether a motion-based RR of Xu or an rPPG-based RR of Frank is closer to a ground truth RR for that training sample; and for each training sample: extracting features of the motion-based respiratory signal of Xu and Jens and the rPPG-based respiratory signal of Jens; providing the features as input to the machine learning model which predicts whether the motion-based RR of Xu or the rPPG-based RR of Frank is more likely to be closer to the ground truth RR; and updating parameters of the machine learning model based on a comparison of the label and the prediction through the substitution of machine learning algorithm structures as differing machine learning algorithm structures are known in the art (Aronovich (Par. 178)) and it would have yielded the predictable result of improving the model accuracy.
Regarding claims 12 and 19, modified Jens discloses the method of claim 5 above, which comprises the device and system of claims 12 and 19. As the claims are similar, claims 12 and 19 are rejected in the same manner as claim 5.
Regarding claim 6, modified Jens further discloses wherein the features include one or more of: a signal noise ratio (Jens (Par. 108, “…The quality index can be calculated based on the respiratory signal itself, or information extracted from the video or other context information, for example, signal to noise ratio, shape of respiratory signal versus expected physiological pattern, motion artefact, and so on”)), a number of peaks, and a skewness.
Regarding claims 13 and 20, modified Jens discloses the method of claim 6 above, which comprises the device and system of claims 13 and 20. As the claims are similar, claims 13 and 20 are rejected in the same manner as claim 6.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ARI SINGH KANE PADDA whose telephone number is (571)272-7228. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Sims can be reached at (571) 272-7540. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ARI S PADDA/ Examiner, Art Unit 3791
/JASON M SIMS/ Supervisory Patent Examiner, Art Unit 3791