DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
The following claims are objected to because of the following informalities and should recite:
Claim 1: lines 22-24,
“ concatenating the filtered 1D signals with the same color channel sequentially to create at a 2D (two-dimension) image for each of the different color channels; and
concatenating the 2D images of the different color channels along a color channel dimension to generate the PAI;”
The same applies to Claim 11.
Claim 6:
line 3: “each of the different color channels”
The same applied to claim 16.
Claim 8:
lines 1-2, “wherein the physiological activity image processing system applies the bandpass filtering to each 1D signal via a bandpass filter”.
The same applies to claim 18.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action. This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
The nonce term “system” in the phrase “color image and signal processing system” for extracting physiological signals according to the color image frames is used in claim(s) 1 & 11, for invokes 35 USC 112(f).
The nonce term “system” in the phrase “physiological activity image processing system” for using the extracted physiological signals to generate a physiological activity image (PAI) embedding the subject's physiological informationt is used in claim(s) 1 & 11, for invokes 35 USC 112(f).
The nonce term “module” in the phrase “report output module” for receiving an inference result with estimated blood oxygen saturation and provided a readable report regarding the subject's blood oxygen saturation status according to the inference result is used in claim(s) 1 & 11, for invokes 35 USC 112(f).
The nonce term “module” in the phrase “face detection module” for detecting the face used in claim(s) 2 & 12 for invokes 35 USC 112(f).
The nonce term “predictor” in the phrase “face landmark predictor” for detecting the facial landmarks used in claim(s) 2 & 12 for invokes 35 USC 112(f).
The term, “module” & “system” & “predictor” are a non-structural generic placeholder that does not include any specific structure for performing the accompany functions. See MPEP 2181.I.A: The following is a list of non-structural generic placeholders that may invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, paragraph 6: "mechanism for," "module for," "device for," "unit for," "component for," "element for," "member for," "apparatus for," "machine for," or "system for." Welker Bearing Co., v. PHD, Inc., 550 F.3d 1090, 1096, 89 USPQ2d 1289, 1293-94 (Fed. Cir. 2008); Massachusetts Inst. of Tech. v. Abacus Software, 462 F.3d 1344, 1354, 80 USPQ2d 1225, 1228 (Fed. Cir. 2006); Personalized Media, 161 F.3d at 704, 48 USPQ2d at 1886–87; Mas-Hamilton Group v. LaGard, Inc., 156 F.3d 1206, 1214-1215, 48 USPQ2d 1010, 1017 (Fed. Cir. 1998).
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f).
Please note that for the purposes of this examination the phrase “color image and signal processing system” is being interpreted to include generic front-end processing performed on a “processor” as described in Paragraphs [0019-0021, 0064, & 0066, FIG. 1] in the specification as performing the claimed function, and equivalents thereof.
Please note that for the purposes of this examination the phrase “physiological activity image processing system” is being interpreted to include generic middle/back-end processing performed on a “processor” as described in Paragraphs [0019, 0022, 0064 & 0066, FIG. 1] in the specification as performing the claimed function, and equivalents thereof.
Please note that for the purposes of this examination the phrase “report output module” is being interpreted to include generic module performed on a “processor” as described in Paragraphs [0064 & 0066] in the specification as performing the claimed function, and equivalents thereof.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-4 & 13-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as failing to set forth the subject matter which the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the applicant regards as the invention.
Claim 1:
“the approximated facial landmarks”-lines 10-11. There is insufficient antecedent basis for this limitation in the claim, as required by MPEP 2173.05(e). For examination purposes, the Examiner assumes the identified facial landmarks. Accordingly, proper antecedent basis is required.
“the same color channel”-line 22. There is insufficient antecedent basis for this limitation in the claim, as required by MPEP 2173.05(e). For examination purposes, the Examiner assumes a common color channel. Accordingly, proper antecedent basis is required.
“the subject's blood oxygen saturation status”-lines 32-33. There is insufficient antecedent basis for this limitation in the claim, as required by MPEP 2173.05(e). For examination purposes, the Examiner assumes a blood oxygen saturation status of the subject. Accordingly, proper antecedent basis is required.
The above rejections to claim 1 apply to claim 11 for substantially identical claim limitations recited in the claim. Appropriate correction is required.
Claim 3:
“the facial representation”. There is insufficient antecedent basis for this limitation in the claim, as required by MPEP 2173.05(e). For examination purposes, the Examiner assumes a facial representation of the subject. Accordingly, proper antecedent basis is required.
The above rejections to claim 3 apply to claim 13 for substantially identical claim limitations recited in the claim. Appropriate correction is required.
The dependent claims of the above rejected claims are rejected due to their dependency.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-13, & 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al (An Algorithmic Benchmark for Contactless Blood Oxygen Saturation Measurement from Facial Videos (2022)) in view of Wang et al (US20200151852A1) in view of Akamatsu et al (Blood Oxygen Saturation Estimation from Facial Video via DC and AC components of Spatio-temporal Map - 2022) in view of Dcruz et al (US20220254502A1).
Claim 1: Cheng discloses: A system for remote deep learning-based blood oxygen saturation monitoring, comprising: (¶Abstract pg. 1 & [Introduction pg. 1-2] In this system, includes the process and evaluation utilizing deep learning baselines for remote, contactless blood oxygen saturation (SpO2) measurement from facial videos captured by visible RGB cameras.)
a camera configured to capture color image frames of a subject; ([Abstract] - utilize a spatial-temporal representation to encode SpO2 information recorded by conventional RGB cameras [Introduction pg. 1-2] & [Performance on Different Subject Scenarios and Acquisition Devices pg. 4-5] - (1) Logitech C310 web camera (960 x 720, 25fps), (2) HUAWEI P9 frontal camera (1920 x 1080, 30fps), and (3) RealSense F200 RGB-D camera (1920 x 1080, 30fps) to capture color facial video frames. [Dataset 3.3 pg. 3]: The dataset contains 2378 RGB and 752 near-infrared (NIR) facial videos of 107 subjects (79 males and 28 females) recorded by four acquisition devices (web camera, smartphone frontal camera, RGB D camera, and NIR camera).)
a color image and signal processing system configured to extract physiological signals according to the color image frames of the subject by ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the front end of the Spatial-Temporal Map (STMAp) generation pipeline):
executing light intensity analysis for the color image frames of the subject; ([Introduction pg. 1-2] & [2.3 Deep Learning-Based Remote Vital Signs Monitoring pg. 2] – captured subtle pixel intensity variations across video frames caused by the blood volume and hemoglobin concentration changes (rPPG) principle.)
identifying a location of a face within the color image frames to detect facial images; ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] & [Conclusion pg. 6]: the deep learning benchmarks for remote SpO2 measurement from facial videos in the VIPL-HR public database. We encoded the facial videos into STMaps, low dimensional spatial-temporal representations containing physiological information of the subject, and directly used them as the model inputs for training and testing. We then investigated the model performances using different STMap color spaces, on different subject scenarios, acquisition devices, and over different SpO2 ranges. The proposed deep learning methods outperform the conventional Ratio of Ratios technique in all cases, setting a solid baseline for upcoming research.)
resizing the detected facial images; creating facial patches from the resized facial images;
selecting the facial patches; ([Methods 3.1 Spatial-temporal Maps Generation pg. 2]: randomly sampled 225 consecutive frames and used a face detector (OpenFace [4]) to obtain the subject’s face location. The facial frames were downsampled to 128 x 128 using an average pooling filter (kernel size = 16 and stride = 16) to reduce noise and image dimension. Each frame was then split into 64 patches (8 x 8), and the average value of the color channels within each patch was extracted into a temporal sequence.) and
setting different color channels and then extracting the physiological signals from the selected facial patches; ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] the temporal sequence is extracted from the patches across the multiple color spaces, including the RGB as well as the transformed YUV and YCrCb map.)
a physiological activity image processing system configured to use the extracted physiological signals to generate a physiological activity image (PAI) embedding the subject's physiological information by ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the middle/back end of the Spatial-Temporal Map (STMAp) generation pipeline, see also section 2.3 & 2.4 pg. 2 – the 2D image encoding spatial and temporal physiological information):
forming 1D (one-dimension) signals from the extracted physiological signals; ([3.1 Spatial-temporal Maps Generation pg. 2-3] for each path the average color is extracted, creating 1D temporal sequence over time that encodes the blood volume fluctuations (rPPG/physiological information)
concatenating the 1D signals with the same color channel sequentially to create at a 2D (two-dimension) image for each of the color channels ([3.1 Spatial-temporal Maps generation pg. 2-3] & The 1D temporal extraction is formed into the 2D map per color channels); and
concatenating the 2D images of different color channels along a color channel dimension to generate the PAI; ([3.1 Spatial-temporal Maps Generation pg. 3] -these signals are combines across the color channels. Specifically, the c color dimensions for each face patch were concatenated to produce a final spatial temporal representation. [4.1 Performance and different color spaces, pg. 4] When combining different color spaces such as the (RGB + YUV) the 2D maps of each color are concatenated along the color channel dimension c to construct the STMap fed into the 2D CNN.)
a blood oxygen saturation estimator configured to receive the PAI for inference, resulting in an estimated blood oxygen saturation level of the subject, wherein the blood oxygen saturation estimator has a deep learning model set up for SpO2 estimation; and ([Introduction pg. 1 right color last paragraph to first paragraph left column on pg. 3], [2.4 Spatial-temporal representation for vital signs estimation pg. 2], [3.2 SpO2 Estimation Using CNNs pg. 3] & [3.4 Evaluation Metrics pg. 3]: SpO2 estimation is define via the regression of using 2D CNN. The three state of the art of deep learning is refer to in Table 1 on pg. 3. The STMap is passed directly as the input tensor into the 2D CNN model. The interface on the STMAp though the 2D CNNN yields a single predicted SpO2 value in percentage for the subject.)
report an inference result with estimated blood oxygen saturation (¶3.2 SpO2 Estimation Using CNN pg. 3] & [3.4 Evaluation Metrics pg. 3]: the SpO2 estimation uses the 2D CNN model and there is an outputted percentage of the SpO2).
Cheng fails to disclose: identifying facial landmarks and interpolating the approximated facial landmarks; and selecting the facial patches based on the interpolated facial landmarks; and
However, Wang in the context of detecting facial landmarks, interpolating facial landmarks, & obtaining patches centered on the interpolated facial landmarks, ¶Abstract, FIG. 9, discloses, identifying facial landmarks (¶0031, ¶0042, ¶0080, ¶0093, ¶0099) and interpolating the approximated facial landmarks (¶0042-0043, ¶0048, ¶0070: bicubic/binary interpolating ); and selecting the facial patches based on the interpolated facial landmarks (¶0044, ¶0069);
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the color image and signal processing system of Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face, as suggested by Wang ¶0043.
Cheng fails to explicitly disclose: the bandpass filtering such that
applying bandpass filtering to each 1D signal;
concatenating the filtered 1D signals with the same color channel sequentially to create at a 2D (two-dimension) image for each of the color channels; and
However, Akamatsu in the field of blood oxygen saturation estimation from facial video of spatio-temporal map using CNN field of endeavor, discloses: applying bandpass filtering to each 1D signal;
concatenating the filtered 1D signals with the same color channel sequentially to create at a 2D (two-dimension) image for each of the color channels; and ([3.1 DC and AC components of Spatio-Temporal Map pg. 2])
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the converting of 1D signals into 2D color channel images of modified Cheng to incorporate the teachings of Akamatsu bandpass filter. The motivation to do this yield predictable results such as to improve the blood oxygen saturation estimation performance, as suggested by Akamatsu [Introduction pg. 1].
Cheng fails to disclose: and a report output module configured to receive an inference result with estimated blood oxygen saturation and provided a readable report regarding the subject's blood oxygen saturation status according to the inference result.
However, Dcruz in the context of non-contact physiological monitoring system using facial videos, discloses, a report output module (¶0108, ¶0110) configured to receive an inference result with estimated blood oxygen saturation (¶0027, ¶0029 ¶0033, ¶0045, ¶0055, ¶0088) and provided a readable report (¶0029, ¶0088, ¶0084) regarding the subject's blood oxygen saturation status according to the inference result (¶0027: a non-invasive and non-contact report will be quickly and correctively produced for health characteristics such as blood oxygen levels (Spo2) of the user which may provide warning for any abnormal range. ¶0055: The result shown to user consists of SPO2 readings in (%) and also health alert with textual wordings to help user understand the alert.).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the output of the interference result of modified Cheng to incorporate the teachings of Dcruz. The motivation to do this yield predictable results such as providing clear visual understanding to help and alter the user of percentage values, as suggested by Dcruz, ¶0055.
Claim 2: Cheng as modified discloses all the elements above in claim 1, Cheng discloses, wherein the color image and signal processing system comprises a face detection model for face detection. ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] & [Conclusion pg. 6]: the deep learning benchmarks for remote SpO2 measurement from facial videos in the VIPL-HR public database. We encoded the facial videos into STMaps, low dimensional spatial-temporal representations containing physiological information of the subject, and directly used them as the model inputs for training and testing. We then investigated the model performances using different STMap color spaces, on different subject scenarios, acquisition devices, and over different SpO2 ranges. The proposed deep learning methods outperform the conventional Ratio of Ratios technique in all cases, setting a solid baseline for upcoming research.)
Cheng fails to disclose: the color image and signal processing system comprises a facial landmark predictor for facial landmarks detection.
However, Wang is relied upon above discloses, ¶Abstract, FIG. 9, discloses, the color image and signal processing system comprises a facial landmark predictor (¶0042-0044, ¶0048, ¶0069-0070: bicubic/binary interpolating) for facial landmarks detection (¶0031, ¶0042, ¶0080, ¶0093, ¶0099).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the color image and signal processing system of modified Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face, as suggested by Wang ¶0043.
Claim 3: Cheng as modified discloses all the elements above in claim 2, Cheng discloses, wherein the face detection model is configured to process the color image frames for detecting a bounding box of the face representation within the color image frames, (FIG. 1 - ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] & [Conclusion pg. 6]: the deep learning benchmarks for remote SpO2 measurement from facial videos in the VIPL-HR public database. We encoded the facial videos into STMaps, low dimensional spatial-temporal representations containing physiological information of the subject, and directly used them as the model inputs for training and testing. We then investigated the model performances using different STMap color spaces, on different subject scenarios, acquisition devices, and over different SpO2 ranges. The proposed deep learning methods outperform the conventional Ratio of Ratios technique in all cases, setting a solid baseline for upcoming research.))
Cheng fails to disclose:
wherein the facial landmark predictor applies a detection algorithm to identify key facial points and to determine the facial landmarks, defining regions of interest (ROIs) on the face representation.
However, Wang is relied upon above discloses, ¶Abstract, FIG. 9, discloses, wherein the facial landmark predictor (¶0042-0044, ¶0048, ¶0069-0070: bicubic/binary interpolating) applies a detection algorithm (¶0042-0044, ¶0048, ¶0069-0070: bicubic/binary interpolating) to identify key facial points and to determine the facial landmarks, defining regions of interest (ROIs) on the face representation (¶0031, ¶0042, ¶0080, ¶0093, ¶0099).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the facial landmark predictor modified Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face, as suggested by Wang ¶0043.
Claim 5: Cheng as modified discloses all the elements above in claim 1, Cheng discloses, the color image and signal processing system, ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the front end of the Spatial-Temporal Map (STMAp) generation pipeline)
Cheng fails to disclose: wherein the color image and signal processing system selects the facial patches near predetermined ROIs as defined by the identifying the location of the face and the facial landmarks, and wherein the predetermined ROIs include areas around eyes, nose, mouth, or combinations thereof.
However, Wang is relied upon above discloses, selects the facial patches near predetermined ROIs as defined by the identifying the location of the face and the facial landmarks (¶0042, ¶0044, ¶0069), and wherein the predetermined ROIs include areas around eyes, nose, mouth, or combinations thereof (¶0036, ¶0042, ¶0044).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the color and signal processing system modified Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face.
Claim 6: Cheng as modified discloses all the elements above in claim 1, Cheng discloses, wherein the color image and signal processing system extracts the physiological signals from the selected facial patches via extracting a mean pixel intensity value for each of different color channels within the facial patch. ([2.3 Deep Learning-Based Remote Vital Signs Monitoring 2.4 Spatial-temporal Representation for Vital Signs Estimation pg. 2] & [Methods 3.1 Spatial-temporal Maps Generation pg. 2-3])
Claim 7: Cheng as modified discloses all the elements above in claim 6, Cheng discloses, wherein the mean pixel intensity values are packaged as the physiological signals by the color image and signal processing system, completing the extracting the physiological signals ([2.3 Deep Learning-Based Remote Vital Signs Monitoring 2.4 Spatial-temporal Representation for Vital Signs Estimation pg. 2] & [Methods 3.1 Spatial-temporal Maps Generation pg. 2-3]).
Claim 8: Cheng as modified discloses all the elements above in claim 1, Cheng discloses: the physiological activity image processing system. ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the middle/back end of the Spatial-Temporal Map (STMAp) generation pipeline, see also section 2.3 & 2.4 pg. 2 – the 2D image encoding spatial and temporal physiological information)
Cheng fails to disclose: wherein the physiological activity image processing system applies bandpass filtering to each 1D signal via a bandpass filter with any order, low cutoff frequency, or high cutoff frequency to remove noise and retain physiological-related information.
However, Akamatsu is relied upon above discloses, wherein the physiological activity image processing system applies bandpass filtering to each 1D signal via a bandpass filter with any order, low cutoff frequency, or high cutoff frequency to remove noise and retain physiological-related information ([2. Preliminary: Ratio-Of-Ratio pg 1-2] & [3.1 DC and AC components of Spatio-Temporal Map pg. 2]).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the physiological activity image processing system of modified Cheng to incorporate the teachings of Akamatsu bandpass filter. The motivation to do this yield predictable results such as to improve the blood oxygen saturation estimation performance, as suggested by Akamatsu [Introduction pg. 1].
Claim 9: Cheng as modified discloses all the elements above in claim 1, Cheng discloses: wherein the deep learning model of the blood oxygen saturation estimator is trained using a training set comprising processed PAIs for a group of testers and SpO2 ground truth values of the group of the testers, such that the SpO2 estimation is presented based on the SpO2 ground truth values ([3.2 SpO2 Estimation Using CNNs pg. 3], [3.3 Dataset pg. 3] & 3.5 Training Settings pg. 4], Table 2-3 & FIG. 3).
Claim 10: Cheng as modified discloses all the elements above in claim 9, Cheng discloses: wherein the deep learning model of the blood oxygen saturation estimator is further configured to extract features from the PAIs in the training set and establish a functional relationship with the SpO2 ground truth values (2.4 Spatial-temporal representation for vital sign estimation pg. 2, see also ([3.2 SpO2 Estimation Using CNNs pg. 3], [3.3 Dataset pg. 3] & 3.5 Training Settings pg. 4], Table 2-3 & FIG. 3)), thereby mapping the PAI from the physiological activity image processing system to the SpO2 estimation ([Introduction pg. 1 last para right col to pg. 2 left col first para.] & [3.2 SpO2 Estimation Using CNNs pg. 3], [3.3 Dataset pg. 3] & 3.5 Training Settings pg. 4], Table 2-3 & FIG. 3).
Claim 11: Cheng discloses: A method for remote deep learning-based blood oxygen saturation monitoring, comprising: (¶Abstract pg. 1 & [Introduction pg. 1-2] In this system, includes the process and evaluation utilizing deep learning baselines for remote, contactless blood oxygen saturation (SpO2) measurement from facial videos captured by visible RGB cameras.)
capturing color image frames of a subject using a camera; ([Abstract] - utilize a spatial-temporal representation to encode SpO2 information recorded by conventional RGB cameras [Introduction pg. 1-2] & [Performance on Different Subject Scenarios and Acquisition Devices pg. 4-5] - (1) Logitech C310 web camera (960 x 720, 25fps), (2) HUAWEI P9 frontal camera (1920 x 1080, 30fps), and (3) RealSense F200 RGB-D camera (1920 x 1080, 30fps) to capture color facial video frames. [Dataset 3.3 pg. 3]: The dataset contains 2378 RGB and 752 near-infrared (NIR) facial videos of 107 subjects (79 males and 28 females) recorded by four acquisition devices (web camera, smartphone frontal camera, RGB D camera, and NIR camera).)
extracting physiological signals according to the color image frames of the subject using a color image and signal processing system by steps of: ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the front end of the Spatial-Temporal Map (STMAp) generation pipeline)
executing light intensity analysis for the color image frames of the subject; ([Introduction pg. 1-2] & [2.3 Deep Learning-Based Remote Vital Signs Monitoring pg. 2] – captured subtle pixel intensity variations across video frames caused by the blood volume and hemoglobin concentration changes (rPPG) principle.)
identifying a location of a face within the color image frames to detect facial images; ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] & [Conclusion pg. 6]: the deep learning benchmarks for remote SpO2 measurement from facial videos in the VIPL-HR public database. We encoded the facial videos into STMaps, low dimensional spatial-temporal representations containing physiological information of the subject, and directly used them as the model inputs for training and testing. We then investigated the model performances using different STMap color spaces, on different subject scenarios, acquisition devices, and over different SpO2 ranges. The proposed deep learning methods outperform the conventional Ratio of Ratios technique in all cases, setting a solid baseline for upcoming research.)
resizing the detected facial images; creating facial patches from the resized facial images;
selecting the facial patch ([Methods 3.1 Spatial-temporal Maps Generation pg. 2]: randomly sampled 225 consecutive frames and used a face detector (OpenFace [4]) to obtain the subject’s face location. The facial frames were downsampled to 128 x 128 using an average pooling filter (kernel size = 16 and stride = 16) to reduce noise and image dimension. Each frame was then split into 64 patches (8 x 8), and the average value of the color channels within each patch was extracted into a temporal sequence.); and
setting different color channels and then extracting the physiological signals from the selected facial patches ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] the temporal sequence is extracted from the patches across the multiple color spaces, including the RGB as well as the transformed YUV and YCrCb map.);
using the extracted physiological signals to generate a physiological activity image (PAI) embedding the subject's physiological information, using a physiological activity image processing system, by steps of: ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the middle/back end of the Spatial-Temporal Map (STMAp) generation pipeline, see also section 2.3 & 2.4 pg. 2 – the 2D image encoding spatial and temporal physiological information)
forming 1D (one-dimension) signals from the extracted physiological signals; ([3.1 Spatial-temporal Maps Generation pg. 2-3] for each path the average color is extracted, creating 1D temporal sequence over time that encodes the blood volume fluctuations (rPPG/physiological information)
concatenating the 1D signals with the same color channel sequentially to create at a 2D (two-dimension) image for each of the color channels ([3.1 Spatial-temporal Maps generation pg. 2-3] & The 1D temporal extraction is formed into the 2D map per color channels); and
concatenating the 2D images of different color channels along a color channel dimension to generate the PAI ([3.1 Spatial-temporal Maps Generation pg. 3] -these signals are combines across the color channels. Specifically, the c color dimensions for each face patch were concatenated to produce a final spatial temporal representation. [4.1 Performance and different color spaces, pg. 4] When combining different color spaces such as the (RGB + YUV) the 2D maps of each color are concatenated along the color channel dimension c to construct the STMap fed into the 2D CNN.);
receiving the PAI for inference by a blood oxygen saturation estimator, resulting in an estimated blood oxygen saturation level of the subject, wherein the blood oxygen saturation estimator has a deep learning model set up for SpO2 estimation ([Introduction pg. 1 right color last paragraph to first paragraph left column on pg. 3], [2.4 Spatial-temporal representation for vital signs estimation pg. 2], [3.2 SpO2 Estimation Using CNNs pg. 3] & [3.4 Evaluation Metrics pg. 3]: SpO2 estimation is define via the regression of using 2D CNN. The three state of the art of deep learning is refer to in Table 1 on pg. 3. The STMap is passed directly as the input tensor into the 2D CNN model. The interface on the STMAp though the 2D CNNN yields a single predicted SpO2 value in percentage for the subject.); and
receiving an inference result with estimated blood oxygen saturation (¶3.2 SpO2 Estimation Using CNN pg. 3] & [3.4 Evaluation Metrics pg. 3]: the SpO2 estimation uses the 2D CNN model and there is an outputted percentage of the SpO2).
Cheng fails to disclose: identifying facial landmarks and interpolating the approximated facial landmarks; and selecting the facial patches based on the interpolated facial landmarks
However, Wang in the context of detecting facial landmarks, interpolating facial landmarks, & obtaining patches centered on the interpolated facial landmarks, ¶Abstract, FIG. 9, discloses, identifying facial landmarks (¶0031, ¶0042, ¶0080, ¶0093, ¶0099) and interpolating the approximated facial landmarks (¶0042-0043, ¶0048, ¶0070: bicubic/binary interpolating ); and selecting the facial patches based on the interpolated facial landmarks (¶0044, ¶0069);
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the color image and signal processing system of Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face, as suggested by Wang ¶0043.
Cheng fails to explicitly disclose: the bandpass filtering such that
applying bandpass filtering to each 1D signal;
concatenating the filtered 1D signals with the same color channel sequentially to create at a 2D (two-dimension) image for each of the color channels
However, Akamatsu in the field of blood oxygen saturation estimation from facial video of spatio-temporal map using CNN field of endeavor, discloses: applying bandpass filtering to each 1D signal;
concatenating the filtered 1D signals with the same color channel sequentially to create at a 2D (two-dimension) image for each of the color channels; and ([3.1 DC and AC components of Spatio-Temporal Map pg. 2])
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the converting of 1D signals into 2D color channel images of modified Cheng to incorporate the teachings of Akamatsu bandpass filter. The motivation to do this yield predictable results such as to improve the blood oxygen saturation estimation performance, as suggested by Akamatsu [Introduction pg. 1].
Cheng fails to disclose:
and providing a readable report regarding the subject's blood oxygen saturation status according to the inference result, using a report output model.
However, Dcruz in the context of non-contact physiological monitoring system using facial videos, discloses, providing a readable report (¶0029, ¶0088, ¶0084) regarding the subject's blood oxygen saturation status according to the inference result (¶0027, ¶0029 ¶0033, ¶0045, ¶0055, ¶0088), using a report output model (¶0108, ¶0110). (¶0027: a non-invasive and non-contact report will be quickly and correctively produced for health characteristics such as blood oxygen levels (Spo2) of the user which may provide warning for any abnormal range. ¶0055: The result shown to user consists of SPO2 readings in (%) and also health alert with textual wordings to help user understand the alert.)
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the output of the interference result of modified Cheng to incorporate the teachings of Dcruz. The motivation to do this yield predictable results such as providing clear visual understanding to help and alter the user of percentage values, as suggested by Dcruz, ¶0055.
Claim 12: Cheng as modified discloses all the elements above in claim 11, Cheng discloses:
wherein the color image and signal processing system comprises a face detection model for face detection. ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] & [Conclusion pg. 6]: the deep learning benchmarks for remote SpO2 measurement from facial videos in the VIPL-HR public database. We encoded the facial videos into STMaps, low dimensional spatial-temporal representations containing physiological information of the subject, and directly used them as the model inputs for training and testing. We then investigated the model performances using different STMap color spaces, on different subject scenarios, acquisition devices, and over different SpO2 ranges. The proposed deep learning methods outperform the conventional Ratio of Ratios technique in all cases, setting a solid baseline for upcoming research.)
Cheng fails to disclose: wherein the color image and signal processing system comprises a a facial landmark predictor for facial landmarks detection.
However, Wang is relied upon above discloses, ¶Abstract, FIG. 9, discloses, the color image and signal processing system comprises a facial landmark predictor (¶0042-0044, ¶0048, ¶0069-0070: bicubic/binary interpolating) for facial landmarks detection (¶0031, ¶0042, ¶0080, ¶0093, ¶0099).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the color image and signal processing system of modified Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face, as suggested by Wang ¶0043.
Claim 13: Cheng as modified discloses all the elements above in claim 12, Cheng discloses, further comprising: processing the color image frames for detecting a bounding box of the face representation within the color image frames using the face detection model (FIG. 1 - ([Methods 3.1 Spatial-temporal Maps Generation pg. 2] & [Conclusion pg. 6]: the deep learning benchmarks for remote SpO2 measurement from facial videos in the VIPL-HR public database. We encoded the facial videos into STMaps, low dimensional spatial-temporal representations containing physiological information of the subject, and directly used them as the model inputs for training and testing. We then investigated the model performances using different STMap color spaces, on different subject scenarios, acquisition devices, and over different SpO2 ranges. The proposed deep learning methods outperform the conventional Ratio of Ratios technique in all cases, setting a solid baseline for upcoming research.); and
Cheng fails to disclose: and executing a detection algorithm by the facial landmark predictor to identify key facial points and to determine the facial landmarks, defining regions of interest (ROIs) on the face representation.
However, Wang is relied upon above discloses, ¶Abstract, FIG. 9, discloses, executing a detection algorithm (¶0042-0044, ¶0048, ¶0069-0070: bicubic/binary interpolating) by the facial landmark predictor (¶0042-0044, ¶0048, ¶0069-0070: bicubic/binary interpolating) to identify key facial points and to determine the facial landmarks, defining regions of interest (ROIs) on the face representation (¶0031, ¶0042, ¶0080, ¶0093, ¶0099).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the facial landmark predictor modified Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face, as suggested by Wang ¶0043.
Claim 15: Cheng as modified discloses all the elements above in claim 11, Cheng discloses, the color image and signal processing system, ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the front end of the Spatial-Temporal Map (STMAp) generation pipeline)
Cheng fails to disclose: wherein the color image and signal processing system selects the facial patches near predetermined ROIs as defined by the identifying the location of the face and the facial landmarks, and wherein the predetermined ROIs include areas around eyes, nose, mouth, or combinations thereof.
However, Wang is relied upon above discloses, selects the facial patches near predetermined ROIs as defined by the identifying the location of the face and the facial landmarks (¶0042, ¶0044, ¶0069), and wherein the predetermined ROIs include areas around eyes, nose, mouth, or combinations thereof (¶0036, ¶0042, ¶0044).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the color and signal processing system modified Cheng to incorporate the teachings of Wang. The motivation to do this yield predictable results such as improving the measurements by extending coverage of two or more regions of the face, as suggested by Wang ¶0043.
Claim 16: Cheng as modified discloses all the elements above in claim 11, Cheng discloses, wherein the color image and signal processing system extracts the physiological signals from the selected facial patches via extracting a mean pixel intensity value for each of different color channels within the facial patch ([2.3 Deep Learning-Based Remote Vital Signs Monitoring 2.4 Spatial-temporal Representation for Vital Signs Estimation pg. 2] & [Methods 3.1 Spatial-temporal Maps Generation pg. 2-3]).
Claim 17: Cheng as modified discloses all the elements above in claim 16, Cheng discloses, wherein the mean pixel intensity values are packaged as the physiological signals by the color image and signal processing system, completing the extracting the physiological signals ([2.3 Deep Learning-Based Remote Vital Signs Monitoring 2.4 Spatial-temporal Representation for Vital Signs Estimation pg. 2] & [Methods 3.1 Spatial-temporal Maps Generation pg. 2-3]).
Claim 18: Cheng as modified discloses all the elements above in claim 11, Cheng discloses: the physiological activity image processing system ([3.1 Spatial-temporal Maps Generation pg. 2-3]: the middle/back end of the Spatial-Temporal Map (STMAp) generation pipeline, see also section 2.3 & 2.4 pg. 2 – the 2D image encoding spatial and temporal physiological information)
Cheng fails to disclose: wherein the physiological activity image processing system applies bandpass filtering to each 1D signal via a bandpass filter with any order, low cutoff frequency, or high cutoff frequency to remove noise and retain physiological-related information.
However, Akamatsu is relied upon above discloses, wherein the physiological activity image processing system applies bandpass filtering to each 1D signal via a bandpass filter with any order, low cutoff frequency, or high cutoff frequency to remove noise and retain physiological-related information. ([2. Preliminary: Ratio-Of-Ratio pg 1-2] & [3.1 DC and AC components of Spatio-Temporal Map pg. 2]).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the physiological activity image processing system of modified Cheng to incorporate the teachings of Akamatsu bandpass filter. The motivation to do this yield predictable results such as to improve the blood oxygen
Claim 19: Cheng as modified discloses all the elements above in claim 11, Cheng discloses:, further comprising: training the deep learning model of the blood oxygen saturation estimator using a training set comprising processed PAIs for a group of testers and SpO2 ground truth values of the group of the testers, such that the SpO2 estimation is presented based on the SpO2 ground truth values ([3.2 SpO2 Estimation Using CNNs pg. 3], [3.3 Dataset pg. 3] & 3.5 Training Settings pg. 4], Table 2-3 & FIG. 3).
Claim 20: Cheng as modified discloses all the elements above in claim 19, Cheng discloses: further comprising: extracting features from the PAIs in the training set and establishing a functional relationship with the SpO2 ground truth values, thereby mapping the PAI from the physiological activity image processing system to the SpO2 estimation ([Introduction pg. 1 last para right col to pg. 2 left col first para.] & [3.2 SpO2 Estimation Using CNNs pg. 3], [3.3 Dataset pg. 3] & 3.5 Training Settings pg. 4], Table 2-3 & FIG. 3).
Claims 4 & 14 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al (An Algorithmic Benchmark for Contactless Blood Oxygen Saturation Measurement from Facial Videos (2022)) in view of Wang et al (US20200151852A1) in view of Akamatsu et al (Blood Oxygen Saturation Estimation from Facial Video via DC and AC components of Spatio-temporal Map - 2022) in view of Dcruz et al (US20220254502A1), as applied to claim 1 & 11, respectively, in further view of Akinyelu and Blignaut (Convolutional Neural Network-Based Technique for Gaze Estimation on Mobile Devices. Front Artif Intell. 2022 Jan 26).
Claim 4: Cheng as modified discloses all the elements above in claim 1, Cheng fails to disclose: wherein the color image and signal processing system is further configured to resize the detected facial images first and then resize the facial landmarks to maintain alignment with the resized facial images for consistency.
However, Akinyelu and Blignaut disclose in the context of convolutional networks for facial detection and landmarks discloses: wherein the color image and signal processing system (¶Abstract) is further configured to resize the detected facial images first ([Input to the Convolutional Neural Network pg. 6]: the original frame dimensions are then resized as the primary pre-processing step) and then resize the facial landmarks (Fig. 3 -[Input to the Convolutional Neural Network & Model Architecture- pg. 6]: following the extraction step, 39-point facial landmarks from the cropped face images and then resized the resulting facial landmark representations.) to maintain alignment with the resized facial images for consistency (FIG. 3 - [Input to the Convolutional Neural Network & Model Architecture- pg. 6]: both the face image and facial landmark image input are scaled to the same dimensions can be fed into parallel branches of the dual stream CNN).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the color image signal processing system of modified Cheng to incorporate the step-wise teachings taught by Akinyelu and Blignaut. The motivation to do this yield predictable results such as improving the performance of the model, as suggested by Akinyelu and Blignaut [Results and Discussion, pg. 8].
Claim 14: Cheng as modified discloses all the elements above in claim 11, Cheng fails to disclose: further comprising: resizing the detected facial images first and then resizing the facial landmarks to maintain alignment with the resized facial images for consistency.
However, Akinyelu and Blignaut disclose in the context of convolutional networks for facial detection and landmarks discloses: further comprising: resizing the detected facial images first ([Input to the Convolutional Neural Network pg. 6]: the original frame dimensions are then resized as the primary pre-processing step) and then resizing the facial landmarks (Fig. 3 -[Input to the Convolutional Neural Network & Model Architecture- pg. 6]: following the extraction step, 39-point facial landmarks from the cropped face images and then resized the resulting facial landmark representations.) to maintain alignment with the resized facial images for consistency (FIG. 3 - [Input to the Convolutional Neural Network & Model Architecture- pg. 6]: both the face image and facial landmark image input are scaled to the same dimensions can be fed into parallel branches of the dual stream CNN).
It would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method of modified Cheng to incorporate the step-wise teachings taught by Akinyelu and Blignaut. The motivation to do this yield predictable results such as improving the performance of the model, as suggested by Akinyelu and Blignaut [Results and Discussion, pg. 8].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nicholas Robinson whose telephone number is (571)272-9019. The examiner can normally be reached M-F 9:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.A.R./Examiner, Art Unit 3798
/PASCAL M BUI PHO/Supervisory Patent Examiner, Art Unit 3798
