DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Amendments made to specification have overcome all previously held objections
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 9, 21-22, and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Wu (US 12245845 B2) and Jacquel (US 10667723 B2).
With respect to claim 1, Wu teaches a method for determining one or more physiological parameters of a subject (“Some example embodiments may relate to methods, apparatuses, and/or systems for contact-free monitoring of heart rate(s) using videos of people.” Page 13 Field), the method comprising: generating video data, the video data being reproducible as a plurality of frames, each of the plurality of frames depicting at least a portion of the subject (“Certain embodiments are directed to a method of remote photoplethysmography (rPPG) and/or a rPPG system. The method may include or the system may be caused to perform: receiving one or more videos that include visual frames of at least one subject…” page 14 col.3 paragraph 2);for each of the plurality of the frames, identifying a plurality of regions of interest (ROIs) based at least in part on excluding one or more potentially signal-disruptive facial features including eyes of the subject, lips of the subject, facial hair of the subject, or any combination thereof (“The non-skin regions on the face (such as eyebrows, nostril, forehead hair, and glasses) as well as the regions dominated by the specular reflections has little or no contribution to the pulse signal extraction, and the possible non-rigid motion in mouth region (e.g., the talking motion) or in eye region (e.g., blinking) add additional distortions to the rPPG signal. Thus, a rPPG system according to certain embodiments is configured to first reject those non-skin pixels in the facial ROI before any pulse extraction.” Page 19 col. 14 last paragraph), each of the plurality of ROIs including skin pixels (“…system 700 may include a skin detection block 710 configured to detect exposed skin from the input video(s) 705. In an embodiment, the skin detection block 710 may include a new unsupervised learning scheme to detect the skin pixels on a face accurately” page 19 col 13 last paragraph and col 14 line 1); for each ROI in each of the plurality of frames, extracting pixels(“According to an embodiment, in the skin detection block 710, an ellipsoid-shaped skin classifier may be learned from the face pixel colors in the first few frames of the video 705 and may be deployed in the subsequent frames to detect facial skins. This step enhances the quality of the extracted pulse signal as the skin surface contains the highest pulse signal-to-noise ratio (SNR).” Page 19 col. 14 paragraph 1); determining color intensity variations for at least a portion of the pixels across at least a portion of the plurality of frames (“In an embodiment, the system 700 may also include a spatial averaging block 715 configured to spatially average the three color channels in red, green, and blue inside the detected facial skin region, to produce a RGB face color signal. A pulse color mapping block 720 may be configured to map the face color temporal measurement to a color direction that generates the highest pulse SNR.” Page 19 col. 14 lines 7-14); and using the determined color intensity variations, determining the one or more physiological parameters for the subject (see figure 7 element 740).
Wu does not teach, the plurality of ROIs including at least two noncontiguous ROIs; traversing combinations of the plurality of ROIs to generate ROI-combination regions; for each ROI in each of the plurality of frames, extracting pixels, and filtering undesired pixels based, at least in part, on a confidence level check;
Jacquel teaches the plurality of ROIs including at least two noncontiguous ROIs (see Figure 6A); traversing combinations of the plurality of ROIs to generate ROI-combination regions (“In an embodiment, two different flood-filled contiguous regions are each used to extract the same or a different vital sign.” Page 38 col. 22 lines 50-52 and “As discussed above with respect to FIGS. 5A and 5B, different ROI's at different locations on the patient exhibit the same vital signs in different ways, such as one region exhibiting respiration rate more strongly than another.” Page 38 col. 22 lines 57-61); for each ROI in each of the plurality of frames, extracting pixels, and filtering undesired pixels based, at least in part, on a confidence level check (“The forehead region 1016A encounters the sensor 1044 and automatically excludes it from the region, based on different characteristics of the pixels along the sensor. The flood fill method automatically defines this boundary, excluding pixels that are less likely to contribute a physiologic signal” page 39 col.23 and lines 48-53);
Jacquel is analogous art in the same field of endeavor as the claimed invention. Jacquel is directed towards a system that uses a remote videos people to gain physiological insights (“Video-based monitoring is a new field of patient monitoring that uses a remote video camera to detect physical attributes of the patient. This type of monitoring may also be called “non-contact” monitoring in reference to the remote video sensor, which does not contact the patient. The remainder of this disclosure offers solutions and improvements in this new field.” Page 28 col.1 lines 56-62). A person of ordinary skill, before the effective filing date of the claimed invention, would have found it obvious to combine Wu and Jacquel by utilizing Jacquel’s ROI pixel elimination scheme inside of Wu’s broader rPPG method, with the expectation that doing so would improve the quality of Wu’s physiological readings by eliminating obstructions and identifying low confidence areas (“In such a dynamic situation, the boundary (such as 1017A or 1019A) moves frequently to accept or reject neighboring pixels or areas, and the size and/or location of the flood fill region changes quickly. The size or location of the region can be calculated, and the variability of the size or location calculated over time. If the variability exceeds a threshold, the physiologic signals measured from the flood fill region can be flagged as having low confidence, and/or a confidence metric can be reduced, and/or an indicator or alarm can be triggered. This same approach can be used by tracking the location of the seed point—if the location changes rapidly, a low-confidence flag can be set and/or a confidence metric reduced. Conversely, if the size of the flood fill region and/or the location of the seed point are stable, then a confidence metric can be increased and/or a high-confidence flag set.” Page 39 col 24 lines 34-49).
With respect to claim 2, Wu and Jacquel teach the method of claim 1. Wu further teaches wherein the method is performed using a remote photoplethysmography (rPPG) system (“Certain embodiments are directed to a method of remote photoplethysmography (rPPG) and/or a rPPG system.” Page 14 col 3 lines 10-11), and Jacquel teaches wherein the video data is generated using one or more cameras of a mobile device associated with the subject (“As used herein, the term “non-contact” refers to monitors whose measuring device (such as a detector) is not in physical contact with the patient. Examples include cameras, accelerometers mounted on a patient bed without contacting the patient, radar systems viewing the patient, and others. “Video-based” monitoring is a sub-set of non-contact monitoring, employing one or more cameras as the measuring device.” Page 30 col 5 lines 62-67 and col 6 lines 1-2).
With respect to claim 5, Wu and Jacquel teach the method of claim 1. Wu further teaches wherein the identifying the plurality of regions of interest includes accounting for a subject's movements and excluding the one or more potentially signal-disruptive facial features (“The non-skin regions on the face (such as eyebrows, nostril, forehead hair, and glasses) as well as the regions dominated by the specular reflections has little or no contribution to the pulse signal extraction, and the possible non-rigid motion in mouth region (e.g., the talking motion) or in eye region (e.g., blinking) add additional distortions to the rPPG signal. Thus, a rPPG system according to certain embodiments is configured to first reject those non-skin pixels in the facial ROI before any pulse extraction.” Page 19 col. 14 last paragraph).
With respect to claim 9, Wu and Jacquel teach the method of claim 1. Wu further teaches wherein the color intensity variations of the pixels are determined via a plane orthogonal to skin (POS) method (“As discussed above, for rPPG analysis, the POS can be employed to map facial videos to pulse signals. This method first extracts the intensity variations in RGB channels via spatial averaging within the facial region in the video frames…”col. 21 lines 15-20) and wherein the POS method compensates for subject motion (“The motion residue term in Eq. (8) is negligible when the illumination source is single, as the POS direction is orthogonal to the color direction of the motion-induced intensity change, and the specular change is suppressed via “alpha tuning”. However, if the video is captured in an uncontrolled environment, the motion residue term is often non-negligible, and sometimes can be more significant than the pulse term” col. 17 lines 48-55 and equation 8), skin variance (“Without loss of generality, it may be assumed that the face color signal … may be mapped to the POS direction” col. 17 lines 32-34)
With respect to claim 21, Wu and Jacquel teach the method of claim 1. Jacquel further teaches wherein the determining the one or more physiological parameters includes accounting a skin tone of the subject (“Certain embodiments may be configured to perform skin tone learning and pruning. The non-skin regions on the face (such as eyebrows, nostril, forehead hair, and glasses) as well as the regions dominated by the specular reflections has little or no contribution to the pulse signal extraction, and the possible non-rigid motion in mouth region (e.g., the talking motion) or in eye region (e.g., blinking) add additional distortions to the rPPG signal. Thus, a rPPG system according to certain embodiments is configured to first reject those non-skin pixels in the facial ROI before any pulse extraction.” Page 19 col 14 lines 58-67).
With respect to claim 22, Wu and Jacquel teach the method of claim 1. Jacquel teaches it further comprising: analyzing the skin of the subject, the facial features of the subject, or both (“Certain embodiments may be configured to perform skin tone learning and pruning. The non-skin regions on the face (such as eyebrows, nostril, forehead hair, and glasses) as well as the regions dominated by the specular reflections has little or no contribution to the pulse signal extraction, and the possible non-rigid motion in mouth region (e.g., the talking motion) or in eye region (e.g., blinking) add additional distortions to the rPPG signal. Thus, a rPPG system according to certain embodiments is configured to first reject those non-skin pixels in the facial ROI before any pulse extraction.” Page 19 col 14 lines 58-67); and detecting, based at least in part on the analyzing, changes in a physiological state of the subject (“Certain embodiments may be configured to perform skin tone learning and pruning. The non-skin regions on the face (such as eyebrows, nostril, forehead hair, and glasses) as well as the regions dominated by the specular reflections has little or no contribution to the pulse signal extraction, and the possible non-rigid motion in mouth region (e.g., the talking motion) or in eye region (e.g., blinking) add additional distortions to the rPPG signal. Thus, a rPPG system according to certain embodiments is configured to first reject those non-skin pixels in the facial ROI before any pulse extraction.” Page 19 col 14 lines 58-67)
With respect to claim 30, Wu and Jacquel teach the method of claim 1. Wu further teaches wherein the video data includes RGB data and the method further comprises: generating, based on the RGB data, a photoplethysmograph-like (PPG-like) signal; and determining pulse extraction via a projection plane orthogonal to skin of the subject (“Certain embodiments include motion compensation via multi-channel adaptive processing. In this aspect, once the skin pixels are detected in each frame, a temporal RGB sequence Ć(t) may be generated by spatially averaging the RGB values of the detected skin pixels and temporally normalized in each color channel. Ć(t) may then be linearly mapped to a specific color direction in the RGB space to generate a 1-D pulse signal. Without loss of generality, it may be assumed that the face color signal Ć(t) may be mapped to the POS direction, which is one of the most robust color features representing the highest relative pulse strength.” Page 21 col 17 lines 25-36).
Claims 8 are rejected under 35 U.S.C. 103 as being unpatentable over Wu and Jacquel as applied to claim 1 above, and further in view of Przybyło (Jaromir Przybyło, A deep learning approach for remote heart rate estimation, Biomedical Signal Processing and Control, Volume 74, 2022, 103457) and Tohma (Tohma, A.; Nishikawa, M.; Hashimoto, T.; Yamazaki, Y.; Sun, G. Evaluation of Remote Photoplethysmography Measurement Conditions toward Telemedicine Applications. Sensors 2021, 21, 8357.).
With respect to claim 8, Wu and Jacquel teach the method of claim 1, but do not teach wherein the generated video data has a resolution about 540p and the video data was generated at least about 1 year prior to the determining the one or more physiological parameters for the subject.
Przybyło teaches wherein video was generated at least about 1 year prior to determining the one or more physiological parameters of the subject (“To assess the generalization potential of the LSTM neural network, videos (collected for several years) from our previous work were used as test data.” Page 6 col 2. Paragraph 2 lines 1-3).
Przybyło is analogous art in the same field of endeavor as the claimed invention. Przybyło is directed towards PPG signal analysis using machine learning (“The basic principle of both methods is the same, but the sources of errors are different. Assuming the PPG signal is less susceptible to noise, it can be used as a reference for training the LSTM network. Thus, the purpose of LSTM is to eliminate artifacts resulting from, for example, rigid or nonrigid subject movements or fluctuations of scene lighting” page 4 col 2. Paragraph 2). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the teachings of Wu, Jacquel, and Przybyło by utilizing Przybyło’s pre-filmed videos and LSTM correction teachings in the combined system of Wu and Jacquel, with the expectation that doing so would allow for motion and luminance correction (“The basic principle of both methods is the same, but the sources of errors are different. Assuming the PPG signal is less susceptible to noise, it can be used as a reference for training the LSTM network. Thus, the purpose of LSTM is to eliminate artifacts resulting from, for example, rigid or nonrigid subject movements or fluctuations of scene lighting” page 4 col 2. Paragraph 2).
Tohma teaches where generated video data has a resolution about 540p (“A single RGB camera … with a maximum frame rate of 539 fps was used for the rPPG measurements. The camera was connected to a PC via USB 3.0, and the video image was captured at 720 × 540 resolution and saved in an uncompressed format” page 3 paragraph 1 lines 1-4).
Tohma is analogous art in the same field of endeavor as the claimed invention. Tohma is directed toward rPPG analysis (“A single RGB camera … with a maximum frame rate of 539 fps was used for the rPPG measurements. The camera was connected to a PC via USB 3.0, and the video image was captured at 720 × 540 resolution and saved in an uncompressed format” page 3 paragraph 1 lines 1-4). A person of ordinary skill before the effective filing date of the claimed invention would have found it easy to substitute 540 resolution videos in place of the combined system’s (Wu and Jacquel teachings) input video data resolution, with the expectation that doing so would not affect the operation of the combined system.
Claims 11 is rejected under 35 U.S.C. 103 as being unpatentable over Wu and Jacquel as applied to claim 1 above, and further in view of Maman (US 20230000376 A1).
With respect to claim 11, Wu and Jacquel teach the method of claim 1. wherein the one or more physiological parameters include heart rate (HR) (“an rPPG system for robust estimation during fitness exercise of heart rate (HR) or pulse rate (PR) and interbeat interval (IBI) signal is provided, whereby the IBI signal can support the estimation of heart/pulse rate variability (HRV/PRV).” Col. 13 lines 52-56), heart rate variability (HRV) (“an rPPG system for robust estimation during fitness exercise of heart rate (HR) or pulse rate (PR) and interbeat interval (IBI) signal is provided, whereby the IBI signal can support the estimation of heart/pulse rate variability (HRV/PRV).” Col. 13 lines 52-56) and , interbeat interval (IBI) (“an rPPG system for robust estimation during fitness exercise of heart rate (HR) or pulse rate (PR) and interbeat interval (IBI) signal is provided, whereby the IBI signal can support the estimation of heart/pulse rate variability (HRV/PRV).” Col. 13 lines 52-56),
Jacquel additional teaches oxygen saturation (SpO2) (“The vital signs measured from the video signal can be used to trigger alarms based on physiologic limits (for example, high or low heart rate, SpO2, or respiration rate alarms).” Page 43 col 32 lines 44-47), respiration rate (RR) (“The vital signs measured from the video signal can be used to trigger alarms based on physiologic limits (for example, high or low heart rate, SpO2, or respiration rate alarms).” Page 43 col 32 lines 44-47).
Maman additionally teaches wherein the one or more physiological parameters include heart rate (HR) (“The processed video signals are then calculated to determine a heart rate (HR)” paragraph 0110), heart rate variability (HRV) (“The output is obtained for each observed video frame and constructing the overlapping time series of pulse. These time series must be averaged to produce mean final rPPG trace suitable for HRV processing.” Paragraph 0108), oxygen saturation (SpO2), respiration rate (RR) (“calculating respiratory rate (RR)” paragraph 0109), standard deviation of RR intervals (SDRR) (“The following parameters may be calculated according to information provided in F. Shaffer and J. P. Ginsberg (An Overview of Heart Rate Variability Metrics and Norms, Front Public Health. 2017; 5: 258), which is hereby incorporated by reference as if fully set forth herein: SDRR, RMSSD, triangle (HRV triangular index), and TINN.” Paragraph 0114), root mean square of successive differences (RMSSD) (“The following parameters may be calculated according to information provided in F. Shaffer and J. P. Ginsberg (An Overview of Heart Rate Variability Metrics and Norms, Front Public Health. 2017; 5: 258), which is hereby incorporated by reference as if fully set forth herein: SDRR, RMSSD, triangle (HRV triangular index), and TINN.” Paragraph 0114), low-frequency spectrum (LF) (“The VLF is calculated at 806. The LF peak is calculated at 808.” Paragraph 0117 and “LF power is calculated at 810” paragraph 0118), high- frequency spectrum (HF) (“The HF peak is calculated at 812. HF power is calculated at 814.” Paragraph 0118), a ratio of high frequency to low frequency (HF/LF) (“The ratio of LF to HF is calculated at 816.” Paragraph 0118), or any combination thereof.
Maman is analogous art in the same field of endeavor as the claimed invention. Maman is directed towards a rppg system (“The present invention is of a system and method for physiological measurements as determined from optical data, and in particular, for such a system and method for determining such measurements from video data of a subject.” Paragraph 0001). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the system of Wu and Jacquel with the substantially similar rppg system of Maman by utilizing Maman’s notification strategy and physiological measurements in conjunction with Jacquel’s trend based notification system, with the expectation that doing so would result in improved physiological measurements (“The presently claimed invention overcomes these difficulties by providing a new system and method for improving the accuracy of pulse rate detection. Various aspects contribute to the greater accuracy, including but not limited to pre-processing of the camera output/input, extracting the pulsatile signal from the preprocessed camera signals, followed by post-filtering of the pulsatile signal. This improved information may then be used for such analysis as HRV determination, which is not possible with inaccurate methods for optical pulse rate detection.” Paragraph 0009).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Wu and Jacquel as applied to claim 1 above, and further in view of Przybyło (Jaromir Przybyło, A deep learning approach for remote heart rate estimation, Biomedical Signal Processing and Control, Volume 74, 2022, 103457).
With respect to claim 12, Wu and Jacquel teach the method of claim 1, but do not teach wherein the generated video data was generated at least about 1 day prior to the determining the one or more physiological parameters for the subject.
Przybyło teaches wherein the generated video data was generated at least about 1 day prior to the determining the one or more physiological parameters for the subject (“To assess the generalization potential of the LSTM neural network, videos (collected for several years) from our previous work were used as test data.” Page 6 col 2. Paragraph 2 lines 1-3).
Przybyło is analogous art in the same field of endeavor as the claimed invention. Przybyło is directed towards PPG signal analysis using machine learning (“The basic principle of both methods is the same, but the sources of errors are different. Assuming the PPG signal is less susceptible to noise, it can be used as a reference for training the LSTM network. Thus, the purpose of LSTM is to eliminate artifacts resulting from, for example, rigid or nonrigid subject movements or fluctuations of scene lighting” page 4 col 2. Paragraph 2). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the teachings of Wu and Jacquel, and Przybyło by utilizing Przybyło’s pre-filmed videos and LSTM correction teachings in the combined system of Wu and Jacquel, with the expectation that doing so would allow for motion and luminance correction (“The basic principle of both methods is the same, but the sources of errors are different. Assuming the PPG signal is less susceptible to noise, it can be used as a reference for training the LSTM network. Thus, the purpose of LSTM is to eliminate artifacts resulting from, for example, rigid or nonrigid subject movements or fluctuations of scene lighting” page 4 col 2. Paragraph 2).
Claims 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wu and Jacquel as applied to claim 1 above, and further in view of Kaiser (CN 112151172 A) and Rothman (US 20180247713 A1).
With respect to claim 15, Wu and Jacquel teach the method of claim 1, but do not explicitly teach the rest of the claim limitations. Kaiser teaches presentation, via a mobile device, of one or more vital signs of the subject (see figure 27), contextual information associated with the one or more vital signs (see figures 9-10, risk factors and arrows), and historical health trends of the subject (see figures 9-10 vital arrows and 28- 29 graphs).
Kaiser is analogous art in the same field of endeavor as the claimed invention. Kaiser is directed towards a medical information display interface that uses a mobile phone to display vitals and additional medical information (see figure 27) A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the system of Wu and Jacquel with the mobile information display system of Kaiser by utilizing the teachings of Kaiser to display the physiological parameters and subject vitals garnered by utilizing Wu in combination with Jacquel, with the expectation that doing so would enable the system to detect changes indicative of malady or worsening health and quickly notify staff (“The invention relates to evaluating patient risk in a medical institution, and especially relates to evaluating patient risk based on data obtained from the medical device. More specifically, the present disclosure relates to evaluating a plurality of risk of the patient in the medical institution, and notifying a plurality of risks of the patient to the nursing staff.” Technical Field page 2)
Rothman teaches comparisons of the health trends of the subject with others (“By comparing patients based on their disease, treatment/surgery, or affliction, the patient's health score may be better interpreted” paragraph 0113).
Rothman is analogous art in the same field of endeavor as the claimed invention. Rothman is directed towards wearable health sensors and the interpretation of their results (“The methods and systems described herein leverage wearable physiological sensors and a network of supporting technology to provide adaptive complimentary self-assessment and automated health scoring. Monitoring and tracking the health and wellbeing of patients in their homes, assisted living, nursing homes, hospice care, rehabilitation centers, and other healthcare facilities can be vital to identifying changes in health indicators.” Paragraph 004). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the system of Wu, Jacquel, Kaiser, and Rothman by utilizing the teachings of Rothman (namely its interpretation of health scores) with the visually displayed scores of Kaiser, with the expectation that doing so would improve interpretation of health scores thus improving the system’s diagnostic capabilities (“By comparing patients based on their disease, treatment/surgery, or affliction, the patient's health score may be better interpreted” paragraph 0113).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Wu and Jacquel as applied to claim 1 above, and further in view of Maman and Tzvieli (US 20200221956 A1)
With respect to claim 18, Wu and Jacquel teach the method of claim 1. Jacquel teaches generating, based at least in part on the one or more physiological parameters, recommendations for the subject determining that at least one of the one or more physiological parameters has changed an amount above a predetermined threshold for the subject (“if not, the method continues to 706, where the video SpO2 value is compared to a threshold to identify changes. If the video SpO2 value crosses the threshold, the method includes sounding an alarm (such as an audible sound and/or a visible alert) at 707, and prompting re-calibration at 701. If not, the method returns to continue measuring at 704. The threshold used to detect a change at 706 can be set by the caregiver to identify changes in video SpO2 that may indicate a clinically significant change in the patient's physiology, for further diagnosis or treatment” page 35 col 16 lines 33-43) generating, based on the determined change, a notification (“if not, the method continues to 706, where the video SpO2 value is compared to a threshold to identify changes. If the video SpO2 value crosses the threshold, the method includes sounding an alarm (such as an audible sound and/or a visible alert) at 707, and prompting re-calibration at 701. If not, the method returns to continue measuring at 704. The threshold used to detect a change at 706 can be set by the caregiver to identify changes in video SpO2 that may indicate a clinically significant change in the patient's physiology, for further diagnosis or treatment” page 35 col 16 lines 33-43) and automatically transmitting the notification to a computer system associated with a healthcare provider (“if not, the method continues to 706, where the video SpO2 value is compared to a threshold to identify changes. If the video SpO2 value crosses the threshold, the method includes sounding an alarm (such as an audible sound and/or a visible alert) at 707, and prompting re-calibration at 701. If not, the method returns to continue measuring at 704. The threshold used to detect a change at 706 can be set by the caregiver to identify changes in video SpO2 that may indicate a clinically significant change in the patient's physiology, for further diagnosis or treatment” page 35 col 16 lines 33-43), but does not teach any further limitations.
Maman teaches causing a reminder notification to be presented to the subject to remind the subject to record a video (“either by separately activating camera 114, or by recording such data by issuing a command through user app interface 104” paragraph 0041).
Maman is analogous art in the same field of endeavor as the claimed invention. Maman is directed towards a rppg system (“The present invention is of a system and method for physiological measurements as determined from optical data, and in particular, for such a system and method for determining such measurements from video data of a subject.” Paragraph 0001). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the system of Wu and Jacquel with the substantially similar rppg system of Maman by utilizing Maman’s notification strategy and physiological measurements in conjunction with Jacquel’s trend based notification system, with the expectation that doing so would result in improved physiological measurements and thusly better notifications (“The presently claimed invention overcomes these difficulties by providing a new system and method for improving the accuracy of pulse rate detection. Various aspects contribute to the greater accuracy, including but not limited to pre-processing of the camera output/input, extracting the pulsatile signal from the preprocessed camera signals, followed by post-filtering of the pulsatile signal. This improved information may then be used for such analysis as HRV determination, which is not possible with inaccurate methods for optical pulse rate detection.” Paragraph 0009).
Tzvieli further teaches all computer functions related to PPG analysis being done on using a subject’s device (“In other embodiments, essentially all functions attributed to the computer herein may be performed by a processor on a wearable device (e.g., smartglasses to which the head-mounted devices are coupled) and/or some other device carried by the user, such as a smartwatch or smartphone” paragraph 0184).
Tzvieli is analogous art in the same field of endeavor as the claimed invention. Tzvieli is directed towards a system that uses PPG signals to identify physiological parameters (“remote photoplethysmography (rPPG)” paragraph 0051 and In some embodiments, a system configured to detect an abnormal medical event includes a computer and several head-mounted devices that are used to measure photoplethysmographic signals (PPG signals) indicative of blood flow at various regions on a user's head. Optionally, the system may include additional components, such as additional sensors that may be used to measure the user and/or the environment. ….Some examples of abnormal medical events that may be detected by embodiments described herein include an ischemic stroke, a migraine, a headache, cellulitis (soft tissue infection), dermatitis (skin infection), and an ear infection.” Paragraph 0163). A person of ordinary skill in the art before the effective filing date of the claimed invention would have found it obvious to combine the system of Wu, Jacquel and Maman with the substantially similar system of Tzvieli, by utilizing Tzvieli’s explicit teachings of pixel extraction to determine light intensity as well as its teachings of various medical ailments able to be determined by utilizing PPG systems in concert with Wu, Jacquel and Maman’s explicitly taught rPPG system and face and skin detection algorithms, with the expectation that doing so would result in a more adaptable system that is able to detect additional ailments like a stroke (“Calculating the value indicative of the risk that the user has suffered from a stroke may be done in different ways in different embodiments. In one embodiment, the value indicative of the risk is indicative difference between of extents of physiological changes in the right and left sides of the body, which are calculated based on M.sub.R and M.sub.L, respectively. For example, the value may be indicative of a difference in blood flow between the right and left sides of the body, a difference in the temperature between the right and left sides, and/or a difference in skin color and/or extent of changes to skin color between the right and left sides.” Paragraph 0335) increasing the capabilities of the combined system.
Claims 27 is rejected under 35 U.S.C. 103 as being unpatentable over Wu and Jacquel as applied to claim 1 above, and further in view of McDuff (US 20150282724 A1).
With respect to claim 27, Wu and Jacquel teach the method of claim 1. McDuff further teaches performing signal processing on the color intensity variations to extract blood volume pulses (BVP), wherein the BVP is extracted via (i) a signal extraction (“The attached Source Code takes multiple signals from different color band images and recovers the blood volume pulse (BVP)” paragraph 0143) and peak detection module (“As an intermediate step the code performs peak detection on the BVP waveform and computes the inter-beat intervals (IBIs)” paragraph 0143), wherein the signal extraction and peak detection module compensates for amplitude variation (“Identify the diastolic peak (or inflection) as the maximum following the systolic peak in each pulse cycle in the inverted second derivative pulse wave. Calculate the systolic-diastolic peak-to-peak times (SD-PPT) for each beat. Classify SD-PPT estimates that fall beyond one standard deviation from the mean as outliers and do not include these in the estimate of the final mean SD-PPT.” paragraph 0084) and morphology changes of photoplethysmographic complexes (“Step 9 of the Illustrative BVP Extraction Algorithm (above) is desirable the ICA may scale the source signals arbitrarily. If the scaling is by a negative number, the ICA flips the source signal, which is undesirable because the flip causes (if not corrected) less accurate readings. Step 9 corrects for this. For an inverted BVP signal, the mean trough amplitude is likely to be greater than the mean peak amplitude due to the shape of the BVP waveform. Therefore, to detect and correct for an inverted source signal (which has been inverted by ICA), a computer: (a) calculates the mean absolute peak amplitude μ.sub.peakamp and mean absolute trough amplitude μ.sub.troughamp of the source signal; and (b) inverts the source signal (that is, multiplies it by −1), if μ.sub.peakamp<μ.sub.troughamp.” paragraph 0080).
McDuff is analogous art in the same field of endeavor to the claimed invention. McDuff is directed towards photoplethysmography (“The present invention relates generally to photoplethysmography.” Paragraph 0002). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the system of Wu and Jacquel with McDuff by utilizing McDuff’s teachings of BVP calculations in conjunction or in addition to the other produced subject physiological parameters with the expectation that doing so would enable the combined system to further assess emotion responses (“The BVP amplitude displays moment-by-moment HRV and may offer significant insight into individual emotional responses” page 57 col. 2 paragraph 4 lines 1-3 in Peper, Erik & Harvey, Richard & Lin, I-Mei & Tylova, H. & Moss, Donald. (2007). Is There More to Blood Volume Pulse Than Heart Rate Variability, Respiratory Sinus Arrhythmia, and Cardiorespiratory Synchrony?. Biofeedback. 35. 54-61.) and physical responses such as changes in temperature (“Changes in BVP represent changes in the blood volume. If BVP amplitude increases, it reflects an increase in vasodilation, which leads to an increase in peripheral temperature; if the BVP amplitude decreases, it reflects a decrease in peripheral circulation and a decrease in peripheral temperature. BVP amplitude changes are very rapid and precede changes in temperature, as temperature is a slow-averaging signal” page 58 col.2 paragraph 3 in Peper, Erik & Harvey, Richard & Lin, I-Mei & Tylova, H. & Moss, Donald. (2007). Is There More to Blood Volume Pulse Than Heart Rate Variability, Respiratory Sinus Arrhythmia, and Cardiorespiratory Synchrony?. Biofeedback. 35. 54-61.) or blood pressure (“The shape of the BVP waveform may be an indicator of cardiovascular variables such as blood pressure” page 57 col. 1 paragraph 2 lines 1-2 in Peper, Erik & Harvey, Richard & Lin, I-Mei & Tylova, H. & Moss, Donald. (2007). Is There More to Blood Volume Pulse Than Heart Rate Variability, Respiratory Sinus Arrhythmia, and Cardiorespiratory Synchrony?. Biofeedback. 35. 54-61.).
9. Claims 50-52 are rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Jacquel and Tak (KR 20100036601 A)
With respect to claim 50, Wu teach a method for determining one or more physiological parameters of a subject, the method comprising: generating video data, the video data being reproducible as a plurality of frames, each of the plurality of frames depicting at least a portion of the subject (“Certain embodiments are directed to a method of remote photoplethysmography (rPPG) and/or a rPPG system. The method may include or the system may be caused to perform: receiving one or more videos that include visual frames of at least one subject…” page 14 col.3 paragraph 2) ; stabilizing, each of the plurality of frames (“The non-skin regions on the face (such as eyebrows, nostril, forehead hair, and glasses) as well as the regions dominated by the specular reflections has little or no contribution to the pulse signal extraction, and the possible non-rigid motion in mouth region (e.g., the talking motion) or in eye region (e.g., blinking) add additional distortions to the rPPG signal. Thus, a rPPG system according to certain embodiments is configured to first reject those non-skin pixels in the facial ROI before any pulse extraction.” Page 19 col. 14 last paragraph); processing each of the plurality of frames; and estimating, via one or both of a classical approach module and a deep learning network module, the one or more physiological parameters of the subject (“In certain embodiments, additional approaches to privacy protective transform may include using deep neural network-based approach to “change” the identifying facial features to a different person, as if performing a “forgery”, while preserving the skin region in terms of temporal variation.” Page 23 col 20 lines 66-67 and page 24 col 21 lines 1-4). Wu does not teach any further limitations explicitly.
Jacquel teaches the estimating being performed using one or more multi-scale spatial-temporal maps generated from the video data (“In an embodiment, the skin tone filter dynamically updates over time, continually refreshing to identify the candidate skin pixels in the changing video stream. In an embodiment, an initial calibration period is utilized first, to determine initial parameters for the skin tone filter, and to map the face (including noting differences such as bandages, sensors, etc on the face). After that initial calibration, the skin tone filter dynamically updates as the environment changes (subject to a maximum refresh rate), or at a set frequency such as once per second or faster.” Page 42 col 29 lines 1-11).
Jacquel is analogous art in the same field of endeavor as the claimed invention. Jacquel is directed towards a system that uses a remote videos people to gain physiological insights (“Video-based monitoring is a new field of patient monitoring that uses a remote video camera to detect physical attributes of the patient. This type of monitoring may also be called “non-contact” monitoring in reference to the remote video sensor, which does not contact the patient. The remainder of this disclosure offers solutions and improvements in this new field.” Page 28 col.1 lines 56-62). A person of ordinary skill, before the effective filing date of the claimed invention, would have found it obvious to combine Wu and Jacquel by utilizing Jacquel’s ROI pixel elimination scheme inside of Wu’s broader rPPG method, with the expectation that doing so would improve the quality of Wu’s physiological readings by eliminating obstructions and identifying low confidence areas (“In such a dynamic situation, the boundary (such as 1017A or 1019A) moves frequently to accept or reject neighboring pixels or areas, and the size and/or location of the flood fill region changes quickly. The size or location of the region can be calculated, and the variability of the size or location calculated over time. If the variability exceeds a threshold, the physiologic signals measured from the flood fill region can be flagged as having low confidence, and/or a confidence metric can be reduced, and/or an indicator or alarm can be triggered. This same approach can be used by tracking the location of the seed point—if the location changes rapidly, a low-confidence flag can be set and/or a confidence metric reduced. Conversely, if the size of the flood fill region and/or the location of the seed point are stable, then a confidence metric can be increased and/or a high-confidence flag set.” Page 39 col 24 lines 34-49).
Tak teaches stabilizing, via a motion artifact confidence weight (MACW) module, each of a plurality of frames (“receiving an image signal decoded in units of frames; Detecting a brightness level of the input video signal; Detecting a motion level of the input video signal by comparing motion detection information of the frame with a previous frame; Setting a noise reduction weight for each detected motion level according to the detected brightness level; And performing noise filtering of the video signal by applying the set noise reduction weight” pages 3 (last paragraph) and 4 (first paragraph) ); processing, via a luminance artifact confidence weight (LACW) module, each of a plurality of frames (“receiving an image signal decoded in units of frames; Detecting a brightness level of the input video signal; Detecting a motion level of the input video signal by comparing motion detection information of the frame with a previous frame; Setting a noise reduction weight for each detected motion level according to the detected brightness level; And performing noise filtering of the video signal by applying the set noise reduction weight” pages 3 (last paragraph) and 4 (first paragraph) );
Tak is analogous art reasonably pertinent to the problem of motion and luminance artifacts in video frames faced by the inventor. Tak discloses noise reduction methods for image signals (“The present invention relates to an image processing system, and more particularly, to an apparatus and method for removing image noise applying motion blurring and an adaptive noise reduction algorithm according to a distribution of luminance levels of an image signal.” page 1 paragraph 2). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the teachings of Tak with the combined system of Wu and Jacquel by utilizing its noise reduction teachings in combination with the video acquisition systems of the combined system to generate video frames with less artifacts making for more accurate physiological result findings (“An object of the present invention is to provide an apparatus and method for removing image noise, which minimizes motion blurring of an image signal.” Page 2 paragraph 6).
With respect to claim 51, Wu, Jacquel and Tak teach the method of claim 50. Wu further teaches wherein the estimating the one or more physiological parameters of the subject includes both the classical approach module and the deep learning network module (“In certain embodiments, additional approaches to privacy protective transform may include using deep neural network-based approach to “change” the identifying facial features to a different person, as if performing a “forgery”, while preserving the skin region in terms of temporal variation.” Page 23 col 20 lines 66-67 and page 24 col 21 lines 1-4), and wherein the method further comprises integrating data derived from the classical approach module and data derived from the deep learning network module (“In certain embodiments, additional approaches to privacy protective transform may include using deep neural network-based approach to “change” the identifying facial features to a different person, as if performing a “forgery”, while preserving the skin region in terms of temporal variation.” Page 23 col 20 lines 66-67 and page 24 col 21 lines 1-4).
With respect to claim 52, Wu, Jacquel and Tak teach the method of claim 50. Tak further teaches wherein the MACW module accommodates for movement of the subject during a capture of the generated video data (“receiving an image signal decoded in units of frames; Detecting a brightness level of the input video signal; Detecting a motion level of the input video signal by comparing motion detection information of the frame with a previous frame; Setting a noise reduction weight for each detected motion level according to the detected brightness level; And performing noise filtering of the video signal by applying the set noise reduction weight” pages 3 (last paragraph) and 4 (first paragraph) ), wherein the processing each of the plurality of frames includes quantifying luminance discrepancies within the video data (“receiving an image signal decoded in units of frames; Detecting a brightness level of the input video signal; Detecting a motion level of the input video signal by comparing motion detection information of the frame with a previous frame; Setting a noise reduction weight for each detected motion level according to the detected brightness level; And performing noise filtering of the video signal by applying the set noise reduction weight” pages 3 (last paragraph) and 4 (first paragraph) ), wherein the processing the plurality of frames includes introducing a rectification factor associated with the luminance discrepancies (“receiving an image signal decoded in units of frames; Detecting a brightness level of the input video signal; Detecting a motion level of the input video signal by comparing motion detection information of the frame with a previous frame; Setting a noise reduction weight for each detected motion level according to the detected brightness level; And performing noise filtering of the video signal by applying the set noise reduction weight” pages 3 (last paragraph) and 4 (first paragraph) ).
Allowable Subject Matter
10. Claims 55 and 56 are allowable over prior art due to the amended limitations specifically the limitation of “wherein the LACW module quantifies luminance discrepancies by computing a concatenated matrix across a plurality of color spaces and introduces a rectification factor derived from the concatenated matrix;”, when taken in consideration of the entirety of claim 55. Claim 56 depends on claim 55 and thus has also been found allowable.
Response to Arguments
11. Applicant’s arguments filed 01/20/2026 have been fully considered.
On page 11, the applicant argues that amendments to claim 1 have introduced limitations not taught by the previous combination of Maman and Tzvieli and that accordingly claim 1 should be rendered allowable. The examiner views this as a moot point due to updated rejections made utilizing a new combination of prior art (see above). On page 12, applicant continues by arguing that because claim 1 is allowable its dependent claims 2, 5, 8, 9, 11, 12, 15, 18, 21, 22, 27, and 30 are also in a condition for allowance. The examiner again finds this moot in face of claim 1 receiving an updated rejection.
With respect to independent claims 50 and 55 applicant makes substantially similar arguments revolving around the introduction of amendments overcoming prior art and thus making the independent claims and their dependent claims allowable. In the case of claim 50 and its dependents 51 and 52, similarly to the above claim 1 arguments, the examiner finds these arguments moot due to updated rejections made using a new combination of art. However, the examiner does agree with the applicant that the amendments made to claim 55 have overcome not only the previously identified prior art but also upon further consideration seem allowable. The examiner thusly agrees with the applicant that its dependent claim, 56, is also allowable due to its dependence on the allowable claim.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REBECCA C WILLIAMS whose telephone number is (571)272-7074. The examiner can normally be reached M-F 7:30am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew W Bee can be reached at (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/REBECCA COLETTE WILLIAMS/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights