DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 03/23/2026 have been fully considered but they are not persuasive.
Applicant remarks on pages 13-14 that prior art Wang (US 20200178809 A1) comprises a PPG heart monitoring system and not a ballistocardiographic heart rate monitoring system that relies on primarily on depth video data, as claimed. However, Wang—2017 indicates that the measurements include ballistocardiographic data.
Furthermore, the ballistocardiography feature is only set forth in the preamble and hence if all the features in the body of the claim are met, then the features of the BCG heart monitoring system is disclosed by Wang. That is, the preamble disclosing ballistocardiography appears to be rather an intended purpose of the system than structurally limiting. Pitney Bowes, Inc. v. Hewlett-Packard Co., 182 F.3d 1298, 1305, 51 USPQ2d 1161, 1165 (Fed. Cir. 1999). See also Rowe v. Dror, 112 F.3d 473, 478, 42 USPQ2d 1550, 1553 (Fed. Cir. 1997) ("where a patentee defines a structurally complete invention in the claim body and uses the preamble only to state a purpose or intended use for the invention, the preamble is not a claim limitation"). See MPEP 2111.02(II).
Applicant further remarks on page 14 that the RGB video of [0107] in Wang does not constitute the claimed “optical video depth data”. However, the imaging unit is disclosed as configured to acquire image data of a scene, said image data comprising a time-sequence of image frames in [0021] and being a depth camera in [0047].
Applicant further remarks on pages 14-15 that Wang’s affinity matrix for all patch pixels in [0110] does not constitute the “depth signal data matrix” as claimed because Wang does not include depth video and process the depth video for ballistocardiography. However, as noted above, the imaging unit is a depth camera and acquires a time-series of image frames for processing to obtain PPG signal.
Therefore, the claim stand rejected.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 9-12 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, et al., US 20200178809 A1, in view of Wang, W. (2017). Robust and automatic remote photoplethysmography. [Ph.D. Thesis 1 (Research TU/e /Graduation TU/e), Electrical Engineering]. Technische Universiteit Eindhoven, hereafter referred to as “Wang-2017”.
Regarding claim 9, Wang teaches a heart rate monitoring system (the abstract discloses a “camera-based vital signs monitoring using remote PPG. A Device (10) for determining a physiological parameter”. [0002] and [0004] suggest that the physiological parameter is a heart rate. Also see fig. 1 for the system 1) comprising:
an optical depth sensor (imaging unit 2 of fig. 1 and [0066] comprising a depth camera according to [0050]); and
an image processor ([0071] discloses a processor 12) configured to:
receive optical depth video data of at least one subject ([0107] states that “In a given non-limiting example an RGB video is provided as the image data of a scene comprising a time-sequence image frames”, the video being of a subject [0066]);
identify a region of interest of the at least one subject from the optical depth video data ([0107] further states “Given a video registered by an RBG camera viewing the scene including a living-skin, I(x, c, t) is used to denote the intensity of a pixel at a location x of an image, in the channel c, recorded at time t”. The region of interest comprises the skin of the subject which the camera is viewing);
segment the region of interest into multiple areas ([0109] discloses using spectral clustering to “build a fully connected affinity/similarity graph for all the patches using I.sub.n (t) and decompose it into uncorrelated subspaces, where a subspace can be used as an independent weighting mask to discriminate patches or pixels with different colors”);
identify pixel intensity with respect to time ([0107] states that “Given a video registered by an RBG camera viewing the scene including a living-skin, I(x, c, t) is used to denote the intensity of a pixel at a location x of an image, in the channel c, recorded at time t”) in the multiple areas ([0109] discloses the patches and subspaces for which the affinity/similarity graph representing pixel intensities) to produce a depth signal data matrix including multiple spatial channels ([0110] discloses creating an affinity matrix for all patch pixels and [0111] states that “In a next step 105, 106 each weighted image can be condensed into one or more spatial color representations given by a statistical parameter value”. Of note, [0041] and [0042] describe that the affinity/similarity matrix is generated using clustering to determine principal components);
decompose the depth signal data matrix into a low-rank spatial-temporal eigenvector matrix to produce refined depth signal data streams ([0110] discloses decomposing an affinity matrix into orthogonal subspaces using singular value decomposition (SVD). [0110] further states that “Since each eigenvector describes a group of patches having a similar color feature, a number of K, preferably K top-ranked, eigenvectors can be used to create the weighting maps, where K can be defined, either automatically (using s(t)), or manually” and [0123] states “Referring again to FIG. 7, in the next step 109, a physiological parameter can be extracted based on said candidate signals obtained in the previous step. To extract the physiological parameter from the candidate signals T, known algorithms used in photoplethysmography can be applied”. Of note, Wang uses only a low-rank number of k eigenvectors, that is the top rank); and
While [0128] states that “a long-term pulse-signal or physiological parameter signal H can be derived by concatenating sections h, e.g. by overlap-adding h, (preferably after removing its mean and normalizing its standard deviation) estimated in different time windows or short sequences, e.g., using a sliding window. The physiological parameter of the subject, such as the pulse rate, can then be extracted based thereon and provided as an output in step 110 of FIG. 7”, Wang does not explicitly teach that the system 1 of Wang is a ballistocardiographic heart rate monitoring system; to project the refined depth signal data streams onto a selected pulsatile direction and produce, based on the projection of the refined depth signal data stream, a first cardiac pulse signal for the at least one subject.
However, within the same field of endeavor, Wang-2017 teaches a remote photoplethysmography for pulse extraction (page iii) including acquiring ballistocardiographic signals (section 6.3.1 on page 123), and designing a projection function for pulse retrieval (first paragraph of page iv). Wang-2017 further teaches a blood volume pulse (BPV) signature based method (section 2.3.2.1 on page 34), which section notes that PBV method chooses to directly retrieve the pulse from the pulsatile component by restricting all color variations to the pulsatile direction. It does so by projecting (zero-mean) color variation signals (refined depth signal data streams) onto a single direction z (claimed selected pulsatile direction) to create an estimate of a pulse signal (claimed cardiac pulse signal). Section 2.3.2.1 on pages 34-35 describe the steps as outlined below:
PNG
media_image1.png
468
844
media_image1.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to configure Wang with a ballistocardiographic heart rate monitoring system; to project the refined depth signal data streams onto a selected pulsatile direction and produce, based on the projection of the refined depth signal data stream, a first cardiac pulse signal for the at least one subject, as taught by Wang-2017, to provide an optimal algorithm for pulse retrieval (lines 14-17 of page 35 under section 2.3.1.1), with a reasonable expectation of success, as Wang is also tasked with improving the extraction of physiological parameters of a subject using robust algorithms ([0012]), even when compared to conventional ground truth ECG contact-based measurements ([0129]).
Regarding claim 10, Wang in view of Wang-2017 teaches all the limitations of claim 9 above.
Wang further teaches wherein the selected pulsatile direction comprises an optimum pulsatile direction at which cross power spectral density, comprising spectral coherence values as a function of frequency, is maximized ([0125] states that “due to the use of both a central tendency (e.g. the mean) and a dispersion-related measure (e.g. the variance) for the spatial pixel combination, multiple T.sub.i (and thus P.sub.i) may contain useful pulsatile content. Therefore, it would be advantageous to combine several candidate signals instead of selecting just one of them”, where T.sub.i denotes a candidate signal ([0122]) and P.sub.i is the remote-PPG signal or physiological parameter ([0124]) and [0126] states that “Advantageously, in order to arrive at a clean output, higher weights can be given to components that are more likely to be pulse-related during the combination. However it has to be considered that the frequency amplitude may not directly be used to determine the weights or selected the components, because a large amplitude may not be due to pulse but due to motion artifacts. In view of a relation of pulsatile energy and motion energy it is thus proposed to estimate an intensity signal of each T.sub.i and use an energy ratio between pulsatile components and intensity components as weights”)
Regarding claim 11, Wang in view of Wang-2017 teaches all the limitations of claim 9 above.
Wang further teaches wherein the image processor is further configured to apply bandpass temporal filtering to eliminate excessively high frequency and excessively low frequency data to reduce noise in the first cardiac pulse signal ([0127] states, with respect to equation 12 that “B optionally denotes a band for filtering such as a heart-rate and for eliminating clearly non-pulsatile components which can e.g. be defined as [40, 240] beats per minute (bpm) according to the resolution of Fp.sub.i.”).
Regarding claim 12, Wang in view of Wang-2017 teaches all the limitations of claim 9 above.
Wang further teaches wherein the eigenvector matrix comprises data representing magnitude and direction of each spatial channel contributing to eigenvectors of the eigenvector matrix ([0110] states that “Since each eigenvector describes a group of patches having a similar color feature, a number of K, preferably K top-ranked, eigenvectors can be used to create the weighting maps, where K can be defined, either automatically (using s(t)), or manually”. That is, each patch is represented in magnitude and direction by its respective eigenvector that makes up the affinity matrix in [0110]).
Regarding claim 15, Wang in view of Wang-2017 teaches all the limitations of claim 9 above.
Wang further teaches wherein the image processor is further configured to detect a respiration rate of the at least one subject ([0034] states that “As used herein, a physiological parameter of the subject can refer to a physiological parameter indicative of a vital sign of the subject, such as a pulse, respiration rate or blood oxygen saturation of the subject”).
Regarding claim 16, Wang in view of Wang-2017 teaches all the limitations of claim 9 above.
Wang further teaches wherein the at least one subject comprises a plurality of subjects, and the image processor is configured to produce a different first cardiac pulse signal for each subject of the plurality of subjects ([0076] states that “A system 1 as illustrated in FIG. 1 may, e.g., be located in a hospital, healthcare facility, elderly care facility or the like. Apart from the monitoring of patients, the solutions proposed herein may also be applied in other fields such as neonate monitoring, general surveillance applications, security monitoring or so-called lifestyle environments, such as fitness equipment, a wearable, a handheld device like a smartphone, or the like”, meaning that the determined physiological parameter in claim 9 above is performed for each monitored patient of the plurality of patients).
Regarding claim 17, Wang teaches a non-transitory computer readable medium comprising computer-readable instructions, that when executed by a processor, cause the processor to perform operations ([0028] discloses “a non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method disclosed herein to be performed”), the operations comprising:
identifying a region of interest of the at least one subject from the optical depth video data ([0107] further states “Given a video registered by an RBG camera viewing the scene including a living-skin, I(x, c, t) is used to denote the intensity of a pixel at a location x of an image, in the channel c, recorded at time t”. The region of interest comprises the skin of the subject which the camera is viewing);
segmenting the region of interest into multiple areas ([0109] discloses using spectral clustering to “build a fully connected affinity/similarity graph for all the patches using I.sub.n (t) and decompose it into uncorrelated subspaces, where a subspace can be used as an independent weighting mask to discriminate patches or pixels with different colors”);
identifying pixel intensity with respect to time ([0107] states that “Given a video registered by an RBG camera viewing the scene including a living-skin, I(x, c, t) is used to denote the intensity of a pixel at a location x of an image, in the channel c, recorded at time t”) in the multiple areas ([0109] discloses the patches and subspaces for which the affinity/similarity graph representing pixel intensities) to produce a depth signal data matrix including multiple spatial channels ([0110] discloses creating an affinity matrix for all patch pixels and [0111] states that “In a next step 105, 106 each weighted image can be condensed into one or more spatial color representations given by a statistical parameter value”. Of note, [0041] and [0042] describe that the affinity/similarity matrix is generated using clustering to determine principal components);
decomposing the depth signal data matrix into a low-rank spatial-temporal eigenvector matrix to produce refined depth signal data streams ([0110] discloses decomposing an affinity matrix into orthogonal subspaces using singular value decomposition (SVD). [0110] further states that “Since each eigenvector describes a group of patches having a similar color feature, a number of K, preferably K top-ranked, eigenvectors can be used to create the weighting maps, where K can be defined, either automatically (using s(t)), or manually” and [0123] states “Referring again to FIG. 7, in the next step 109, a physiological parameter can be extracted based on said candidate signals obtained in the previous step. To extract the physiological parameter from the candidate signals T, known algorithms used in photoplethysmography can be applied”. Of note, Wang uses only a low-rank number of k eigenvectors, that is the top rank); and
While [0128] states that “a long-term pulse-signal or physiological parameter signal H can be derived by concatenating sections h, e.g. by overlap-adding h, (preferably after removing its mean and normalizing its standard deviation) estimated in different time windows or short sequences, e.g., using a sliding window. The physiological parameter of the subject, such as the pulse rate, can then be extracted based thereon and provided as an output in step 110 of FIG. 7”, Wang does not explicitly teach ballistocardiographic heart rate monitoring; and projecting the refined depth signal data streams onto a selected pulsatile direction and producing, based on the projection of the refined depth signal data streams, a first cardiac pulse signal for the at least one subject.
However, within the same field of endeavor, Wang-2017 teaches a remote photoplethysmography for pulse extraction (page iii) including acquiring ballistocardiographic signals (section 6.3.1 on page 123), and designing a projection function for pulse retrieval (first paragraph of page iv). Wang-2017 further teaches a blood volume pulse (BPV) signature based method (section 2.3.2.1 on page 34), which section notes that PBV method chooses to directly retrieve the pulse from the pulsatile component by restricting all color variations to the pulsatile direction. It does so by projecting (zero-mean) color variation signals (refined depth signal data streams) onto a single direction z (claimed selected pulsatile direction) to create an estimate of a pulse signal (claimed cardiac pulse signal). Section 2.3.2.1 on pages 34-35 describe the steps as outlined below:
PNG
media_image1.png
468
844
media_image1.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to configure Wang ballistocardiographic heart rate monitoring; and projecting the refined depth signal data streams onto a selected pulsatile direction and producing, based on the projection of the refined depth signal data streams, a first cardiac pulse signal for the at least one subject, as taught by Wang-2017, to provide an optimal algorithm for pulse retrieval (lines 14-17 of page 35 under section 2.3.1.1), with a reasonable expectation of success, as Wang is also tasked with improving the extraction of physiological parameters of a subject using robust algorithms ([0012]), even when compared to conventional ground truth ECG contact-based measurements ([0129]).
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, et al., US 20200178809 A1, in view of Wang, W. (2017). Robust and automatic remote photoplethysmography. [Ph.D. Thesis 1 (Research TU/e /Graduation TU/e), Electrical Engineering]. Technische Universiteit Eindhoven, hereafter referred to as “Wang-2017”, as applied to claim 9 above, and further in view of Jacquel, et al., US 20170238842 A1.
Regarding claim 13, Wang in view of Wang-2017 teaches all the limitations of claim 9 above.
Wang further teaches wherein the image processor is further configured to detect a torso area of the at least one subject in the optical depth video data ([0068] states that “The imaging unit 2 may include a camera (also referred to as detection unit or remote PPG sensor) for acquiring an image data, said image data comprising a time-sequence of image frames… The image data represents a scene, preferably comprising skin areas 23, 24, 25 of the subject 20 from which a physiological parameter of the subject 20 can be derived. Exemplary skin areas that are usually not covered by a blanket 26 or clothing are the forehead 23, the cheeks 24 or the hands or arms 25”, hence while Wang does not explicitly mention detecting the torso, by virtue of the system being configured for detecting the forehead, cheeks and hands, the system is also configured for detecting a torso of the subject. Examiner notes that the torso detection is an intended use of the system and according to MPEP 2114(II), “A claim containing a "recitation with respect to the manner in which a claimed apparatus is intended to be employed does not differentiate the claimed apparatus from a prior art apparatus" if the prior art apparatus teaches all the structural limitations of the claim”).
PNG
media_image2.png
414
508
media_image2.png
Greyscale
Wang in view of Wang-2017 does not teach wherein the identifying of the region of interest comprises removing edge areas from the detected torso area.
However, within the same field of endeavor, Jacquel further teaches medical monitoring, and in particular non-contact, video-based monitoring of pulse rate, respiration rate, motion, and oxygen saturation including capturing images of a patient, producing intensity signals from the images, filtering those signals to focus on a physiologic component, and measuring a vital sign from the filtered signals (see abstract), using a system 200 of [0067], the system comprising a video camera configured to record multiple sequential image frames (such as image frames 300A and 300B) that each include the head region 314 and chest/torso region 316 according to [0068] and [0069]. [0083] states that “referring to FIG. 3A, cells or groups of pixels at the edges of the forehead region 330 can be added or removed from the combined signal during motion as they enter and exit the forehead region” and hence teaching the limitation “wherein the identifying of the region of interest comprises removing edge areas from the detected torso area”. Fig. 3A is reproduced below for reference.
PNG
media_image3.png
358
500
media_image3.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to configure modified Wang, wherein the identifying of the region of interest comprises removing edge areas from the detected torso area, as taught by Jacquel, as such modification enables the monitoring system to continue to track the vital sign through movement of the patient, even as the patient moves or rotates with respect to the camera according to [0083], producing a vitals signs with improved quality ([0158]), with a reasonable expectation of success as Wang also strives to improve signal quality of vital sign according to [0098] of Wang.
Regarding claim 14, Wang in view of Wang-2017 teaches all the limitations of claim 9 above.
Wang further teaches wherein the image processor is further configured to detect a head area of the at least one subject in the optical depth video data ([0068] states that “The image data represents a scene, preferably comprising skin areas 23, 24, 25 of the subject 20 from which a physiological parameter of the subject 20 can be derived. Exemplary skin areas that are usually not covered by a blanket 26 or clothing are the forehead 23, the cheeks 24 or the hands or arms 25”),
Wang in view of Wang-2017 does not teach wherein the identifying of the region of interest comprises removing edge areas from the detected head area.
However, within the same field of endeavor, Jacquel further teaches medical monitoring, and in particular non-contact, video-based monitoring of pulse rate, respiration rate, motion, and oxygen saturation including capturing images of a patient, producing intensity signals from the images, filtering those signals to focus on a physiologic component, and measuring a vital sign from the filtered signals (see abstract), using a system 200 of [0067], the system comprising a video camera configured to record multiple sequential image frames (such as image frames 300A and 300B) that each include the head region 314 and chest/torso region 316 according to [0068] and [0069]. [0083] states that “referring to FIG. 3A, cells or groups of pixels at the edges of the forehead region 330 can be added or removed from the combined signal during motion as they enter and exit the forehead region” and hence teaching the limitation “wherein the identifying of the region of interest comprises removing edge areas from the detected head area”. Fig. 3A is reproduced below for reference.
PNG
media_image3.png
358
500
media_image3.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to configure modified Wang, wherein the identifying of the region of interest comprises removing edge areas from the detected head area, as taught by Jacquel, as such modification enables the monitoring system to continue to track the vital sign through movement of the patient, even as the patient moves or rotates with respect to the camera according to [0083], producing a vitals signs with improved quality ([0158]), with a reasonable expectation of success as Wang also strives to improve signal quality of vital sign according to [0098] of Wang.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Farouk A Bruce whose telephone number is (408)918-7603. The examiner can normally be reached Mon-Fri 8-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christopher Koharski can be reached at (571) 272-7230. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FAROUK A BRUCE/ Examiner, Art Unit 3797
/CHRISTOPHER KOHARSKI/ Supervisory Patent Examiner, Art Unit 3797
Prosecution Insights