DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/29/2024, 12/12/2024, 08/18/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Allowable Subject Matter
Claims 4 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, 5-10, and 13-15 is/are rejected under 35 U.S.C. 102(a) as being taught by Wu et al. (US Patent Number 2021/0386307-A1, hereinafter “Wu”).
Regarding claim 1, Wu teaches: A method comprising: accessing a video of a user’s face captured by a camera, the video comprising a sequence of image frames; accessing, for each image frame in the video, ([0036], "Contact-free monitoring of the heart rate using videos of human faces or other exposed skin is a user-friendly approach compared to conventional contact based ones, such as the use of electrodes, chest belts, and/or finger clips")
(1) one or more facial landmarks of the user’s face determined by a facial landmark detection (FLD) model and ([0047], "To determine the cheek regions for conducting spatial averaging 140, an example embodiment may construct two conservative regions that do not contain facial structures and are most upfront in order to avoid strong motion-induced specular illumination changes. Certain embodiments may then use identified facial landmarks to facilitate the construction of the cheek regions. In one embodiment, each cheek region may be constructed to be a polygon that has a safe margin to the facial structures protected by the landmarks.")
(2) a corresponding determined position in the image for each facial landmark; (Fig. 8(a); [0077], "A facial region of interest (ROI) may be located with the facial landmarks estimated with an ensemble of regression trees. Certain embodiments may then follow a ROI selection process to include the cheek and forehead regions of a face."; [0078], "FIG. 8 illustrates a face landmark localization and skin classification result example. FIG. 8(a) depicts an example cropped video frame, and FIG. 8(b) depicts facial landmark localization result (dots) and the estimated face ROI (the transparent area). FIG. 8(d) illustrates a scatter plot of the skin pixels in face ROI in their top three dominating principle component directions according to eigenvectors of S.")
determining, based on the one or more facial landmarks and corresponding positions, a motion of the user’s face in the captured video; extracting, from the determined motion of the user’s face, a corrected motion signal of the user’s face in the video; adjusting, based on the extracted corrected motion signal of the user’s face, one or more determined positions of one or more facial landmarks in one or more image frames of the video; ([0040], "In certain embodiments, the system and method may include correction mechanisms to account for a variety of noise sources, including subject movement or facial motion, changes in ambient environment, or natural variations in the subject's complexion. In certain embodiments, the system and method may further include features to protect the privacy and personal data of the subjects."; [0077], "Certain embodiments can provide a highly precise motion compensation in order to generate a clean face color signal. For example, in an embodiment, the skin detection block 710, as depicted in the example of FIG. 7, may include a DNN face detector trained with a Single-Shot-Multibox detector (SSD) and a ResNet framework to obtain a rectangular face detection region. The SSD-ResNet can provide a better detection result compared with traditional methods, such as the Viola-Jones detector, especially for face profiles. In an embodiment, a CSR-DCF algorithm may be used to track the face region. A facial region of interest (ROI) may be located with the facial landmarks estimated with an ensemble of regression trees. Certain embodiments may then follow a ROI selection process to include the cheek and forehead regions of a face.")
determining, based at least in part on the adjusted positions of the facial landmarks in the sequential images of the video, an rPPG signal; (Fig. 13; [0119], "FIG. 13 illustrates an example flow diagram of a remote photoplethysmography (rPPG) method, according to an example embodiment. For instance, the method of FIG. 13 may be configured to measure pulse rate and/or pulse rate variability of one or more subjects. In certain embodiments, the method of FIG. 14 may be performed by the system of FIG. 7 and/or apparatus of FIG. 6, discussed above.")
and determining, based on the determined rPPG signal, one or more vital signs of the user. ([0038], "As discussed below, certain embodiments provide a remote photoplethysmography (rPPG) system that is robust for purposes of pulse rate and/or pulse rate variability extraction from fitness face video, such as where the subject is exercising and/or the video contains large subject motions. Some embodiments provide an online learning scheme for precise subject- and scene-specific skin detection, and can use motion information as a cue to adaptively remove the motion-induced artifacts in the corrupt rPPG signal. In an embodiment, for pulse rate variability extraction, the accurate heart rate estimation is provided as feedback for a second-level pulse signal filtering. In certain embodiments, after the pulse filtering processing, the inter-beat intervals and/or pulse rate variability can be precisely estimated.")
Regarding claim 2, Wu teaches: The method of Claim 1, wherein extracting, from the determined motion of the user’s face, a corrected motion signal of the user’s face comprises filtering the determined motion of the user’s face. ([0040], "In certain embodiments, the system and method may include correction mechanisms to account for a variety of noise sources, including subject movement or facial motion, changes in ambient environment, or natural variations in the subject's complexion. In certain embodiments, the system and method may further include features to protect the privacy and personal data of the subjects."; [0077], "Certain embodiments can provide a highly precise motion compensation in order to generate a clean face color signal. For example, in an embodiment, the skin detection block 710, as depicted in the example of FIG. 7, may include a DNN face detector trained with a Single-Shot-Multibox detector (SSD) and a ResNet framework to obtain a rectangular face detection region. The SSD-ResNet can provide a better detection result compared with traditional methods, such as the Viola-Jones detector, especially for face profiles. In an embodiment, a CSR-DCF algorithm may be used to track the face region. A facial region of interest (ROI) may be located with the facial landmarks estimated with an ensemble of regression trees. Certain embodiments may then follow a ROI selection process to include the cheek and forehead regions of a face.")
Regarding claim 3, Wu teaches: The method of Claim 2, wherein filtering the determined motion of the user’s face comprises applying a dynamic filter with an adaptive alpha parameter α that is based on the rate of the change of the determined motion. ([0097], "where p∈R.sup.3×1 denotes the projection vector of the POS algorithm, u.sub.p denotes the unit color direction of the pulse component, and um,k denotes the unit color direction of the kth motion component. The motion residue term in Eq. (8) is negligible when the illumination source is single, as the POS direction is orthogonal to the color direction of the motion-induced intensity change, and the specular change is suppressed via “alpha tuning”. However, if the video is captured in an uncontrolled environment, the motion residue term is often non-negligible, and sometimes can be more significant than the pulse term.")
Regarding claim 5, Wu teaches: The method of Claim 1, wherein extracting, from the determined motion of the user’s face, a corrected motion signal of the user’s face comprises: providing, to a trained face-alignment model, the determined motion of the user’s face; and outputting, by the trained face-alignment model, the corrected motion signal. ([0080], "The direct use of a pre-trained skin detection model may generate high false positive skin detection results, when the model accounts for all possible skin-tone variations, while high false negative results in a test instance, when the model fails to include the specific skin color. To address this problem, certain embodiments include a hypothesis testing scheme to detect skin pixels based on a scenario-tailored learning of the probability distribution of the skin pixels, which is detailed below.")
Regarding claim 6, Wu teaches: The method of Claim 1, further comprising: defining a reference image of the user’s face, the reference image comprising one or more reference facial landmarks; determining, for each of one or more subsequent frames in the sequence of image frames, a landmark difference between the facial landmarks in that frame and the corresponding reference facial landmarks in the reference image; and transforming, based on the landmark difference, the respective subsequent frames. ([0044], "According to certain embodiments, each video may be divided into small temporal segments with one frame overlapping for successive segments. In an embodiment, the frame in the middle of the segment may be used as the reference for optical flow based motion compensation. This would ensure that two frames being aligned do not have significant occlusion due to long separation in time. FIG. 2 illustrates an example of face images from a same segment before and after optical flow based motion compensation using the same reference, according to an example embodiment.")
Regarding claim 7, Wu teaches: The method of Claim 6, wherein the reference image comprises the first image in the sequence of images. ([0044]), "According to certain embodiments, each video may be divided into small temporal segments with one frame overlapping for successive segments. In an embodiment, the frame in the middle of the segment may be used as the reference for optical flow based motion compensation. This would ensure that two frames being aligned do not have significant occlusion due to long separation in time. FIG. 2 illustrates an example of face images from a same segment before and after optical flow based motion compensation using the same reference, according to an example embodiment.")
Regarding claim 8, Wu teaches: The method of Claim 6, wherein the reference image comprises an image in which the user’s face is oriented such that the user is looking directly at the camera. (Fig. 8 (a))
Regarding claim 9, Wu teaches: The method of Claim 6, wherein: determining the landmark difference comprises determining a difference in an orientation of the user’s face relative to the camera; and transforming, based on the landmark difference, the respective subsequent frames comprises transforming a perspective of the subsequent frame to match a perspective of the reference image. ([0044]), "According to certain embodiments, each video may be divided into small temporal segments with one frame overlapping for successive segments. In an embodiment, the frame in the middle of the segment may be used as the reference for optical flow based motion compensation. This would ensure that two frames being aligned do not have significant occlusion due to long separation in time. FIG. 2 illustrates an example of face images from a same segment before and after optical flow based motion compensation using the same reference, according to an example embodiment."; Examiner's Note - Motion maps to orientation)
Regarding claim 10, Wu teaches: The method of Claim 6, further comprising determining, in the reference image, a light intensity corresponding to each of the reference landmarks, wherein: determining the landmark difference comprises determining a difference in light intensity between one or more reference landmarks in the reference image and one or more corresponding facial landmarks in the subsequent frame; and transforming, based on the landmark difference, the respective subsequent frame comprises transforming the lighting intensity of the subsequent frame to match the lighting intensity of the reference image. ([0058], "In an embodiment, the generating of the skin color signals 530 may include estimating a skin or face color for each frame by taking a spatial average over pixels of a cheek of the face(s) for the R, G, and B channels, respectively. According to some embodiments, the generating of the skin color signals 530 may include concatenating segments of the video(s) into color signals. When concatenating the segments, in an embodiment, the last point of the current segment and the first point of the next segment may have different intensities because they correspond to the same frame whose motion compensation were conducted with respect to two different references. Thus, according to an embodiment, the generating of the skin color signals 530 may include calculating the difference of the intensity between the two points and using the resulting value to bias the signal of the next segment in order to maintain the continuity. The skin or face color signals may contain color change due to the heartbeat, and illumination change due to face motions such as tilting. In an embodiment, the green channel may be used because it corresponds to the absorption peak of (oxy-) hemoglobin that changes periodically as the heartbeat, and source separation methods such as ICA may also be used to separate the heartbeat component.")
Regarding claim 13, Wu teaches: The method of Claim 1, wherein the one or more vital signs of the user comprise one or more of a blood oxygenation, a heart rate, a respiratory rate, or a blood pressure. ([0054], "A robust spectrogram based frequency estimator may then be applied to extract the final heart rate trace.")
Regarding claim 14, claim 14 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Wu further teaching on: One or more non-transitory computer readable storage media storing instructions and coupled to one or more processors that are operable to execute the instructions to: access a video of a user’s face captured by a camera (Fig. 6)
Regarding claim 15, claim 15 has been analyzed with regard to claim 6 and is rejected for the same reasons of obviousness as used above as well as in accordance with Wu further teaching on: One or more non-transitory computer readable storage media storing instructions and coupled to one or more processors that are operable to execute the instructions to: access a video of a user’s face captured by a camera (Fig. 6)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 11-12, 16-17, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (US Patent Number 2021/0386307-A1, hereinafter “Wu”) in view of Wang et al. (US Patent Number 2020/0342210 -A1, hereinafter “Wang”).
Regarding claim 11, Wu does not teach: The method of Claim 1, wherein the extraction and the adjustment are performed by a face alignment model, the face alignment having been selected from a plurality of face-alignment models based on one or more evaluation scores of the face-alignment models.
However, Wang does teach: The method of Claim 1, wherein the extraction and the adjustment are performed by a face alignment model, the face alignment having been selected from a plurality of face-alignment models based on one or more evaluation scores of the face-alignment models. (Wang, [0029], "The system may be also caused to conduct T (T≥1) stages of model set updating operation. In each stage of the T stages of model set updating operation, the system may be caused to conduct a first performance evaluation to each candidate model of the image processing model set with respect to the test image, and update the image processing model set by excluding at least one model from the image processing model set based on the first performance evaluation"; [0500], "Each of the plurality of face alignment model may determine a shape for a face (or an image of the face) included in an inputted image (e.g., test image), and may preferably be operated on a face having a postural angle within the corresponding postural angle range. The term “shape” in the present disclosure generally refers to a set of landmarks for describing key parts (e.g., eyes, nose, mouse, eyebrows) of a face. The postural angle range may be predetermined when training the corresponding face alignment model.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify contact free heart rate monitoring through exposed human faces to include a face alignment model a because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, heart rate monitoring through exposed faces as modified by face alignment model can yield a predictable result of better allowing the hear rate monitoring to better determine data adjustments from face alignment and angle. Thus, a person of ordinary skill would have appreciated including in contact free heart rate monitoring the ability to do face alignment model since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 12, Wu in view of Wang teaches: The method of Claim 11, wherein the one or more evaluation scores comprise one or more of a circular radius score, a mean offset score, or an impacted pixels score. (Wang, [0029], "The system may be also caused to conduct T (T≥1) stages of model set updating operation. In each stage of the T stages of model set updating operation, the system may be caused to conduct a first performance evaluation to each candidate model of the image processing model set with respect to the test image, and update the image processing model set by excluding at least one model from the image processing model set based on the first performance evaluation"; [0500], "Each of the plurality of face alignment model may determine a shape for a face (or an image of the face) included in an inputted image (e.g., test image), and may preferably be operated on a face having a postural angle within the corresponding postural angle range. The term “shape” in the present disclosure generally refers to a set of landmarks for describing key parts (e.g., eyes, nose, mouse, eyebrows) of a face. The postural angle range may be predetermined when training the corresponding face alignment model."; Examiner's Note - In prior art, the confidence score is based off the pixels of the face and the accuracy of the model, mapping to a impacted pixels score)
Regarding claim 16, Wu teaches: A method comprising: generating, for each frame in a reference video of a reference subject, a plurality of ground-truth facial landmarks; accessing, for each frame in a test video of a test subject, a plurality of facial landmark detection (FLD) facial landmarks determined by an FLD model; ([0047], "To determine the cheek regions for conducting spatial averaging 140, an example embodiment may construct two conservative regions that do not contain facial structures and are most upfront in order to avoid strong motion-induced specular illumination changes. Certain embodiments may then use identified facial landmarks to facilitate the construction of the cheek regions. In one embodiment, each cheek region may be constructed to be a polygon that has a safe margin to the facial structures protected by the landmarks.")
However, Wang does teach: (Wang, [0029], "The system may be also caused to conduct T (T≥1) stages of model set updating operation. In each stage of the T stages of model set updating operation, the system may be caused to conduct a first performance evaluation to each candidate model of the image processing model set with respect to the test image, and update the image processing model set by excluding at least one model from the image processing model set based on the first performance evaluation"; [0500], "Each of the plurality of face alignment model may determine a shape for a face (or an image of the face) included in an inputted image (e.g., test image), and may preferably be operated on a face having a postural angle within the corresponding postural angle range. The term “shape” in the present disclosure generally refers to a set of landmarks for describing key parts (e.g., eyes, nose, mouse, eyebrows) of a face. The postural angle range may be predetermined when training the corresponding face alignment model.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify contact free heart rate monitoring through exposed human faces to include a face alignment model a because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, heart rate monitoring through exposed faces as modified by face alignment model can yield a predictable result of better allowing the hear rate monitoring to better determine data adjustments from face alignment and angle. Thus, a person of ordinary skill would have appreciated including in contact free heart rate monitoring the ability to do face alignment model since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 17, Wu in view of Wang teaches: The method of Claim 16, wherein the reference subject comprises a mannequin head. (Wang, [0029], "The system may be also caused to conduct T (T≥1) stages of model set updating operation. In each stage of the T stages of model set updating operation, the system may be caused to conduct a first performance evaluation to each candidate model of the image processing model set with respect to the test image, and update the image processing model set by excluding at least one model from the image processing model set based on the first performance evaluation"; [0500], "Each of the plurality of face alignment model may determine a shape for a face (or an image of the face) included in an inputted image (e.g., test image), and may preferably be operated on a face having a postural angle within the corresponding postural angle range. The term “shape” in the present disclosure generally refers to a set of landmarks for describing key parts (e.g., eyes, nose, mouse, eyebrows) of a face. The postural angle range may be predetermined when training the corresponding face alignment model.")
Regarding claim 19, Wu in view of Wang teaches: The method of Claim 16, wherein the one or more scoring criteria comprises one or more of a circular radius score, a mean offset score, or an impacted pixels score. (Wang, [0029], "The system may be also caused to conduct T (T≥1) stages of model set updating operation. In each stage of the T stages of model set updating operation, the system may be caused to conduct a first performance evaluation to each candidate model of the image processing model set with respect to the test image, and update the image processing model set by excluding at least one model from the image processing model set based on the first performance evaluation"; [0500], "Each of the plurality of face alignment model may determine a shape for a face (or an image of the face) included in an inputted image (e.g., test image), and may preferably be operated on a face having a postural angle within the corresponding postural angle range. The term “shape” in the present disclosure generally refers to a set of landmarks for describing key parts (e.g., eyes, nose, mouse, eyebrows) of a face. The postural angle range may be predetermined when training the corresponding face alignment model."; Examiner's Note - In prior art, the confidence score is based off the pixels of the face and the accuracy of the model, mapping to a impacted pixels score)
Regarding claim 20, Wu in view of Wang teaches: The method of Claim 16, wherein evaluating the FLD model comprises evaluating a transformed landmark determined by a facial alignment model applied to an output of the FLD model. (Wang, [0029], "The system may be also caused to conduct T (T≥1) stages of model set updating operation. In each stage of the T stages of model set updating operation, the system may be caused to conduct a first performance evaluation to each candidate model of the image processing model set with respect to the test image, and update the image processing model set by excluding at least one model from the image processing model set based on the first performance evaluation")
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jinsu Hwang whose telephone number is (703)756-1370. The examiner can normally be reached Mon -Thu 10am-8am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JINSU HWANG/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights