DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzvieli (PGPUB: 20200221956 A1) in view of Zhang (CN 116327154 A).
Regarding claims 1 and 9. Tzvieli teaches a measuring device, comprising:
a capturing unit configured to capture a biological subject to sequentially obtain a plurality of images (see Fig. 1, paragraph 92 and 109, inward-facing camera 802a is located above the nose bridge and captures images of an area 803a on the user's forehead. Inward-facing cameras 802b and 802c are located on the left and right sides of the smartglasses 800, respectively; they capture images that include areas 803b and 803c on the left and right sides of the user's face, respectively, a facial blood flow pattern may refer to time series data, such as a sequence of images representing a progression of a pulse wave in the area. Different extents of blood flow may produce different sequences of representative images, which depend on the structure of the facial blood vessels of the user); and
a time point estimating unit configured to each of the plurality of images is obtained in accordance with a periodic variation that appears in a sequence of a plurality of pixel values each obtained from a corresponding one of the plurality of images (see Fig. 1, paragraph 131 and 141, the computer 828 calculates a value indicative of skin color at different times based on the second set of images, and calculates the extent of CHF based on a length of a duration following the period, in which the difference between the skin color and a baseline skin color, calculated based on the first set of images, was above a threshold. Optionally, a feature value indicative of the length of the duration may be utilized to calculate the extent of CHF; the various types of feature values that may be generated based on the images 821 by the computer 828. In one embodiment, at least some of the feature values may be derived directly from values of pixels in the images 821. Optionally, at least some of the feature values are values of pixels from the images 821. Optionally, one or more of the feature values may be the values of the pixels themselves or some simple function of the pixels, such as the average of pixels at certain regions in each of the images; one or more of the feature values may represent a difference between values of pixels at one time t at a certain location on the face and values of pixels at a different location at some other time t+x (which can help detect different arrival times of a pulse wave)).
However, Tzvieli does not estimate a time point of images.
Zhang teaches that obtaining the target image sequence, the target image sequence comprises a multi-frame image of the target object. wherein the target image sequence comprises a multi-frame image of the target object in the first time period. The target image sequence may be an image containing the respective time points of the same target object in the video stream of which the acquisition time is the first time period. The embodiment does not limit the acquisition mode of the target image sequence, wherein the acquisition time of the multi-frame image is continuous, for example, the first time period can be 30 s, for the time of the target object is 30 s of the video stream, when the frame rate is 30 frames per second, the obtained target image sequence comprises 900 frame image. The image in the target image sequence may be an image containing any of the skin object region, such as neck or hand region (see page 6 and 7, lines 27-33 and 1-4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tzvieli by Zhang to obtain obtaining the target image sequence, the target image sequence comprises a multi-frame image of the target object. wherein the target image sequence comprises a multi-frame image of the target object in the first time period. The target image sequence may be an image containing the respective time points of the same target object in the video stream of which the acquisition time is the first time period, in order to provide estimate a time point of images. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Claim(s) 2-3, 5-8, and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzvieli (PGPUB: 20200221956 A1) in view of Zhang (CN 116327154 A), and further in view of Roberts (PGPUB: 20170187455 A1).
Regarding claim 2. The combination teaches the measuring device according to claim 1, wherein the capturing unit sequentially obtains a plurality of capture images at a frame rate set to a blink that a lighting fixture makes regularly (see Tzvieli, Fig. 1, paragraph 94, the images 821 are captured at a frame rate of at least 30 frames per second (fps). In another example, the images 821 are captured at a frame rate of at least 100 fps. In still another example, the images 821 are captured at a frame rate of at least 256 fps. Images taken by inward-facing cameras may have various resolutions. In one example, the images 821 have a resolution of at least 8×8 pixels. In another example, the images 821 have a resolution of at least 32×32 pixels. In yet another example, the images 821 have a resolution of at least 640×480 pixels), and
the plurality of images are at least partially the plurality of capture images (see Tzvieli, Fig. 1, paragraph 90, the system includes smartglasses 805, which have various components coupled thereto. These components include the inward-facing cameras 806a and 806b, which capture images that include portions of the left and right sides of the forehead, respectively).
However the combination does not expressly teach a frame rate set to cause aliasing due to a blink.
Roberts teaches that the apparent blinking, as seen by the camera, may either be due to a blinking frequency that is slower than the camera frame rate, or it may be due to sub-sampling aliasing produced by the camera of a light that blinks faster than the human flicker perception frequency (see paragraph 48).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by Roberts to obtain that the apparent blinking, as seen by the camera, may either be due to a blinking frequency that is slower than the camera frame rate, or it may be due to sub-sampling aliasing produced by the camera of a light that blinks faster than the human flicker perception frequency, in order to provide a frame rate set to cause aliasing due to a blink. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claim 3. The combination teaches the measuring device according to claim 2, further comprising
a pixel value calculating unit configured to calculate, from each of the plurality of images, a first representative pixel value among pixel values of two or more pixels in a first region of interest in order to calculate each of a plurality of the first representative pixel values from a corresponding one of the plurality of images (see page 12, lines 5-9, the image in the target image sequence comprises a human face region of interest of the target object, the human face region of interest of the target object may be any region of the face skin the pixel value of the skin can be a grey value, a brightness value or intensity value and so on. For example, it can select the forehead and cheek region with more rPPG information as face region of interest),
wherein the plurality of pixel values are the plurality of first representative pixel values (see Page 12, lines 19-22, the initial signal comprises information of period change of the skin pixel value caused by blood flow generated by heart beat, and may also include environmental factors such as illumination change and the face information caused by the interference information).
Regarding claim 5. The combination teaches the measuring device according to claim 2,
wherein the periodic variation includes a frequency component due to the blink (see Tzvieli, Fig. 1, paragraph 94, the images 821 are captured at a frame rate of at least 30 frames per second (fps). In another example, the images 821 are captured at a frame rate of at least 100 fps. In still another example, the images 821 are captured at a frame rate of at least 256 fps. Images taken by inward-facing cameras may have various resolutions. In one example, the images 821 have a resolution of at least 8×8 pixels. In another example, the images 821 have a resolution of at least 32×32 pixels. In yet another example, the images 821 have a resolution of at least 640×480 pixels).
Regarding claim 6. The combination teaches the measuring device according to claim 3,
wherein the first region of interest includes an image of a body surface of the biological subject (see Tzvieli, paragraph 99, due to the proximity of the one or more inward-facing cameras to the face, in some embodiments, there may be an acute angle between the optical axis of an inward-facing camera and the area captured by images taken by said camera (e.g., when the area is on, and/or includes a portion of, the forehead or a cheek)), and
the measuring device further comprises a biological signal calculating unit configured to calculate a biological signal from the plurality of first representative pixel values (see Tzvieli, paragraph 101, in order to incorporate information about respiration into detection of CHF, some embodiments include the sensor 822, which is physically coupled to the smartglasses, and configured to measure a respiration signal 823, which is a signal indicative of the user's respiration rate. Various types of sensors may be utilized for this purpose).
Regarding claim 7. The combination teaches the measuring device according to claim 3,
wherein the first region of interest does not include the image of the body surface of the biological subject, the pixel value calculating unit calculates, from each of the plurality of images (see Tzvieli, paragraph 153, the system includes an inward-facing head-mounted thermal camera configured to take measurements of a region on the forehead of the user, where the measurements of the region on the forehead of the user are indicative of the stress level of the user. In still another example, the system includes an inward-facing head-mounted thermal camera configured to take measurements of a region on the nose of the user, where the measurements of the region on the nose of the user are indicative of the stress level of the user), a second representative pixel value among pixel values of two or more pixels in a second region of interest in order to calculate each of a plurality of the second representative pixel values from a corresponding one of the plurality of images (see Tzvieli, paragraph 118, the computer 828 is configured to calculate the extent of CHF based on: a facial blood flow patterns recognizable in the images 821, and respiration rate recognizable in the respiration signal 823) (see Tzvieli, paragraph 211, at least some of the feature values may be pixel values obtained by inward-facing head-mounted cameras. Optionally, the pixel values may be provided as input to functions in order to generate at feature values that are low-level image-based features; one or more of the feature values may be derived from multiple images taken at different times, such as volume local binary patterns (VLBP), cuboids, and/or optical strain-based features. In one example, one or more of the feature values may represent a difference between values of pixels at one time t and values of other pixels at a different region at some other time t+x (which, for example, can help detect different arrival times of a pulse wave)), and
the measuring device further comprises a biological signal calculating unit configured to calculate a biological signal from the plurality of the second representative pixel values (see Tzvieli, paragraph 211, at least some of the feature values may be pixel values obtained by inward-facing head-mounted cameras. Optionally, the pixel values may be provided as input to functions in order to generate at feature values that are low-level image-based features; one or more of the feature values may be derived from multiple images taken at different times, such as volume local binary patterns (VLBP), cuboids, and/or optical strain-based features. In one example, one or more of the feature values may represent a difference between values of pixels at one time t and values of other pixels at a different region at some other time t+x (which, for example, can help detect different arrival times of a pulse wave)).
Regarding claim 8. The combination teaches the measuring device according to claim 6,
wherein the biological signal indicates a pulse wave signal (see Tzvieli, paragraph 111, sentences of the form “a facial blood flow pattern recognizable in the images (of an area comprising skin on the user's head)” refer to effects of blood volume changes due to pulse waves that may be extracted from one or more images of the area).
Regarding claim 10. The combination teaches the measuring device according to claim 7,
wherein the biological signal indicates a pulse wave signal (see Tzvieli, paragraph 111, sentences of the form “a facial blood flow pattern recognizable in the images (of an area comprising skin on the user's head)” refer to effects of blood volume changes due to pulse waves that may be extracted from one or more images of the area).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzvieli (PGPUB: 20200221956 A1) in view of Zhang (CN 116327154 A), in view of Roberts (PGPUB: 20170187455 A1), and further in view of JONES (EP 3278722 B1).
Regarding claim 4. The combination teaches the measuring device according to claim 3,
wherein the time point estimating unit calculates a period at the periodic variation in accordance with the frame rate (see Tzvieli, Fig. 1, paragraph 94, the images 821 are captured at a frame rate of at least 30 frames per second (fps). In another example, the images 821 are captured at a frame rate of at least 100 fps. In still another example, the images 821 are captured at a frame rate of at least 256 fps. Images taken by inward-facing cameras may have various resolutions. In one example, the images 821 have a resolution of at least 8×8 pixels. In another example, the images 821 have a resolution of at least 32×32 pixels. In yet another example, the images 821 have a resolution of at least 640×480 pixels) and a frequency of the blink (see Roberts, paragraph 49, the composite waveform, as described above, includes two parts. In some embodiments the first part may include a lower frequency time varying amplitude component that, when subsampled by a low frame rate camera, results in alias induced flicker (i.e. blinking, twinkle)).
However, the combination does not expressly teach:
determines that a frame drop has occurred if a signal having the frequency does not match the plurality of first representative pixel values.
JONES teaches that since the pulse wave is detected by simple and low-load processing, drop of a frame (so-called drop frame) caused by the computer processing capability can be suppressed even in the real-time processing (see paragraph 101); The pulse wave signal 52 before correction is subjected to an influence of the drop in brightness as illustrated in FIG. 8(c), and the pulse wave signal Qm in the period 42 also drops (see paragraph 184); In the after correction pulse wave signal 53, since the influence by the change in the brightness has been removed, an appropriate pulse wave can be obtained even in the period 42 when the brightness drops (see paragraph 187).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by JONES to obtain the pulse wave signal 52 before correction is subjected to an influence of the drop in brightness as illustrated in FIG. 8(c), and the pulse wave signal Qm in the period 42 also drops; in the after correction pulse wave signal 53, since the influence by the change in the brightness has been removed, an appropriate pulse wave can be obtained even in the period 42 when the brightness drops, in order to provide to determine that a frame drop has occurred if a signal having the frequency does not match the plurality of first representative pixel values. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN JIA whose telephone number is (571)270-5536. The examiner can normally be reached 9:00 am-7:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIN JIA/Primary Examiner, Art Unit 2663
Prosecution Insights