Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are currently pending in the application filed 09/19/2024
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 09/19/2024, 01/10/2025, 04/02/2025, and 06/26/2025 have been considered by the Examiner
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 6, 7, 9, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Visvikis (US 12,220,220 B2).
Regarding claim 1, Visvikis teaches:
A system, comprising:
(Visvikis, [Col 4 Line 64]; “The present invention further relates to a system for measuring respiratory parameters of a subject comprising:”)
one or more image sensors configured to collect image data of a user in an environment over a period of time;
(Visvikis, [Col 4 Line 66]; “an acquisition module configured to control a range imaging sensor for the acquisition of at least one raw image comprising at least one portion of the torso of the subject, wherein each point of the raw image represents the distance between the range imaging sensor and the subject”)
(Visvikis, [Col 4 Line 22]; “the range imaging sensor is a time-of-flight (ToF) camera”)
Examiner Note: The applicant’s specification at [0032] states that “image data of the user may be captured using image sensors” and that depth sensors such as time-of-flight sensors located on a device facing the user may be used to obtain the data. Visvikis’s range imaging sensor, embodied as a time-of-flight camera that acquires raw images in which each point represents the distance to the subject, is a sensor that collects image data of the user, and reads on the claimed “one or more image sensors configured to collect image data of a user.” The raw images acquired across the monitoring period read on “image data… over a period of time.”
and processing circuitry (Visvikis, [Col 18 Line 18]; "the system is a data processing system such as dedicated circuitry or a general-purpose computer device (i.e. a processor), configured for receiving the raw images and executing the operations"), configured to: obtain the image data;
(Visvikis, [Col 5 Line 19]; “an input module configured to receive a set of acquisitions derived from a range imaging sensor, said set of acquisitions comprising at least one raw image comprising at least one portion of a torso of the subject”)
generate, from the image data, point cloud data comprising points representative of the user in the environment as captured in the image data over the period of time;
(Visvikis, [Col 2 Line 61]; “The present method is based on the analysis of 3D surface images (i.e. 3D point cloud)”)
(Visvikis, [Col 2 Line 46]; “generating a surface image of at least one portion of a surface of the torso of the subject by surface interpolation of the raw image”)
Examiner Note: Visvikis generates a 3D point cloud (surface image) from the raw image data by surface interpolation. The 3D point cloud, which represents the surface of the user’s torso as captured across the acquisition period, reads on the claimed “point cloud data comprising points representative of the user… over the period of time.”
select one or more regions of interest in the point cloud data across the period of time, wherein the selected one or more regions of interest contain points in the point cloud data that correspond to upper-body landmarks of the user;
(Visvikis, [Col 12 Line 7]; “the position and dimensions of the region of interest are automatically defined on the torso of the patient by means of a deep learning algorithm”)
(Visvikis, [Col 12 Line 42]; “The patient torso ROI may be then defined by the surface connecting the identified joints of shoulders and hipbones”)
Examiner Note: The applicant’s specification at [0041] selects the region of interest from landmark points such as the shoulders, the neck-to-shoulder connection, and the waist. Visvikis defines its region of interest on the torso from upper-body key points (shoulders, hips, chest) identified in the point cloud, which reads on selecting a region of interest containing points that correspond to “upper-body landmarks of the user.”
measure respective motion of the points contained in the selected one or more regions of interest in the point cloud data over the period of time;
(Visvikis, [Col 3 Line 12]; “the thoracic region motion (chest wall motion) may be tracked by the analysis of the 3D spatial information provided by the range imaging sensor”)
(Visvikis, [Col 11 Line 37]; “derive the respiratory motion dynamics on a point by point basis”)
Examiner Note: Visvikis tracks the motion of the chest-wall surface within the region of interest on a point-by-point basis across the acquisition, which reads on measuring “respective motion of the points contained in the selected one or more regions of interest… over the period of time.”
based on the respective motion of the points, generate an estimated breath signal of the user;
(Visvikis, [Col 2 Line 50]; “estimating a respiratory signal as a function of time calculated as the spatial average, in a given region of interest defined on the torso of the subject, of the differences between the depth values of the surface image at a given time and the depth values of a reference surface image”)
Examiner Note: Visvikis estimates a respiratory signal as a function of time from the depth differences (the measured point motion) within the region of interest. This respiratory signal reads on the claimed “estimated breath signal of the user.”
and provide a final breath signal based, at least in part, on the estimated breath signal.
(Visvikis, [Col 2 Line 60]; “providing as output said respiratory parameters”)
(Visvikis, [Col 18 Line 16]; “Said output module may be a display or a microphone connected wirelessly or not to the system”)
While Visvikis discloses estimating a respiratory signal from the point cloud and providing that respiratory signal as output, Visvikis does not expressly recite the claimed two-tier arrangement in which a “final breath signal” is provided “based, at least in part, on the estimated breath signal.” However, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide Visvikis’s estimated respiratory signal as the final breath signal. The reason is that Visvikis already estimates the respiratory signal and outputs it for monitoring, so providing the estimated signal as the final output signal merely carries out Visvikis’s express output step. The applicant’s own specification at [0085] describes providing the estimated breath signal directly as the signal that is provided, and the claim term “based, at least in part, on” does not foreclose the final breath signal from being the estimated breath signal itself. Thus, it would have been obvious to provide a final breath signal based at least in part on Visvikis’s estimated breath signal, because doing so accomplishes Visvikis’s stated purpose of outputting the respiratory signal to a user or monitor, with a reasonable expectation of success.
Regarding claim 3, Visvikis teaches:
wherein to select the region of interest, the processing circuitry is configured to:
(Visvikis, [Col 12 Line 7]; "the position and dimensions of the region of interest are automatically defined on the torso of the patient by means of a deep learning algorithm")
select a first reference point and a second reference point from the point cloud data corresponding to respective ones of the upper-body landmarks of the user as an upper boundary and lower boundary;
(Visvikis, [Col 12 Line 31]; "this algorithm is able to accurately predict the location of various human 'key points' (joints and landmarks) such as elbows, knees, neck, shoulder, hips, chest etc.")
(Visvikis, [Col 12 Line 42];"The patient torso ROI may be then defined by the surface connecting the identified joints of shoulders and hipbones")
measure a first distance from the first reference point to the second reference point;
(Visvikis, [Col 12 Line 54]; "The width and length are defined as the distance between the minimum and maximum coordinates of the pixels within the ROI along x and y axis respectively")
Examiner Note: The first reference point and the second reference point are the upper boundary and lower boundary of the region of interest, so the first distance between them is the vertical extent of the region of interest. Visvikis defines this vertical extent as the "length" of the region of interest, computed as the distance between the minimum and maximum coordinates of the pixels along the y axis]). The length measured along the y axis therefore reads on the first distance measured from the first reference point to the second reference point.
select a third reference point and a fourth reference point corresponding to different ones of the upper-body landmarks of the user as a left boundary and a right boundary;
(Visvikis, [Col 12 Line 31] "this algorithm is able to accurately predict the location of various human "key points" (joints and landmarks) such as elbows, knees, neck, shoulder, hips, chest etc.")
measure a second distance from the third reference point to the fourth reference point;
(Visvikis, [Col 12 Line 54]; "The width and length are defined as the distance between the minimum and maximum coordinates of the pixels within the ROI along x and y axis respectively")
Examiner Note: The third reference point and the fourth reference point are the left boundary and right boundary of the region of interest, so the second distance between them is the horizontal extent of the region of interest. Visvikis defines this horizontal extent as the "width" of the region of interest, computed as the distance between the minimum and maximum coordinates of the pixels along the x axis (Visvikis, [Col 12 Line 54]). The width measured along the x axis therefore reads on the second distance measured from the third reference point to the fourth reference point, which is a measurement distinct from and orthogonal to the first distance.
determine height and width for the region of interest to create an area of interest based on the first distance and the second distance;
(Visvikis, [Col 12 Line 52]; "the size of the ROI is the area measured as the product of the width for the length of the ROI")
Examiner Note: Visvikis defines the region of interest from torso landmark joints, using the shoulders and hipbones as upper and lower references, and computes the width and length of the region of interest as the distance between the minimum and maximum coordinates of the pixels along the x and y axes. Visvikis’s shoulder and hipbone joints read on the first and second reference points (upper/lower boundary), the lateral extremes of the torso along the x axis read on the third and fourth reference points (left/right boundary), and the measured x and y distances used to size the region of interest read on the claimed first distance, second distance, and the resulting height and width.
select a depth (Visvikis, [Col 12 Line 48] "calculating the difference between the current depth surface and reference depth surface") for the area of interest to create the region of interest (Visvikis, [Col 12 Line 63]; "using surfaces (3D cloud of point) for the estimation of the respiratory parameters"), wherein the depth is selected such that when the region of interest is centered between the first and second reference points and the third and fourth reference points at least part of the points in the point cloud data corresponding to the upper-body landmarks of the user are contained within the region of interest (Visvikis, [Col 12 Line 49]; "for all pixels in the ROI").
(Visvikis, [Col 12 Line 60]; “it is possible to just multiply the surface dimension by the depth difference (between actual and reference surface)”)
Examiner Note: Visvikis does not recite selecting a "depth," but it builds the region of interest as a 3D torso surface and evaluates the depth surface for all pixels within it, so the torso point-cloud points are contained within the region across the depth dimension. Under applicant's specification at [0041], item (8), the depth is selected so the region of interest formed from the area of interest and the depth encompasses at least some upper-body point-cloud points, which reads on Visvikis's depth-evaluated 3D torso region.
Regarding claim 6, Visvikis teaches:
wherein the one or more image sensors comprise a pair of stereo cameras (Visvikis, [Col 8 Line 58]; "A variety of range imaging sensors or cameras ") ; and wherein the processing circuitry is further configured to determine disparity values between different portions of the image data obtained from the pair of stereo cameras (Visvikis, [Col 8 Line 59]; "range imaging technics such as stereo triangulation") , wherein the point cloud data is generated based on the disparity values ((Visvikis, [Col 12 Line 61] "The present method is based on the analysis of 3D surface images (i.e. 3D point cloud)").
Examiner Note: Visvikis discloses obtaining the range image by stereo triangulation, which is performed with a pair of stereo cameras, so the stereo-triangulation technique reads on the claimed pair of stereo cameras. Visvikis does not use the word "disparity," but stereo triangulation operates by measuring the positional difference between the two camera views of the same scene and triangulating that difference into depth, and applicant's specification at [0102] and [0103] defines disparity values as the values calculated between the first and second image streams and used to estimate depth. Reading the claim term in light of that definition, the positional difference Visvikis triangulates between its two stereo views equates to the claimed disparity values, and the depth surface Visvikis builds from that triangulation equates to the point cloud data generated based on the disparity values.
Regarding claim 7, Visvikis teaches:
performing, by a device comprising processing circuitry: (Visvikis, [Col 18 Line 18]; "the system is a data processing system such as dedicated circuitry or a general-purpose computer device (i.e. a processor), obtaining point cloud data of an environment that includes a user of the device for a period of time;
(Visvikis, [Col 5 Line 19]; “an input module configured to receive a set of acquisitions derived from a range imaging sensor, said set of acquisitions comprising at least one raw image”) (Visvikis, [Col 5 Line 61]; “The present method is based on the analysis of 3D surface images (i.e. 3D point cloud)”)
selecting one or more regions of interest in the point cloud data across the period of time, wherein the selected one or more regions of interest contain points in the point cloud data that correspond to upper-body landmarks of the user;
(Visvikis, [Col 12 Line 42]; “The patient torso ROI may be then defined by the surface connecting the identified joints of shoulders and hipbones”)
measuring respective motion of the points contained in the selected one or more regions of interest in the point cloud data over the period of time;
(Visvikis, [Col 3 Line 13]; “the thoracic region motion (chest wall motion) may be tracked by the analysis of the 3D spatial information”)
(Visvikis,[Col 11 Line 37]; “derive the respiratory motion dynamics on a point by point basis”)
based on the respective motion of the points, generating an estimated breath signal of the user;
(Visvikis, [Col 2 Line 50]; “estimating a respiratory signal as a function of time calculated as the spatial average, in a given region of interest defined on the torso of the subject, of the differences between the depth values of the surface image at a given time and the depth values of a reference surface image”)
and providing a final breath signal based, at least in part, on the estimated breath signal.
(Visvikis, [Col 2 Line 60]; “providing as output said respiratory parameters”)
While Visvikis discloses estimating a respiratory signal from the point cloud and providing that respiratory signal as output, Visvikis does not expressly recite the claimed two-tier arrangement in which a “final breath signal” is provided “based, at least in part, on the estimated breath signal.” However, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide Visvikis’s estimated respiratory signal as the final breath signal. The reason is that Visvikis already estimates the respiratory signal and outputs it for monitoring, so providing the estimated signal as the final output signal merely carries out Visvikis’s express output step. The applicant’s own specification at [0085] describes providing the estimated breath signal directly as the signal that is provided, and the claim term “based, at least in part, on” does not foreclose the final breath signal from being the estimated breath signal itself. Thus, it would have been obvious to provide a final breath signal based at least in part on Visvikis’s estimated breath signal, because doing so accomplishes Visvikis’s stated purpose of outputting the respiratory signal to a user or monitor, with a reasonable expectation of success.
Regarding claim 9, Visvikis teaches:
wherein selecting the one or more regions of interest, comprises: selecting a first reference point and a second reference point from the point cloud data corresponding to respective ones of the upper-body landmarks of the user as an upper boundary and lower boundary;
(Visvikis, [Col 12 Line 31]; "this algorithm is able to accurately predict the location of various human 'key points' (joints and landmarks) such as elbows, knees, neck, shoulder, hips, chest etc.")
(Visvikis, [Col 12 Line 42]; "The patient torso ROI may be then defined by the surface connecting the identified joints of shoulders and hipbones")
Examiner Note: Visvikis does not use the term "reference point," but applicant's specification at [0041] defines a reference/landmark point as a body landmark such as a shoulder or waist point. Visvikis's shoulder joint equates to the first reference point (upper boundary) and its hipbone joint = the second reference point (lower boundary).
measuring a first distance from the first reference point to the second reference point;
(Visvikis, [Col 12 Line 54] "The width and length are defined as the distance between the minimum and maximum coordinates of the pixels within the ROI along x and y axis respectively")
Examiner Note: Visvikis does not say "first distance," but applicant's specification at [0041] defines the first distance as the measured span between the first and second landmark points. Visvikis's distance between the minimum and maximum pixel coordinates along the vertical (y) axis, spanning shoulder to hipbone, equates to the claimed first distance.
selecting a third reference point and a fourth reference point corresponding to different ones of the upper-body landmarks of the user as a left boundary and a right boundary;
(Visvikis, [Col 12 Line 31]; "this algorithm is able to accurately predict the location of various human "key points" (joints and landmarks) such as elbows, knees, neck, shoulder, hips, chest etc.")
Examiner Note: The claim requires different landmarks for the lateral pair. Visvikis's laterally outermost torso joints (left and right), distinct from the shoulder/hipbone pair used above, = the third and fourth reference points serving as the left and right boundaries, consistent with [0041] using left/right body points as reference points.
measuring a second distance from the third reference point to the fourth reference point;
(Visvikis, [0115]; "The width and length are defined as the distance between the minimum and maximum coordinates of the pixels within the ROI along x and y axis respectively")
Examiner Note: Visvikis's distance between the minimum and maximum pixel coordinates along the horizontal (x) axis = the claimed second distance between the third and fourth reference points; [0041] defines this second distance as the span between the lateral landmark pair.
determining height and width for the region of interest to create an area of interest based on the first distance and the second distance;
(Visvikis, [Col 12 Line 53]; "the size of the ROI is the area measured as the product of the width for the length of the ROI")
Examiner Note: Visvikis's length (y axis) equates to the claimed height and its width (x axis) = the claimed width. The region computed as the product of width and length equates to the claimed area of interest formed from the first and second distances, matching applicant's [0041], which forms the area of interest from the two measured distances.
selecting a depth for the area of interest to create the region of interest, wherein the selection of the depth is such that when the region of interest is centered between the first and second reference points and the third and fourth reference points at least part of the points in the point cloud data corresponding to the upper-body landmarks of the user are contained within the region of interest.
(Visvikis, [Col 12 Line 60]; “it is possible to just multiply the surface dimension by the depth difference (between actual and reference surface)”)
Examiner Note: As with claim 3, Visvikis bounds the region of interest using a depth difference together with the width and length so that the torso surface points are contained within the region of interest, reading on selecting a depth such that the upper-body landmark points are contained within the region of interest.
Regarding claim 11, Visvikis teaches:
wherein providing the final breath signal comprises sending the final breath signal to a different device over a wireless connection between the device and the different device, wherein the different device includes a display that provides a visualization based on the final breath signal.
(Visvikis, [Col 18 Line 38]; “Output (respiratory features, alerts) are communicated to computing platform or user systems (computer, or tablet computer via wireless or wired network)”)
(Visvikis, [Col 18 Line 16]; “Said output module may be a display or a microphone connected wirelessly or not to the system”)
Examiner Note: Visvikis communicates the respiratory output to a separate computing platform or user system, such as a computer or tablet computer, over a wireless network, where the output is displayed. Sending the respiratory signal to the separate computer or tablet over the wireless network, which then displays it, reads on sending the final breath signal to a different device over a wireless connection, where the different device includes a display that provides a visualization based on the final breath signal.
Claims 2, 4, 5, 8, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Visvikis (US 12,220,220 B2) in view of the Genc patent (US 9,706,945 B2).
Regarding claim 2, Visvikis teaches:
wherein to measure the respective motion of the points contained in the selected one or more regions of interest in the point cloud data over the period of time, the processing circuitry is configured to: divide the one or more regions of interest into two or more smaller subregions; independently measure the respective motion of the points contained in the two or more subregions over the period of time; wherein to generate an estimated breath signal of the user, the processing circuitry is configured to: based on the respective motion of the points in each of the two or more subregions, generate respective subregion breath signals;
(Visvikis, [Col 14 Line 41]; “the thoracic region and on the abdominal region are defined as region of interest… so as to monitor these parameters separately in the thorax and in the abdomen”)
Examiner Note: Visvikis divides the torso region of interest into separate thoracic and abdominal subregions and measures the respiratory motion and parameters in each subregion separately. This reads on dividing the region of interest into two or more subregions, independently measuring the respective motion in each, and generating respective subregion breath signals.
Visvikis fails to teach
and combine the subregion breath signals according to a weighting method to generate the estimated breath signal.
Genc teaches:
and combine the subregion breath signals according to a weighting method to generate the estimated breath signal.
(the Genc patent, [Col 6 Line 7]; “larger weights may be applied to different signal quality indicators to determine the overall respiration signal quality… the combined effects (weighted or otherwise) of different quality indicators in one measure”)
Examiner Note: The Genc patent combines multiple component respiration signal measures into one overall respiration signal by applying weights. Applying the Genc patent’s weighted combination to Visvikis’s separate thoracic and abdominal subregion signals reads on combining the subregion breath signals according to a weighting method to generate the estimated breath signal.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine Visvikis and the Genc patent. Visvikis separately monitors the thoracic and abdominal subregions of the torso but does not expressly teach combining the resulting subregion signals using a weighting method. The Genc patent teaches combining component respiration signal measures by applying larger weights to higher-quality components to form one overall respiration signal (the Genc patent, [Col 6 Line 13]; "the final respiration signal quality indicator may include the combined effects (weighted or otherwise) of different quality indicators in one measure"). The motivation for the combination is to weight Visvikis's higher-quality subregion signal more heavily when forming a single combined breath signal, thereby reducing the influence of noisier subregions and improving the overall signal quality, with a reasonable expectation of success.
Regarding claim 4, Visvikis fails to teach:
wherein to provide the final breath signal, the processing circuitry is configured to: compare the estimated breath signal to a model breath signal to generate a similarity score between the estimated breath signal and the model breath signal; and compare the similarity score to a similarity threshold,
Genc teaches:
wherein to provide the final breath signal, the processing circuitry is configured to: compare the estimated breath signal to a model breath signal to generate a similarity score between the estimated breath signal and the model breath signal; and compare the similarity score to a similarity threshold,
(the Genc patent, [Col 5 Line 56]; “a correlation vector is generated from the stored PSD templates… if the mean… of the generated correlation vector is greater than a pre-selected threshold, then the PSD… is determined… to not be noisy”)
Examiner Note: The applicant’s specification at [0036] and [0052] compares the estimated breath signal to a model breath signal to produce a similarity score and compares that score to a similarity threshold. The Genc patent generates a correlation vector by comparing a current respiration power spectral density (the estimated signal) against stored PSD templates (the model signal) and compares a statistical descriptor (the mean) of that correlation vector to a pre-selected threshold. The Genc patent’s correlation value reads on the claimed similarity score, the stored PSD template reads on the model breath signal, and the pre-selected threshold reads on the similarity threshold.
where the processing circuitry is configured to: provide a default breath signal as the final breath signal when the similarity score does not satisfy the similarity threshold; provide the estimated breath signal as the final breath signal when the similarity score satisfies the similarity threshold.
(the Genc patent, [Col 2 Line 21]; “If the respective power spectral density for the current window is not noisy, a previous power spectral density template is updated… If the respective power spectral density for the current window is noisy, the previous power spectral density template is not updated”)
Examiner Note: The Genc patent uses the estimated PSD when it satisfies the threshold (not noisy) to update and provide the current template, and retains the previous template (the default) when the estimated PSD fails the threshold (noisy). Using the estimated signal when the threshold is satisfied and the previous, default signal when it is not reads on providing the estimated breath signal when the similarity score satisfies the threshold and a default breath signal when it does not.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine Visvikis and the Genc patent. Visvikis estimates a respiratory signal but does not expressly teach validating the estimate against a model signal and substituting a default signal when the estimate is unreliable. The Genc patent teaches comparing a current respiration estimate against stored templates using a correlation and threshold test and retaining the previous default template when the current estimate is too noisy (the Genc patent, [Col 5 Line 59]; "if the mean… of the generated correlation vector is greater than a pre-selected threshold, then the PSD… is determined… to not be noisy"). The motivation for the combination is to reject unreliable breath estimates and fall back to a previously validated default, thereby improving the accuracy and robustness of the provided respiration signal, with a reasonable expectation of success.
Regarding claim 5, the combination of Visvikis and the Genc patent teaches:
wherein the model breath signal and the default breath signal are respectively determined based on a plurality of previous estimated breath signals of the user stored in a memory.
(the Genc patent, [Col 5 Line 51]; “a set of the most recent good PSD templates (e.g., 3, 4, or 5 of the most recent good PSD templates) are stored in a memory (e.g., memory 24)”)
Examiner Note: The applicant’s specification at [0052] and [0099] computes the model and default breath signals from previous estimated breath signals stored in memory. The Genc patent stores a set of the most recent good PSD templates (the previous estimated respiration signals) in memory and uses them as the model and default. Determining the model and default signals from the plurality of stored prior templates reads on the claimed limitation.
Regarding claim 8, Visvikis teaches:
wherein measuring the respective motion of the points contained in the selected one or more regions of interest in the point cloud data over the period of time, comprises: dividing the one or more regions of interest into two or more smaller subregions; independently measuring the respective motion of the points contained in the two or more subregions over the period of time; wherein generating the estimated breath signal of the user, comprises: based on the respective motion of the points in each of the two or more subregions, generating respective subregion breath signals;
(Visvikis, [Col 14 Line 41]; “the thoracic region and on the abdominal region are defined as region of interest… so as to monitor these parameters separately in the thorax and in the abdomen”)
Visvikis fails to teach:
and combining the subregion breath signals according to a weighting method to generate the estimated breath signal.
Genc teaches:
and combining the subregion breath signals according to a weighting method to generate the estimated breath signal.
(the Genc patent, [Col 6 Line 8]; “larger weights may be applied to different signal quality indicators to determine the overall respiration signal quality… the combined effects (weighted or otherwise) of different quality indicators in one measure”)
Examiner Note: As with claim 2, applying the Genc patent’s weighted combination to Visvikis’s separate thoracic and abdominal subregion signals reads on combining the subregion breath signals according to a weighting method to generate the estimated breath signal.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine Visvikis and the Genc patent. Visvikis separately monitors the thoracic and abdominal subregions of the torso but does not expressly teach combining the resulting subregion signals using a weighting method. The Genc patent teaches combining component respiration signal measures by applying larger weights to higher-quality components to form one overall respiration signal (the Genc patent, [Col 6 Line 13]; "the final respiration signal quality indicator may include the combined effects (weighted or otherwise) of different quality indicators in one measure"). The motivation for the combination is to weight Visvikis's higher-quality subregion signal more heavily when forming a single combined breath signal, thereby reducing the influence of noisier subregions and improving the overall signal quality, with a reasonable expectation of success.
Regarding claim 10, Visvikis fails to teach:
wherein providing the final breath signal, comprises: comparing the estimated breath signal to a model breath signal to generate a similarity score between the estimated breath signal and the model breath signal; and comparing the similarity score to a similarity threshold; providing a default breath signal as the final breath signal responsive to determining that the similarity score does not satisfy the similarity threshold according to the comparing; providing the estimated breath signal as the final breath signal responsive to determining that the similarity score satisfies the similarity threshold according to the comparing.
Genc teaches:
wherein providing the final breath signal, comprises: comparing the estimated breath signal to a model breath signal to generate a similarity score between the estimated breath signal and the model breath signal; and comparing the similarity score to a similarity threshold; providing a default breath signal as the final breath signal responsive to determining that the similarity score does not satisfy the similarity threshold according to the comparing; providing the estimated breath signal as the final breath signal responsive to determining that the similarity score satisfies the similarity threshold according to the comparing.
(the Genc patent, [Col 5 Line 56]; “a correlation vector is generated from the stored PSD templates… if the mean… of the generated correlation vector is greater than a pre-selected threshold, then the PSD… is determined… to not be noisy”)
(the Genc patent, [Col 1 Line 41]; “If the respective power spectral density for the current window is not noisy, a previous power spectral density template is updated… If the respective power spectral density for the current window is noisy, the previous power spectral density template is not updated”)
Examiner Note: As with claim 4, the Genc patent’s correlation value, stored PSD template, and pre-selected threshold read on the claimed similarity score, model breath signal, and similarity threshold, and the Genc patent’s use of the estimated PSD when not noisy and the previous (default) template when noisy reads on providing the estimated breath signal when the threshold is satisfied and a default breath signal when it is not.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine Visvikis and the Genc patent. Visvikis estimates a respiratory signal but does not expressly teach validating the estimate against a model signal and substituting a default signal when the estimate is unreliable. The Genc patent teaches comparing a current respiration estimate against stored templates using a correlation and threshold test and retaining the previous default template when the current estimate is too noisy (the Genc patent, [Col 5 Line 59]; "if the mean… of the generated correlation vector is greater than a pre-selected threshold, then the PSD… is determined… to not be noisy"). The motivation for the combination is to reject unreliable breath estimates and fall back to a previously validated default, thereby improving the accuracy and robustness of the provided respiration signal, with a reasonable expectation of success.
Claims 12, 13, 14, 15, 17, 18, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Visvikis (US 12,220,220 B2) in view of Tzvieli (US 10,136,856 B2).
Regarding claim 12,
Visvikis teaches:
an upper body of the user in an environment
Visvikis fails to teach:
A device, comprising: a frame, configured to be worn on a head of a user;
(Visvikis, [Col 4 Line 66]; “an acquisition module configured to control a range imaging sensor for the acquisition of at least one raw image comprising at least one portion of the torso of the subject”)
Tzvieli teaches:
A device, comprising: a frame, configured to be worn on a head of a user;
(Tzvieli, [Col 5 Line 9]; “at least two thermal cameras that are physically coupled to a frame worn on a user's head”)
two or more image sensors integrated in or coupled to the frame and configured to collect image data of at least an upper body of the user in an environment when the frame is worn on the head of the user over a period of time;
(Tzvieli, [Col 5 Line 9]; “at least two thermal cameras that are physically coupled to a frame worn on a user's head”)
Examiner Note: Tzvieli teaches a head-worn frame that physically holds at least two cameras directed at the wearer. Visvikis teaches collecting raw image data of the user’s torso (upper body) using an image/range sensor. In the combination, the two or more head-worn cameras of Tzvieli collect the upper-body image data taught by Visvikis, reading on “two or more image sensors integrated in or coupled to the frame and configured to collect image data of at least an upper body of the user.”
Visvikis teaches:
a controller for the device, comprising processing circuitry configured to: obtain the image data; generate, from the image data, point cloud data containing points representative of the user in the environment as captured in the image data over the period of time;
(Visvikis, [Col 5 Line 19]; “an input module configured to receive a set of acquisitions derived from a range imaging sensor”)
(Visvikis, [Col 2 Line 61]; “The present method is based on the analysis of 3D surface images (i.e. 3D point cloud)”)
identify one or more regions of interest in the point cloud data encompassing points in the point cloud data that correspond to upper-body landmarks of the user;
(Visvikis, [Col 12 Line 42]; “The patient torso ROI may be then defined by the surface connecting the identified joints of shoulders and hipbones”)
measure respective motion of the points encompassed in the one or more regions of interest in the point cloud data over the period of time;
(Visvikis, [Col 3 Line 13]; “the thoracic region motion (chest wall motion) may be tracked by the analysis of the 3D spatial information”)
(Visvikis, Col 5 Line 37]; “derive the respiratory motion dynamics on a point by point basis”)
generate an estimated breath signal of the user;
(Visvikis, [Col 3 Line 33]; “estimating a respiratory signal as a function of time calculated as the spatial average… of the differences between the depth values of the surface image at a given time and the depth values of a reference surface image”)
and provide a final breath signal, based at least in part, on the estimated breath signal.
(Visvikis, [Col 2 Line 60]; “providing as output said respiratory parameters”)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine Visvikis and Tzvieli. Visvikis teaches the complete point-cloud breath-estimation chain but mounts its range imaging sensor near the patient rather than on the user, stating that “the range imaging sensor may be advantageously placed on top of the patient bed”. Tzvieli teaches a frame configured to be worn on a user’s head that physically holds first and second cameras directed at the wearer. The motivation for the combination is to mount Visvikis’s image sensors on Tzvieli’s head-worn frame so that the sensors follow the user and remain directed at the user’s body without a fixed bed or ceiling installation. Tzvieli expressly teaches that head-coupling the cameras causes them to remain pointed at the user during head movements and removes the need for image registration and object tracking (Tzvieli, [Col 8 Line 52]; “a thermal camera moves with the user's head… thus, there may be no need for image registration and/or object tracking”). Because both references collect non-contact respiration data from cameras directed at the same person, the combination yields a portable, self-aligning wearable that performs Visvikis’s breath estimation, with a reasonable expectation of success since each reference already performs camera-based respiration sensing.
Regarding claim 13, The combination of Visvikis and Tzvieli teaches:
wherein the processing circuitry of the controller is further configured to convert the point cloud data from image coordinate system to a world coordinate system; and wherein the identification of the one or more regions of interest in point cloud data from the transformed point cloud across the period of time, is based on the converted point cloud data in the world coordinate system.
(Visvikis, [Col 10 Line 58]; “a calibration step consisting in the application of a rotation matrix to the raw image so as to align the patient torso in the raw image with the xy plane of the time-of-flight camera”)
(Visvikis, [Col 12 Line 58]; “the relation between camera's natural units (pixel positions in the image) and real-world units (in mm)”)
Examiner Note: Visvikis applies a rotation matrix to transform the raw image from the camera (image) coordinate frame into an aligned frame referenced to real-world units (mm), and defines the region of interest in that transformed, calibrated frame. Converting from the camera’s pixel coordinates to real-world units and identifying the region of interest in that frame reads on converting the point cloud from an image coordinate system to a world coordinate system and identifying the region of interest based on the converted point cloud data. This rejection is made consistent with the interpretation set forth in the 35 U.S.C. 112(b) rejection of claim 13 above.
Regarding claim 14, The combination of Visvikis and Tzvieli teaches:
wherein to identify the one or more regions of interest, the processing circuitry of the controller is configured to: select a first reference point and a second reference point… as an upper boundary and lower boundary; measure a first distance…; select a third reference point and a fourth reference point… as a left boundary and a right boundary; measure a second distance…; determine height and width for the one or more regions of interest to create an area of interest based on the first distance and the second distance;
(Visvikis, [Col 12 Line 42]; “The patient torso ROI may be then defined by the surface connecting the identified joints of shoulders and hipbones”)
(Visvikis, [Col 12 Line 54]; “The width and length are defined as the distance between the minimum and maximum coordinates of the pixels within the ROI along x and y axis respectively”)
Examiner Note: For the reasons set forth with respect to claims 3 and 9, Visvikis defines the region of interest from the shoulder and hipbone landmark joints and sizes it from the x- and y-axis distances between the minimum and maximum pixel coordinates, reading on the claimed reference points, distances, and resulting height and width.
select a depth for the area of interest to create the one or more regions of interest, wherein the depth is selected such that when the one or more regions of interest are centered between the first and second reference points and the third and fourth reference points at least part of the points in the point cloud data corresponding to the upper-body landmarks of the user are contained within the one or more regions of interest.
(Visvikis, [Col 12 Line 60]; “it is possible to just multiply the surface dimension by the depth difference (between actual and reference surface)”)
Examiner Note: Visvikis bounds the region of interest using a depth difference together with the width and length so that the torso surface points are contained within the region of interest, reading on the claimed depth selection.
Regarding claim 15, The combination of Visvikis and Tzvieli teaches:
wherein to identify the one or more regions of interest in the point cloud data across the period of time, the processing circuitry of the controller is configured to apply a trained machine learning model to identify the upper-body landmarks of the user.
(Visvikis, [Col 12 Line 28]; “the deep learning algorithm for real time tracking of the skeleton model is based on Deep Neural Net model to perform Human Pose Estimation”)
(Visvikis, [Col 12 Line 8]; “the position and dimensions of the region of interest are automatically defined on the torso of the patient by means of a deep learning algorithm”)
Examiner Note: Visvikis applies a trained deep neural network performing human pose estimation to identify the body key points (joints and landmarks) used to define the region of interest. Visvikis’s trained deep neural network reads on the claimed “trained machine learning model to identify the upper-body landmarks of the user.”
Regarding claim 17, The combination of Visvikis and Tzvieli teaches:
wherein the device further comprises a display visible to the user and wherein the final breath signal is provided to generate a visualization on the display corresponding to a breath cycle state of the user.
(Visvikis, [Col 18 Line 16]; “Said output module may be a display or a microphone connected wirelessly or not to the system”)
(Visvikis, [Col 13 Line 14]; “a curve of the lung volume as a function of time V (k), which presents a peak for each inhalation and a valley for each exhalation”)
Examiner Note: Visvikis provides a display output and generates a lung-volume-versus-time curve having a peak for each inhalation and a valley for each exhalation. The displayed curve, whose peaks and valleys correspond to the inhalation and exhalation states of the breath cycle, reads on a visualization generated on the display corresponding to a breath cycle state of the user.
Regarding claim 18, The combination of Visvikis and Tzvieli teaches:
wherein the one or more image sensors comprise a pair of stereo cameras; and wherein the processing circuitry of the controller is further configured to determine disparity values between different portions of the image data obtained from the pair of stereo cameras, wherein the point cloud data is generated based on the disparity values.
(Visvikis, [Col 8 Line 59]; “range imaging technics such as stereo triangulation, sheet of light triangulation, structured light, interferometry, coded aperture”)
Examiner Note: As with claim 6, Visvikis discloses stereo triangulation, which uses a pair of stereo cameras and determines disparity between the two views to recover depth, from which the point cloud is generated. This reads on a pair of stereo cameras whose disparity values are used to generate the point cloud data.
Regarding claim 19, The combination of Visvikis and Tzvieli teaches:
wherein the one or more image sensors comprise one or more depth sensors configured to collect depth data corresponding to the image data; and wherein the point cloud data is generated based on the depth data.
(Visvikis, [Col 8 Line 64]; “the range imaging sensor is a time-of-flight (ToF) camera”)
(Visvikis, [Col 9 Line 16]; “ToF cameras may combine a single or multiple laser beams… to produce 1-D or 2-D arrays of depth values”)
Examiner Note: Visvikis’s range imaging sensor is a time-of-flight depth camera that produces arrays of depth values, from which the 3D point cloud is generated. Visvikis’s ToF depth sensor and its depth-value output read on the claimed “one or more depth sensors configured to collect depth data” used to generate the point cloud data.
Regarding claim 20, The combination of Visvikis and Tzvieli teaches:
wherein the image data is obtained at the device from the one or more image sensors via a wireless communication.
(Visvikis, [Col 16 Line 61]; “a communication module that, through physical connection or wireless connection between the range imaging sensor Sri and the system… controls the image acquisition and the reception of the acquired raw images”)
Examiner Note: Visvikis receives the acquired raw images from the range imaging sensor over a wireless connection via its communication module. Receiving the raw image data from the sensor over the wireless connection reads on obtaining the image data from the image sensors via wireless communication.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Visvikis (US 12,220,220 B2) and Tzvieli (US 10,136,856 B2) and further in view of the Genc patent (US 9,706,945 B2).
Regarding claim 16,
Visvikis fails to teach:
wherein to provide the final breath signal, the processing circuitry is configured to: compare the estimated breath signal to a model breath signal to generate a similarity score between the estimated breath signal and the model breath signal stored in the device; and compare the similarity score to a similarity threshold, where the processing circuitry of the controller is configured to: provide a default breath signal as the final breath signal when the similarity score does not satisfy the similarity threshold; provide the estimated breath signal as the final breath signal when the similarity score satisfies the similarity threshold.
Genc patent teaches:
wherein to provide the final breath signal, the processing circuitry is configured to: compare the estimated breath signal to a model breath signal to generate a similarity score between the estimated breath signal and the model breath signal stored in the device; and compare the similarity score to a similarity threshold, where the processing circuitry of the controller is configured to: provide a default breath signal as the final breath signal when the similarity score does not satisfy the similarity threshold; provide the estimated breath signal as the final breath signal when the similarity score satisfies the similarity threshold.
(the Genc patent, [Col 5 Line 56]; “a correlation vector is generated from the stored PSD templates… if the mean… of the generated correlation vector is greater than a pre-selected threshold, then the PSD… is determined… to not be noisy”)
(the Genc patent, [Col 1 Line 41]; “If the respective power spectral density for the current window is not noisy, a previous power spectral density template is updated… If the respective power spectral density for the current window is noisy, the previous power spectral density template is not updated”)
Examiner Note: The Genc patent’s correlation value, stored PSD template held in memory, and pre-selected threshold read on the claimed similarity score, model breath signal stored in the device, and similarity threshold, and the Genc patent’s use of the estimated PSD when not noisy and the previous, default template when noisy reads on providing the estimated breath signal when the threshold is satisfied and a default breath signal when it is not.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to further combine the Genc patent with the combination of Visvikis and Tzvieli, for the same reasons set forth with respect to claim 4: to reject unreliable breath estimates and fall back to a previously validated default stored in the device, thereby improving the accuracy and robustness of the provided respiration signal, with a reasonable expectation of success.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHIVANGI SARKAR whose telephone number is (571)272-7262. The examiner can normally be reached M-F: 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHIVANGI SARKAR/Examiner, Art Unit 2666
/Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights