DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 4, 8, 11, 14-15, 18, and 20 are objected to because of the following informalities:
In claims 1, 8, 15, “the face” in lines 3, 8, and 4-5, respectively, should read “a face” and “the presence” in lines 4, 8, and 5, respectively, should read “a presence”.
In claims 4, 11, and 18, line 2, the “one or more image frames” should read the “plurality of image frames” as in claims 1, 8, and 15.
In claim 7, “wherein,” should read “further comprising:” or the “increasing a color depth of video capture” should be changed to “a color depth of video capture is increased”.
In claim 8, line 6, “camera” should read “the camera”.
In claim 14, the “wherein, prior to capturing the video, increasing” should read “wherein the processing device is further configured to:
prior to capturing the video, increase” or the “increasing a color depth of video capture” should be changed to “a color depth of video capture is increased”.
In claim 20, line 2, “comprising” should read “comprises” and the “increasing a color depth of video capture” in line 5 should be changed to “a color depth of video capture is increased”.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder (i.e., a module) that is coupled with functional language (i.e.,, “to receive” and “to determine”) without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are:
• “a processing device…configured to …” in claim 8, line 5. Interpreted as a hardware (PG Pub [0064]).
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-6, 8, 12-13, 15, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Guazzi et al. (Non-contact measurement of oxygen saturation with an RGB camera. Biomedical Optics Express, August 11, 2015), hereinafter Guazzi, in view of Panasyuk et al (US 20070024946), hereinafter Panasyuk.
Regarding claim 1, Guazzi teaches a method of estimating peripheral oxygen saturation (SpO2) in a human subject (“IX: Ratio of ratios,” Fig. 1) (“Fig. 2. Bland-Altman plot of all results for oxygen saturation.”, p. 11. Table 2: “Oxygen saturation results for the non-contact (Camera) estimates;” p 11; Fig. 3: “Oxygen saturation results for Subject 3. Top-left: results with the camera using the method described. Bottom-left: reference oxygen saturation from the pulse oximeter.”; p. 12), the method comprising:
capturing, by a camera (“a 3-CCD (JAI AT-200CL) RGB camera” 2.4. Dataset, p. 9), video comprising a plurality of image frames ("2.4. Dataset The algorithm was tested using the video recordings of five volunteers" p. 9; “updated every frame” p. 13, 1st para.) of the face and/or a hand palm of the human subject (“Fig. 5. A comparison of different ROI selection methods and of the effect of the inclusion
criteria on the ROI selection in Sophia, carried out on all subjects considered. From left to right: the method introduced by [14], using a fixed region manually set onto the lower part of the face; the method introduced by [15], using a fixed region placed on the cheek of the subject;” p. 13) in the presence of an ambient light source (“The present work introduces a novel method for Skin-Oxygen Photoplethysmographic Image Analysis (Sophia), which allows oxygen saturation changes to be tracked accurately over time. Sophia uses broad-band lighting and an RGB camera”; p. 3);
identifying, within the plurality of image frames, a plurality of skin pixels corresponding pixels that spatially correspond to skin of the face and/or the hand palm within each of the plurality of image frames (“Once a search area has been defined in which all the skin to be imaged has been included, the area is divided into N contiguous nxn pixel regions of interest i (n=40) and the algorithm is intiated (Step II).”; p. 6; “whereas the ROI-extraction method taken from [17] was updated every frame as it was understood to be dynamic. Once the raw red and blue signals were acquired by taking the average of the pixel intensities for these two colour channels in the respective ROIs,”; p. 13, 1st para.);
computing, for each of the plurality of skin pixels (“the position-dependence”; p. 4), a time-dependent signal (“timeseries x(t), consisting of successive samples from one of the R;G;B
colour channels”; p. 6, Eq. (10); “a timeseries”; p. 7, 3rd complete para.; “the normalised channel amplitude timeseries” 3.1. Post-processing; p. 10) corresponding to blood volume change (“The pulsatile, cardiac-synchronous signals which are caused by colour changes due to the in and outflow of blood will have a relative phase shift between them that is uniquely determined by the maximum pulse transit time within the area of skin observed.”; p. 7; 1st complete para.) for each color channel of the skin pixel (“In Step VI, the normalised AC amplitude is determined for each colour channel as per Equation 7 in all M regions of interest”; p. 8; Fig. 1);
and
computing an overall SpO2 estimate (“The present work introduces a novel method for Skin-Oxygen Photoplethysmographic Image Analysis (Sophia), which allows oxygen saturation changes to be tracked accurately over time." p. 3, the last complete para.; Figs. 2-4 and 6-7; Table 3).
Guazzi does not explicitly teach generating, based on the time-dependent signal, a plurality of SpO2 estimates for each of the plurality of skin pixels at each frame of the captured video and computing an overall SpO2 estimate from the plurality of SpO2 estimates.
However, in the medical imaging devices and methods field of endeavor, Panasyuk discloses hyperspectral/multispectral imaging in determination, assessment and monitoring of systemic physiology and shock, which is analogous art. Panasyuk teaches generating, based on the time-dependent signal (“near-real time information" [0080]. “By revealing changes in tissue … oxygen delivery, oxygen extraction, S.sub.HSIO.sub.2 … that correlate with adverse outcomes, the MHSI approach is additionally able to provide information about patient survivability" [0096]), a plurality of SpO2 estimates (“hyperspectral tissue oxygen saturation …S.sub.HSIO.sub.2” [0051]; “an oxygen saturation value” Claim 62) for each of the plurality of skin pixels at each frame of the captured video (“changes in tissue … S.sub.HSIO.sub.2” [0096]. “Oxygen saturation images in which the brightness of each pixel is proportionate to the intensity of the S.sub.HSIO.sub.2 for that pixel” [0196]; “62. …calculating an oxygen saturation value for each pixel in the image.”); and
computing an overall SpO2 estimate from the plurality of SpO2 estimates (“Oxyhemoblobin (OxyHb), deoxyhemoglobin (DeoxyHb) … can be presented …as scalars reflecting a mean value across a region of interest (ROI), or the oxyhemoblobin and deoxyhemoglobin coefficients can be used to calculate hyperspectral tissue oxygen saturation (S.sub.HSIO.sub.2=OxyHb/(OxyHb+DeoxyHb)) ... This information can be presented as black and white or false color images…This presentation can be used to represent oxyhemoglobin and deoxyhemoglobin values for any pixel in the ROI, to present the average oxyhemoglobin and deoxyhemoglobin values over the entire ROI.” [0051]).
Therefore, based on Panasyuk’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi to have the step of generating, based on the time-dependent signal, a plurality of SpO2 estimates for each of the plurality of skin pixels at each frame of the captured video, and computing an overall SpO2 estimate from the plurality of SpO2 estimates as taught by Panasyuk, in order to facilitate the image analysis for contactless SpO2 tracking in tissue over time.
Regarding claim 5, Guazzi modified by Panasyuk teaches the method of claim 1, wherein Guazzi teaches computing, for each of the plurality of skin pixels, the time-dependent signal corresponding to blood volume change for each color channel of the skin pixel comprises computing the time-dependent signal for each of a red-channel, a green-channel, and a blue-channel of each skin pixel (“The tracking of oxygen saturation with RGB cameras (visible light cameras with three channels: red, green and blue respectively)” p. 3, 2nd complete para.; “For the purpose of initialization, x(t) is taken as the green signal from the RGB camera over
a period of 12-seconds… the per-frame spatial average of the green channel in each of the N regions of interest”; pp. 6-7. “The breathing rate estimate is found by taking the average of the power spectral density (PSD) of the detrended blue signal across all N regions of interest and then searching for a peak in the expected physiological range (between 0.1 and 0.7 Hz, corresponding to 6 to 42 breaths per minute). While the green channel usually has a high heart rate SNR as a result of its poor absorption by melanin but high absorption by blood, the blue channel is used to determine the breathing rate for the opposite reason.” p. 7, 1st para.; “The amplitude of the waveform is taken as the normalised amplitude for that channel and the ratio of ratios is calculated as the ratio of the blue amplitude to the red amplitude (Step IX).”; 2.2. Implementation; p. 8).
Regarding claim 6, Guazzi modified by Panasyuk teaches the method of claim 1, wherein Guazzi teaches applying a statistical inference within each of the plurality of image frames of the captured video (“If any region of interest may be used, then an average of the regions of interest may be taken instead, as previous research has shown that averaging over a large number of physiologically relevant pixels will improve the definition of the resulting PPG signal”; p. 5).
Guazzi does not explicitly teach that computing the overall SpO2 estimate from the plurality of SpO2 estimates comprises applying the statistical inference to the plurality of SpO2 estimates computed over all of the identified skin pixels.
However, in the medical imaging devices and methods field of endeavor, Panasyuk discloses hyperspectral/multispectral imaging in determination, assessment and monitoring of systemic physiology and shock, which is analogous art. Panasyuk teaches that the statistical inference (“a mean value” [0051]) is being applied to the plurality of SpO2 estimates computed over all of the identified skin pixels (“Oxyhemoblobin (OxyHb), deoxyhemoglobin (DeoxyHb) … can be presented …as scalars reflecting a mean value across a region of interest (ROI), or the oxyhemoblobin and deoxyhemoglobin coefficients can be used to calculate hyperspectral tissue oxygen saturation (S.sub.HSIO.sub.2=OxyHb/(OxyHb+DeoxyHb)) ... This information can be presented as black and white or false color images…This presentation can be used to represent oxyhemoglobin and deoxyhemoglobin values for any pixel in the ROI, to present the average oxyhemoglobin and deoxyhemoglobin values over the entire ROI.” [0051] “Oxygen saturation images in which the brightness of each pixel is proportionate to the intensity of the S.sub.HSIO.sub.2 for that pixel” [0196]; “62. …calculating an oxygen saturation value for each pixel in the image.”).
Therefore, based on Panasyuk’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi to have the step of computing the overall SpO2 estimate from the plurality of SpO2 estimates that comprises applying the statistical inference to the plurality of SpO2 estimates computed over all of the identified skin pixels, as taught by Panasyuk, in order to facilitate the image analysis for contactless SpO2 tracking in tissue over time.
Regarding claim 8, Guazzi teaches a device (a processing device used for the “Skin-Oxygen Photoplethysmographic Image Analysis (Sophia)” p. 3, the last complete para. and the “3-CCD (JAI AT-200CL) RGB camera”; 2.4. Dataset; p. 9) to estimate peripheral oxygen saturation (SpO2) in a human subject (Abstract; “the video recordings of five volunteers”; 2.4. Dataset, p. 9), the device comprising:
a housing (a housing of the “3-CCD (JAI AT-200CL) RGB camera”; 2.4. Dataset; p. 9);
a camera (“2.4. Dataset The algorithm was tested using the video recordings of five volunteers who had undergone a controlled study in a hypoxic environment. The recordings were made with a 3-CCD (JAI AT-200CL) RGB camera placed about 1.5 m from the subject, shooting at 16 frames per second.”; p. 9);
a processing device (a processing device of the “3-CCD (JAI AT-200CL) RGB camera”; 2.4. Dataset; p. 9, and a processing device used for the “Skin-Oxygen Photoplethysmographic Image Analysis (Sophia)” p. 3, the last complete para.), the processing device being communicatively coupled to camera (the processing device of the camera has to be communicatively coupled to the camera’s optic components to make video recordings), wherein the processing device is configured to:
cause the camera to capture video comprising a plurality of image frames ("2.4. Dataset The algorithm was tested using the video recordings of five volunteers" p. 9; “updated every frame” p. 13, 1st para.) of the face and/or a hand palm of the human subject (“Fig. 5. A comparison of different ROI selection methods and of the effect of the inclusion criteria on the ROI selection in Sophia, carried out on all subjects considered. From left to right: the method introduced by [14], using a fixed region manually set onto the lower part of the face; the method introduced by [15], using a fixed region placed on the cheek of the subject;” p. 13) in the presence of an ambient light source (“The present work introduces a novel method for Skin-Oxygen Photoplethysmographic Image Analysis (Sophia), which allows oxygen saturation changes to be tracked accurately over time. Sophia uses broad-band lighting and an RGB camera”; p. 3, the last complete para.);
identify, within the plurality of image frames, a plurality of skin pixels corresponding pixels that spatially correspond to skin of the face and/or the hand palm within each of the plurality of image frames (“Once a search area has been defined in which all the skin to be imaged has been included, the area is divided into N contiguous nxn pixel regions of interest i (n=40) and the algorithm is intiated (Step II).”; p. 6; “whereas the ROI-extraction method taken from [17] was updated every frame as it was understood to be dynamic. Once the raw red and blue signals were acquired by taking the average of the pixel intensities for these two colour channels in the respective ROIs,”; p. 13, 1st para.);
compute, for each of the plurality of skin pixels (“the position-dependence”; p. 4), a time-dependent signal (“timeseries x(t), consisting of successive samples from one of the R;G;B
colour channels”; p. 6, Eq. (10); “a timeseries”; p. 7, 3rd complete para.; “the normalised channel amplitude timeseries” 3.1. Post-processing; p. 10), corresponding to blood volume change (“The pulsatile, cardiac-synchronous signals which are caused by colour changes due to the in and outflow of blood will have a relative phase shift between them that is uniquely determined by the maximum pulse transit time within the area of skin observed.”; p. 7; 1st complete para.) for each color channel of the skin pixel (“In Step VI, the normalised AC amplitude is determined for each colour channel as per Equation 7 in all M regions of interest”; p. 8; Fig. 1); and
compute an overall SpO2 estimate (“The present work introduces a novel method for Skin-Oxygen Photoplethysmographic Image Analysis (Sophia), which allows oxygen saturation changes to be tracked accurately over time." p. 3, the last complete para.; Figs. 2-4 and 6-7; Table 3); and
a display device (required to display the results) configured to display the overall SpO2 (Figs. 2-4) (“Fig. 2. Bland-Altman plot of all results for oxygen saturation”; p. 11; “Fig. 3. Oxygen saturation results for Subject 3. Top-left: results with the camera using the method described.”; p. 12; “Fig. 4. Oxygen saturation results for Subject 4. Top-left: results with the camera using the method described.”; p. 12).
Guazzi does not explicitly teach the device comprising:
a housing;
a camera disposed at least partially in the housing;
a processing device disposed within the housing configured to:
generate, based on the time-dependent signal, a plurality of SpO2 estimates for each of the plurality of skin pixels at each frame of the captured video and compute an overall SpO2 estimate from the plurality of SpO2 estimates.
However, in the medical imaging devices and methods field of endeavor, Panasyuk discloses hyperspectral/multispectral imaging in determination, assessment and monitoring of systemic physiology and shock, which is analogous art. Panasyuk teaches the device (10) comprising:
a housing (a housing of the “Portable apparatus 10”);
a camera (36) disposed at least partially in the housing;
a processing device (38) disposed within the housing (seen in Fig. 1) (“Portable apparatus 10 comprises an optical acquisition system 36 and a diagnostic processor 38.” [0109]; Fig. 1) configured to:
generate, based on the time-dependent signal (“near-real time information" [0080]. “By revealing changes in tissue … oxygen delivery, oxygen extraction, S.sub.HSIO.sub.2 … that correlate with adverse outcomes, the MHSI approach is additionally able to provide information about patient survivability" [0096]), a plurality of SpO2 estimates (“hyperspectral tissue oxygen saturation …S.sub.HSIO.sub.2” [0051]; “an oxygen saturation value” Claim 62) for each of the plurality of skin pixels at each frame of the captured video (“changes in tissue … S.sub.HSIO.sub.2” [0096]. “Oxygen saturation images in which the brightness of each pixel is proportionate to the intensity of the S.sub.HSIO.sub.2 for that pixel” [0196]; “62. …calculating an oxygen saturation value for each pixel in the image.”); and
compute an overall SpO2 estimate from the plurality of SpO2 estimates (“Oxyhemoblobin (OxyHb), deoxyhemoglobin (DeoxyHb) … can be presented …as scalars reflecting a mean value across a region of interest (ROI), or the oxyhemoblobin and deoxyhemoglobin coefficients can be used to calculate hyperspectral tissue oxygen saturation (S.sub.HSIO.sub.2=OxyHb/(OxyHb+DeoxyHb)) ... This information can be presented as black and white or false color images…This presentation can be used to represent oxyhemoglobin and deoxyhemoglobin values for any pixel in the ROI, to present the average oxyhemoglobin and deoxyhemoglobin values over the entire ROI.” [0051]).
Therefore, based on Panasyuk’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi to employ the device comprising: a housing; a camera disposed at least partially in the housing; a processing device disposed within the housing configured to:
generate, based on the time-dependent signal, a plurality of SpO2 estimates for each of the plurality of skin pixels at each frame of the captured video and compute an overall SpO2 estimate from the plurality of SpO2 estimates, as taught by Panasyuk, in order to facilitate the image analysis for contactless SpO2 tracking in tissue over time.
Regarding claim 12, Guazzi modified by Panasyuk teaches the device of claim 8, wherein Guazzi teaches computing, for each of the plurality of skin pixels, the time-dependent signal corresponding to blood volume change for each color channel of the skin pixel comprises computing the time-dependent signal for each of a red-channel, a green-channel, and a blue-channel of each skin pixel (“The tracking of oxygen saturation with RGB cameras (visible light cameras with three channels: red, green and blue respectively)” p. 3, 2nd complete para.; “For the purpose of initialization, x(t) is taken as the green signal from the RGB camera over
a period of 12-seconds… the per-frame spatial average of the green channel in each of the N regions of interest”; pp. 6-7. “The breathing rate estimate is found by taking the average of the power spectral density (PSD) of the detrended blue signal across all N regions of interest and then searching for a peak in the expected physiological range (between 0.1 and 0.7 Hz, corresponding to 6 to 42 breaths per minute). While the green channel usually has a high heart rate SNR as a result of its poor absorption by melanin but high absorption by blood, the blue channel is used to determine the breathing rate for the opposite reason.” p. 7, 1st para.; “The amplitude of the waveform is taken as the normalised amplitude for that channel and the ratio of ratios is calculated as the ratio of the blue amplitude to the red amplitude (Step IX).”; 2.2. Implementation; p. 8).
Regarding claim 13, Guazzi modified by Panasyuk teaches the device of claim 8, wherein Guazzi teaches applying a statistical inference within each of the plurality of image frames of the captured video (“If any region of interest may be used, then an average of the regions of interest may be taken instead, as previous research has shown that averaging over a large number of physiologically relevant pixels will improve the definition of the resulting PPG signal”; p. 5).
Guazzi does not explicitly teach that computing the overall SpO2 estimate from the plurality of SpO2 estimates comprises applying the statistical inference to the plurality of SpO2 estimates computed over all of the identified skin pixels.
However, in the medical imaging devices and methods field of endeavor, Panasyuk discloses hyperspectral/multispectral imaging in determination, assessment and monitoring of systemic physiology and shock, which is analogous art. Panasyuk teaches that the statistical inference (“a mean value” [0051]) is being applied to the plurality of SpO2 estimates computed over all of the identified skin pixels (“Oxyhemoblobin (OxyHb), deoxyhemoglobin (DeoxyHb) … can be presented …as scalars reflecting a mean value across a region of interest (ROI), or the oxyhemoblobin and deoxyhemoglobin coefficients can be used to calculate hyperspectral tissue oxygen saturation (S.sub.HSIO.sub.2=OxyHb/(OxyHb+DeoxyHb)) ... This information can be presented as black and white or false color images…This presentation can be used to represent oxyhemoglobin and deoxyhemoglobin values for any pixel in the ROI, to present the average oxyhemoglobin and deoxyhemoglobin values over the entire ROI.” [0051] “Oxygen saturation images in which the brightness of each pixel is proportionate to the intensity of the S.sub.HSIO.sub.2 for that pixel” [0196]; “62. …calculating an oxygen saturation value for each pixel in the image.”).
Therefore, based on Panasyuk’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi to have the step of computing the overall SpO2 estimate from the plurality of SpO2 estimates that comprises applying the statistical inference to the plurality of SpO2 estimates computed over all of the identified skin pixels, as taught by Panasyuk, in order to facilitate the image analysis for contactless SpO2 tracking in tissue over time.
Regarding claim 15, Guazzi teaches a non-transitory computer-readable medium having instructions stored thereon (required for the “algorithm (Sophia) and the “post-processing techniques”; Table 1, p. 10), which, when executed by a processing device (“The present work introduces a novel method for Skin-Oxygen Photoplethysmographic Image Analysis (Sophia), which allows oxygen saturation changes to be tracked accurately over time. Sophia uses broad-band lighting and an RGB camera”; p. 3) “3.1. Post-processing…Table 1. Skin colour type (Fitzpatrick scale I through to VI), and proportion of data used per subject. Both the algorithm (Sophia) and post-processing techniques applied to the results reject data that are detected as noise or outliers.”; p. 10. A processing device required for the “algorithm (Sophia) and post-processing techniques”; p. 10) that is operatively coupled to a camera (“a 3-CCD (JAI AT-200CL) RGB camera” para 2.4.; p. 9), causes the processing device to:
cause the camera to capture video comprising a plurality of image frames ("2.4. Dataset The algorithm was tested using the video recordings of five volunteers" p. 9; “updated every frame” p. 13, 1st para.) of the face and/or a hand palm of a human subject (“Fig. 5. A comparison of different ROI selection methods and of the effect of the inclusion criteria on the ROI selection in Sophia, carried out on all subjects considered. From left to right: the method introduced by [14], using a fixed region manually set onto the lower part of the face”; p. 13) in the presence of an ambient light source (“The present work introduces a novel method for Skin-Oxygen Photoplethysmographic Image Analysis (Sophia), which allows oxygen saturation changes to be tracked accurately over time. Sophia uses broad-band lighting and an RGB camera”; p. 3);
identify, within the plurality of image frames, a plurality of skin pixels corresponding pixels that spatially correspond to skin of the face and/or the hand palm within each of the plurality of image frames (“Once a search area has been defined in which all the skin to be imaged has been included, the area is divided into N contiguous nxn pixel regions of interest i (n=40) and the algorithm is intiated (Step II).”; p. 6; “whereas the ROI-extraction method taken from [17] was updated every frame as it was understood to be dynamic. Once the raw red and blue signals were acquired by taking the average of the pixel intensities for these two colour channels in the respective ROIs,”; p. 13, 1st para.);
compute, for each of the plurality of skin pixels (“the position-dependence”; p. 4), a time-dependent signal (“timeseries x(t), consisting of successive samples from one of the R;G;B
colour channels”; p. 6, Eq. (10); “a timeseries”; p. 7, 3rd complete para.; “the normalised channel amplitude timeseries” 3.1. Post-processing; p. 10) corresponding to blood volume change (“The pulsatile, cardiac-synchronous signals which are caused by colour changes due to the in and outflow of blood will have a relative phase shift between them that is uniquely determined by the maximum pulse transit time within the area of skin observed.”; p. 7; 1st complete para.) for each color channel of the skin pixel (“In Step VI, the normalised AC amplitude is determined for each colour channel as per Equation 7 in all M regions of interest”; p. 8; Fig. 1); and
compute an overall SpO2 estimate (“The present work introduces a novel method for Skin-Oxygen Photoplethysmographic Image Analysis (Sophia), which allows oxygen saturation changes to be tracked accurately over time." p. 3, the last complete para.; Figs. 2-4 and 6-7; Table 3).
Guazzi does not explicitly teach generating, based on the time-dependent signal, a plurality of SpO2 estimates for each of the plurality of skin pixels at each frame of the captured video and computing an overall SpO2 estimate from the plurality of SpO2 estimates.
However, in the medical imaging devices and methods field of endeavor, Panasyuk discloses hyperspectral/multispectral imaging in determination, assessment and monitoring of systemic physiology and shock, which is analogous art. Panasyuk teaches generating, based on the time-dependent signal (“near-real time information" [0080]. “By revealing changes in tissue … oxygen delivery, oxygen extraction, S.sub.HSIO.sub.2 … that correlate with adverse outcomes, the MHSI approach is additionally able to provide information about patient survivability" [0096]), a plurality of SpO2 estimates (“hyperspectral tissue oxygen saturation …S.sub.HSIO.sub.2” [0051]; “an oxygen saturation value” Claim 62) for each of the plurality of skin pixels at each frame of the captured video (“changes in tissue … S.sub.HSIO.sub.2” [0096]. “Oxygen saturation images in which the brightness of each pixel is proportionate to the intensity of the S.sub.HSIO.sub.2 for that pixel” [0196]; “62. …calculating an oxygen saturation value for each pixel in the image.”); and
computing an overall SpO2 estimate from the plurality of SpO2 estimates (“Oxyhemoblobin (OxyHb), deoxyhemoglobin (DeoxyHb) … can be presented …as scalars reflecting a mean value across a region of interest (ROI), or the oxyhemoblobin and deoxyhemoglobin coefficients can be used to calculate hyperspectral tissue oxygen saturation (S.sub.HSIO.sub.2=OxyHb/(OxyHb+DeoxyHb)) ... This information can be presented as black and white or false color images…This presentation can be used to represent oxyhemoglobin and deoxyhemoglobin values for any pixel in the ROI, to present the average oxyhemoglobin and deoxyhemoglobin values over the entire ROI.” [0051]).
Therefore, based on Panasyuk’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi to have the step of generating, based on the time-dependent signal, a plurality of SpO2 estimates for each of the plurality of skin pixels at each frame of the captured video, and computing an overall SpO2 estimate from the plurality of SpO2 estimates as taught by Panasyuk, in order to facilitate the image analysis for contactless SpO2 tracking in tissue over time.
Regarding claim 19, Guazzi modified by Panasyuk teaches the non-transitory computer-readable medium of claim 15, wherein Guazzi teaches computing, for each of the plurality of skin pixels, the time-dependent signal corresponding to blood volume change for each color channel of the skin pixel comprises computing the time-dependent signal for each of a red-channel, a green-channel, and a blue-channel of each skin pixel (“The tracking of oxygen saturation with RGB cameras (visible light cameras with three channels: red, green and blue respectively)” p. 3, 2nd complete para.; “For the purpose of initialization, x(t) is taken as the green signal from the RGB camera over a period of 12-seconds… the per-frame spatial average of the green channel in each of the N regions of interest”; pp. 6-7. “The breathing rate estimate is found by taking the average of the power spectral density (PSD) of the detrended blue signal across all N regions of interest and then searching for a peak in the expected physiological range (between 0.1 and 0.7 Hz, corresponding to 6 to 42 breaths per minute). While the green channel usually has a high heart rate SNR as a result of its poor absorption by melanin but high absorption by blood, the blue channel is used to determine the breathing rate for the opposite reason.” p. 7, 1st para.; “The amplitude of the waveform is taken as the normalised amplitude for that channel and the ratio of ratios is calculated as the ratio of the blue amplitude to the red amplitude (Step IX).”; 2.2. Implementation; p. 8).
Claims 2, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Guazzi and Panasyuk as applied to claims 1, 8, and 15, and further in view of Wane et al (WO 2021240014), hereinafter, Wane.
Regarding claim 2, Guazzi modified by Panasyuk teaches the method of claim 1.
Guazzi modified by Panasyuk further does not teach that generating the plurality of SpO2 estimates comprises, for each of the plurality of skin pixels, applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal, and wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers.
However, in the medical image analysis field of endeavor, Wane discloses a temperature detection system, which is analogous art. Wane teaches that generating the plurality of SpO2 estimates comprises, for each of the plurality of skin pixels (“the plurality of light emitters and/or the plurality of light detectors are arranged in a linear array or a 2-dimensional matrix of pixels” [0170]), applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal (“The SCO layer, as applied on or otherwise in thermal contact with the target surface, is also compatible with a direct determination of other parameters of the target surface such as a blood oxygen level in the case that the target surface is the skin of a human or animal body. Indeed, the SCO layer for example allows light illumination to propagate through it and reach the target surface. It is therefore possible to determine coherent measurements of temperature and blood oxygen levels at the same precise region of the target surface.” [0065]. Inputs have to include parameters as claimed in order to predict SpO2), wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers (“one or more hidden layers each comprising a further plurality of neurons” [0149]).
Therefore, based on Wane’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have the step of generating the plurality of SpO2 estimates that comprises, for each of the plurality of skin pixels, applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal, and wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers, as taught by Wane, in order to improve the image analysis thereby providing better SpO2 estimates.
Regarding claim 9, Guazzi modified by Panasyuk teaches the device of claim 8.
Guazzi modified by Panasyuk further does not teach that generating the plurality of SpO2 estimates comprises, for each of the plurality of skin pixels, applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal, and wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers.
However, in the medical image analysis field of endeavor, Wane discloses a temperature detection system, which is analogous art. Wane teaches that generating the plurality of SpO2 estimates comprises, for each of the plurality of skin pixels (“the plurality of light emitters and/or the plurality of light detectors are arranged in a linear array or a 2-dimensional matrix of pixels” [0170]), applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal (“The SCO layer, as applied on or otherwise in thermal contact with the target surface, is also compatible with a direct determination of other parameters of the target surface such as a blood oxygen level in the case that the target surface is the skin of a human or animal body. Indeed, the SCO layer for example allows light illumination to propagate through it and reach the target surface. It is therefore possible to determine coherent measurements of temperature and blood oxygen levels at the same precise region of the target surface.” [0065]. Inputs have to include parameters as claimed in order to predict SpO2), wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers (“one or more hidden layers each comprising a further plurality of neurons” [0149]).
Therefore, based on Wane’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have the step of generating the plurality of SpO2 estimates that comprises, for each of the plurality of skin pixels, applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal, and wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers, as taught by Wane, in order to improve the image analysis thereby providing better SpO2 estimates.
Regarding claim 16, Guazzi modified by Panasyuk teaches the non-transitory computer-readable medium of claim 15.
Guazzi modified by Panasyuk further does not teach that generating the plurality of SpO2 estimates comprises, for each of the plurality of skin pixels, applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal, and wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers.
However, in the medical image analysis field of endeavor, Wane discloses a temperature detection system, which is analogous art. Wane teaches that generating the plurality of SpO2 estimates comprises, for each of the plurality of skin pixels (“the plurality of light emitters and/or the plurality of light detectors are arranged in a linear array or a 2-dimensional matrix of pixels” [0170]), applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal (“The SCO layer, as applied on or otherwise in thermal contact with the target surface, is also compatible with a direct determination of other parameters of the target surface such as a blood oxygen level in the case that the target surface is the skin of a human or animal body. Indeed, the SCO layer for example allows light illumination to propagate through it and reach the target surface. It is therefore possible to determine coherent measurements of temperature and blood oxygen levels at the same precise region of the target surface.” [0065]. Inputs have to include parameters as claimed in order to predict SpO2), wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers (“one or more hidden layers each comprising a further plurality of neurons” [0149]).
Therefore, based on Wane’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have the step of generating the plurality of SpO2 estimates that comprises, for each of the plurality of skin pixels, applying as inputs to a machine learning model a plurality of parameters derived from the corresponding time-dependent signal, and wherein the machine learning model comprises a multilayer perceptron model comprising at least three hidden layers, as taught by Wane, in order to improve the image analysis thereby providing better SpO2 estimates.
Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Guazzi and Panasyuk as applied to claims 1, 8, and 15, and further in view of Jones et al (US 20220133241), hereinafter, Jones.
Regarding claim 3, Guazzi modified by Panasyuk teaches the method of claim 1.
Guazzi modified by Panasyuk further does not teach that prior to capturing the video, one or more performance settings of the camera are disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance, de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancements.
However, in the medical image analysis field of endeavor, Jones discloses a temperature detection system, which is analogous art. Jones teaches that prior to capturing the video, one or more performance settings of the camera (110 or 204) are disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance (“AWB” [0257] ), de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancement (“FIG. 12B shows a flowchart illustrating an example process 1250 for resetting or adjusting operational settings of the camera device 110 or 204. The data acquisition module 502 can adjust, or reset, basic settings, automatic settings and/or color correction settings of the camera device 110 or 204. Adjusting the basic settings can include adjusting, or resetting, the sensor frame duration, the sensor exposure time, the sensor sensitivity and/or the flash mode to corresponding predefined values. Adjusting the automatic settings can include disabling one or more automatic settings of the camera device 110 or 204, such as auto-white balance (AWB),” [0257]).
Therefore, based on Jones’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have, prior to capturing the video, one or more performance settings of the camera disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance, de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancement, as taught by Jones, in order to improve the image analysis thereby providing better SpO2 estimates.
Regarding claim 10, Guazzi modified by Panasyuk teaches the device of claim 8.
Guazzi modified by Panasyuk further does not teach that prior to capturing the video, one or more performance settings of the camera are disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance, de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancements.
However, in the medical image analysis field of endeavor, Jones discloses a temperature detection system, which is analogous art. Jones teaches that prior to capturing the video, one or more performance settings of the camera (110 or 204) are disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance (“AWB” [0257] ), de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancement (“FIG. 12B shows a flowchart illustrating an example process 1250 for resetting or adjusting operational settings of the camera device 110 or 204. The data acquisition module 502 can adjust, or reset, basic settings, automatic settings and/or color correction settings of the camera device 110 or 204. Adjusting the basic settings can include adjusting, or resetting, the sensor frame duration, the sensor exposure time, the sensor sensitivity and/or the flash mode to corresponding predefined values. Adjusting the automatic settings can include disabling one or more automatic settings of the camera device 110 or 204, such as auto-white balance (AWB),” [0257]).
Therefore, based on Jones’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have, prior to capturing the video, one or more performance settings of the camera disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance, de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancement, as taught by Jones, in order to improve the image analysis thereby providing better SpO2 estimates.
Regarding claim 17, Guazzi modified by Panasyuk teaches the non-transitory computer-readable medium of claim 15.
Guazzi modified by Panasyuk further does not teach that prior to capturing the video, one or more performance settings of the camera are disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance, de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancements.
However, in the medical image analysis field of endeavor, Jones discloses a temperature detection system, which is analogous art. Jones teaches that prior to capturing the video, one or more performance settings of the camera (110 or 204) are disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance (“AWB” [0257] ), de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancement (“FIG. 12B shows a flowchart illustrating an example process 1250 for resetting or adjusting operational settings of the camera device 110 or 204. The data acquisition module 502 can adjust, or reset, basic settings, automatic settings and/or color correction settings of the camera device 110 or 204. Adjusting the basic settings can include adjusting, or resetting, the sensor frame duration, the sensor exposure time, the sensor sensitivity and/or the flash mode to corresponding predefined values. Adjusting the automatic settings can include disabling one or more automatic settings of the camera device 110 or 204, such as auto-white balance (AWB),” [0257]).
Therefore, based on Jones’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have, prior to capturing the video, one or more performance settings of the camera disabled, wherein the one or more performance settings are selected from bad pixel correction, Bayer domain hardware noise reduction, high-order lens-shading compensation, auto-white-balance, de-mosaic filtering, 3×3 color transformation, color artifact suppression, downscaling, or edge enhancement, as taught by Jones, in order to improve the image analysis thereby providing better SpO2 estimates.
Claims 4, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Guazzi and Panasyuk as applied to claims 1, 8, and 15, and further in view of Islam (US 20240130621), hereinafter, Islam.
Regarding claim 4, Guazzi modified by Panasyuk teaches the method of claim 1, wherein Guazzi teaches that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm (“Fig. 5. A comparison of different ROI selection methods and of the effect of the inclusion criteria on the ROI selection in Sophia, carried out on all subjects considered. From left to right: the method introduced by [14], using a fixed region manually set onto the lower part of the face” p. 13) (“Once a search area has been defined in which all the skin to be imaged has been included, the area is divided into N contiguous nxn pixel regions of interest i (n=40) and the algorithm is intiated (Step II).”; p. 6; “whereas the ROI-extraction method taken from [17] was updated every frame as it was understood to be dynamic. Once the raw red and blue signals were acquired by taking the average of the pixel intensities for these two colour channels in the respective ROIs,”; p. 13, 1st para.).
Guazzi modified by Panasyuk further does not teach that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks to determine bounds of the skin.
However, in the medical image analysis field of endeavor, Islam discloses a camera based system with processing using artificial intelligence for detecting anomalous occurrences and improving performance, which is analogous art. Islam teaches that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks (“to identify different landmarks on the face” [0424]) to determine bounds of the skin (“the Camera-based system combined with Processing (CSP) may observe multiple pieces of information on a person's face. For example, the CSP may evaluate the facial blood flow, … and vital signs/physiological parameters to decipher the state of a person (FIG. 86A). The CSP may first use face-tracking to identify different landmarks on the face and potentially build a bounding box around the face. One algorithm that may be used that is open-source available is MediaPipe,” [0424]).
Therefore, based on Islam’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have identifying the plurality of skin pixels within the one or more image frames that comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks to determine bounds of the skin, as taught by Islam, in order to improve the image analysis thereby providing better SpO2 estimates.
Regarding claim 11, Guazzi modified by Panasyuk teaches the device of claim 8, wherein Guazzi teaches that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm (“Fig. 5. A comparison of different ROI selection methods and of the effect of the inclusion criteria on the ROI selection in Sophia, carried out on all subjects considered. From left to right: the method introduced by [14], using a fixed region manually set onto the lower part of the face” p. 13) (“Once a search area has been defined in which all the skin to be imaged has been included, the area is divided into N contiguous nxn pixel regions of interest i (n=40) and the algorithm is intiated (Step II).”; p. 6; “whereas the ROI-extraction method taken from [17] was updated every frame as it was understood to be dynamic. Once the raw red and blue signals were acquired by taking the average of the pixel intensities for these two colour channels in the respective ROIs,”; p. 13, 1st para.).
Guazzi modified by Panasyuk further does not teach that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks to determine bounds of the skin.
However, in the medical image analysis field of endeavor, Islam discloses a camera based system with processing using artificial intelligence for detecting anomalous occurrences and improving performance, which is analogous art. Islam teaches that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks (“to identify different landmarks on the face” [0424]) to determine bounds of the skin (“the Camera-based system combined with Processing (CSP) may observe multiple pieces of information on a person's face. For example, the CSP may evaluate the facial blood flow, … and vital signs/physiological parameters to decipher the state of a person (FIG. 86A). The CSP may first use face-tracking to identify different landmarks on the face and potentially build a bounding box around the face. One algorithm that may be used that is open-source available is MediaPipe,” [0424]).
Therefore, based on Islam’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have identifying the plurality of skin pixels within the one or more image frames that comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks to determine bounds of the skin, as taught by Islam, in order to improve the image analysis thereby providing better SpO2 estimates.
Regarding claim 18, Guazzi modified by Panasyuk teaches the non-transitory computer-readable medium of claim 15, wherein Guazzi teaches that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm (“Fig. 5. A comparison of different ROI selection methods and of the effect of the inclusion criteria on the ROI selection in Sophia, carried out on all subjects considered. From left to right: the method introduced by [14], using a fixed region manually set onto the lower part of the face” p. 13) (“Once a search area has been defined in which all the skin to be imaged has been included, the area is divided into N contiguous nxn pixel regions of interest i (n=40) and the algorithm is intiated (Step II).”; p. 6; “whereas the ROI-extraction method taken from [17] was updated every frame as it was understood to be dynamic. Once the raw red and blue signals were acquired by taking the average of the pixel intensities for these two colour channels in the respective ROIs,”; p. 13, 1st para.);
Guazzi modified by Panasyuk further does not teach that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks to determine bounds of the skin.
However, in the medical image analysis field of endeavor, Islam discloses a camera based system with processing using artificial intelligence for detecting anomalous occurrences and improving performance, which is analogous art. Islam teaches that identifying the plurality of skin pixels within the one or more image frames comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks (“to identify different landmarks on the face” [0424]) to determine bounds of the skin (“the Camera-based system combined with Processing (CSP) may observe multiple pieces of information on a person's face. For example, the CSP may evaluate the facial blood flow, … and vital signs/physiological parameters to decipher the state of a person (FIG. 86A). The CSP may first use face-tracking to identify different landmarks on the face and potentially build a bounding box around the face. One algorithm that may be used that is open-source available is MediaPipe,” [0424]).
Therefore, based on Islam’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to have identifying the plurality of skin pixels within the one or more image frames that comprises detecting the face and/or the hand palm and extracting face landmarks and/or hand landmarks to determine bounds of the skin, as taught by Islam, in order to improve the image analysis thereby providing better SpO2 estimates.
Claims 7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Guazzi and Panasyuk as applied to claims 1, 8, and 15, and further in view of Zeng et al (US 20050008223), hereinafter, Zeng.
Regarding claim 7, Guazzi modified by Panasyuk teaches the method of claim 1.
Guazzi modified by Panasyuk further does not teach prior to capturing the video, increasing a color depth of video capture to 10 or more bits per color channel.
However, in the RGB color space processing field of endeavor, Zeng discloses representing extended color gamut information, which is analogous art. Zeng teaches prior to capturing the video, increasing a color depth of video capture to 10 or more bits per color channel (“To increase, or extend, the color gamut for a particular color space, or to increase the accuracy to represent color values, the number of bits, or the bit depth, of each color channel is typically increased. However, increasing the bit depth of a color space's channels significantly increases memory or storage space requirements. For example, an 8-bit per-channel RGB color space image file uses 24 bits to represent a pixel, whereas a 12-bit per-channel RGB color space image file uses 36 bits to represent a pixel. Increasing the bit depth of an RGB color space from 8 bits to 12 bits thus results in a 50% increase in the amount of memory or storage space needed.” [0003]).
Therefore, based on Zeng’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to employ the step of increasing a color depth of video capture to 10 or more bits per color channel prior to capturing the video, as taught by Zeng, in order to increase the accuracy of representing color values thereby providing better SpO2 estimates.
Regarding claim 14, Guazzi modified by Panasyuk teaches the device of claim 8. Guazzi modified by Panasyuk further does not teach prior to capturing the video, increasing a color depth of video capture to 10 or more bits per color channel.
However, in the RGB color space processing field of endeavor, Zeng discloses representing extended color gamut information, which is analogous art. Zeng teaches prior to capturing the video, increasing a color depth of video capture to 10 or more bits per color channel (“To increase, or extend, the color gamut for a particular color space, or to increase the accuracy to represent color values, the number of bits, or the bit depth, of each color channel is typically increased. However, increasing the bit depth of a color space's channels significantly increases memory or storage space requirements. For example, an 8-bit per-channel RGB color space image file uses 24 bits to represent a pixel, whereas a 12-bit per-channel RGB color space image file uses 36 bits to represent a pixel. Increasing the bit depth of an RGB color space from 8 bits to 12 bits thus results in a 50% increase in the amount of memory or storage space needed.” [0003]).
Therefore, based on Zeng’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to employ the step of increasing a color depth of video capture to 10 or more bits per color channel prior to capturing the video, as taught by Zeng, in order to increase the accuracy of representing color values thereby providing better SpO2 estimates.
Regarding claim 20, Guazzi modified by Panasyuk teaches the non-transitory computer-readable medium of claim 15, wherein Guazzi teaches that computing the overall SpO2 estimate from the plurality of SpO2 estimates comprises applying a statistical inference within each of the plurality of image frames of the captured video (“If any region of interest may be used, then an average of the regions of interest may be taken instead, as previous research has shown that averaging over a large number of physiologically relevant pixels will improve the definition of the resulting PPG signal”; p. 5).
Guazzi does not explicitly teach that the statistical inference is being applied to the plurality of SpO2 estimates computed over all of the identified skin pixels.
However, in the medical imaging devices and methods field of endeavor, Panasyuk discloses hyperspectral/multispectral imaging in determination, assessment and monitoring of systemic physiology and shock, which is analogous art. Panasyuk teaches that the statistical inference (“a mean value” [0051]) is being applied to the plurality of SpO2 estimates computed over all of the identified skin pixels (“Oxyhemoblobin (OxyHb), deoxyhemoglobin (DeoxyHb) … can be presented …as scalars reflecting a mean value across a region of interest (ROI), or the oxyhemoblobin and deoxyhemoglobin coefficients can be used to calculate hyperspectral tissue oxygen saturation (S.sub.HSIO.sub.2=OxyHb/(OxyHb+DeoxyHb)) ... This information can be presented as black and white or false color images…This presentation can be used to represent oxyhemoglobin and deoxyhemoglobin values for any pixel in the ROI, to present the average oxyhemoglobin and deoxyhemoglobin values over the entire ROI.” [0051] “Oxygen saturation images in which the brightness of each pixel is proportionate to the intensity of the S.sub.HSIO.sub.2 for that pixel” [0196]; “62. …calculating an oxygen saturation value for each pixel in the image.”); and
computing an overall SpO2 estimate from the plurality of SpO2 estimates).
Therefore, based on Panasyuk’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi to have the step of generating, based on the time-dependent signal, a plurality of SpO2 estimates for each of the plurality of skin pixels at each frame of the captured video, and computing an overall SpO2 estimate from the plurality of SpO2 estimates as taught by Panasyuk, in order to facilitate the image analysis for contactless SpO2 tracking in tissue over time.
Guazzi modified by Panasyuk further does not teach prior to capturing the video, increasing a color depth of video capture to 10 or more bits per color channel.
However, in the RGB color space processing field of endeavor, Zeng discloses representing extended color gamut information, which is analogous art. Zeng teaches prior to capturing the video, increasing a color depth of video capture to 10 or more bits per color channel (“To increase, or extend, the color gamut for a particular color space, or to increase the accuracy to represent color values, the number of bits, or the bit depth, of each color channel is typically increased. However, increasing the bit depth of a color space's channels significantly increases memory or storage space requirements. For example, an 8-bit per-channel RGB color space image file uses 24 bits to represent a pixel, whereas a 12-bit per-channel RGB color space image file uses 36 bits to represent a pixel. Increasing the bit depth of an RGB color space from 8 bits to 12 bits thus results in a 50% increase in the amount of memory or storage space needed.” [0003]).
Therefore, based on Zeng’s teachings, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Guazzi and Panasyuk to employ the step of increasing a color depth of video capture to 10 or more bits per color channel prior to capturing the video, as taught by Zeng, in order to increase the accuracy of representing color values thereby providing better SpO2 estimates.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXEI BYKHOVSKI whose telephone number is (571)270-1556. The examiner can normally be reached on Monday-Friday: 8:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXEI BYKHOVSKI/
Primary Examiner, Art Unit 3798