DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on April 12, 2023, October 9, 2024, and April 9, 2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements have been considered by the examiner.
Claim Objections
Claim 1 is objected to because of the following informalities:
“and first age information” should read “first age information” in line 18”; and
“the state code date” should read “the state code data” in line 54.
Appropriate correction is required.
Claim 2 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 3 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 4 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 5 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 6 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 7 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 8 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 9 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim 10 is objected to because of the following informality: “The system for predicting a blood pressure based on video data” should read “The system for predicting the blood pressure based on the video data” in lines 1-2.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “first master control module”, “light module”, “master control module”, “first communication module”, “second master control module”, “validity verification module”, “parameter verification module”, second communication module” in claim 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “the first master control module” in line 6. There is insufficient antecedent basis for this limitation in the claim. It is unclear if the “first master control module” is meant to be the same module recited as “a master control module” in line 4, or if these are different modules. Further clarification is required to understand the difference, or lack thereof, between the first master control module and the master control module.
Claim 1 recites “perform shooting processing” in line 7. It is unclear what is meant by the term “shooting processing”, as this is not a commonly used term in the art. What does it mean to “perform shooting processing? The specification mentioned “shooting processing” in [0007], [0020], and [0051-0053], but does not define what shooting processing entails, which rendered the recitation of “shooting processing in claim 1, and claim 2 (line 3), indefinite. Further clarification is required.
Claim 1 recites “perform play processing” in line 10. It is unclear what is meant by the term “play processing”, as this is not a commonly used term in the art. What does it mean to “perform play processing” in the claim? The specification mentions “play processing” in [0055], but does not define the term, which renders the recitation of “play processing” in the claim indefinite. Further clarification is required.
Claim 1 recites the limitation "the parameter integrity verification processing" in line 35. There is insufficient antecedent basis for this limitation in the claim.
Claim 1 recites “set state code data to be normal state code information” in lines 47-48. It is unclear what is meant by the “state code data” and “state code information”, and the claims and specification do not describe what these data are and how they are acquired. Additionally, it is unclear was is meant by the term “return data” in line 48. For the purpose of examination, these claim limitations are interpreted to be any type of matching/corresponding data and output/transmission of data.
Claim 1 recites “acquire, when the state code data is the normal state code information” in lines 54-55. It is unclear what is meant by the term “acquire”, as the claim does not described what is being acquired. Is this different than the following limitation reciting “obtain the heart rate data, the diastolic pressure data and the systolic pressure data”? Further clarification is required.
Claim 2 recites “generate first video data” in line 5. It is unclear if this “first video data” is meant to be the same first video data as the “first video data” recited in claim 1, or if these are different data. Further clarification is required.
Claim 7 recites “the second signal data is equivalent to the first signal data” in line 4. It is unclear if there is a process being performed to equalize the second signal data and the first signal data, or if this claim is merely describing the nature of the data. Additionally, claim 7 describes equivalence between device token information, device type information, age information, gender information, height information, and weight information. These recitations of equivalent data are indefinite, as it is unclear if the data are required to be equivalent in order for the system to perform the claimed functions, or if there is some aspect of the system that equalizes these data.
Claim 10 recites “the first sub blood pressure prediction module” in lines 6-7. There is insufficient antecedent basis for this limitation in the claim. Additionally, lines 20-21 recite “the corresponding first sub blood pressure prediction module”. It is unclear if this is meant to refer to the same module as lines 6-7 and 18-19 or a different module.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US 20200260956 A1 (Lee et al.).
Regarding claim 1, Lee teaches a video data based system for a blood pressure prediction (Abstract; [0010-0011]),
wherein the system for predicting a blood pressure based on video data comprises a first device and a cloud server ([0156]; [0160] “The mobile terminal 100 may operate a web storage or a cloud server on the internet which performs a storage function of the memory 770.”), wherein:
the first device comprises a master control module ([0053] “a controller 120”), a first camera ([0018] “camera sensor”; [0170] “The image obtainer 810 may obtain an image that may be moving picture data”), a light module ([0053] “The collector 110 collects images of a plurality of objects and pieces of blood pressure information of the plurality of objects”), a display screen ([0088] “display”; [0150]) and a first communication module ([0067] “a communication interface (not shown) capable of transmitting the generated blood pressure estimation model to the external server.”);
the first master control module is configured to call the first camera and the light module to perform shooting processing on an epidermal area of a test object for a first duration, to generate a first video data ([0011] “an image signal corresponding to a finger or face photographed by the camera sensor”; [0053-0054]);
the display screen is configured to receive the first video data sent by the master control module to perform play processing ([0141] “the mobile terminal 100 according to an embodiment may further include a user input interface 740, a sensing unit 750, an audio/video (A/V) input interface 760, and a memory 770, in addition to the processor 700, the communication interface 710, and the output interface 730.”; [0149]);
the first master control module is further configured to perform extraction processing of light source channel data on the first video data according to light source information to generate first channel data ([0021] “The data processing module may be further configured to transform an RGB image of the image signal into an HSV image”); then to perform remote photoplethysmography signal data conversion processing on the first channel data to generate first signal data ([0011] “PPG signal”); and then to send the first signal data to the display screen to perform signal waveform display processing according to a display duration ([0137] “a finger of an examination subject is put on a rear camera 110 of a mobile terminal 100 to measure a pulse wave signal. The mobile terminal 100 obtains an image of the finger through a smartphone”);
the first master control module is further configured to encapsulate the first signal data, first device token information, first device type information, and first age information, first gender information, first height information and first weight information of the test object
to a first data packet according to a first protocol ([0016] “The user information may include a height, a gender, an age, and a weight”; [0017] “a bio information measurement device; an authentication module configured to perform user authentication”; [0118]), and send the first data packet to the cloud server by means of the first communication module ([0086] “The blood pressure estimation system 3 according to the present embodiment may further include a communication interface (not shown) capable of receiving the blood pressure estimation model from the external server.”);
the cloud server comprises a second master control module ([0086] “The blood pressure estimation model may be stored in the controller 320 or may be stored in an external server”), a validity verification module, a parameter verification module, a data pre-processing module, an artificial intelligence blood pressure prediction module ([0063-0065] “The controller 120 generates the blood pressure estimation model through machine learning by using the corrected images and the pieces of blood pressure information as learning data.”) and a second communication module ([0121] “The open API server 410 communicates with open API clients mounted on the various electronic apparatuses 100 through 300.”);
the second master control module is configured to perform data analysis processing on the first data packet according to the first protocol ([0093] “Operations S420 through S440 are a process for correcting the image captured by the image capturer 310, and are thus performed by the controller 320.”), to obtain second signal data, second device token information, second device type information, second age information, second gender information, second height information and second weight information ([0110] “user information including respective heights and weights of objects are matched with a reference blood pressure value,”);
the validity verification module is configured to perform validity verification processing on the second device token information according to a valid token list ([0123] “The authentication server 420 performs user authentication, based on the user information received from the electronic apparatuses 100 through 300.”);
the parameter verification module is configured to, when the validity verification processing is successful, perform parameter verification processing on the second signal data, the second device type information, the second age information, the second gender information, the second height information and the second weight information ([0124] “The data processing server 430 estimates heart rate information, stress index information, cardiovascular disease dangerousness index information, blood pressure information, and the like, based on the user information and the bio information measurement signal, and transmits estimated medical information to the electronic apparatuses 100 through 300 through the network”);
the second master control module is further configured to, when the parameter integrity verification processing is successful, perform heart rate calculation processing according to the second signal data, to generate heart rate data ([0087] “The controller 320 may estimate a heart rate of the examination subject from the corrected images”);
the data pre-processing module is configured to perform input data preparation processing of a
blood pressure prediction module on the second signal data, the second age information, the second gender information, the second height information and the second weight information according to identifier information of the prediction module, to generate input data of a model ([0110-0119] “The open API-based medical information providing system 400 receives user information and a bio information measurement signal from the electronic apparatuses 100 through 300 connected to the open API-based medical information providing system 400 through a network and respectively including bio information measurement apparatuses according to a call of the open API protocol, performs user authentication, estimates heart rate information, stress index information, cardiovascular disease dangerousness index information, blood pressure information, and the like, based on the user information and the bio information measurement signal, and transmits estimated medical information to the electronic apparatuses 100 through 300 through the network.”; [0124]);
the artificial intelligence blood pressure prediction module is configured to perform blood pressure prediction operation processing on the input data of the model according to the identifier information of the prediction module, to generate diastolic pressure data and systolic pressure data ([0059] “The machine learning or the neural network may be a group of algorithms that learn a method of recognizing an object from a certain image input to the neural network, based on an artificial intelligence (AI).”; [0061] “the blood pressure estimation model may estimate a systolic blood pressure and a diastolic blood pressure of the examination subject from an input image of a portion of the body of the examination subject”); and
the second master control module is further configured to set state code data to be normal state code information, then constitute return data according to the heart rate data, the diastolic pressure data and the systolic pressure data ([0087] “The controller 320 may estimate a heart rate of the examination subject from the corrected images”), then encapsulate the return data and the state code data to a second data packet according to the first protocol, and send the second data packet to the first device by means of the second communication module ([0095-0097] “The controller 320 transforms the HSV image having the preset value as its V channel value of each frame into the RGB image”; [0101-0102]; [0086] “he controller 320 estimates the blood pressure of the examination subject by using the corrected images and a blood pressure estimation model. The blood pressure estimation model is generated according to the blood pressure estimation model generation method according to another embodiment of the disclosure. The blood pressure estimation model may be stored in the controller 320 or may be stored in an external server. The blood pressure estimation system 3 according to the present embodiment may further include a communication interface (not shown) capable of receiving the blood pressure estimation model from the external server”); and
the first master control module is further configured to perform data analysis processing on the second data packet according to the first protocol, to obtain the return data and the state code date; acquire, when the state code data is the normal state code information, obtain the heart rate data, the diastolic pressure data and the systolic pressure data from the return data; and then send the heart rate data, the diastolic pressure data and the systolic pressure data to the display screen to perform heart rate and blood pressure data display processing ([0061] “the blood pressure estimation model is a model generated from correspondences between respective images of a plurality of objects and pieces of blood pressure information of the plurality of objects, and is used to estimate a blood pressure of an examination subject, based on an image of a portion of the body of the examination subject. In other words, the blood pressure estimation model may estimate a systolic blood pressure and a diastolic blood pressure of the examination subject from an input image of a portion of the body of the examination subject.”; [0088]; [0159]; [0166] “The notification module 773 may generate a signal for notifying that an event has been generated in the mobile terminal 100.”; [0167] “The notification module 773 may output a notification signal in the form of a video signal via the display 731”).
Regarding claim 2, Lee teaches the system for predicting a blood pressure based on video data of claim 1, wherein the first master control module is configured to call the light module to irradiate the epidermal area of the test object and perform shooting processing on the epidermal area for the first duration by means of the first camera after a lens of the first camera covers the epidermal area, to generate first video data ([0137] “a finger of an examination subject is put on a rear camera 110 of a mobile terminal 100 to measure a pulse wave signal. The mobile terminal 100 obtains an image of the finer through a smartphone rear camera and a built-in flash that are generally embedded in the mobile terminal 100. The mobile terminal 100 extracts the pulse wave signal from the obtained image by using several signal processing techniques, for example, by using an infinite impulse response (IIR) filter.”; [0144] “While obtaining a plurality of images of an examination subject through a camera sensor and a built-in flash, the processor 700 may record time information in each of the obtained plurality of images”).
Regarding claim 3, Lee teaches the system for predicting a blood pressure based on video data of claim 1, wherein the first master control module is configured to, when the light source information is red light, perform extraction processing of red light channel data on the first video data, to generate the first channel data; when the light source information is green light, perform extraction processing of green light channel data on the first video data, to generate first channel data; when the light source information is red and green light, perform extraction processing of red light channel data on the first video data, to generate first red light channel data, perform extraction processing of green light channel data on the first video data, to generate first green light channel data, and encapsulate the first red light channel data and the first green light channel data to the first channel data ([0069]; [0080] “Each RGB image includes a red channel image, a green channel image, and a blue channel image. Because a human body absorbs light differently according to the wavelengths of the light, the accuracy of a blood pressure estimation model changes according to from what image from among a red channel image, a green channel image, and a blue channel image the blood pressure estimation model has been generated.”; [0099] “Accordingly, the green channel image is used to estimate the blood pressure of the examination subject from the blood pressure estimation model. Thus, the controller 320 extracts the green channel image from the RGB image in operation S450.”; [0102] “As described above, the blood pressure estimation model generated using the blood pressure estimation model generation method according to another embodiment of the disclosure is the blood pressure estimation model generated from the green channel image, and the blood pressure estimation model has high accuracy compared to a blood pressure estimation model generated from a red channel image or a blue channel image”).
Regarding claim 8, Lee teaches the system for predicting a blood pressure based on video data of claim 1, wherein the validity verification module is configured to inquire the valid token list according to the second device token information, and when the second device token information satisfies the valid token list, the validity verification processing is successful ([0017]; [0121-0123] “The authentication server 420 performs user authentication, based on the user information received from the electronic apparatuses 100 through 300.”).
Regarding claim 9, Lee teaches the system for predicting a blood pressure based on video data of claim 1, wherein the parameter verification module is configured to examine whether none of the second signal data, the second device type information, the second age information, the second gender information, the second height information and the second weight information is null, and when none of the second signal data, the second device type information, the second age information, the second gender information, the second height information and the second weight information is null, the parameter verification processing is successful ([0016-0017] “The user information may include a height, a gender, an age, and a weight.”; [0110] “user information including respective heights and weights of objects are matched with a reference blood pressure value, thereby training a blood pressure estimation model 1000 as shown in FIG. 10A.”; [0118] “the electronic apparatuses 100 through 300 may transmit personal information of a user, for example, a height, an age, and a weight, and a PPG signal, an ECG signal, or an image signal from which a pulse wave signal may be estimated, which is measured by a bio signal measuring apparatus, according to an open API protocol.”; [0128]; Fig. 12 shows successful data processing and output when the personal information of the user from authenticated devices is present input into the system (not null).).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over US 20200260956 A1 (Lee et al.) in view of US 20170238805 A1 (Addison et al.).
Regarding claim 4, Lee teaches the system for predicting a blood pressure based on video data of claim 3.
Lee does not explicitly teach wherein the first master control module is configured to, when the light source information is red light, perform frame image extraction processing on the first video data, to obtain a plurality of first frame image data; count a quantity of first red pixel points with a pixel value meeting a red light pixel threshold range in each of the first frame image data, to generate a first aggregate, and perform a summation calculation on pixel values of all the first red pixel points, to generate a first pixel value sum, and then take a ratio of the first pixel value sum to the first aggregate as first frame red light channel data corresponding to each of the first frame image data; and then rank all the first frame red light channel data in a chronological order, to generate the first channel data;
the first master control module is configured to, when the light source information is green light, perform frame image extraction processing on the first video data, to obtain a plurality of second frame image data; count a quantity of first green pixel points with a pixel value meeting a green light pixel threshold range in each of the second frame image data, to generate a second aggregate, and perform a summation calculation on pixel values of all the first green pixel points, to generate a second pixel value sum, and then take a ratio of the second pixel value sum to the second aggregate as first frame green light channel data corresponding to each of the second frame image data; and then rank all the first frame green light channel data in a chronological order, to generate the first channel data; and
the first master control module is configured to, when the light source information is red and green light, perform frame image extraction processing on the first video data, to obtain a plurality of third frame image data; count a quantity of second red pixel points with a pixel value meeting a red light pixel threshold range in each of the third frame image data, to generate a third aggregate, and perform a summation calculation on pixel values of all the second red pixel points, to generate a third pixel value
sum, and then take a ratio of the third pixel value sum to the third aggregate as second frame red light channel data corresponding to each of the third frame image data, and then rank all the second frame red light channel data in a chronological order, to generate the first red light channel data; count a quantity of second green pixel points with a pixel value meeting a green light pixel threshold range in each of the third frame image data, to generate a fourth aggregate, and perform a summation calculation on pixel values of all the second green pixel points, to generate a fourth pixel value sum, and then take a ratio of the fourth pixel value sum to the fourth aggregate as second frame green light channel data corresponding to each of the third frame image data, and then rank all the second frame green light channel data in a chronological order, to generate the first green light channel data; and then perform multi-channel data encapsulation processing on the first red light channel data and the first green light channel data, to generate the first channel data.
However,
Addison teaches wherein the first master control module is configured to, when the light source information is red light, perform frame image extraction processing on the first video data ([0048] “extracting from the video signal time-varying red, green, and blue signals”), to obtain a plurality of first frame image data; count a quantity of first red pixel points with a pixel value meeting a red light pixel threshold range in each of the first frame image data ([0050] “extracting the red, green, and blue signals comprises selecting pixels within the image frame exhibiting a modulation that is at the primary frequency and that has an amplitude above a threshold”), to generate a first aggregate, and perform a summation calculation on pixel values of all the first red pixel points, to generate a first pixel value sum, and then take a ratio of the first pixel value sum to the first aggregate as first frame red light channel data corresponding to each of the first frame image data ([0045]; [0084] “Pixel signals can be combined by summing or averaging or weighted averaging. In an embodiment, the combined pixel signals are obtained by averaging the Red (or Blue, or Green) color values of the pixels within the region, so that regions of different sizes can be compared against each other.”); and then rank all the first frame red light channel data in a chronological order, to generate the first channel data ([0084] “The Combined Forehead plot shows the combined pixel signals from all three identified regions 1A, 2A, and 3A, meaning that the Red components from all three regions are combined together and plotted over time, as are the Green components and the Blue components”; [0100]);
the first master control module is configured to, when the light source information is green light, perform frame image extraction processing on the first video data ([0048] “extracting from the video signal time-varying red, green, and blue signals”), to obtain a plurality of second frame image data; count a quantity of first green pixel points with a pixel value meeting a green light pixel threshold range in each of the second frame image data ([0050] “extracting the red, green, and blue signals comprises selecting pixels within the image frame exhibiting a modulation that is at the primary frequency and that has an amplitude above a threshold”), to generate a second aggregate, and perform a summation calculation on pixel values of all the first green pixel points, to generate a second pixel value sum, and then take a ratio of the second pixel value sum to the second aggregate as first frame green light channel data corresponding to each of the second frame image data ([0045]; [0084] “Pixel signals can be combined by summing or averaging or weighted averaging. In an embodiment, the combined pixel signals are obtained by averaging the Red (or Blue, or Green) color values of the pixels within the region, so that regions of different sizes can be compared against each other.”); and then rank all the first frame green light channel data in a chronological order, to generate the first channel data ([0084] “The Combined Forehead plot shows the combined pixel signals from all three identified regions 1A, 2A, and 3A, meaning that the Red components from all three regions are combined together and plotted over time, as are the Green components and the Blue components”; [0100]); and
the first master control module is configured to, when the light source information is red and green light, perform frame image extraction processing on the first video data, ([0048] “extracting from the video signal time-varying red, green, and blue signals”), to obtain a plurality of third frame image data; count a quantity of second red pixel points with a pixel value meeting a red light pixel threshold range in each of the third frame image data, to generate a third aggregate, and perform a summation calculation on pixel values of all the second red pixel points, to generate a third pixel value
sum, and then take a ratio of the third pixel value sum to the third aggregate as second frame red light channel data corresponding to each of the third frame image data ([0088] “Though many embodiments herein are described with reference to pixels and pixel values, this is just one example of a detected light intensity signal. The light intensity signals that are detected, measured, or analyzed may be collected from larger regions or areas, without differentiating down to groups of pixels or individual pixels.”), and then rank all the second frame red light channel data in a chronological order, to generate the first red light channel data; count a quantity of second green pixel points with a pixel value meeting a green light pixel threshold range in each of the third frame image data, to generate a fourth aggregate, and perform a summation calculation on pixel values of all the second green pixel points ([0050] “extracting the red, green, and blue signals comprises selecting pixels within the image frame exhibiting a modulation that is at the primary frequency and that has an amplitude above a threshold”), to generate a fourth pixel value sum, and then take a ratio of the fourth pixel value sum to the fourth aggregate as second frame green light channel data corresponding to each of the third frame image data ([0101] “The method includes segmenting a first image frame into a plurality of regions at 512, and then, for each region, extracting from the video signal a time-varying color signal at 513. In an example, three time-varying color signals are extracted from each region, corresponding to red, green, and blue pixel values”), and then rank all the second frame green light channel data in a chronological order, to generate the first green light channel data; and then perform multi-channel data encapsulation processing on the first red light channel data and the first green light channel data, to generate the first channel data ([0084] “The Combined Forehead plot shows the combined pixel signals from all three identified regions 1A, 2A, and 3A, meaning that the Red components from all three regions are combined together and plotted over time, as are the Green components and the Blue components”; [0100]).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified the system taught by Lee to include processing specific to the red and green light data. One would have been motivated to make this modification because analyzing video data over time using red and green pixels allows cardiac vital signs to be measured from the skin of the patient and the light color changes over time indicate changes in the patient’s physiology in a non-contact manner, as suggested by Addison ([0079]; [0082]; [0101]).
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over US 20200260956 A1 (Lee et al.) in view of US 20170238805 A1 (Addison et al.), further in view of US 20160242700 A1 (Ferber et al.).
Regarding claim 5, Lee teaches the system for predicting a blood pressure based on video data of claim 4, wherein the first master control module is configured to, when the light source information is the red light, perform remote photoplethysmography signal filtering processing on the first channel data, to generate first red light filter data, and perform remote photoplethysmography signal denoise processing on the first red light filter data, to generate first red light signal data ([0056] “The controller 120 corrects the images collected by the collector 110. The controller 120 transforms the collected images from RGB format to HSV format to produce HSV images, and inversely transforms the HSV images of which V values have been changed to 1 back to RGB images”; [0080] “Each RGB image includes a red channel image, a green channel image, and a blue channel image. Because a human body absorbs light differently according to the wavelengths of the light, the accuracy of a blood pressure estimation model changes according to from what image from among a red channel image, a green channel image, and a blue channel image the blood pressure estimation model has been generated.”);
the first master control module is configured to, when the light source information is the green light, perform remote photoplethysmography signal filtering processing on the first channel data, to generate first green light filter data, and then perform remote photoplethysmography signal denoise processing on the first green light filter data, to generate first green light signal data ([0096-0097]; [0099] “he green channel image is used to estimate the blood pressure of the examination subject from the blood pressure estimation model. Thus, the controller 320 extracts the green channel image from the RGB image in operation S450.”); and
the first master control module is configured to, when the light source information is the red and green light, perform red light channel data extraction processing on the first channel data, to generate second red light channel data, and perform green light channel data extraction processing on the first channel data, to generate second green light channel data; then respectively perform remote photoplethysmography signal filtering processing on the second red light channel data and the second green light channel data, to generate second red light filter data and second green light filter data; and then respectively perform remote photoplethysmography signal denoise processing on the second red light filter data and the second green light filter data, to generate second red light signal data and second green light signal data ([0080-0082] “The controller 120 generates the blood pressure estimation model through machine learning using each of the green channel images as an input and using, as a target, blood pressure information of an object from which the each green channel image has been obtained. The controller 120 may store the generated blood pressure estimation model therein.”; [0097] “The controller 320 transforms the HSV image having the preset value as its V channel value of each frame into the RGB image”; [0102] “the blood pressure estimation model generated using the blood pressure estimation model generation method according to another embodiment of the disclosure is the blood pressure estimation model generated from the green channel image, and the blood pressure estimation model has high accuracy compared to a blood pressure estimation model generated from a red channel image or a blue channel image.”).
Lee in view of Addison does not explicitly teach performing remote photoplethysmography signal band-pass filtering.
However,
Ferber teaches performing remote photoplethysmography signal band-pass filtering ([0098] “bandpass filter”; [0156]).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified the system taught by Lee in view of Addison to include band-pass filtering. One would have been motivated to make this modification because bandpass filtering is capable of separating signal variation from light intensity in determining cardiac parameters using PPG signals to calculate blood pressure non-invasively, as suggested by Ferber ([0125], [0156]).
Regarding claim 6, Lee teaches the system for predicting a blood pressure based on video data of claim 5, wherein the first master control module is configured to, when the light source information is the red light, intercept a latest data segment with a duration being the display duration from the first red light signal data, so as to generate first red light display data; then perform waveform image data conversion processing on the first red light display data, to generate first red light waveform image data; and then send the first red light waveform image data to the display screen to perform first red light waveform display processing ([0080] “Each RGB image includes a red channel image, a green channel image, and a blue channel image.”; [0125-0126] “when the bio information measurement signal is an image signal obtained by a camera sensor, the data processing server 430 may transform an RGB image of the image signal into an HSV image, change the V channel value of the HSV image to a preset value, transform the HSV image having the preset value as its V channel value into an RGB image, and estimate blood pressure information by using the RGB image and a previously machine-learned blood pressure estimation model. “Selectively, the data processing server 430 may transform the RGB image into an optical flow image, extract a motion vector from the optical flow image, extract a blood flow rate, based on the extracted motion vector, and estimate blood pressure information by using data about the extracted blood flow rate and the previously machine-learned blood pressure estimation model.”; [0144] “While obtaining a plurality of images of an examination subject through a camera sensor and a built-in flash, the processor 700 may record time information in each of the obtained plurality of images, extract pulse wave signals from the plurality of images, correct the extracted pulse wave signals by using the recorded time information, calculate an HRV from the corrected pulse wave signals, and estimate a stress index by using the calculated HRV”);
the first master control module is configured to, when the light source information is the green light, intercept the latest data segment with a duration being the display duration from the first green light signal data, to generate first green light display data; then perform waveform image data conversion processing on the first green light display data, to generate first green light waveform image data; and then send the first green light waveform image data to the display screen to perform first green light waveform display processing ([0069] “operation S260 of generating a blood pressure estimation model through machine learning using the extracted green channel images as an input and using the collected pieces of blood pressure information as a target.”; [0125-0126] “when the bio information measurement signal is an image signal obtained by a camera sensor, the data processing server 430 may transform an RGB image of the image signal into an HSV image, change the V channel value of the HSV image to a preset value, transform the HSV image having the preset value as its V channel value into an RGB image, and estimate blood pressure information by using the RGB image and a previously machine-learned blood pressure estimation model. “Selectively, the data processing server 430 may transform the RGB image into an optical flow image, extract a motion vector from the optical flow image, extract a blood flow rate, based on the extracted motion vector, and estimate blood pressure information by using data about the extracted blood flow rate and the previously machine-learned blood pressure estimation model.”; [0144] “While obtaining a plurality of images of an examination subject through a camera sensor and a built-in flash, the processor 700 may record time information in each of the obtained plurality of images, extract pulse wave signals from the plurality of images, correct the extracted pulse wave signals by using the recorded time information, calculate an HRV from the corrected pulse wave signals, and estimate a stress index by using the calculated HRV”); and
the first master control module is configured to, when the light source information is the red and green light, intercept the latest data segment with a duration being the display duration from the second red light signal data, to generate second red light display data, then perform waveform image data conversion processing on the second red light display data, to generate second red light waveform image data; intercept the latest data segment with a duration being the display duration from the second green light signal data, to generate second green light display data, then perform waveform image data conversion processing on the second green light display data, to generate second green light waveform image data; and then send the second red light waveform image data to the display screen to perform second red light waveform display processing and send the second green light waveform image data to the display screen to perform second green light waveform display processing ([0015]; [0080]; [0125-0126] “when the bio information measurement signal is an image signal obtained by a camera sensor, the data processing server 430 may transform an RGB image of the image signal into an HSV image, change the V channel value of the HSV image to a preset value, transform the HSV image having the preset value as its V channel value into an RGB image, and estimate blood pressure information by using the RGB image and a previously machine-learned blood pressure estimation model. “Selectively, the data processing server 430 may transform the RGB image into an optical flow image, extract a motion vector from the optical flow image, extract a blood flow rate, based on the extracted motion vector, and estimate blood pressure information by using data about the extracted blood flow rate and the previously machine-learned blood pressure estimation model.”; [0144] “While obtaining a plurality of images of an examination subject through a camera sensor and a built-in flash, the processor 700 may record time information in each of the obtained plurality of images, extract pulse wave signals from the plurality of images, correct the extracted pulse wave signals by using the recorded time information, calculate an HRV from the corrected pulse wave signals, and estimate a stress index by using the calculated HRV”).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over US 20200260956 A1 (Lee et al.) in view of US 20200196881 A1 (Zemel, Marc).
Regarding claim 7, Lee teaches the system for predicting a blood pressure based on video data of claim 1;
the second signal data is equivalent to the first signal data, the second device token information is equivalent to the first device token information, the second device type information is equivalent to the first device type information, the second age information is equivalent to the first age information, the second gender information is equivalent to the first gender information; the second height information is equivalent to the first height information, and the second weight information is equivalent to the first weight information; ([0110] “data about this blood flow rate, for example, y1, y2, and y1/y2, and user information including respective heights and weights of objects are matched with a reference blood pressure value, thereby training a blood pressure estimation model 1000 as shown in FIG. 10A.”; [0111-0112]; [0116] “various electronic apparatuses 100 through 300 are connected to an open API-based medical information providing system 400 through a network.”; [0119] “The open API-based medical information providing system 400 receives user information and a bio information measurement signal from the electronic apparatuses 100 through 300 connected to the open API-based medical information providing system 400 through a network and respectively including bio information measurement apparatuses according to a call of the open API protocol, performs user authentication”);
the first communication module is configured to access an Internet via a mobile communication network, a wireless local area network or a wired local area network ([0147] “a wireless local area network (WLAN) (e.g., Wi-Fi) communication interface”; [0160] “The mobile terminal 100 may operate a web storage or a cloud server on the internet which performs a storage function of the memory 770.”); and
the second communication module is configured to access the Internet via a mobile communication network, a wireless local area network or a wired local area network ([0147]; [0160]).
Lee does not explicitly teach the first protocol comprises a Hyper Text Transfer Protocol (HTTP) and a Hyper Text Transfer Protocol over Secure Socket Layer (HTTPS).
However,
Zemel teaches the first protocol comprises a Hyper Text Transfer Protocol (HTTP) and a Hyper Text Transfer Protocol over Secure Socket Layer (HTTPS) ([0153] “HyperText Transfer Protocol (HTTP); Secure Hypertext Transfer Protocol (HTTPS)”).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified the system taught by Lee to include the first protocol comprising a HTTP and HTTPS. One would have been motivated to make this modification because HTTP and HTTPS are conventional internet information severs and can provide secure communications of blood pressure data, as suggested by Zemel (Abstract, [0153], [0158]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over US 20200260956 A1 (Lee et al.) in view of US 20160242700 A1 (Ferber et al.).
Regarding claim 10, Lee teaches the system for predicting a blood pressure based on video data of claim 1, wherein the data pre-processing module comprises a plurality of sub pre-processing modules, and the artificial intelligence blood pressure prediction module comprises a plurality of sub blood pressure prediction modules ([0124-0127]);
the data pre-processing module is configured to select a corresponding first sub preprocessing module to perform input data preparation processing of the first sub blood pressure prediction module on the second signal data, the second age information, the second gender information, the second height information and the second weight information according to identifier information of the prediction module, to generate input data of a first model ([0016-0017] “The user information may include a height, a gender, an age, and a weight.”; [0123] “The authentication server 420 performs user authentication, based on the user information received from the electronic apparatuses 100 through 300.”);
the first sub pre-processing module is configured to perform baseline drift elimination processing on the second signal data, to generate first process signal data, then perform denoise processing on the first process signal data, to generate second process signal data ([0137] “The mobile terminal 100 extracts the pulse wave signal from the obtained image by using several signal processing techniques, for example, by using an infinite impulse response (IIR) filter”); and then encapsulate the second signal data, the second age information, the second gender information, the second height information, the second weight information and the standard signal data to be the input data of the first model according to a requirement for input data format of the first sub blood pressure prediction module ([0118] “he electronic apparatuses 100 through 300 may transmit personal information of a user, for example, a height, an age, and a weight, and a PPG signal, an ECG signal, or an image signal from which a pulse wave signal may be estimated, which is measured by a bio signal measuring apparatus, according to an open API protocol.”); and
the artificial intelligence blood pressure prediction module is configured to select the corresponding first sub blood pressure prediction module to perform a first blood pressure prediction operation on the input data of the first model according to the identifier information of the prediction module, to generate diastolic pressure data and systolic pressure data ([0059] “The machine learning or the neural network may be a group of algorithms that learn a method of recognizing an object from a certain image input to the neural network, based on an artificial intelligence (AI).”; [0061] “the blood pressure estimation model may estimate a systolic blood pressure and a diastolic blood pressure of the examination subject from an input image of a portion of the body of the examination subject”; [0127]).
Lee does not explicitly teach performing standard sampling and normalization processing on the second process signal data, to generate standard signal data.
However,
Ferber teaches performing standard sampling and normalization processing on the second process signal data, to generate standard signal data ([0198-0201] “filtered signals (e.g., to remove noise from the signals), and/or normalized signal values (e.g., between 0-1).”; [0247]).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to have modified the system taught by Lee to include normalization processing on the second process signal data. One would have been motivated to make this modification because processing and normalizing PPG signals reduces noise and allows signals to be compared in values 0-1, as suggested by Ferber ([0200]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EVELYN GRACE PARK whose telephone number is (571)272-0651. The examiner can normally be reached Monday - Friday, 9AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Robert (Tse) Chen can be reached at (571)272-3672. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EVELYN GRACE PARK/Examiner, Art Unit 3791
/TSE W CHEN/Supervisory Patent Examiner, Art Unit 3791