DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-7 and 9 are rejected. Claim 8 is canceled.
Response to Arguments
Claim Objections
The previous claim objections have been withdrawn in view of the amendment.
Claim Rejections - 35 USC § 112
Applicant's arguments filed 5/22/26 have been fully considered but they are not persuasive.
Applicant asserts that claims 3-7 are amended to remove the limitation of “ML prediction model” or replace the limitations by the limitations of “one of a trained neural network, SVM, and Random Forest Classifier. However, these are still machine learning algorithms. Applicant has claimed black-box algorithms without any clear description of what is inside the boxes to determine activity type, attention level, and detect objects in the image/video signal.
The previous 112(b) rejection of claims 1, 2, and 8 has been withdrawn in view of the amendment.
Claim Rejections - 35 USC § 101
Applicant's arguments filed 5/22/26 have been fully considered but they are not persuasive.
Regarding the different types of sensing and signal generating components, Applicant asserts that it is an additional element that is neither well understood, routine, nor conventional in the state of the art. However, the two or more of one or more EEG electrodes, one or more PPG sensor, optical sensor, audio receiver, and IMU amount to nothing more than pre-solution activity of data gathering. This has been shown as well-understood, routine, and conventional activity under Step 2B. The signal receiving and processing device is recited at a high-level of generality and amount to nothing more than parts of a generic computer. Merely including instructions to implement an abstract idea on a computer does not integrate a judicial exception into practical application.
Applicant asserts that determination of the subject’s level of engagement in a task assigned to the subject integrates the claimed invention to the practical application of machine-determination of a human subject’s level of engagement in a task assigned to her/him. However, the Examiner disagrees. This step is directed to the abstract idea. Merely including instructions to implement an abstract idea on a computer does not integrate a judicial exception into practical application.
Claim Rejections - 35 USC § 103
Applicant's arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 3-7 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
For the limitations of
in claim 3 “employing one of a trained neural network, a Support Vector Machine (SVM), and a Random Forest classifier to predict the activity type of the participating activity from the representative pattern of the brainwave plot”;
in claim 3 “employing one of a trained neural network, SVM, and Random Forest classifier, based on frequency analysis on the representative pattern of the brainwave plot, to predict the attention level”;
in claim 4 “employing one of a trained neural network, a SVM, and a Random Forest classifier to predict the activity type of the participating activity from the motion artifact information”;
in claim 4 “employing one of the trained neural network, the SVM, and the Random Forest classifier, based on pulse frequency and heart rate variability analysis, to predict the attention level from the PPG signal generated and received through only a single channel of the PPG sensors”;
in claim 5 “ML-based object detection using a trained neural network to detect objects in the image/video signal”;
in claim 5 “employing a ML prediction model one of a trained neural network, SVM, and Random Forest classifier to predict the activity type of the participating activity from the selected-detected objects”;
in claim 5 “employing one of a trained neural network, SVM, and Random Forest classifier, based on analysis of image characteristics, the selected-detected objects, and frame-to-frame changes, to predict the attention level from the image/video signal”;
in claim 6 “employing one of a trained neural network, SVM, and Random Forest classifier to predict the activity type of the participating activity from the context, intents, and entities of the speech contents”;
in claim 6 “employing one of a trained neural network, SVM, and Random Forest classifier to predict the activity type of the participating activity from the extracted features of the audio signal”;
in claim 6 “employing one of a trained neural network, SVM, and Random Forest classifier to predict the attention level from the degree of relevance of the subject's dialogue and the response speed of the subject”;
in claim 6 “employing one of a trained neural network, SVM, and Random Forest classifier to predict the attention level from the extracted features of the audio signal”; and
in claim 7 ”employing one of a trained neural network, SVM, and Random Forest classifier to predict the activity type of the participating activity from the inertial measurement signal.”
these are computer implemented functional limitations due to being one of a trained neural network, SVM, and Random Forest classifier. ¶22, ¶24, ¶28, ¶30, ¶33, ¶35, ¶38, ¶41, ¶48, ¶49, ¶51, and ¶57 of the specification fail to disclose how these machine learning algorithms are used to recognize a participating activity and compute an attention level of a subject from multi-modal signals. MPEP 2161.01(I) states the following:
Claims may lack written description when the claims define the invention in functional language
specifying a desired result but the specification does not sufficiently describe how the function is
performed or the result is achieved. For software, this can occur when the algorithm or
steps/procedure for performing the computer function are not explained at all or are not
explained in sufficient detail (simply restating the function recited in the claim is not necessarily
sufficient). In other words, the algorithm or steps/procedure taken to perform the function must
be described with sufficient detail so that one of ordinary skill in the art would understand how
the inventor intended the function to be performed. See MPEP §§ 2163.02 and 2181, subsection
IV.”
Applicant has claimed black-box algorithms without any clear description of what is inside the boxes to determine activity type, attention level, and detect objects in the image/video signal. These are functional results and the description is lacking information, for instance, how are these model trained, what are the inputs to each model, what is the order of the inputs into each model, and how are the weights determined for each model? Therefore, the claims with the limitations of “employing one of a trained neural network, SVM, and Random Forest classifier” are rejected under 112(a) for failing to meet the written description requirement.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7 and 9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception, specifically an abstract idea.
Step 1
The claimed invention in claims 1-7 and 9 are directed to statutory subject matter as the claims recite a method for recognizing a participating activity and computing an attention level of a subject.
Step 2A, Prong One
Regarding claim 1, the recited steps are directed to a mental process of performing concepts in a human mind or by a human using a pen and paper (see MPEP 2106.04(a)(2) subsection (III)).
Regarding claim 1, the limitations of “ an activity recognition to predict an activity type of a participating activity being performed by the subject using the pre-processed multi-modal signals, comprising fusing all prediction results of activity recognitions from the multi-modal signals under one of early fusion strategy, intermediate fusion strategy, and decision fusion strategy depending on signal types of the multi-modal signals received for more accurate prediction results; executing, by the signal receiving and processing device, an attention level computation to predict the subject's attention level in performing the participating activity using the pre-processed multi-modal signals comprising fusing all prediction results of attention level computations from the multi- modal signals under the early fusion strategy, the intermediate fusion strategy, or the decision fusion strategy as used in the activity recognition; and executing, by the signal receiving and processing device, determination of the subject's level of engagement in a task assigned to the subject based on the predicted activity type of the participating activity performed by the subject and the predicted subject's attention level in performing the participating activity” are a process, as drafted, covers performance of the limitation that can be performed by a human mind (including an observation, evaluation, judgment, opinion) under the broadest reasonable standard. For example, these limitations are nothing more than a medical professional receiving print outs of the pre-processed multi-modal signals to predict an activity type and an attention level of a subject.
Step 2A, Prong Two
For claim 1, the judicial exception is not integrated into a practical application. In particular, claim 1 recites “two or more of one or more EEG electrodes, one or more PPG sensors, an optical sensor, an audio receiver, an inertial measurement unit (IMU), and a signal receiving and processing device.” The two or more of one or more EEG electrodes, one or more PPG sensor, optical sensor, audio receiver, and IMU amount to nothing more than pre-solution activity of data gathering. The signal receiving and processing device is recited at a high-level of generality and amount to nothing more than parts of a generic computer. Merely including instructions to implement an abstract idea on a computer does not integrate a judicial exception into practical application.
Step 2B
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into
a practical application, the additional elements of the two or more of one or more EEG electrodes, one or more PPG sensor, optical sensor, audio receiver, and IMU amount to nothing more than mere pre-solution activity of data gathering, which does not amount to an inventive concept. Moreover, the two or more of one or more EEG electrodes, one or more PPG sensor, optical sensor, audio receiver, and IMU are recited at a high level of generality and are well-understood, routine, and conventional structures as evidenced by:
US 20210052198: ¶5-conventional PERs may also collect information on vital signs, such as heart rate and heart rate variability with IMU information. Typical sources for measuring heart rate and heart rate variability are electrocardiogram (ECG) monitors attached to the user's chest using electrodes and photoplethysmography (PPG), IMUs;
US 20220180991: ¶68-one or more of the following: the microphonic sensor 120, the Inertial Measurement Unit (IMU) sensor 121, the electrodes 122 or the photoplethysmography sensors 123; and
US 20220338801: ¶30-a conventional multimodal sensing system includes different combinations of sensing modalities such as EEG, EMG, ECG, eye tracking, facial expressions, GSR, etc.
Further, simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry, as discussed in Alice Corp., 573 U.S. at 225, 110 USPQ2d at 1984 (see MPEP § 2106.05(d)).
Regarding dependent claims 2-7 and 9, the limitations of claim 1 further define the limitations already indicated as being directed to the abstract idea.
Regarding claim 2, the limitation of “discarding one or more signal segments in the multi-model multi-modal signals having amplitudes below a minimum signal amplitude threshold or having continuous active durations shorter than a minimum signal active duration threshold” are a process, as drafted, covers performance of the limitation that can be performed by a human mind (including an observation, evaluation, judgment, opinion) under the broadest reasonable standard. For example, these limitations are nothing more than a medical professional discarding one or more signal segments in the multi-modal signals based on a simple comparison for each and filtering the noise from the data on paper. Regarding the remaining “if” limitations: MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim.
Regarding claim 3, Applicant includes multiple trained neural networks, Support Vector Machines (SVM), and Random Forest classifiers which are nothing more than the computer implementation/automation of an abstract mental process of analyzing the activity type from a brainwave plot and further analyzing the brainwave plot to predict the attention level.
Regarding claim 4, Applicant includes multiple trained neural networks, Support Vector Machines (SVM), and Random Forest classifiers which are nothing more than the computer implementation/automation of an abstract mental process of analyzing activity type from the activity data and artifact information, analyzing pulse frequency and HRV data to predict the attention level, and analyzing fNIRS data to predict the attention level.
Regarding claim 5, Applicant includes multiple trained neural networks, Support Vector Machines (SVM), and Random Forest classifiers which are nothing more than the computer implementation/automation of an abstract mental process of analyzing activity type from image/video data and analyzing attention level from image/video data.
Regarding claim 6, Applicant includes multiple trained neural networks, Support Vector Machines (SVM), and Random Forest classifiers which are nothing more than the computer implementation/automation of an abstract mental process of analyzing activity type from speech contents and audio data, as well as analyzing the attention level from audio data.
Regarding claim 7, Applicant includes a trained neural network, Support Vector Machine (SVM), or Random Forest classifier, which is nothing more than the computer implementation/automation of an abstract mental process of analyzing the activity type from inertial measurement data and analyzing the activity type to evaluate the attention level.
Regarding claim 9, includes multiple trained neural networks, Support Vector Machines (SVM), and Random Forest classifiers which are nothing more than the computer implementation/automation of an abstract mental process of analyzing activity type and attention level.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Siddharth (NPL “On Assessing Driver Awareness of Situational Criticalities: Multi-modal Bio-Sensing and Vision-Based Analysis, Evaluations, and Insights” published in 2020) in view of Chua (US 20190092337 filed on 7/11/18).
Regarding claim 1, Siddharth teaches a method for recognizing a participating activity and computing an attention level of a subject from multi-modal signals, comprising: receiving the multi-modal signals comprising two or more of an electroencephalogram (EEG) signal generated and received through one or more EEG electrodes attached to the subject's body (page 4, ¶4-EEG electrodes), a photoplethysmography (PPG) signal generated and received through one or more PPG sensors attached to the subject's body (page 6, ¶2-the PPG signal was recorded using an armband (Biovotion) that measures PPG), an image/video signal generated and received through an optical sensor configured to capture the subject's body (page 7, last ¶-we extracted the face region from the frontal body image of the person captured by the camera for each frame), pre-processing the received multi-modal signals (page 3, 2nd to last ¶-pre-process the data); executing, by the signal receiving and processing device, an activity recognition to predict an activity type of a participating activity being performed by the subject using the pre-processed multi-modal signals (page 2, last ¶-test if the modalities with low-temporal resolution (but easily wearable) namely PPG and GSR can work as well as EEG and vision modality for assessing driver’s attention; page 12, ¶2-annotated by two annotators for low/high driver attention), comprising fusing all prediction results of activity recognitions from the multi-modal signals under one of early fusion strategy, intermediate fusion strategy, and decision fusion strategy depending on signal types of the multi-modal signals received for more accurate prediction results (Siddharth, page 8, ¶4-extract relevant features from the faces for driver attention and hazardous conditions detection; page 2, last ¶-test if (and when) the fusion of features from different sensor modalities boost the classification performance over using each modality independently for attention and hazardous/non-hazardous event classification); an attention level computation to predict the subject's attention level in performing the participating activity using pre-processed multi-modal signals (page 2, last ¶-test if (and when) the fusion of features from different sensor modalities boost the classification performance over using each modality independently for attention) comprising fusing all prediction results of attention level computations from the multi-modal signals under the early fusion strategy, the intermediate fusion strategy, or the decision fusion strategy as used in the activity recognition (Siddharth, page 2, last ¶-test if (and when) the fusion of features from different sensor modalities boost the classification performance over using each modality independently for attention and hazardous/non-hazardous event classification); and executing, by the signal receiving and processing device, determination of the subject's level of engagement in a task assigned to the subject based on the predicted activity type of the participating activity performed by the subject and the predicted subject's attention level in performing the participating activity (page 11, Section 5, ¶1-multi-modal evaluation results for driver attention analysis and hazardous/non-hazardous instances classification; Fig. 11 shows the EEG and Face data vs. the EEG and Face LSTM trend; page 16, ¶1-multi-modality boosts performance over using individual modalities for hazardous/non-hazarodus incident classification (as it did for driver attention analysis) while further improvement in performance is observed by using higher temporal resolution using LSTMs).
While Siddharth recites signal processing (page 2, last ¶), Siddharth does not explicitly recite receiving, by a signal receiving and processing device, simultaneously during a measurement time duration, wherein at least two of the EEG electrodes, PPG sensors, fNIRS, optical sensor, audio receiver, and IMU are available and electrically connected to the signal receiving and processing device; by the signal receiving and processing device, and executing, by the signal receiving and processing device.
Chua relates to vehicle-based operator monitoring systems, methods, and apparatuses. In particular, systems, methods, and apparatuses capture information regarding the operator's physical and/or physiological characteristics, analyze the information, determine a level of operator fatigue or health state, and/or provide warnings based at least in part on the information (¶2).
receiving, by a signal receiving and processing device, simultaneously during a measurement time duration (¶76-an automated operator system may be configured to continuously monitor operator actions, expressions, responses, physiological data, etc. during travel, as well as automatically generating one or more warnings or alerts to the operator or other responsible party and/or system in response to determination of one of a variety of operator states (e.g., fatigue)) wherein at least two of the EEG electrodes, PPG sensors, fNIRS, optical sensor, audio receiver, and IMU are available and electrically connected to the signal receiving and processing device (¶152-audio recording/voice transcription, electroencephalogram (EEG), functional Near Infrared Spectroscopy (fNIRS), inertial sensors, and accelerometers; ¶151-the information monitoring system 112 collects, determines, or otherwise perceives the real-time characteristics of the operator. As noted above, the information monitoring system 112 maintains a direct connection (e.g., integral with or otherwise hardwired) to the core platform; Fig. 3); by the signal receiving and processing device (¶151-the information monitoring system 112 may include a dedicated controller (e.g., processor) or share a controller (e.g., controller 300 ) of the information collection system 106), and executing, by the signal receiving and processing device (¶90-the data is combined and time synchronized by the core processor 102).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include receiving, by a signal receiving and processing device, simultaneously during a measurement time duration, wherein at least two of the EEG electrodes, PPG sensors, fNIRS, optical sensor, audio receiver, and IMU are available and electrically connected to the signal receiving and processing device; by the signal receiving and processing device, and executing, by the signal receiving and processing device of Chua in order to determine a level of operator fatigue or health state, and/or provide warnings based at least in part on the information (Chua, ¶2).
Regarding claim 5, the combination of Siddharth and Chua teaches the method of claim 1, wherein the activity recognition comprises an image/video activity recognition (Siddharth, page 4, right col., 2nd to last ¶-extract the face region from the frontal body image of the person captured by the camera for each frame) and the attention level computation comprises an image/video attention level computation (Siddharth, Fig. 9 on page 7-single modality attention classification, face); wherein the image/video activity recognition comprising: performing one of feature-based object detection, attribute-based object detection, and ML-based object detection using a trained neural network to detect objects in the image/video signal (Siddharth, page 5, ¶1-these face localized points are then used to calculate 30 different features based on the distances such as between center of the eyebrow to the midpoint of the eye, between the midpoint of nose and corners of the lower lip, between the midpoints of two eyebrows, etc. and angles between such line segments); selecting the detected objects using an object detection confidence system (Siddharth, page 4, right col., 2nd to last ¶-Viola-Jones object detector with Haar-like features [44] to detect the most likely face candidate); and employing one of a trained neural network, SVM, and Random Forest classifier (Chua, ¶93-support vector machines or artificial neural networks) to predict the activity type of the participating activity from the selected-detected objects (Siddharth, page 5, ¶1-map the variation in these features across a trial (which may directly correspond to driver’s attention and driving condition)); wherein the image/video attention level computation comprising: for static activity type, employing one of a trained neural network, SVM, and Random Forest classifier (Chua, ¶93-support vector machines or artificial neural networks), based on analysis of image characteristics, the selected-detected objects, and frame-to-frame changes to predict the attention level from the image/video signal (Siddharth, page 5, left col., ¶1-code the facial expressions and map them to different emotional states [47]. Our goal was to use face localized points similar to the ones used in FACS without identifying the facial expression such as anger, happiness, etc. since they are not highly relevant in driving domain and short time intervals); and for dynamic activity type, comparing the image/video signal to an image scene model for the activity type of the participating activity to estimate the attention level (Siddharth, page 5, ¶1-map the variation in these features across a trial (which may directly correspond to driver’s attention and driving condition)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include employing one of a trained neural network, a Support Vector Machine (SVM), and a Random Forest classifier of Chua in order to determine a level of operator fatigue or health state, and/or provide warnings based at least in part on the information (Chua, ¶2).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Siddharth in view of Chua as applied to claim 1 above, and further in view of Murray (NPL “Adaptive Filtering Methods for Identifying Cross-Frequency Couplings in Human EEG” published in 2013) and Adams (US 20160317097 filed on 12/17/15).
Regarding claim 2, the combination of Siddharth and Chua teaches the method of claim 1. However, the combination of Siddharth and Chua does not teach wherein the pre-processing of the received multi-modal signals comprising: discarding one or more signal segments in the multi-model multi-modal signals having amplitudes below a minimum signal amplitude threshold or having continuous active durations shorter than a minimum signal active duration threshold; if the EEG signal is received: reducing AC electrical frequency interferences in the EEG signal using one or more notch filters; if the PPG signal is received: reducing AC electrical frequency interferences in the PPG signal using one or more notch filters; if the PPG signal and inertial measurement signal are received: determining changes of physical movement of the subject from the inertial measurement signal during the measurement time duration; for any time segment during which the change of physical movement of the subject exceeds a maximum change of movement threshold, discarding the corresponding PPG signal segment; if the image/video signal and inertial measurement signal are received: determining changes of physical movement of the subject from the inertial measurement signal during the measurement time duration; for any time segment during which the change of physical movement of the subject exceeds a maximum change of movement threshold, discarding the corresponding image/video signal segment; and if the audio signal is received: filtering out background ambient noise of the audio signal.
Murray teaches wherein the pre-processing of the received multi-modal signals comprising: discarding one or more signal segments in the multi-model multi-modal signals having amplitudes below a minimum signal amplitude threshold or having continuous active durations shorter than a minimum signal active duration threshold (page 3, left col., ¶1-a threshold of 680 mV for artifact rejection was used); if the EEG signal is received: reducing AC electrical frequency interferences in the EEG signal using one or more notch filters (page 3, left col., ¶2-then, signals from these electrodes were re-sampled from 500 Hz to 250 Hz and the power line interference at 60 Hz was canceled with a narrow notch filter; MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim).
Murray relates to adaptive frequency tracking appears to improve the measurements of cross-frequency couplings through precise extraction of neuronal oscillations (Abstract).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include wherein the pre-processing of the received multi-modal signals comprising: discarding one or more signal segments in the multi-model multi-modal signals having amplitudes below a minimum signal amplitude threshold or having continuous active durations shorter than a minimum signal active duration threshold; if the EEG signal is received: reducing AC electrical frequency interferences in the EEG signal using one or more notch filters of Murray in order for more reliable analysis (Murray, page 2, left col., ¶2).
While Siddharth teaches PPG signals (Table 1 on page 11)), the combination fails to teach if the PPG signal is received: reducing AC electrical frequency interferences in the PPG signal using one or more notch filters.
Adams teaches if the PPG signal is received: reducing AC electrical frequency interferences in the PPG signal using one or more notch filters (¶70-the ALP filter 700 may be used as an adaptive notch filter for filtering the optical signal detected by an optical sensor, such as the sensor 104 described above, from artifacts caused by the motion, which motion is detected by an accelerometer, e.g. the accelerometer 110 described above; ¶70; MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim).
Adams relates to the field of digital signal processing, in particular to digital signal processing for tracking a heartbeat frequency in a noisy environment (¶2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include if the PPG signal is received: reducing AC electrical frequency interferences in the PPG signal using one or more notch filters of Adams in order for more accurate heart rate measurements (Adams, ¶5).
The Examiner notes that the remaining “if” statement limitations are not required, as independent claim 1 only required two data signals. See below:
if the PPG signal and inertial measurement signal are received: determining changes of physical movement of the subject from the inertial measurement signal during the measurement time duration; for any time segment during which the change of physical movement of the subject exceeds a maximum change of movement threshold, discarding the corresponding PPG signal segment (MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim); if the image/video signal and inertial measurement signal are received: determining changes of physical movement of the subject from the inertial measurement signal during the measurement time duration; for any time segment during which the change of physical movement of the subject exceeds a maximum change of movement threshold, discarding the corresponding image/video signal segment (MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim); and if the audio signal is received: filtering out background ambient noise of the audio signal (MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Siddharth in view of Chua as applied to claim 1 above, and further in view of Newlon (WO 2020037332 filed on 10/1/19).
Regarding claim 3, the combination of Siddharth and Chua teaches the method of claim 1, wherein the activity recognition comprises an EEG activity recognition (Siddharth, page 2, right col., last ¶-cognitive processes pertaining to attention and mental load such as while driving, processed EEG data, we employed two distinct and novel methods to extract EEG features that capture the interplay between various brain regions to map human cognition), employing one of a trained neural network, a Support Vector Machine (SVM), and a Random Forest classifier (Chua, ¶93-support vector machines or artificial neural networks), and the attention level computation comprises an EEG attention level computation (Siddharth, page 8, left col., ¶1-EEG performs the best among the four modalities for driver attention classification).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include employing one of a trained neural network, a Support Vector Machine (SVM), and a Random Forest classifier of Chua in order to determine a level of operator fatigue or health state, and/or provide warnings based at least in part on the information (Chua, ¶2).
However, the combination of Siddharth and Chua does not teach wherein the EEG activity recognition comprising: converting the EEG signal to a brainwave plot; identifying a representative pattern of the brainwave plot; and to predict the activity type of the participating activity from the representative pattern of the brainwave plot; and wherein the EEG attention level computation comprising: based on frequency analysis on the representative pattern of the brainwave plot to predict the attention level.
Newlon teaches wherein the EEG activity recognition comprising: converting the EEG signal to a brainwave plot (¶11-the brainwave signal may be indicative of an electrical activity of a brain of the learner, and may comprise an electroencephalography (EEG) signal); identifying a representative pattern of the brainwave plot (¶14-analyzing the at least one characteristic of the brainwave signal comprises analyzing one of a waveform, a frequency, a frequency distribution, an amplitude, and a periodicity of the brainwave signal); and to predict the activity type of the participating activity from the representative pattern of the brainwave plot (¶79-analyzing one or more outputs of EEG algorithm's) that measure different cognitive states such as focus or relaxation, such that the neurofeedback training reinforces one or more of these states. The algorithm(s) to measure these states may be developed by generating machine learning based models of EEG signals that predict the likelihood that a user is in one of these states; ¶677-the algorithm may be based on or driven by advanced machine learning techniques or artificial intelligence-based techniques); and wherein the EEG attention level computation comprising: based on frequency analysis on the representative pattern of the brainwave plot to predict the attention level (¶79-generating machine learning based models of EEG signals that predict the likelihood that a user is in one of these states; ¶78-analyzing one or more frequency band(s) of the brainwaves. For example, the lower frequency bands may be associated with relaxation and daydreaming, the middle frequency bands may be associated with focused thinking and problem solving, and the higher frequency bands may be indicative of anxiety, hyper vigilance, and agitation).
Newlon relates generally to a brain-machine interface, and more particularly, to neuro-feedback training systems and methods for personalized learning and teaching experience using biometric data of a user (¶2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include wherein the EEG activity recognition comprising: converting the EEG signal to a brainwave plot; identifying a representative pattern of the brainwave plot; and to predict the activity type of the participating activity from the representative pattern of the brainwave plot; and wherein the EEG attention level computation comprising: based on frequency analysis on the representative pattern of the brainwave plot to predict the attention level of Newlon in order for personalized learning and teaching experience using biometric data of a user (Newlon, ¶2).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Siddharth in view of Chua as applied to claim 1 above, and further in view of Li (CN 111345800 published on 6/30/20) and Olivier (US 20170238875 filed on 8/6/15).
Regarding claim 4, the combination of Siddharth and Chua teaches the method of claim 1,
wherein the activity recognition comprises an PPG activity recognition (Siddharth, page 4, left col., ¶2- heart-rate variability (HRV) has shown to be a good measure for classifying cognitive states such as emotional valence and stress, the PPG data so obtained was then scaled between 0 and 1 and then a peak-detection algorithm [38] was applied to find the inter-beat intervals (RR) for the calculation of HRV) and the attention level computation comprises an PPG attention level computation (Siddharth, page 7, Fig. 9-single modality classification performance, PPG); wherein the PPG activity recognition comprising: extracting motion artifact information from the PPG signal (Siddharth, page 4, ¶2-a moving-average filter with a window length of 0.25 seconds for filtering the noise in the PPG data), and employing one of a trained neural network, a SVM, and a Random Forest classifier (Chua, ¶93-support vector machines or artificial neural networks).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include employing one of a trained neural network, a Support Vector Machine (SVM), and a Random Forest classifier of Chua in order to determine a level of operator fatigue or health state, and/or provide warnings based at least in part on the information (Chua, ¶2).
However, the combination of Siddharth and Chua does not teach to predict the activity type of the participating activity from the motion artifact information; and wherein the PPG attention level computation comprising: based on pulse frequency and heart rate variability analysis to predict the attention level from the PPG signal generated and received through only a single channel of the PPG sensors; or if the measurement signal of oxygen concentration of neurons in the subject's brain generated and received through the fNIRS is received, predicting the attention level from computation of changes of oxygen concentration of neurons extracted from the measurement signal of oxygen concentration of neurons in the subject's brain generated and received through the fNIRS.
Li teaches wherein the PPG attention level computation comprising: based on pulse frequency and heart rate variability analysis to predict the attention level from the PPG signal generated and received through only a single channel of the PPG sensors (page 4, ¶3-use the input feature matrix and output feature matrix of multiple PPG signal sample sequences to build a sample set of random forest decision tree; build a random forest decision tree model, the input of the random forest decision tree model is the input feature matrix, and the output of the random forest decision tree model To predict the value of attention, use the sample set for machine learning to obtain a trained decision tree model; page 2, ¶10-preprocess the PPG signal sequence to obtain multiple PPG signal subsequences, use the time domain, frequency, and nonlinear characteristics of the PPG signal subsequence to construct the feature vector of the PPG signal subsequence; page 5, ¶5-the current ECG signal measurement attention is mainly based on heart rate variability (HRV), which considers the changes in the heartbeat cycle, while the PPG signal is formed by the heartbeat (vibration) propagating along the arteries and blood flow to the outer periphery. When the blood pressure is relatively normal, the PPG fluctuations and ECG fluctuations have different waveforms, but the frequencies are close. Therefore, the time domain characteristics of the PPG signal can also collect such characteristics) or if the measurement signal of oxygen concentration of neurons in the subject's brain generated and received through the fNIRS is received, predicting the attention level from computation of changes of oxygen concentration of neurons extracted from the measurement signal of oxygen concentration of neurons in the subject's brain generated and received through the fNIRS (MPEP 2111.04: The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. For example, assume a method claim requires step A if a first condition happens and step B if a second condition happens. If the claimed invention may be practiced without either the first or second condition happening, then neither step A or B is required by the broadest reasonable interpretation of the claim).
Li relates to the field of learning attention detection, and specifically relates to a learning attention detection method and system in a MOOC environment (page 1, ¶2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include wherein the PPG attention level computation comprising: based on pulse frequency and heart rate variability analysis to predict the attention level from the PPG signal generated and received through only a single channel of the PPG sensors; or if the measurement signal of oxygen concentration of neurons in the subject's brain generated and received through the fNIRS is received, predicting the attention level from computation of changes of oxygen concentration of neurons extracted from the measurement signal of oxygen concentration of neurons in the subject's brain generated and received through the fNIRS of Li in order to reflects the change of attention through the change of similar continuous pulse period intervals (Li, page 3, ¶6).
While the combination of Siddharth, Chua, and Li teaches a PPG activity recognition (Siddharth, page 4, left col., ¶2- heart-rate variability (HRV) has shown to be a good measure for classifying cognitive states such as emotional valence and stress, the PPG data so obtained was then scaled between 0 and 1 and then a peak-detection algorithm [38] was applied to find the inter-beat intervals (RR) for the calculation of HRV), the combination fails to teach to predict the activity type of the participating activity from the motion artifact information.
Olivier teaches to predict the activity type of the participating activity from the motion artifact information (¶14-a model that predicts HR changes based on an inferred activity level (typically from an accelerometer channel) to predict a likely HR trajectory under conditions where the HR signal can not be accurately separated from the motion artifact signal, allowing for a smooth crossing of the predicted HR and motion frequencies during exercise).
Olivier relates to the field of non-invasive monitoring of physiological parameters. More specifically, a system and method is introduced by which the accuracy of a heart rate prediction from sensor data can be improved under conditions where movement distorts the signal (¶1).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include predicting the activity type of the participating activity from the motion artifact information of Olivier in order to provide more accurate heart rate predictions or how it can be utilized to infer the physiological load for different exercise or rest states (Olivier, ¶43).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Siddharth in view of Chua as applied to claim 1 above, and further in view of Bi (CN 112869743 published 6/1/21).
Regarding claim 7, the combination of Siddharth and Chua teaches the method of claim 1, wherein the activity recognition comprises an inertial measurement activity recognition (Chua, ¶47- motion tracking of arm movements is also used as indicators of operator activity). However, the combination of Siddharth and Chua does not teach the attention level computation comprises an inertial measurement attention level computation; wherein the inertial measurement activity recognition comprising: employing one of a trained neural network, SVM, and Random Forest classifier to predict the activity type of the participating activity from the inertial measurement signal; wherein the inertial measurement attention level computation comprising: comparing the inertial measurement signal to a movement model for the activity type of the participating activity to estimate the attention level.
Bi teaches the attention level computation comprises an inertial measurement attention level computation (page 2, ¶1-motion intent analysis models in the two attention states to predict whether there is motion intent in the two attention states. Recognizing the state of attention during the exercise task through the EEG signals can obtain real-time feedback of the neurological attention state); wherein the inertial measurement activity recognition comprising: employing one of a trained neural network, SVM, and Random Forest classifier to predict the activity type of the participating activity from the inertial measurement signal (page 9, ¶3-a new adaptive system model is proposed, which firstly judges the current concentration state of the person before estimating the initial intention of the movement; page 5, ¶7-Support Vector Machine (SVM)); wherein the inertial measurement attention level computation comprising: comparing the inertial measurement signal to a movement model for the activity type of the participating activity to estimate the attention level (page 2, ¶4-S1. Recognize the attention state during the motor task, and judge whether there is cognitive distraction in the attention state).
Bi relates to the technical field of neuroscience, and particularly relates to a method for neural analysis of the initiation intention of a movement considering cognitive distraction (page 1, ¶2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Siddharth to include the attention level computation comprises an inertial measurement attention level computation; wherein the inertial measurement activity recognition comprising: employing one of a trained neural network, SVM, and Random Forest classifier to predict the activity type of the participating activity from the inertial measurement signal; wherein the inertial measurement attention level computation comprising: comparing the inertial measurement signal to a movement model for the activity type of the participating activity to estimate the attention level of Bi in order to help to understand the neural activity of the human body during exercise (Bi, page 7, ¶4).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL “Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces”: 3.1. EEG and Eye Movement: Eye-movement signals allow us to determine what is attracting a user’s attention and observe their subconscious behaviors.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LAURA HODGE whose telephone number is (571) 272-7101. The examiner can normally be reached M-F: 8:00 am-5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, UNSU JUNG can be reached at (571) 272-8506. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.N.H./Examiner, Art Unit 3792
/AMANDA L STEINBERG/Examiner, Art Unit 3792
Prosecution Insights