DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of Applicant's claim for priority to the following application(s):
* PCT/US2022/026714 filed on 28 April 2022
* 63180810 filed on 28 April 2021
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on the following date(s) is/are entered and considered by Examiner:
* 14 June 2024
* 04 March 2024
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-22 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter.
Claim 1 recites a device comprising a plurality of logic limitations.
When read in light of the Specification as originally filed, the broadest reasonable interpretation of this claim would include software per se, e.g. mere collection of data and/or instructions.
Claims 2-15 fail to remedy the deficiencies of parent claim 1, and are also therefore rejected for at least the same rationale as applied to parent claim 1 above, and incorporated herein.
Claim(s) 16-22 recite(s) substantially similar limitations as those of claim(s) 1-15 above, and are therefore rejected for substantially similar rationale as applied above, and incorporated herein.
Claim(s) 1-22 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract idea without significantly more.
Claim 1 recites:
A device, comprising:
a first modality processing logic to process a first data modality from a first type of sensor to output a first data representation comprising a first set of features;
a second modality processing logic to process a second data modality from a second type of sensor to output a second data representation comprising a second set of features;
modality combination logic to process the first and second data representations to output a combined data representation comprising products of the first and second set of features;
relevance determination logic to identify the relevance of each of the products of the first and second features to a mental health diagnosis; and
diagnosis determination logic to determine a mental health diagnosis based on the relevance of the products of the first and second set of features to the mental health diagnosis.
Step 1:
The claim as a whole falls within at least one statutory category, i.e. a process, machine, manufacture, or composition of matter.
Step 2A Prong One:
The highlighted portion, as drafted, is a process that, under its broadest reasonable interpretation, falls under “Certain methods of organizing human activity” because the step of performing the various recited steps to diagnose a patient are traditionally performed by a physician when treating a patient, i.e. fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions). MPEP 2106.04(a)(2)(II)
The highlighted portion, as drafted, is a process that, under its broadest reasonable interpretation, falls under “Mental processes”.
The claim does not recite any computer and/or other structure for performing the recited steps. Therefore, the steps may be performed in the human mind either mentally or with pen and paper, e.g. by looking at data and thinking about the results.
Accordingly, these limitations have been found to be directed towards concepts performed in the human mind (including an observation, evaluation, judgment, opinion). MPEP 2106.04(a)(2)(III)
The different categories of abstract ideas are being considered together as one single abstract idea. MPEP 2106.04(II)(B)
Dependent claim(s) recite(s) additional subject matter which further narrows or defines the abstract idea embodied in the claims (such as claim(s) 2-15 reciting limitations further defining the abstract idea, which may be performed in the mind but for recitation of generic computer components, and/or may be a method of managing relationship or interactions between people).
Step 2A Prong Two:
This judicial exception is not integrated into a practical application. In particular, the claim recites the following additional element(s), if any:
The claim does not recite any additional element.
The additional element(s) do(es) not integrate the abstract idea into a practical application, other than the abstract idea per se, because no additional element was recited.
Dependent claim(s) recite(s) additional subject matter which amount to limitation(s) consistent with the additional element(s) in the independent claims (such as claim(s) 2, 13 reciting various sensors, including a 3D camera, used for data collection, claim 5-12 reciting various machine learning components, additional limitation(s) which amount(s) to invoking computers as a tool to perform the abstract idea without materially affecting how the abstract idea is performed).
Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation and do not impose a meaningful limit to integrate the abstract idea into a practical application.
Accordingly, the additional elements do not integrate the judicial exception into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Accordingly, the claim recites an abstract idea.
Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than mere instructions to apply an exception, add insignificant extra-solution activity to the abstract idea, and/or generally link the abstract idea to a particular technological environment or field of use.
The claim recites no additional elements.
Dependent claims recite additional subject matter which amount to limitations consistent with the additional elements in the independent claims. MPEP 2106.05(d)(II)(ii))
Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
The claim is not patent eligible.
Claim(s) 16-22 recite(s) substantially similar limitations as those of claim(s) 1-15 above, and are therefore rejected for substantially similar rationale as applied above, and incorporated herein.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-5, 8-22 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Garcia-Ceja (Mental health monitoring with multimodal sensing and machine learning: A survey, cited by Applicant on 04 March 2024).
Claim 1: Garcia-Ceja discloses:
A device (page 3 paragraph 5 illustrating a device), comprising:
a first modality processing logic to process a first data modality from a first type of sensor (Wearable devices like smartphones, smart watches and fitness bands, have a vast variety of embedded sensors. These can include communication devices (WIFI, Bluetooth, etc.), inertial sensors (accelerometer, gyroscope, etc.), physiological sensors (heart rate, dermal activity, etc.) and ambient sensors (ambient pressure, temperature, etc.) to name a few. This opens the possibility of multimodal sensing applications in the healthcare domain. By combining the data from subsets of those sensors, it is possible to infer contextual information such as physical activity, location, mood and social relationships, pg 2 para 4, different types of devices and sensing modalities have been used to monitor for a wide variety of different mental conditions, pg 3 para 5, detect/recognize the current mental state based on real-time and/or previous data collected through different sensing modalities, pg 8 para 4, the three sensing modalities that have been used in automatic mental monitoring systems. External, wearable and software/social media sensing, the sensor data is not sensing the mental state itself, but can be described as the sensing of a behavior that is emerging from underlying physiological alterations [i.e. process multiple data modalities from multiple types of sensors], pg 9 para 3) to output a first data representation comprising a first set of features (after collecting the data is the Exploratory Data Analysis (EDA) and preprocessing. EDA is an approach that helps to better understand the data. Many machine learning algorithms require compact representations of the data instead of sensor raw signals. These representations are often in the form of feature vectors which are numerical n-dimensional vectors that represent an object. The process of building feature vectors from the original data is called feature extraction and is one of the most important steps for mental states prediction, some of the common extracted features for mental states detection are: arithmetic mean, standard deviation, min, max, skewness, kurtosis, root mean square, power spectrum density, energy, correlation coefficient, etc. By using this approach, each mental state sample can be represented by its corresponding feature vector, pg 15 para 4 to pg 16 para 1, The smart watch could collect physiological signals like heart rate and galvanic skin response and send them to the smartphone via Bluetooth for further processing. The smartphone would then, be in charge of aggregating the data and preprocessing it for feature extraction [i.e. output multiple data representation comprising multiple set of features], pg 19 para 5);
a second modality processing logic to process a second data modality from a second type of sensor (Wearable devices like smartphones, smart watches and fitness bands, have a vast variety of embedded sensors.These can include communication devices (WiFi, Bluetooth, etc.), inertial sensors (accelerometer, gyroscope, etc.), physiological sensors (heart rate, dermal activity, etc.) and ambient sensors (ambient pressure, temperature, etc.) to name a few. This opens the possibility of multimodal sensing applications in the healthcare domain. By combining the data from subsets of those sensors, it is possible to infer contextual information such as physical activity, location, mood and social relationships, pg 2 para 4, different types of devices and sensing modalities have been used to monitor for a wide variety of different mental conditions, pg 3 para 5, detect/recognize the current mental state based on real-time and/or previous data collected through different sensing modalities, pg 8 para 4, the three sensing modalities that have been used in automatic mental monitoring systems. External, wearable and software/social media sensing, the sensor data is not sensing the mental state itself, but can be described as the sensing of a behavior that is emerging from underlying physiological alterations [i.e. process multiple data modalities from multiple types of sensors], pg 9 para 3) to output a second data representation comprising a second set of features (after collecting the data is the Exploratory Data Analysis (EDA) and preprocessing. EDA is an approach that helps to better understand the data. Many machine learning algorithms require compact representations of the data instead of sensor raw signals. These representations are often in the form of feature vectors which are numerical n-dimensional vectors that represent an object. The process of building feature vectors from the original data is called feature extraction and is one of the most important steps for mental states prediction, some of the common extracted features for mental states detection are: arithmetic mean, standard deviation, min, max, skewness, kurtosis, root mean square, power spectrum density, energy, correlation coefficient, etc. By using this approach, each mental state sample can be represented by its corresponding feature vector, pg 15 para 4 to pg 16 para 1, The smart watch could collect physiological signals like heart rate and galvanic skin response and send them to the smartphone via Bluetooth for further processing. The smartphone would then, be in charge of aggregating the data and preprocessing it for feature extraction [i.e. output multiple data representation comprising multiple set of features], pg 19 para 5);
modality combination logic to process the first and second data representations to output a combined data representation comprising products of the first and second set of features (By combining the data from subsets of those sensors, it is possible to infer contextual information such as physical activity, location, mood and social relationships, pg 2 para 4, With the further miniaturization of sensors, they began to be embedded in smaller devices such as bracelets and watches. Since bracelets and watches are in constant contact with the users' skin, physiological signals such as heart rate, galvanic skin response, body temperature, etc., can now be collected by such devices. By combining sensor data from smartphones and smart watches, it is possible to capture more details about users' behavior, pg 11 para 1, heart rate (hr) measurements are often used in combination with Electrodermal Activity (EDA) measurements. It can also be seen that movement sensors (accelerometers, gyroscopes) are likely to be combined with other types of sensors. App, social media and EEG were used in isolation (from the sampled works), pg 12 para 4, there are many types of data sources from which physiological and behavioral data can be collected to make inferences. These data sources can be from software or hardware. Each data source has its own format, measuring units, sampling rate, etc., hence, requiring different preprocessing steps. Predictive models require their input data to be in a predefined format, use a late fusion approach which consists of training classifiers for each sensor and then combining their results to get the final classification, pg 20 para 5 to pg 21 para 1, a multi-view stacking approach which is a type of late fusion method for activity recognition from different types of sensors and also obtained better results than using aggregation [i.e. process multiple data representations to output a combined data representation comprising products of the multiple set of features], pg 21 para 2));
relevance determination logic to identify the relevance of each of the products of the first and second features to a mental health diagnosis (Found significant correlations between physiological responses and subjective evaluations, pg 5 Table 2, The preprocessing step consists of applying filters and transformations to the raw data in order to make it suitable for further analysis. Filtering methods can be applied to reduce noise and remove outliers. Not all features from the feature vector may be relevant or add a significant value to the prediction. In order to determine the importance of each feature, some feature selection algorithms can be applied, thus, reducing the dimensionality of the data [i.e. identify the relevance of each of the products of the multiple features to a mental health diagnosis], pg 15 para 4 to pg 16 para 1); and
diagnosis determination logic to determine a mental health diagnosis based on the relevance of the products of the first and second set of features to the mental health diagnosis (leveraging ubiquitous sensing technologies for mental health care applications, thus, allowing the automatic continuous monitoring of different mental conditions such as depression, anxiety, stress, and so on. This paper surveys recent research works in mental health monitoring systems (MHMS) using sensor data and machine learning, abstract, illustrates the current use of technology (multimodal sensing and machine learning) for automatic and adaptive mental health monitoring, pg 3 para 2, monitoring human mental health by using multimodal sensing devices. The possibility of capturing patient data in real-lime and in real-life conditions represents a feasible tool for mental health monitoring and treatment, pg 3 para 4, a state-change detection algorithm without explicitly recognizing the new state, i.e., detect when there is a change from a default state such that this could trigger a notification to visit a doctor for an exact diagnosis. Given the recent advances in wearable devices, communications and information technologies, it is becoming possible to envision systems capable of detecting mental states in an automatic and timely manner, pg 8 para 4, Clinical validation (the data represents what is it supposed to measure and that it is clinical meaningful) is important as one aim of EMA's is to offer just-in-time interventions based on the collected sensor data, the variation of a mental and behavioral states, and not the current state, that indicates a debut or relapse of a mental health disorder [i.e. determine a mental health diagnosis based on the relevance of the products of the multiple set of features to the mental health diagnosis], pg 21 para 4).
Claim 2: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first and second sensor type each comprise one of:
a camera, a microphone, a MRI scanner, a user interface, a keyboard, an EEG detector, or a plate reader (several important aspects of mental health monitoring systems (MHMS) such as human computer interfaces, practical implementations, legal issues, business models, collection of relevant data through sensors like voice, motion and location, pg 3 para 5, sensing types can be wearable, external and software/social media, pg 3 para 6 to pg 4 para 1, they analyzed electroencephalogram (EEG) signals to analyze the predictability of epileptic seizures, pg 8 para 5, Examples of these type of sensors are video cameras, depth vision cameras, high quality microphones, motion sensors, etc., pg 9 para 4, Information about uploaded photos, social network posts, likes, comments, etc., pg 10 Table 3).
Claim 3: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first and second modality processing logic each further comprise a first and second modality preprocessing logic (Collecting data from more sensors and at higher sampling rates might provide more information for the analyses but will cause devices to decrease their battery life and perhaps discouraging users to participate. At the predictive phase, some or all preprocessing can be performed locally on the devices or send the data to a central server, pg 15 para 1 ).
Claim 4: Garcia-Ceja discloses:
The device of claim 3, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first and second modality preprocessing logic comprises a feature dimensionality reduction model (Another common preprocessing technique is dimensionality reduction which is used to visualize high dimensional data and to reduce the number of variables to make the model training process more computationally efficient. Two common dimensionality reduction techniques are Principal Component Analysis (PCA) and Multidimensional scaling (MOS), pg 15 para 4 to pg 16 para 1).
Claim 5: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first and second modality processing logic comprises at least one of:
a feed-forward neural network, a convolutional neural network, a long short-term memory network (LSTM), or a transformer (An Exploratory Data Analysis (EDA) can also be useful to detect outliers and identify missing values due to sensor malfunctioning. The preprocessing step consists of applying filters and transformations to the raw data in order to make it suitable for further analysis. Filtering methods can be applied to reduce noise and remove outliers. Examples of transformations are scaling, quantization, binarization, and so on, pg 15 para 4 to pg 16 para 1, Some of the supervised learning classifiers that have been used for mental states detection are: Decision trees, Ada Boost, Support Vector Machines (SVM), Naive Bayes, Markov Models, Logistic Regression, K-Nearest Neighbors (KNN), Random Forest, Artificial Neural Networks, Linear Discriminant Analysis, Hidden Markov Models, etc. Decision trees are very common due to its simplicity and interpretability, pg 16 para 3).
Claim 8: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the diagnosis determination logic comprises a supervised machine learning model (In supervised learning, the algorithms are presented with a set of training examples (also known as instances). Some of the supervised learning classifiers that have been used for mental states detection are: Decision trees, AdaBoost, Support Vector Machines (SVM), Naïve Bayes, Markov Models, Logistic Regression, K-Nearest Neighbors (KNN), Random Forest, Artificial Neural Networks, Linear Discriminant Analysis, Hidden Markov Models, etc., pg 16 para 3).
Claim 9: Garcia-Ceja discloses:
The device of claim 8, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the supervised machine learning model comprises a random forest, support vector machine, Bayesian Decision List, linear regression, logistic regression, naive Bayes, linear discriminant analysis, decision tree, k-nearest neighbor, or neural network (Some of the statistical methods used to find associations and differences are linear regression models, correlation analysis, I-tests, analysis of variance (ANOVA), etc., pg 8 para 3, Some of the supervised learning classifiers that have been used for mental states detection are: Decision trees, AdaBoost, Support Vector Machines (SVM), Naive Bayes, Markov Models, Logistic Regression, K-Nearest Neighbors (KNN), Random Forest, Artificial Neural Networks, Linear Discriminant Analysis, Hidden Markov Models, etc., pg 16 para 3).
Claim 10: Garcia-Ceja discloses:
The device of claim 8, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the supervised machine learning model is trained using responses to clinical questionnaires as the outcome label (The models are built using previously collected and labeled training data and then used to detect the mental state type of newly unseen observations, pg 8 para 4, Data labeling. In order to learn and find patterns, machine learning algorithms require training data. They rely on the amount and quality of data to generate good predictive models. The data labeling phase consists of tagging the sensor data with their corresponding ground truth state. The data labeling process will have an impact on training the final predictive models. For off-site mental state monitoring it is common to use self-report questionnaires and/or expert evaluations. The self-report questionnaires are usually filled in using a mobile application which presents the questions in periodic intervals (e.g., once, twice a day, etc.) For mental health-care applications, periodic clinical assessments are conducted in person (at the hospital, clinic, etc.) or by phone, pg 15 para 1).
Claim 11: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first and second modality processing logic is trained separately from the relevance determination logic The data labeling process will have an impact on training the final predictive models. There exist several approaches for tagging the data. One of them is by videotaping the sessions and then tagging the data by visual inspection. This approach is convenient for on-site experiments but not for off-site experiments, label activities for elderly monitoring was based on a Bluetooth headset combined with speech recognition software to perform the annotations. Another labeling technique is shadowing, i.e., an observer gathers notes from the participants while keeping his/her presence unknown. For off-site mental state monitoring it is common to use self-report questionnaires and/or expert evaluations. The self-report questionnaires are usually filled in using a mobile application which presents the questions in periodic intervals (e.g., once, twice a day, etc.) For mental health-care applications, periodic clinical assessments are conducted in person {al the hospital, clinic, etc.) or by phone, pg 15 para 1).
Claim 12: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first and second modality processing logic is trained jointly with the relevance determination logic (The data labeling process will have an impact on training the final predictive models. In order to gather more reliable data some works have used a combination of both: in person and phone assessments, pg 15 para 1).
Claim 13: Garcia-Ceja discloses:
The device of claim 2, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the camera is a three dimensional camera (Examples of these type of sensors are depth vision cameras, pg 9 para 4).
Claim 14: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the mental health diagnosis comprises at least one of:
a psychiatric disorder, a depression, a schizophrenia, an anxiety, a panic disorder, a borderline personality disorder, an obsessive compulsive disorder, a post-traumatic stress disorder, an autism spectrum disorder, a mood disorder in epilepsy, a personality disorder, a cognitive change associated with chemotherapy, an attention deficient hyperactivity disorder (ADHD), a neurodevelopmental disorder, a neurodegenerative disorder, an Alzheimer's disease, or a dementia (allowing the automatic continuous monitoring of different mental conditions such as depression, anxiety, stress, and so on, abstract, We focused on research works about mental disorders/conditions such as: depression, anxiety disorders, bipolar disorder, stress, epilepsy, etc., pg 3 para 2, distinguish between symptoms of ADHD and bipolar disorder in children, pg 11 para 1, used deep transfer learning with a convolutional network to perform early Alzheimer's disease diagnostics from brain images, pg 17 para 1, Systematic reviews on the use of machine learning within psychiatric and neuroscientific research emphasizes the need for a theory-driven machine learning approach, pg 21 para 5).
Claim 15: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the mental health diagnosis comprises a quantitative assessment of a severity of the mental health disorder (Found correlations between bipolar states severity and screen on time, number of calls/day, cell tower ids, pg 5 Table 2, Other aspects to take into account during this phase are the type of study: longitudinal, cohort, cross-sectional, qualitative, quantitative etc. In MHMS the type of study is usually performed as longitudinal and quantitative. An explanation for the focus on quantitative studies using sensors and machine learning might be the fact that for numerical analysis, data including measurements is needed to build automatic predictive models, pg 14 para 1 ).
Claim(s) 16-20 recite(s) substantially similar limitations as those of claim(s) 1 above, and are therefore rejected for substantially similar rationale as applied above, and incorporated herein.
Claim 21: Garcia-Ceja discloses:
The device of claim 17, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first type of sensor comprises a camera, the second type of sensor comprises a microphone, and the third type of sensor comprises a user interface configured to receive textual user input (several important aspects of mental health monitoring systems (MHMS) such as human computer interfaces, practical implementations, legal issues, business models, collection of relevant data through sensors like voice, motion and location, pg 3 para 5, sensing types can be wearable, external and software/social media, pg 3 para 6 to pg 4 para 1, they analyzed electroencephalogram (EEG) signals to analyze the predictability of epileptic seizures, pg 8 para 5, Examples of these type of sensors are video cameras, depth vision cameras, high quality microphones, motion sensors, etc., pg 9 para 4, Information about uploaded photos, social network posts, likes, comments, etc., pg 1 0 Table 3)
Claim 22: Garcia-Ceja discloses:
The device of claim 21, as discussed above and incorporated herein.
Garcia-Ceja further discloses:
wherein the first set of features comprises facial features, the second set of features comprises voice features, and the third set of features comprises textual features (several important aspects of mental health monitoring systems (MHMS) such as human computer interfaces, practical implementations, legal issues, business models, collection of relevant data through sensors like voice, motion and location, pg 3 para 5, Alterations of the autonomic nervous system can also be detected in the human voice, as revealed in several studies on Stress, pg 9 para 3, Examples of these type of sensors are video cameras, depth vision cameras, high quality microphones, motion sensors, etc., pg 9 para 4, Information about uploaded photos, social network posts, likes, comments, etc., pg 10 Table 3, using late fusion to combine different sensors types (heart rate, facial expression, user interaction with the system, skin conductance, eye movement and content features of the images), pg 21 para 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garcia-Ceja in view of Zadeh (Tensor fusion network for multimodal sentiment analysis, cited by Applicant on 04 March 2024).
Claim 6: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja does not disclose:
wherein the modality combination logic comprises a tensor fusion model, the tensor fusion model configured to generate the combined data representation based on an outer product of all of the first set of features and all of the second set of features.
Zadeh discloses:
wherein the modality combination logic comprises a tensor fusion model, the tensor fusion model configured to generate the combined data representation based on an outer product of all of the first set of features and all of the second set of features (Tensor Fusion Network (TFN), Inter-modality dynamics are modeled with a new multimodal fusion approach, named Tensor Fusion, which explicitly aggregates uni-modal, bimodal and trimodal interactions. Intra-modality dynamics are modeled through three Modality Embedding Subnetworks, for language, visual and acoustic modalities, respectively, sec 1 para 6, TFN consists of three major components: Modality Embedding Subnetworks take as input unimodal features, and output a rich modality embedding, sec 4 para 1, Since Tensor Fusion is mathematically formed by an outer product, it has no learnable parameters and we empirically observed that although the output tensor is high dimensional, chances of overfilling are low, sec 4.2 para 3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include the modality combination logic comprises a tensor fusion model, the tensor fusion model configured to generate the combined data representation based on an outer product of all of the first set of features and all of the second set of features of Zadeh within the device of Garcia-Ceja with the motivation of improving patient care by decoding the meaningful information (Zadeh sec 4.2 para 4).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garcia-Ceja in view of Lu (20180143966).
Claim 7: Garcia-Ceja discloses:
The device of claim 1, as discussed above and incorporated herein.
Garcia-Ceja does not disclose:
wherein the relevance determination logic comprises at least one of a feed-forward neural network, or an attention model.
Lu discloses:
wherein the relevance determination logic comprises at least one of a feed-forward neural network, or an attention model (Attention-based visual neural encoder-decoder models use a convolutional neural network (CNN) to encode an input image into feature vectors and a long short-term memory network (LSTM) to decode the feature vectors into a sequence of words. Attention-based models leverage either previous hidden state information of the LSTM or previously emitted caption word(s) as input to the attention mechanism, para 0055, Submitting the image context vector and the current hidden state of the decoder to a feed-forward neural network and causing the feed-forward neural network to emit a next caption word, para 0125).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to include relevance determination logic comprises at least one of a feed-forward neural network, or an attention model of Lu within the device of Garcia-Ceja with the motivation of improving patient care by extracting spatial image features during image captioning (Lu; para 0008).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jean (20040039258) discloses symptom processing to diagnose a patient (Abstract) in a manner similar to those disclosed in the instant pending Specification as originally filed.
Smith (20120096391) discloses using a computer to diagnose a patient (Abstract) in a manner similar to those disclosed in the instant pending Specification as originally filed.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRAN N NGUYEN whose telephone number is (571)272-0259. The examiner can normally be reached Monday-Friday 9AM-5PM Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KAMBIZ ABDI can be reached on (571)272-6702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.N.N./ Examiner, Art Unit 3626
/KAMBIZ ABDI/ Supervisory Patent Examiner, Art Unit 3685