CTNF 18/821,503 CTNF 98475 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Priority Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copies have been filed on 20 February, 2026. Information Disclosure Statement The information disclosure statement (IDS) submitted on 30 August, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Drawings 06-22-07 AIA The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 400 (Figure 4 and 6) 500 (Figure 5 and 6) Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance. 07-30-03-h AIA Claim Interpretation 07-30-03 AIA The following is a quotation of 35 U.S.C. 112(f): (f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof. The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph: An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof. 07-30-05 The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked. As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph: (A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function; (B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and (C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function. Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function. Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function. Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are indicated in the table below, along with corresponding structure and/or lack thereof: Claim Limitation Claim Numbers Structure (PGPUB Citation) Acquisition module 1 and 2 Software or hardware module included in the processor or executed by the processor (¶ 0034) Preprocessing module 1, 5, 6, 7, 8, and 9 Software or hardware module included in the processor or executed by the processor (¶ 0034) Classification module 1, 9, and 10 Software or hardware module included in the processor or executed by the processor (¶ 0034) Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1 – 7, 11 – 13, 14 – 16, and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. When reviewing independent claim 1 , and based upon consideration of all of the relevant factors with respect to the claim as a whole, claim 1 is held to claim an abstract idea without reciting elements that amount to significantly more than the abstract idea and is therefore rejected as ineligible subject matter under 35 U.S.C. 101. The Examiner will analyze claim 1, and similar rationale applies to independent claims 13 and 20. The rationale, under MPEP § 2106, for this finding is explained below: The claimed invention (1) must be directed to one of the four statutory categories, and (2) must not be wholly directed to subject matter encompassing a judicially recognized exception, as defined below. The following two step analysis is used to evaluate these criteria. Step 1: Is the claim directed to one of the four patent-eligible subject matter categories: process, machine, manufacture, or composition of matter? When examining the claim under 35 U.S.C. 101, the Examiner interprets that the claims is related to a machine since the claim is directed to an apparatus for diagnosing a disease . Step 2a, Prong 1: Does the claim wholly embrace a judicially recognized exception, which includes laws of nature, physical phenomena, and abstract ideas, or is it a particular practical application of a judicial exception? The Examiner interprets that the judicial exception applies since claim 1 limitations of “ acquire multi-modal data including at least two types of data among text data, speech data, and image data related to a test subject ”, “ visualize data that is not the image data among the multi-modal data and output image datasets including the image data and the visualized data ”, and “ classify whether the test subject has a specified disease based on the image datasets ” are directed to an abstract idea. The claim is related to mental process by the limitations being a recitation of steps which can easily be performed within the mind of a person and each limitation merely comprises actions that do not require anything beyond what is capable of being performed by a human mind. If the claim recites a judicial exception ( i.e., an abstract idea enumerated in MPEP § 2106.04(a), a law of nature, or a natural phenomenon), the claim requires further analysis in Prong Two. Step 2a, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application? The Examiner interprets that claim 1 limitations do not provide additional elements or combination of additional elements to a practical application since the claim is generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). See, MPEP §2106.04(a), Because a judicial exception is not eligible subject matter, Bilski, 561 U.S. at 601, 95 USPQ2d at 1005-06 (quoting Chakrabarty, 447 U.S. at 309, 206 USPQ at 197 (1980)), if there are no additional claim elements besides the judicial exception, or if the additional claim elements merely recite another judicial exception, that is insufficient to integrate the judicial exception into a practical application. See, e.g., RecogniCorp, LLC v. Nintendo Co., 855 F.3d 1322, 1327, 122 USPQ2d 1377 (Fed. Cir. 2017) ("Adding one abstract idea (math) to another abstract idea (encoding and decoding) does not render the claim non-abstract"). OR Genetic Techs. v. Merial LLC, 818 F.3d 1369, 1376, 118 USPQ2d 1541, 1546 (Fed. Cir. 2016) (eligibility "cannot be furnished by the unpatentable law of nature (or natural phenomenon or abstract idea) itself."). For a claim reciting a judicial exception to be eligible, the additional elements (if any) in the claim must "transform the nature of the claim" into a patent-eligible application of the judicial exception, Alice Corp., 573 U.S. at 217, 110 USPQ2d at 1981, either at Prong Two or in Step 2B. If there are no additional elements in the claim, then it cannot be eligible. In such a case, after making the appropriate rejection (see MPEP § 2106.07 for more information on formulating a rejection for lack of eligibility), it is a best practice for the examiner to recommend an amendment, if possible, that would resolve eligibility of the claim. Step 2b: If a judicial exception into a practical application is not recited in the claim, the Examiner must interpret if the claim recites additional elements that amount to significantly more than the judicial exception. The Examiner interprets that the claims do not amount to significantly more since the claim states only the claim limitations which fall within the judicial exception of a mental process. Furthermore, regarding claim 20 the generic computer components of the memory and processor recited as performing generic computer functions that are well-understood, routine and conventional activities amount to no more than implementing the abstract idea with a computerized system. Claims 2 – 7, 11, 12, and 14 - 16 depending on the independent claims include all the limitation of the independent claim. The Examiner finds that claim s 2 – 4, 6, 7, 11, 12, and 14 - 16 does not state significantly more since the claim only recite “separately extract the image data and the speech data from a video data recoded during diagnoses of the patient” in claim 2 ; “wherein the video data includes one of a facial expression or a body movement of the test subject” in claim 3 ; “wherein the text data includes at least one of text data converted from the speech data extracted from the video data and text extracted from social networking services of the text subject” in claim 4 ; “when the image data is color image data of two or more dimensions, the preprocessing module converts the image data into one-dimensional black and white image data.” in claim 5; “extracts an emotional keywork from the text data and visualizes the extracted emotional keyword as a word cloud” in claim 6 ; “ wherein the preprocessing module displays the extracted emotional keyword in the word cloud with a color and a size according to a frequency of appearance of the extracted emotional keyword and an emotional score of the extracted emotional keyword according to an emotional evaluation dictionary. ” in claim 7 ; “ further acquires other data that is at least one of a heart rate, health data, and a life log of each of patients having the specified disease, and classify whether the test subject has the specified disease further based on the other data ” in claim 11 ; “ wherein the specified disease includes at least one of depression, bipolar disorder, anxiety, depressive disorder, and anxiety disorder ” in claim 12; “in the acquiring of the multi-modal data includes separately extracting, from a video data recorded during diagnosis of the test subject, the image data including at least one of a facial expression or a body movement of the test subject and the speech data” in claim 14 ; “generating the text data by converting the speech data extracted from the video data into text; and extracting the text data from social networking services of the test subject” in claim 15 ; “extracting an emotional keyword from the text data; and visualizing the extracted emotional keyword as a word cloud” in claim 16 ; Thus, claims 2 – 4, 6, 7, 11, 12, and 14 - 16 recite the same abstract idea and claim 5 recites an abstract idea of a mathematical concept and therefore are not drawn to the eligible subject matter as they are directed to the abstract idea without significantly more. Therefore, the Examiner interprets that the claims are rejected under 35 U.S.C. 101. Claim Rejections - 35 USC § 112 07-34-01 Claim 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. 07-34-05 AIA Claim 9 recites the limitation " the number of image datasets " in line 3 - 4 . There is insufficient antecedent basis for this limitation in the claim. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1 – 4, 6 – 8, 10 – 17, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kokkera et al (A. Kokkera, N. Varsha and A. V. Vasanth, "Multimodal Approach for Detecting Depression Using Physiological and Behavioural Data," 2023 3rd International Conference on Pervasive Computing and Social Networking (ICPCSN) , Salem, India, 2023, pp. 53-65, doi: 10.1109/ICPCSN58827.2023.00016., hereinafter “Kokkera”) in view of Schriberg et al (U.S. Patent Publication No. 2024/0170109 A1, hereinafter “Schriberg”) . Regarding claim 1 , Kokkera teaches an apparatus for diagnosing a disease, comprising: an acquisition module configured to acquire multi-modal data including at least two types of data among text data, speech data, and image data related to a test subject (Page 56, Section “B. Convolutional Neural Network”: Depression detection through multi-modal data involves combining different types of data, such as text, speech, and facial expressions, to improve the accuracy of depression diagnosis.; Page 57, Section “VI. Database Design”: In the first step, audio, video, and textual data are obtained from the patient. The audio data is transcribed into text using a speech-to-text converter, and the text data is cleaned by removing stop words and irrelevant information. The video data is analyzed to extract facial features using facial landmark detection and tracking techniques.) ; a preprocessing module configured to visualize data that is not the image data among the multi-modal data (Page 56, Section “B. Convolutional Neural Network”, Col. 2, ¶ 3: Similarly, in a system that combines text and audio data, a CNN model can be trained to analyze transcribed speech and textual data to predict the presence or severity of depression.; Page 57, Col. 1, ¶ 4: In the study, the data was transformed into the sentence level by averaging the audio and video modality features for a given sentence time-stamp.) and output image datasets including the image data and the visualized data (Page 57, Col. 1, ¶ 4: In the study, the data was transformed into the sentence level by averaging the audio and video modality features for a given sentence time-stamp… After the audio and video features for a sentence were fed individually to the highway layers, they were concatenated with the corresponding text feature. Finally, the concatenated vector was passed through the LSTM to obtain the output.) ; and a classification module configured to classify whether the test subject has a specified disease based on the image datasets (Page 59, Section “B. Convolutional Neural Network”, Col. 2, ¶ 2: In a multi-modal system, the output of the CNN model can be a binary classification indicating whether the data contains signs of depression or not, or a continuous score indicating the severity of depression.; Page 59, Section “B. Convolutional Neural Network”, Col. 2, ¶ 3: The output of the model can be a binary classification indicating whether the data contains signs of depression or not, or a continuous score indicating the severity of depression.; Page 58, Col. 1, ¶ 2: In the third step, the trained SVM classifier is applied to new, unseen data to predict the depression status of the individual. The extracted features from the new data are inputted into the SVM classifier, which outputs a prediction of whether the individual is depressed or not. ) . Kokkera does not explicitly teach an acquisition module, a preprocessing module, and a classification module. However, Schriberg does teach an acquisition module, a preprocessing module, and a classification module (¶ 0031: Another aspect of the present disclosure provides a system comprising one or more computer processors (emphasis added)) . Schriberg is considered to be analogous art it pertains to mental health diagnostics using computer systems. Therefore, it would have been obvious to one of ordinary skill in the art to combine the multimodal approach for detecting depression using physiological and behavioral data (as taught by Kokkera) and the method for mental health assessment (as taught by Schriberg) before the effective filing date of the claimed invention. The motivation for this combination of references would be the method of Schriberg uses acoustic analysis of the audiovisual signal to improve sentiment analysis of words and phrases (See ¶ 0288). This motivation for the combination of Kokkera and Schriberg is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III). Regarding claim 2 , the Kokkera and Schriberg combination teaches the apparatus of claim 1. Additionally, Kokkera teaches wherein the acquisition module separately extracts the image data and the speech data from a video data recorded during diagnosis of the test subject (Page 55, Section “III. Dataset Overview”: For the audio modality, the audio recordings of clinical interviews from the DAIC-WOZ dataset were pre-processed to extract features such as pitch, energy, and spectral information… Finally, for the video modality, the video recordings of the clinical interviews were pre-processed to extract facial expressions, head movements, and eye gaze patterns.; Page 57, Col. 1, ¶ 4: In the study, the data was transformed into the sentence level by averaging the audio and video modality features for a given sentence time-stamp…; Examiner’s note: The audio and video dataset are comprised of both the audio and video data from the same interviews. The audio data and video data being separately processed to extract features specific to the mediums they are captured (image or speech).) . Regarding claim 3 , the Kokkera and Schriberg combination teaches the apparatus of claim 2. Additionally, Kokkera teaches wherein the video data includes at least one of a facial expression or a body movement of the test subject (Page 55, Section “III. Dataset Overview”: Finally, for the video modality, the video recordings of the clinical interviews were pre-processed to extract facial expressions, head movements, and eye gaze patterns.; Page 56, Section “Video Modality”: The dataset contained facial features from the videos of the participant. The facial features consisted of 68 2D points on the face, 24 AU features that measure facial activity, 68 3D points on the face, 16 features to represent the subject’s gaze, and 10 features to represent the subject’s pose. This made for a total of 388 video features.) . Regarding claim 4 , the Kokkera and Schriberg combination teaches the apparatus of claim 2. Additionally, Kokkera teaches wherein the text data includes at least one of text converted from the speech data extracted from the video data and text extracted from social networking services of the test subject (Page 55, Section “III. Dataset Overview”: For the text modality, the transcriptions of the clinical interviews were pre-processed to extract features such as word frequencies, part-of-speed tags, and sentiment scores.; Page 56, Section “B. Text Modality”: The textual modality contains the transcript for the whole conversation of the patient with the RA in csv format. Individual sentences have been timestamped and further classified on the basis of the speaker. Expressions like laughter, frowning, etc. have been added in angular brackets as and when they occur (for e.g., "laughter"). Differentiation between long/short pauses has not been made. Only word (not phenome) level segmentation has been recorded.; Page 57, Col 1., ¶ 3: Additionally, LSTMs with gating mechanisms have been applied to natural language text data such as social media posts or clinical notes.; Page 57, Col. 2, ¶ 2: In depression detection systems, BiLSTMs with gating mechanisms have been used to analyze social media posts or clinical notes, which can provide valuable insights into a person's mental state.) . Regarding claim 6 , the Kokkera and Schriberg combination teaches the apparatus of claim 1. Kokkera does not teach wherein the preprocessing module extracts an emotional keyword from the text data and visualizes the extracted emotional keyword as a word cloud. However, Schriberg does teach wherein the preprocessing module extracts an emotional keyword from the text data and visualizes the extracted emotional keyword as a word cloud (¶ 0170: The system may additionally provide the clinician with a “word cloud” or “topic cloud” extracted from a text transcript of the patient's speech.) . Schriberg is considered to be analogous art it pertains to mental health diagnostics using computer systems. Therefore, it would have been obvious to one of ordinary skill in the art to combine the multimodal approach for detecting depression using physiological and behavioral data (as taught by Kokkera) and the method for mental health assessment (as taught by Schriberg) before the effective filing date of the claimed invention. The motivation for this combination of references would be the method of Schriberg uses acoustic analysis of the audiovisual signal to improve sentiment analysis of words and phrases (See ¶ 0288). Regarding claim 7 , the Kokkera and Schriberg combination teaches the apparatus of claim 6. Additionally, Schriberg teaches wherein the preprocessing module displays the extracted emotional keyword in the word cloud with a color and a size according to a frequency of appearance of the extracted emotional keyword (¶ 0170 A word cloud may be a visual representation of individual words or phrases, with words and phrases used most frequently designated using larger font sizes, different colors, different fonts, different typefaces, or any combination thereof. Depicting word or phrase frequency in such a way may be helpful as depressed patients commonly say particular words or phrases with larger frequencies than non-depressed patients) and an emotional score of the extracted emotional keyword according to an emotional evaluation dictionary (¶ 0170: They may talk about feeling worthless or feeling like failures, or use absolutist language, such as “always”, “never”, or “completely.” Depressed patients may also use a higher frequency of first-person singular pronouns (e.g., “I”, “me”) and a lower frequency of second- or third-person pronouns when compared to the general population. The system may be able to train a machine learning algorithm to perform semantic analysis of word clouds of depressed and non-depressed people, in order to be able to classify people as depressed or not depressed based on their word clouds.) . Schriberg is considered to be analogous art it pertains to mental health diagnostics using computer systems. Therefore, it would have been obvious to one of ordinary skill in the art to combine the multimodal approach for detecting depression using physiological and behavioral data (as taught by Kokkera) and the method for mental health assessment (as taught by Schriberg) before the effective filing date of the claimed invention. The motivation for this combination of references would be the method of Schriberg uses acoustic analysis of the audiovisual signal to improve sentiment analysis of words and phrases (See ¶ 0288). Regarding claim 8 , the Kokkera and Schriberg combination teaches the apparatus of claim 1. Additionally, Kokkera teaches wherein the preprocessing module applies a Mel spectrogram technique to the speech data to visualize the speech data (Page 55, Section “Audio Modality”: The audio modality comprises 12 Mel-frequency Cepstral Coefficients (MFCCs), extracted every 10 milliseconds, along with other features like pitch tracking, peak slope, maximal dispersion quotients, and glottal source parameters.) . Regarding claim 10 , the Kokkera and Schriberg combination teaches the apparatus of claim 1. Additionally, Kokkera teaches wherein the classification module includes a three-dimensional single network model based on convolution (Page 56, Section “B. Convolutional Neural Network”, Col. 2, ¶ 4: The CNN model's architecture (as shown in Fig. 2) consists of several layers, including four convolutional layers with different filter sizes and activation functions (ReLU), four max pooling layers used to reduce the spatial dimensions of the output from the convolutional layers, a flatten layer to flatten the output of the last max pooling layer, and two dense (fully connected) layers. The first dense layer uses ReLU as its activation function, while the second dense layer uses sigmoid.) . Regarding claim 11 , the Kokkera and Schriberg combination teaches the apparatus of claim 1. Additionally, Kokkera teaches which further acquires other data that is at least one of a heart rate, health data, and a life log of each of patients having the specified disease, and classify whether the test subject has the specified disease further based on the other data (Page 57, Col. 1, ¶ 2: In depression detection systems, LSTMs with gating mechanisms have been widely used to model temporal dependencies in physiological signals like EEG (electroencephalogram) or HRV (heart rate variability).) . Regarding claim 12 , the Kokkera and Schriberg combination teaches the apparatus of claim 1. Additionally, Kokkera teaches wherein the specified disease includes at least one of depression, bipolar disorder, anxiety, depressive disorder, and anxiety disorder (Abstract: Therefore, the use of depression detection systems has become increasingly crucial in identifying and assessing the presence and severity of depression in various settings, such as primary care offices, mental health clinics, and online platforms.) . The rejection of device claim 1 above applies mutatis mutandis to the corresponding limitations of method claim 13 while noting that the rejection above cites to both device and method disclosures. Regarding claim 14 , the Kokkera and Schriberg combination teaches the method of claim 13. Additionally, Kokkera teaches wherein the acquiring of the multi-modal data includes separately extracting, from a video data recorded during the diagnosis of the text subject the image data including at least one of a facial expression or a body movement of the test subject and speech data (Page 55, Section “III. Dataset Overview”: For the audio modality, the audio recordings of clinical interviews from the DAIC-WOZ dataset were pre-processed to extract features such as pitch, energy, and spectral information… Finally, for the video modality, the video recordings of the clinical interviews were pre-processed to extract facial expressions, head movements, and eye gaze patterns.; Page 57, Col. 1, ¶ 4: In the study, the data was transformed into the sentence level by averaging the audio and video modality features for a given sentence time-stamp…; Examiner’s note: The audio and video dataset are comprised of both the audio and video data from the same interviews. The audio data and video data being separately processed to extract features specific to the mediums they are captured (image or speech).) . Regarding claim 15 , the Kokkera and Schriberg combination teaches the method of claim 13. Additionally, Kokkera teaches wherein the acquiring of the multi-modal data includes: Generating the text data by converting the speech data extracted from the video data into text (Page 55, Section “III. Dataset Overview”: For the text modality, the transcriptions of the clinical interviews were pre-processed to extract features such as word frequencies, part-of-speed tags, and sentiment scores.; Page 56, Section “B. Text Modality”: The textual modality contains the transcript for the whole conversation of the patient with the RA in csv format. Individual sentences have been timestamped and further classified on the basis of the speaker. Expressions like laughter, frowning, etc. have been added in angular brackets as and when they occur (for e.g., "laughter"). Differentiation between long/short pauses has not been made. Only word (not phenome) level segmentation has been recorded.;) ; and Extracting the text data from social networking services of the test subject (Page 57, Col 1., ¶ 3: Additionally, LSTMs with gating mechanisms have been applied to natural language text data such as social media posts or clinical notes.; Page 57, Col. 2, ¶ 2: In depression detection systems, BiLSTMs with gating mechanisms have been used to analyze social media posts or clinical notes, which can provide valuable insights into a person's mental state.) ; The rejection of device claim 6 above applies mutatis mutandis to the corresponding limitations of method claim 16 while noting that the rejection above cites to both device and method disclosures. The rejection of device claim 8 above applies mutatis mutandis to the corresponding limitations of method claim 17 while noting that the rejection above cites to both device and method disclosures. The rejection of device claim 10 above applies mutatis mutandis to the corresponding limitations of method claim 19 while noting that the rejection above cites to both device and method disclosures. The rejection of device claim 1 above applies mutatis mutandis to the corresponding limitations of device claim 20 . For the following device limitations of claim 20 see Schriberg’s teaching on: A memory including at least one instruction (¶ 0030: Another aspect of the present disclosure provides a non-transitory computer readable-medium comprising machine-executable instructions that, upon execution by one or more computer processors, implements any of the foregoing methods described in the above or elsewhere herein.) ; and A processor functionally connected to the memory (¶ 0030: Another aspect of the present disclosure provides a non-transitory computer readable-medium comprising machine-executable instructions that, upon execution by one or more computer processors, implements any of the foregoing methods described in the above or elsewhere herein.) , Wherein when executed, the at least one instruction causes the processor to (¶ 0030: Another aspect of the present disclosure provides a non-transitory computer readable-medium comprising machine-executable instructions that, upon execution by one or more computer processors, implements any of the foregoing methods described in the above or elsewhere herein.) … Schriberg is considered to be analogous art it pertains to mental health diagnostics using computer systems. Therefore, it would have been obvious to one of ordinary skill in the art to combine the multimodal approach for detecting depression using physiological and behavioral data (as taught by Kokkera) and the method for mental health assessment (as taught by Schriberg) before the effective filing date of the claimed invention. The motivation for this combination of references would be the method of Schriberg uses acoustic analysis of the audiovisual signal to improve sentiment analysis of words and phrases (See ¶ 0288) . 07-21-aia AIA Claim s 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kokkera et al (A. Kokkera, N. Varsha and A. V. Vasanth, "Multimodal Approach for Detecting Depression Using Physiological and Behavioural Data," 2023 3rd International Conference on Pervasive Computing and Social Networking (ICPCSN) , Salem, India, 2023, pp. 53-65, doi: 10.1109/ICPCSN58827.2023.00016., hereinafter “Kokkera”) in view of Schriberg et al (U.S. Patent Publication No. 2024/0170109 A1, hereinafter “Schriberg”) and further in view of Li et al (Li X, Du M, Zuo S, Zhou M, Peng Q, Chen Z, Zhou J and He Q (2023) Deep convolutional neural networks using an active learning strategy for cervical cancer screening and diagnosis . Front. Bioinform. 3:1101667. doi: 10.3389/fbinf.2023.1101667, hereinafter “Li”) . Regarding claim 9 , the Kokkera and Schriberg combination teaches the apparatus of claim 1. Kokkera does not teach wherein, at least in an operation of training the classification module (Page 56, Section “B. Convolutional Neural Network”, Col. 2, ¶ 2: Similarly, in a system that combines text and audio data, a CNN model can be trained to analyze transcribed speech and textual data to predict the presence or severity of depression .) , the preprocessing module augments the image datasets using a data augmentation or a k-fold training technique when the number of the image datasets is less than a predetermined number. Kokkera does not teach wherein, at least in an operation of training the classification module, the preprocessing module augments the image datasets using a data augmentation or a k-fold training technique when the number of the image datasets is less than a predetermined number. However, Li teaches wherein, at least in an operation of training the classification module, the preprocessing module augments the image datasets using a data augmentation or a k-fold training technique when the number of the image datasets is less than a predetermined number (Page 03, Section “2.3 Image Augmentation”: As the data samples in this study were limited, we used the python library “imgaug” 0.4.0 to perform the image augmentation to increase the data diversity and better simulate the real data variability while preventing overfitting; Examiner’s note: It would be obvious to one skilled in the art to utilize data augmentation techniques or K-fold techniques to optimize the amount of training that can be done from a smaller dataset or when data is limited. This is a common technique in the art as can also be seen in Lee et al (Shown below in the prior art of record)) . Li is considered to be analogous art it pertains to neural network classification systems. Therefore, it would have been obvious to one of ordinary skill in the art to combine the multimodal approach for detecting depression using physiological and behavioral data (as taught by Kokkera) and the deep convolutional neural network using an active learning strategy (as taught by Li) before the effective filing date of the claimed invention. The motivation for this combination of references would be the method of Li uses an active learning strategy to improve the efficiency and accuracy of image labelling (See abstract and page 2). This motivation for the combination of Kokkera, Schriberg, and Li is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III). The rejection of device claim 9 above applies mutatis mutandis to the corresponding limitations of method claim 18 while noting that the rejection above cites to both device and method disclosures . Allowable Subject Matter 07-43 Claim 5 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and if amended to overcome the 35 U.S.C. § 101 rejection. Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Ma et al (U.S. Patent Publication No. 2025/0213174 A1) teaches a method of inputting multi-modal data comprised of video data and a Mel spectrograph to determine an output of a Parkinson’s classification result. Gu et al (Chinese Patent Publication No. 116110578A) teaches a method for depression diagnosis using multimodal data comprising video, audio, and text data which is input into a system which extracts features from the different data types and performs depressive symptom classification. Lee et al (U.S. Patent Publication No. 2020/0320439 A1) teaches a method for data augmentation when training data for a classifier is imbalanced or when the amount of training data is relatively small given the input data dimension. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW JONES whose telephone number is (703)756-4573. The examiner can normally be reached Monday - Friday 8:00-5:00 EST, off Every Other Friday. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /ANDREW B. JONES/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667 Application/Control Number: 18/821,503 Page 2 Art Unit: 2667 Application/Control Number: 18/821,503 Page 3 Art Unit: 2667 Application/Control Number: 18/821,503 Page 4 Art Unit: 2667 Application/Control Number: 18/821,503 Page 5 Art Unit: 2667 Application/Control Number: 18/821,503 Page 6 Art Unit: 2667 Application/Control Number: 18/821,503 Page 7 Art Unit: 2667 Application/Control Number: 18/821,503 Page 8 Art Unit: 2667 Application/Control Number: 18/821,503 Page 9 Art Unit: 2667 Application/Control Number: 18/821,503 Page 10 Art Unit: 2667 Application/Control Number: 18/821,503 Page 11 Art Unit: 2667 Application/Control Number: 18/821,503 Page 12 Art Unit: 2667 Application/Control Number: 18/821,503 Page 13 Art Unit: 2667 Application/Control Number: 18/821,503 Page 14 Art Unit: 2667 Application/Control Number: 18/821,503 Page 15 Art Unit: 2667 Application/Control Number: 18/821,503 Page 16 Art Unit: 2667 Application/Control Number: 18/821,503 Page 17 Art Unit: 2667 Application/Control Number: 18/821,503 Page 18 Art Unit: 2667 Application/Control Number: 18/821,503 Page 20 Art Unit: 2667 Application/Control Number: 18/821,503 Page 21 Art Unit: 2667 Application/Control Number: 18/821,503 Page 22 Art Unit: 2667
Prosecution Insights