DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Status of claims
Canceled:
20, 25
Amended:
1-19, 21-24
New:
none
Pending:
1-19, 21-24
Withdrawn:
none
Examined:
1-19, 21-24
Independent:
1, 10, 19
Priority
As detailed on the 06/05/2023 filing receipt, this application is a 371 of PCT/GB2021/051904 07/23/2021. This application claims foreign priority to as early as 07/23/2020.
Drawings
The drawings filed 01/18/2023 are accepted.
Information Disclosure Statement
The Information Disclosure Statements filed on 01/27/2023, 04/19/2023, 12/26/2023, 04/26/2024, 09/07/2024, 11/11/2024, 05/23/2025 and 02/02/2026 are in compliance with the provisions of 37 CFR 1.97 and have been considered in full. The Information Disclosure Statement filed on 09/10/2024 has been considered in part because references that are lined through are missing page number and/or dates. A signed copy of the list of references cited from each IDS is included with this Office Action.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Analysis of claims in Step 1.
Step 1: Are the claims directed to a 101 process, machine, manufacture, or composition of matter (MPEP 2106.03)?
Independent claim 1 is directed to a 101 process, here a "computer implemented cognitive state estimation method," with process steps such as "receiving…"
Independent claim 10 is directed to a 101 process, here a "computer implemented cognitive state estimation method," with process steps such as "receiving…"
[Step 1: claims 1-18: YES]
In accordance with MPEP § 2106, claims found to recite statutory subject matter (Step 1: YES) are then analyzed to determine if the claims recite any concepts that equate to an abstract idea, law of nature or natural phenomenon (Step 2A, Prong 1). In the instant application, the claims recite the following limitations that equate to an abstract idea:
Mental processes recited include:
Independent Claim 1 recite: "…extracting at least one behavior primitive from the recording… producing a behavior map from the at least one behavior primitive; producing a context map from the at least one context primitive; estimating a cognitive state of the user using data derived from the behavior map and the context map." Extracting behavior primitive, producing maps and estimating a cognitive state are acts of evaluating, analyzing, observing and judging data that could be practically performed in the human mind and/or with pen and paper.
Claim 3 recite: "…estimating a behavior by combining the behavior map and the context map to create a joint behavior and context map." Estimating and combining maps are acts of evaluating, analyzing and judging data that could be practically performed in the human mind and/or with pen and paper.
Claim 4 recite: "…combining the behavior map and the context map comprises multiplying the behavior map and the context map..." Combining maps is an act of evaluating, analyzing and judging data that could be practically performed in the human mind and/or with pen and paper.
Claim 5 recite: "...estimating the behavior of the user comprises using data derived from the behavior map, context map and the joint behavior and context map." Estimating is an act of evaluating, analyzing and judging data that could be practically performed in the human mind and/or with pen and paper.
Claim 6 recite: "... performing a Fourier transform..." Performing a Fourier transform is an act of evaluating, analyzing and judging data that could be practically performed in the human mind and/or with pen and paper.
Claim 7 recites: "…determining if the behavior map or the context map has fewer primitives than the other, and inserting additional all -zero primitives to the map having fewer primitives where primitives are missing, to create a behavior map and a context map of equal size." Determining is an act of evaluating, analyzing, observing and judging data that could be practically performed in the human mind and/or with pen and paper.
Independent Claim 10 recites: "…estimating a cognitive state from data comprising the behavior map, wherein the estimating comprises using at least one neural process on the data." Estimating is an act of evaluating, analyzing and judging data that could be practically performed in the human mind and/or with pen and paper.
Claim 11 recites: “…estimating the cognitive state of the user uses a neural process.”
Mathematical concepts recited include:
Claim 6 recite: "…producing the behavior map comprises performing a Fourier transform on the at least one behavior primitive and wherein producing the context map comprises performing a Fourier transform on the at least one context primitive." Fourier transform are mathematical concepts and/or formulas.
Claim 9 recite: "…estimating the behavior uses a convolutional neural network..." This limitation recites a mathematical concepts and/or formulas.
Claims 1, 3-7, and 10-11 recite mental processes because the limitations are involved with extracting a behavior primitive, producing and combining behavior and context maps, estimating cognitive states and behaviors, performing Fourier Transform and determining if which map has the fewer primitive require observing, evaluating, judging and analyzing data as indicated above. Acts of evaluating and analyzing data could be practically performed in the human mind and/or with pen and paper because they merely require making observations, evaluations, judgments, and opinions (See MPEP 2106.04(a)(2) subsection III). Although, claims 1 and 10 recite a computer implemented method, there are no additional limitations to indicate that anything other than a generic computer is required. However, merely requiring that the steps are carried out with a generic computer does not negate the mental nature of these steps and equates rather to merely using a computer as a tool to perform the mental process. Therefore, under the broadest reasonable interpretation, the indicated claims above can be practically carried out in the human mind or with pen and paper as claimed, which falls under the "Mental processes" grouping of abstract ideas.
Claims 6 and 9 recite mathematical concepts and formulas as indicated above. The limitations of performing Fourier transform and estimating the behavior using a convolutional neural network are mathematical concepts and/or formulas that falls under the “mathematical concepts” grouping of abstract ideas.
As such, claims 1-18 recite an abstract idea (Step 2A, Prong 1: YES).
Claims found to recite a judicial exception under Step 2A, Prong 1 are then further analyzed to determine if the claims as a whole integrate the recited judicial exception into a practical application or not (Step 2A, Prong 2). The above indicated judicial exceptions are not integrated into a practical application because the claims do not recite an additional elements that apply, rely on or use the judicial exception in such a manner to amount to integration into a practical application. For example, there are no limitations that reflect an improvement to technology or applies or uses the recited judicial exception in some other meaningful way. Rather, the instant claims recite additional elements that equate to mere instructions to implement an abstract idea or insignificant extra solution activity. Specifically, the instant claims recite the following additional elements:
Independent Claims 1 and 10 recite "computer implemented," "receiving a recording of a user" and "extracting at least one behavior primitive from the recording"
The elements of claims 1 and 10 as indicated above equate to insignificant extra solutional activities of data gathering. Data gathering serves as input to the recited judicial exception in the claims. Claims 1 and 10 recite a computer-implemented method. The elements of these claims equate to generic computer components. Claims 1 and 10 invoke the computer components merely as tools to execute the abstract idea. The use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. (see MPEP 2106.05(f)). Additionally, the listed additional elements are mere instructions to apply an exception because they recite no more than an idea of a solution or outcome and does not recite a technological solution to a technological problem. (See MPEP 2106.05(f)(1)). As such, as currently recited, the claims do not appear to recite an improvement to technology or apply or use the recited judicial exception in some other meaningful way. Therefore, claims 1-18 are directed to an abstract idea (Step 2A, Prong 2: NO).
Claims found to be directed to a judicial exception are then further evaluated to determine if the claims recite an inventive concept that provides significantly more than the judicial exception itself (Step 2B). The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims recite additional elements that equate to well-understood, routine and conventional activities, insignificant extra-solution activity or mere instructions to implement the abstract idea on a generic computer. The instant claims recite the following additional elements:
Independent Claims 1 and 10 recite "computer implemented," "receiving a recording of a user" and "extracting at least one behavior primitive from the recording."
The additional elements indicated above do not comprise an inventive concept when considered individually or as an ordered combination that transforms the claimed judicial exception into a patent-eligible application of the judicial exception. The limitations equate to mere data gathering activities, which are insignificant extra solutional activities. As explained by the Supreme Court, the addition of insignificant extra-solution activity does not amount to an inventive concept, particularly when the activity is well-understood or conventional. (see MPEP 2106.05(g)). Also, limitations that equate to mere data gathering and outputting via generic computer components, such as receiving data at a computer or outputting data, amount to insignificant extra-solution activity as set forth by the courts in Mayo, 566 U.S. at 79, 101 USPQ2d at 1968 and OIP Techs., Inc, v, Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1092-93 (Fed. Cir. 2015). Also, the use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more as identified by the courts in Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Additionally, it is noted that behavior primitive is interpreted to correspond to facial action units or facial action representation as taught by Zhi ("A comprehensive survey on automatic facial action unit analysis." The Visual Computer 36.5 (2020): 1067-1093. Published online: 26 June 2019). Zhi discloses that methods for extracting facial representation is a known method as disclosed in section 3.2, page 1071. Zhi also discloses extracting facial actions from videos in section 3.2.2, page 1074. Therefore, the claims do not amount to significantly more than the judicial exception itself (Step 2B: No). As such, claims 1-18 are not patent eligible.
Matter belonging to no 101 statutory category -- claims 19 and 21-24
Claims 19 and 21-24 are rejected under 35 USC 101 because the claimed inventions are directed to non-statutory subject matter.
Claim 19 is to "a system," which is not, in all embodiments within a BRI, interpreted as belonging to any one particular category listed in 101. In a BRI, the claim reads on data and/or software comprising no structure other than data and/or software. The claim is not recited as a process, and the claim is not limited to any particular structure as a 101 machine or manufacture. The claim reads on transitory propagating signals which are not proper patentable subject matter because it does not fit within any of the four statutory categories of invention (In re Nuijten, Federal. Circuit, 2006).
In a BRI, none of the recited "processor" is clearly a non-transitory element. In a BRI, each reads on information only in the form software and not clearly stored software.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-14 and 17-18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Song ("Spectral representation of behaviour primitives for depression analysis." IEEE Transactions on Affective Computing 13.2 (2020): 829-844.; as cited on the attached 892 form)
Regarding independent claim 1, Song teaches receiving a recording of a user with “Each clip records a participant doing a series of tasks, including sustained vowel phonation, sustained loud vowel phonation, sustained smiling vowel phonation, speaking out loud while solving a task, counting from 1 to 10, etc. All participants are German speakers and each of them do the same tasks in the same order during the video recording. The length of these videos ranges from 20 minutes to 50 minutes with an average of 25 minutes.” (page 836, col. 2, para. 2).
Song teaches at least one context primitive, each context primitive comprising a time series of context descriptors with “The length of these videos ranges from 20 minutes to 50 minutes with an average of 25 minutes.” (page 836, col. 2, para. 2).
Song teaches extracting at least one behavior primitive from the recording, each behavior primitive comprising a time series of behavior descriptors with “In this paper, we employed the OpenFace 2.0 toolkit to automatically detect intensities of 17 AUs (AU01, AU02, AU04, AU05, AU06, AU07, AU09, AU10, AU12, AU14, AU15, AU17, AU20, AU23, AU25, AU26 and AU45), 6 gaze direction descriptors and 6 head pose descriptors (detailed explanation is in Fig. 7) resulting in a 29-D frame-wise human representation.” (page 837, col. 1, para. 2) and “To achieve this, we first extract a set of automatically detected human behaviour primitives to represent a video, allowing the high dimensional videos to be significantly reduced to a low dimensional multichannel time-series signal (Section 3.1). In Section 3.2, we propose two spectral representations as the video-level descriptors for multi-channel behaviour signals, which can not only encode a time-series data of arbitrary length into fixed-size representations but also retain multi-scale temporal information from the original time-series data. Finally, we show how to apply the generated spectral representations to depression analysis (Section 3.3).” (page 832, col. 2, para. 1).
Song teaches producing a behavior map from the at least one behavior primitive with Fig. 6. Fig. 6. Visualization of the most important facial actions and their face region for depression analysis. (page 837)
Song teaches producing a context map from the at least one context primitive with Table 1. Table 1 Analysis of Human Interpretable Temporal AU Activation Patterns (Page 838).
Song teaches estimating a cognitive state of the user using data derived from the behavior map and the context map with “…we independently evaluate the performance of each human behaviour primitive on depression severity estimation. To do so, we trained separate models from the spectral vectors of each behaviour primitive. The results are reported on the AVEC 2013 dataset.” (page 838, col. 1, para. 1)
Regarding claim 2, Song teaches wherein the behavior comprises a depression score with “For both datasets, the frame rate of videos were set to 30 frames per second with resolution of 640 480, and each clip is labeled with a Beck-Depression Inventory (BDI II) score indicating the depression severity. These scores range from a minimum of 0 to a maximum of 63.” (page 837, col. 1, para. 1)
Regarding claim 3, Song teaches estimating a behavior by combining the behavior map and the context map to create a joint behavior and context map with Table 1. Table 1 Analysis of Human Interpretable Temporal AU Activation Patterns (Page 838).
Regarding claim 4, Song teaches wherein combining the behavior map and the context map comprises multiplying the behavior map and the context map with Table 1. Table 1 Analysis of Human Interpretable Temporal AU Activation Patterns (Page 838).
Regarding claim 5, Song teaches wherein estimating the behavior comprises using data derived from the behavior map, context map and the joint behavior and context map with Fig. 1 (page 831). Fig. 1 depicts the pipeline used to predict depression.
Regarding claim 6, Song teaches wherein producing the behavior map comprises performing a Fourier transform on the at least one behavior primitive and wherein producing the context map comprises performing a Fourier transform on the at least one context primitive with “As long-term temporal dynamics are important asset for depression analysis, the proposed approach first employs Fourier Transform to convert time series behaviour signals to frequency domain as spectral signals, where each component in spectral signal encodes different frequency information of the whole video. As a result, the produced spectral signals contain multi-scale video-level temporal information. However, due to the variation in length of original videos, the length of their corresponding time-series behaviour signals and spectral signals are also variable. To allow spectral signals to be easily processed by standard ML models, we also propose two frequency alignment methods. Additionally, we also propose two spectral representations, i.e., spectral heatmap and spectral vector, to encode aligned spectral signals, allowing them to be learned by CNNs and ANNs, respectively.” (page 841, col. 2, para. 1)
Regarding claim 7, Song teaches wherein the behavior map includes multiple behavior primitives and wherein the context map includes multiple context primitives, the method further comprising determining if the behavior map or the context map has fewer primitives than the other, and inserting additional all -zero primitives to the map having fewer primitives where primitives are missing, to create a behavior map and a context map of equal size with “Zero-padding is a common method often used to increase the frequency resolution after Fourier Transfor mation of a discreet time series. In this method, zeros are appended to the time-series data to increase its length, allowing the DFT of this time-series data to have more fre quency components. In particular, the frequency resolution W of the spectral signal is equal to the number of frames N in the original time-series data. …Then, we add zeros to the behaviour signals extracted from the rest of the videos, making all behaviour signals to have the same length as the longest video. Consequently, the spectral signals of all zero-padded time-series behaviour signals will have the same resolution.” (page 834, col. 1, para. 2)
Regarding claim 8, Song teaches wherein the data comprises stacked 2D data comprising a channel for each of the behavior map and context map with Fig. 1 (page 831). Fig. 1 depicts the pipeline used to predict depression.
Regarding claim 9, Song teaches wherein estimating the behavior uses a convolutional neural network with “To learn from spectral heatmaps, our CNN structure was fixed (illustrated in Section 3.3 and Fig. 5a) for all experiments described in this paper. We also employ ANNs consisting of four fully-connected hidden layers (illustrated in Section 3.3 and Fig. 5b) to learn from spectral vectors, where the size (ranged from 20 to 40) of each hidden layer was optimized for each experiment individually. For all networks, we used Adam [67] as the optimizer and MSE as the loss function. All training hyper-parameters for ANNs and CNNs, e.g., learning rate, beta 1, beta 2 etc, and were optimized on a validation set for each experiment individually. Other hyper-parameters of the network, e.g., the number of layers and the pooling method, were chosen based on the average validation results of multiple experiments.” (page 837, col. 1, para. 3)
Regarding independent claim 10, Song teaches receiving a behavior map, the behavior map produced from a plurality of behavior primitives extracted from a recording of a user with “In this paper, we employed the OpenFace 2.0 toolkit to automatically detect intensities of 17 AUs (AU01, AU02, AU04, AU05, AU06, AU07, AU09, AU10, AU12, AU14, AU15, AU17, AU20, AU23, AU25, AU26 and AU45), 6 gaze direction descriptors and 6 head pose descriptors (detailed explanation is in Fig. 7) resulting in a 29-D frame-wise human representation.” (page 837, col. 1, para. 2) and “To achieve this, we first extract a set of automatically detected human behaviour primitives to represent a video, allowing the high dimensional videos to be significantly reduced to a low dimensional multichannel time-series signal (Section 3.1). In Section 3.2, we propose two spectral representations as the video-level descriptors for multi-channel behaviour signals, which can not only encode a time-series data of arbitrary length into fixed-size representations but also retain multi-scale temporal information from the original time-series data. Finally, we show how to apply the generated spectral representations to depression analysis (Section 3.3).” (page 832, col. 2, para. 1).
Song teaches estimating a cognitive state from data comprising the behavior map, wherein the estimating comprises using at least one neural process on the data with “…we independently evaluate the performance of each human behaviour primitive on depression severity estimation. To do so, we trained separate models from the spectral vectors of each behaviour primitive. The results are reported on the AVEC 2013 dataset.” (page 838, col. 1, para. 1)
Regarding claim 11, Song teaches wherein estimating the cognitive state of the user uses a neural process with “To learn from spectral heatmaps, our CNN structure was fixed (illustrated in Section 3.3 and Fig. 5a) for all experiments described in this paper. We also employ ANNs consisting of four fully-connected hidden layers (illustrated in Section 3.3 and Fig. 5b) to learn from spectral vectors, where the size (ranged from 20 to 40) of each hidden layer was optimized for each experiment individually. For all networks, we used Adam [67] as the optimizer and MSE as the loss function. All training hyper-parameters for ANNs and CNNs, e.g., learning rate, beta 1, beta 2 etc, and were optimized on a validation set for each experiment individually. Other hyper-parameters of the network, e.g., the number of layers and the pooling method, were chosen based on the average validation results of multiple experiments.” (page 837, col. 1, para. 3)
Regarding claim 12, Song teaches wherein the estimating a cognitive state comprises using a convolutional neural network, and the at least one neural process receives an output from the convolutional neural network with “To learn from spectral heatmaps, our CNN structure was fixed (illustrated in Section 3.3 and Fig. 5a) for all experiments described in this paper. We also employ ANNs consisting of four fully-connected hidden layers (illustrated in Section 3.3 and Fig. 5b) to learn from spectral vectors, where the size (ranged from 20 to 40) of each hidden layer was optimized for each experiment individually. For all networks, we used Adam [67] as the optimizer and MSE as the loss function. All training hyper-parameters for ANNs and CNNs, e.g., learning rate, beta 1, beta 2 etc, and were optimized on a validation set for each experiment individually. Other hyper-parameters of the network, e.g., the number of layers and the pooling method, were chosen based on the average validation results of multiple experiments.” (page 837, col. 1, para. 3)
Regarding claim 13, Song teaches wherein using a neural process comprises: using a global latent variable and a decoder to estimate the behavior from the data with Fig. 1 (page 831). Fig. 1 depicts the pipeline used to predict depression.
Regarding claim 14, Song teaches wherein the global latent variable has been determined by: encoding conditions by receiving a plurality of condition descriptor pairs, each condition descriptor pair comprising a true behavior of a user and data derived from a behavior map and/or context map for that user; and aggregating the encoded conditions with Fig. 1 (page 831). Fig. 1 depicts the pipeline used to predict depression.
Regarding claim 17, Song teaches wherein the at least one behavior primitive comprises a face behavior primitive with Fig 6. Fig. 6. Visualization of the most important facial actions and their face region for depression analysis. (page 837)
Regarding claim 18, Song teaches wherein the at least one behavior comprises a voice behavior primitive with “Each clip records a participant doing a series of tasks, including sustained vowel phonation, sustained loud vowel phonation, sustained smiling vowel phonation, speaking out loud while solving a task, counting from 1 to 10, etc. All participants are German speakers and each of them do the same tasks in the same order during the video recording. The length of these videos ranges from 20 minutes to 50 minutes with an average of 25 minutes.” (page 836, col. 2, para. 2).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Song ("Spectral representation of behaviour primitives for depression analysis." IEEE Transactions on Affective Computing 13.2 (2020): 829-844.; as cited on the attached 892 form) as applied to claims 1-14 and 17-18 in the 35 U.S.C. 102(a)(1) section above; in view of Ringeval ("AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition." Proceedings of the 9th International on Audio/visual Emotion Challenge and Workshop. 2019.; as cited on the attached 892 form.)
Song is applied to claims 1-14 and 17-18 as discussed in the 35 U.S.C. 102(a)(1) section above.
Regarding claim 16, Song teaches, wherein the recording of the user comprises at least one of an audio recording of the user and/or a video recording of the user with “Each clip records a participant doing a series of tasks, including sustained vowel phonation, sustained loud vowel phonation, sustained smiling vowel phonation, speaking out loud while solving a task, counting from 1 to 10, etc. All participants are German speakers and each of them do the same tasks in the same order during the video recording. The length of these videos ranges from 20 minutes to 50 minutes with an average of 25 minutes.” (page 836, col. 2, para. 2).
Song does not teach wherein the recording of the user was obtained by an interaction with a virtual agent of claim 15. However, Ringeval teaches this claim limitation.
Regarding claim 15, Ringeval teaches wherein the recording of the user was obtained by an interaction with a virtual agent with “The Detecting Depression with AI Sub-challenge (DDS) is a major extension of the AVEC 2016 DSC [73], where the level of depression severity (PHQ-8 questionnaire) was assessed from audiovisual recordings of patients interacting with a virtual agent conducting a clinical interview and driven by a human as a Wizard-of-Oz (WoZ); DAIC-WOZ corpus [23]. The DAIC data set contains new recordings of the same population with the virtual agent being, this time, wholly driven by AI, i.e., without any human intervention. Those new recordings are used as a test partition for the DDS, and will help to understand how the absence of a human conducting the virtual agent impacts on automatic depression severity assessment.” (page 4, col. 1, para. 4).
It would have been prima facia obvious to combine the teachings of Song and Ringeval to arrive at the claimed invention. A person of ordinary skill in the art would have been motivated to modify the method of Song by to include recording the user by a virtual agent as taught by Ringeval for the benefit of conducting patient interviews without human intervention. Furthermore, there would have been a reasonable expectation of success, since Song and Ringeval teach methods that pertain to obtaining recordings of patients for the analysis of facial emotions.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 19 and 21-24 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ringeval ("AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition." Proceedings of the 9th International on Audio/visual Emotion Challenge and Workshop. 2019.; as cited on the attached 892 form.)
Regarding independent claim 19, Ringeval teaches a processor configured to: receive a recording of a user and at least one context descriptor, each context descriptor comprising a time series of context descriptors; extract at least one behavior primitive from the recording, each behavior primitive comprising a time series of behavior descriptors; produce a behavior map from the at least one behavior primitive; produce a context map from the at least one context descriptor; and estimate a cognitive state of the user using data derived from the behavior map and the context map with Tables 2-3. Tables 2 provides the Number of subjects and duration of the interviews included in the Extended-DAIC database (page 6). And Table 3 provides the Number of subjects and duration of the video chats contained in the SEWA database (page 6). Song also teaches “The traditional approach in affect sensing consists in summarising low-level descriptors (LLDs) of audiovisual signals over time with a set of statistical measures computed over a fixed-duration sliding analysis window. Those descriptors usually include spectral, cepstral, prosodic, and voice quality information for the audio channel, and appearance, geometric, and FAUs information for the video channel.” (page 7, col. 1, para. 3) “As audio features, we compute the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [19], which contains 88 measures covering the aforementioned acoustic dimensions, and used here as baseline. In addition, MFCCs 1-13, including their 1st- and 2nd-order derivatives (deltas and double-deltas) are computed as a set of acoustic LLDs, using the openSMILE2 [20] toolkit. As visual features, we extract the intensities of 17 FAUs for each video frame, along with a confidence measure, using the toolkit openFace3 [4]. Descriptors of pose and gaze are additionally extracted.” (page 7, col. 1, para. 4) and “Early paralinguistic investigations into depressed speech found that patients consistently demonstrated prosodic speech abnormalities such as reduced pitch, reduced pitch range, slower speaking rate, and higher articulation errors [11]. Facial expression and head gestures that can be tracked by computer vision are also good predictors of depression; e.g., a more downward angle of the gaze, less intense smiles, and shorter average duration of smiles have been reported as the most salient facial cues of depression [57]. Further, body expressions, gestures, head movements, and linguistic cues have also been reported to provide relevant cues for depression detection” (page 5, col. 1, para. 6 to page 5, col. 2, para. 1)
Regarding claim 21, Ringeval teaches comprising a mobile computing device configured to generate a virtual agent that interacts with the user and wherein the mobile computing device is configured to produce the recording of the user during interactions with the virtual agent with “Detecting Depression with AI Sub-challenge (DDS) is a major extension of the AVEC 2016 DSC, where the level of depression severity (PHQ-8 questionnaire) was assessed from audiovisual recordings of patients interacting with a virtual agent conducting a clinical interview and driven by a human as a Wizard-of-Oz (WoZ); DAIC-WOZ corpus. The DAIC data set contains new recordings of the same population with the virtual agent being, this time, wholly driven by AI, i.e., without any human intervention. Those new recordings are used as a test partition for the DDS, and will help to understand how the absence of a human conducting the virtual agent impacts on automatic depression severity assessment.” (page 4, col. 1, para. 4) and “Data collected include audio and video recordings, automatically transcribed text using Google Cloud’s speech recognition service, and extensive questionnaire responses. The interviews are conducted by an animated virtual interviewer called Ellie. In the WoZ interviews, the virtual agent is controlled by a human interviewer (wizard) in another room, where as in the AI interviews, the agent acts in a fully autonomous way using different automated perception and behaviour generation modules.” (page 6, col. 1, para. 5).
Regarding claim 22, Ringeval teaches, wherein the virtual agent is configured to assign a task to the user and/or ask the user at least one question with “Taking all those evidences together, it has been proposed to integrate affective computing technology into a computer agent that interviews people and identifies verbal and nonverbal indicators of mental illnesses.” (page 5, col. 2, para. 2)
Regarding claim 23, Ringeval teaches wherein the mobile computing device comprises at least one of a smartphone, tablet, laptop computer, and a desktop computer with “Taking all those evidences together, it has been proposed to integrate affective computing technology into a computer agent that interviews people and identifies verbal and nonverbal indicators of mental illnesses.” (page 5, col. 2, para. 2) and “The SEWA database consists of audiovisual recordings of spontaneous behaviour of participants captured using an in-the-wild recording paradigm. Pairs of friends or relatives from German, Hungarian, and Chinese cultures were recorded through a dedicated videochat platform which utilized participants’ own standard–web-cameras and microphones. After watching a set of commercials, pairs of participants were given the task of discussing the last advert watched (a video clip advertising a water tap) for up to three minutes. The aim of this discussion was to elicit further reactions and opinions about the advert and the product advertised” (page 6, col. 2, para. 1)
Regarding claim 24, Ringeval teaches wherein the virtual agent is configured to deliver digital feedback to the user and wherein the mobile computing device is configured to monitor digitally delivered feedback in real-time with “Taking all those evidences together, it has been proposed to integrate affective computing technology into a computer agent that interviews people and identifies verbal and nonverbal indicators of mental illnesses. Data collected with subjects suffering from post-traumatic stress disorder showed that the automatic evaluation of their level of depression severity…” (page 5, col. 2, para. 2)
Conclusion
No claims are allowed.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KETTIP KRIANGCHAIVECH whose telephone number is (571)272-1735. The examiner can normally be reached 8:30am-5:00pm EDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Larry D. Riggs can be reached at (571) 270-3062. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.K./Examiner, Art Unit 1686
/LARRY D RIGGS II/Supervisory Patent Examiner, Art Unit 1686
Prosecution Insights