DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the Amendment filed on 01/02/2026. Claims 1-5 and 21 are pending in the case. This action is FINAL.
Applicant Response
In Applicant’s response dated 01/02/2026, Applicant amended Claim1 canceled claims 5-20 and added a new claim 21 and argued against all objections and rejections previously set forth in the Office Action dated 10/02/2025.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5 and 21 are rejected under 35 U.S.C. 101 because the claimed invention is directed towards an abstract idea, without significantly more.
Step 1
According to the first part of the analysis, in the instant case, claims 1- 5are directed to a computer-implemented method claim and claim 21 is directed to non-transitory computer-readable media claim. Thus, each of the claims falls within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).).
Regarding Claim 1,
At step 2A, Prong 1, Does the claim recite a judicial exception?
Claim 1 recites the steps of:
… obtaining a multi-modal main model comprising a plurality of single-modality models (This step for collecting and defining data for models which is in the “Mental Process” groping of abstract idea.)
a language model trained to receive text input and to generate textual embeddings, (This step for computing mathematical operation on text is a mathematical concept and is understood to be a recitation of an abstract idea math.);
an acoustic model trained to receive speech input and to generate speech embeddings (This step for computing mathematical operation on speech input is a mathematical concept and is understood to be a recitation of an abstract idea math.);
a vision model trained to receive image and video input and to generate visual embeddings (This step for computing mathematical operation on video is a mathematical concept and is understood to be a recitation of an abstract idea math.);
the computing system obtaining a knowledge module …(This step for collecting and defining data for models which is in the “Mental Process” groping of abstract idea.)
obtains multi-modal knowledge inputs from different external sources and generates knowledge embeddings for the multi-modal knowledge inputs, represents the multi-modal knowledge inputs as structured knowledge units that are formatted into one or more representations, and without having to re- train the plurality of single-modality models of the multi-modal main model in response to receipt of new knowledge (This step recites collecting and defining data, vector representation of knowledge, organizing for models which is in the “Mental Process” and “Mathematical concepts” groping of abstract idea.), and
wherein external attention is applied between the multi-modal main model and the knowledge module, resulting in the knowledge module being configured to enhance the knowledge embeddings with knowledge units that were previously stored (This step recites collecting and defining data, vector representation of knowledge, organizing for models which is in the “Mental Process” and “Mathematical concepts” groping of abstract idea.),
the computing system obtaining one or more transformer layers configured to integrate the knowledge embeddings with the textual embeddings, speech embeddings, and visual embeddings through an encoding process into a shared representational space (This step recites collecting and defining data, vector representation of knowledge, organizing for models which is in the “Mental Process” groping of abstract idea.),; and
the computing system generating the multi-modal machine learning model by compiling the language model, the acoustic model, the vision model, and the knowledge module in parallel, such that the one or more transformer layers receives the textual embeddings, speech embeddings, visual embeddings, and knowledge embeddings as input and generates a final set of embeddings comprising a combined output based on integrating the knowledge embeddings into the textual embeddings, speech embeddings, and visual embeddings (This step recites collecting and defining data, vector representation of knowledge, organizing for models which is in the “Mental Process” and “Mathematical concepts” groping of abstract idea.),
wherein, as a result of using the knowledge units that were previously stored to enhance the knowledge embeddings, the multi-modal main model is configured to adapt to new domains more easily as compared to when the knowledge embeddings are not enhanced. (This step recites collecting and defining data, vector representation of knowledge, organizing for models which is in the “Mental Process” and “Mathematical concepts” groping of abstract idea.),
The claim recites a judicial exception, a mathematical concept applied in the field of machine learning. Most of the limitations fall into the mathematical concepts and mathematical algorithm groupings. Accordingly, the claims recite an abstract idea.
Step 2A prong 2: Does the claim recite additional elements? Do those additional elements, individually and in combination, integrate the judicial exception into a practical application?
Further, the claim does not recite any additional element which could integrate this abstract idea into a practical application, because the additional elements recited of consist of:
a computing system for building a multi-modal machine learning model, that is, generic computer components on which to implement the abstract idea (see MPEP 2106.05(f)); performing, for the transaction, an action based on the first output, which is merely “using a computer or other machinery” as a tool to perform the abstract idea step of generating an output (see MPEP 2106.05(f)).
The additional elements are recited at a high level of generality and do not amount to significantly more than the abstract idea (MPEP 2106.05(f)). Thus, the claim is directed towards the abstract idea.
Step 2B: Do the additional elements, considered individually and in combination, amount to significantly more than the judicial exception?
No, as shown above with respect to integration of the abstract idea into a practical application, the additional element of:
a computing system for building a multi-modal machine learning model, that is, generic computer components on which to implement the abstract idea (see MPEP 2106.05(f)); performing, for the transaction, an action based on the first output, which is merely “using a computer or other machinery” as a tool to perform the abstract idea step of generating an output (see MPEP 2106.05(f)).
Thus, the claims are not patent eligible. Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. Neither can insignificant extra-solution activity. All these additional elements as generically claimed are thus considered well-understood, routine, and conventional. Therefore, these limitations, taken alone or in combination, do not integrate the abstract idea into a practical application or recite significantly more that the abstract idea.
Thus, these independent claims are not patent eligible.
The dependent claims respectively recite a judicial exception in limitations of: “training the multi-modal machine learning model on a representation learning task without using external knowledge, such that the multi-modal machine learning model is configured to generate the final set of embeddings when no knowledge embeddings are available for integration.” (claims 2), “training the multi-modal machine learning model on a representation learning task that utilizes relevant external knowledge such that the multi-modal machine learning model is configured to generate the final set of embeddings when new knowledge embeddings are available for integration.” (claims 3), “training the multi-modal machine learning model on a representation learning task that considers irrelevant or noisy external knowledge such that the multi-modal machine learning model is configured to generate a final set of embeddings by ignoring the irrelevant or noisy external knowledge.” (claims 4), “the computing system applying a new input associated with one or more modalities to the multi-modal machine learning model; the computing system generating a set of embeddings for each of the one or more modalities associated with the new input; the computing system identifying one or more knowledge units that are relevant to the new input; the computing system generating one or more knowledge embeddings based on the one or more knowledge units; and the computing system generating a final set of embeddings by integrating the one or more knowledge embeddings with the set of embeddings for each of the one or more modalities.” (claim 5), “after receiving the new input, the computing system identifying one or more modalities associated with data included in the new input; and based on identifying the one or more modalities associated with the data, the computing system selecting only those single-modality models that correspond to the one or more modalities associated with the data, wherein the computing system only applies the new input to the single-modality models that correspond to the one or more modalities associated with the data.”)
These additional limitations (in claims 2-5, also constitute concepts performed in the human mind which fall within the “Mental Processes” groupings of abstract ideas.
This judicial exception is not integrated into a practical application. Additional elements “computer readable medium comprising: computer program code (in claims 2-5), all amount to no more than adding insignificant extra-solution activity/specifications related to data gathering, data input, or data transmittal. These additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The dependent claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of non-transitory computer readable medium comprising: computer program code are again insignificant extra-solution activity steps that cannot provide an inventive concept. All these additional elements as generically claimed are considered well-understood, routine, and conventional.
Therefore, these limitations, taken alone or in combination, do not integrate the abstract idea into a practical application or recite significantly more that the abstract idea. Thus, all the dependent claims are also not patent eligible.
Examiner Comments
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claims 1-5 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Kollada (Pub: No. US 20240221950 A1, Pub. Date: 2024-07-04) in view of FUERST (Pub: No. US 20210357767 A1, Pub. Date: 2021-11-18)
Regarding independent Claim 1,
KOLLADA teaches a method implemented by a computing system for building a multi-modal machine learning model (see KOLLADA: Fig.5, [0105], describes high-level the trained multi-modal product fusion model), the method comprising:
the computing system obtaining a multi-modal main model comprising a plurality of single-modality models (see KOLLADA: Fig.3A, [0078], “shows a high-level block diagram of an embodiment 300 of a multi-modal product fusion model, such as the multi-modal product fusion model 238 at FIG. 2. Accordingly, in one example, the multi-modal product fusion model 300 may be implemented by a server, such as server 234 at FIG. 2.”), wherein the plurality of single-modality models include:
a language model trained to receive text input and to generate textual embeddings (see KOLLADA: Fig.4, [0100], “The text data 406 is processed to generate text features 426 according to a Bidirectional Encoder Representations from Transformers (BERT) model 416.”);
an acoustic model trained to receive speech input and to generate speech embeddings (see KOLLADA: Fig.4, [0101], “The audio and video encoding subnetworks 432 and 434 may have a neural network architecture. In one example, each of the audio and video subnetworks may be modelled according to a deep network, such as ResNet, or any other suitable convolutional backbone, which may process the input audio and video features to generate corresponding audio and video embeddings 432 and 434.”); and
a vision model trained to receive image and video input and to generate visual embeddings (see KOLLADA: Fig.4, [0101], “The audio and video encoding subnetworks 432 and 434 may have a neural network architecture.”);
the computing system obtaining one or more transformer layers configured to integrate the knowledge embeddings with the textual embeddings, speech embeddings, and visual embeddings through an encoding process into a shared representational space (see KOLLADA: Fig.3A, [0083] “Each modality embedding indicates a robust unimodal representation of the mental health features (i.e. knowledge embeddings) extracted from the corresponding modality data. In order to increase accuracy of mental health diagnosis, the product fusion model 300 includes a product fusion layer 360 that generates a multi-modal representation 370 combining respective unimodal representations of all the modalities. That is, the multi-modal representation 370 is generated by combining all of the modalities and in each modality, all of the unimodal representations are considered for the combination. The multi-modal representation captures unimodal contributions, bimodal interactions, as well as higher order interactions (trimodal, quadmodal, etc.) depending on a number of modalities used for mental health evaluation.”), and
the computing system generating the multi-modal machine learning model by compiling the language model, the acoustic model, the vision model, and the knowledge module in parallel (see KOLLADA: Fig.3B, [0094], “a pre-fusion module 340 may be included between the encoder module 320 and the product fusion layer 360. The pre-fusion module 340 may include a plurality of attention-based subnetworks including a first attention-based subnetwork 342, a second attention-based network 344, and so on up to a Nth attention-based subnetwork 346. In one example, each of the plurality of attention based subnetworks may implement a multihead self-attention based mechanism to generate contextualized unimodal representations that are modified embeddings having context information (i.e. knowledge module), such that the one or more transformer layers receives the textual embeddings, speech embeddings, visual embeddings, and knowledge embeddings as input and generates a final set of embeddings comprising a combined output based on integrating the knowledge embeddings into the textual embeddings, speech embeddings, and visual embeddings, (see KOLLADA: Fig.3B, [0083], “the multi-modal representation 370 is generated by combining all of the modalities and in each modality, all of the unimodal representations are considered for the combination. The multi-modal representation captures unimodal contributions, bimodal interactions, as well as higher order interactions (trimodal, quadmodal, etc.) depending on a number of modalities used for mental health evaluation. The multi-modal representation 370 is generated by computing an outer product of all the unimodal representations from each of the modality data. As a non-limiting example, for a mental health evaluation system acquiring audio modality data, video modality data, text modality data, and EEG modality data, a multi-modal product fusion representation (t) is generated by computing an outer product of unimodal embeddings of all the modalities.”), wherein
as a result of using the knowledge units that were previously stored to enhance the knowledge embeddings, the multi-modal main model is configured to adapt to new domains more easily as compared to when the knowledge embeddings are not enhanced (see KOLLADA: Fig.4, [0104], “he multi-modal product fusion representation 470 can be utilized in a variety of applications, including supervised classification, supervised regression, supervised clustering, etc., Accordingly, the multi-modal product fusion representation 470 is fed into one or more neural networks 480. The neural networks 480 may each be trained to classify one or more mental health conditions or output a regression result for a mental health condition.”)
KOLLADA does not teach the computer implemented method wherein:
the computing system obtaining a knowledge module, wherein the knowledge model:
obtains multi-modal knowledge inputs from different external sources and generates knowledge embeddings for the multi-modal knowledge inputs,
represents the multi-modal knowledge inputs as structured knowledge units that are formatted into one or more representations, and
updates without having to re- train the plurality of single-modality models of the multi-modal main model in response to receipt of new knowledge, ;and
wherein external attention is applied between the multi-modal main model and the knowledge module, resulting in the knowledge module being configured to enhance the knowledge embeddings with knowledge units that were previously stored;
However, FUERST teach the system wherein:
the computing system obtaining a knowledge module (see FUERST: Fig.1, [0066], “the knowledge fusion model 110 is deployed (e.g., in a container, or as part of an IoT cloud-edge framework) and processes new data 202.”), wherein the knowledge model:
obtains multi-modal knowledge inputs from different external sources (see FUERST: Fig.1, [0054], “e knowledge infusion device 108 includes multiple knowledge functions (e.g., a first knowledge function (KF.sub.1), a second knowledge function (KF.sub.2), and so on), a knowledge function abstraction layer, multiple ensembles (e.g., majority ensemble, generative ensemble, tutor for reinforced learning (Tutor4RL), and so on), and a knowledge model abstraction layer. As shown, external knowledge sources 102 may be an application programming interface (API) such as a weather API that is directly accessed by the knowledge functions of the knowledge infusion device 108. Additionally, and/or alternatively, the external knowledge sources 102 may be a knowledge graph that integrates/links into a common knowledge base (e.g., a knowledge graph that integrates various aspects of a smart city).”) and generates knowledge embeddings for the multi-modal knowledge inputs (see FUERST: Fig.6, [0110], “, the computing device generates a knowledge model based on the plurality of strong functions and the plurality of weak functions. For example, to generate the knowledge model, the computing device may map the strong and weak functions to one or more ensemble methods (e.g. a majority ensemble method, a generative ensemble method, or a Tutor for Reinforced Learning (Tutor4RL) ensemble method)”)
represents the multi-modal knowledge inputs as structured knowledge units that are formatted into one or more representations (see FUERST: Fig.1, [0074], “the moving mean value of the uncertainty value is less than the threshold, this may indicate that the ML model and reality (e.g., data distribution, environment properties) still match. In such examples, the knowledge base 104 may be updated with the new output from the knowledge fusion model 110. The knowledge model within the knowledge fusion model 110 may be updated using the new knowledge from the knowledge base 104 (e.g., for a different location context).”)
updates without having to re-train the plurality of single-modality models of the multi-modal main model in response to receipt of new knowledge (see FUERST: Fig.1, [0086], “As a BMS is usually a tightly-integrated and hard to extend system, the present invention directly updates the knowledge graph 104 with new sensing values and the outputs of the knowledge fusion model 110. Other applications and users might update the knowledge graph 104 as well. For example, there can be an interface to update building regulations, schedules and other relevant knowledge. Through these updates (and the updates in sensing and output values), the present invention is able to adapt the model automatically and ensure robustness in its output according to safety regulations.”), wherein
external attention is applied between the multi-modal main model and the knowledge module, resulting in the knowledge module being configured to enhance the knowledge embeddings with knowledge units that were previously stored (see FUERST: Fig.1, [0074], “If the moving mean value of the uncertainty value is greater than the threshold, this may indicate that reality (e.g., data distribution, environment properties) may have shifted away from the ML model and the ML model might not be able to create outputs with a good performance (e.g., with sufficient accuracy). In such examples, a re-training of the ML model (for the RL agent, the RL policy may be reset) may be triggered. The re-training may occur through the continuously updated knowledge model that reflects changes in external 102 and internal knowledge bases 104 through its adaptive knowledge functions.”)
Because both KOLLADA and FUERST are in the same/similar field of Multi-modal input processing, accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of KOLLADA to include the system that is configured to update the knowledge module with new knowledge without having to re-train the language model, acoustic model, or vision model as taught by FUERST. After modification of KOLLADA, the multimodal framework with text, audio and vision model that is integrated using transform base fusion can also incorporate a knowledge adaptor that can be updated independently and integrate with the other input modalities as taught by FUERST. One would have been motivated to make such a combination in enhance extensibility and avoid retraining.
Regarding Claim 2,
As shown above, KOLLADA and FUERST teaches all the limitations of Claim 1. KOLLADA further teaches the method comprising:
training the multi-modal machine learning model on a representation learning task without using external knowledge, such that the multi-modal machine learning model is configured to generate the final set of embeddings when no knowledge embeddings are available for integration (see KOLLADA: Fig.3B, [0085], “After generating the multi-modal representation 370, all the dimensions of the multi-modal representation are concatenated into a single multi-modal vector and fed into a mental health inference module 375. The mental health inference module 375 may be an example of diagnosis determination logic 147, discussed at FIG. 1B. The mental health inference module 375 comprises a feed forward neural network 380 and one or more evaluation subnetworks (not shown).”)
Regarding Claim 3,
As shown above, KOLLADA and FUERST teaches all the limitations of Claim 1. KOLLADA further teaches the method comprising:
training the multi-modal machine learning model on a representation learning task that utilizes relevant external knowledge such that the multi-modal machine learning model is configured to generate the final set of embeddings when new knowledge embeddings are available for integration (see KOLLADA: Fig.2, [0078], “The generated multi-modal representation is subsequently input into the feed forward subnetwork 248 to output a mental health classification result or regression result. In some embodiments, the generated multi-modal representation may be input into the relevance determination logic 145 comprising the post-fusion subnetwork 146 for reducing dimensions of the multi-modal representation.”
Regarding Claim 4,
As shown above, KOLLADA and FUERST teaches all the limitations of Claim 1. KOLLADA further teaches the method comprising:
training the multi-modal machine learning model on a representation learning task that considers irrelevant or noisy external knowledge such that the multi-modal machine learning model is configured to generate a final set of embeddings by ignoring the irrelevant or noisy external knowledge (see KOLLADA: Fig.2, [0080],“ Further, when language or text data is preprocessed, noise may be special characters that do not impart useful meaning and thus, noise removal may include removing characters or texts that may interfere with the analysis of text data. Sensor data may be preprocessed by band pass filtering to include sensor data within an upper and lower threshold. In general, the pre-processing of one or more of the first, second, and up to Nth modality data may include one or more of applying one or more modality specific filters to reduce background noise, selecting modality data that has a quality level above a threshold, normalization, and identifying and excluding outlier data, among other modality specific pre-processing. the pre-processing step to remove noise may be performed at server 234.”)
Regarding Claim 5,
As shown above, KOLLADA and FUERST teaches all the limitations of Claim 1. KOLLADA further teaches the method comprising:
the computing system applying a new input associated with one or more modalities to the multi-modal machine learning model (see KOLLADA: Fig.4, [0096], “The patient response to the plurality of tasks and/or the plurality of queries is captured using an audio sensor 401 (e.g., microphone), a video system 403 (e.g. camera), and a text generating system 405 (e.g., user text input via the user interface, speech to text input by converting spoken language to text.”);
the computing system generating a set of embeddings for each of the one or more modalities associated with the new input (see KOLLADA: Fig.4, [0101], “Audio features 422 and video features 424 are input into respective audio and video encoding subnetworks 432 and 434 to obtain audio embedding 432 and video embedding 434 respectively. The audio and video encoding subnetworks 432 and 434 may have a neural network architecture.”);
the computing system identifying one or more knowledge units that are relevant to the new input (see KOLLADA: Fig.12, [0145], “The generated multi-modal representation is subsequently input into the feed forward subnetwork 248 to output a mental health classification result or regression result. In some embodiments, the generated multi-modal representation may be input into the relevance determination logic 145 comprising the post-fusion subnetwork 146 for reducing dimensions of the multi-modal representation.”);
the computing system generating one or more knowledge embeddings based on the one or more knowledge units (see KOLLADA: Fig.2, [0085], After generating the multi-modal representation 370, all the dimensions of the multi-modal representation are concatenated into a single multi-modal vector and fed into a mental health inference module 375. The mental health inference module 375 may be an example of diagnosis determination logic 147, discussed at FIG. 1B”); and
the computing system generating a final set of embeddings by integrating the one or more knowledge embeddings with the set of embeddings for each of the one or more modalities (see KOLLADA: Fig., [0103], “The audio, video, and text embeddings are fused by computing an outer product of the audio, video, and text embeddings at a product fusion layer 460. The outer product of the audio, video, and text embeddings is high-dimensional and captures unimodal contributions as well as bimodal and trimodal interactions.”).
Regarding independent Claim 21,
KOLLADA teaches a computer system comprising:
one or more processors; and one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system (see KOLLADA: Fig.6, [0119], “a flowchart illustrating a high-level method 600 for training a product fusion model for mental health evaluation, such as product fusion model 300 at FIG. 3A. The method 600 may be executed by a processor 104 according to instructions stored in non-transitory memory 106.”), to:
access a multi-modal main model comprising a plurality of single-modality models (see KOLLADA: Fig.3A, [0078], “shows a high-level block diagram of an embodiment 300 of a multi-modal product fusion model, such as the multi-modal product fusion model 238 at FIG. 2. Accordingly, in one example, the multi-modal product fusion model 300 may be implemented by a server, such as server 234 at FIG. 2.”), wherein the plurality of single-modality models include:
(i) a language model trained to receive text input and to generate textual embeddings (see KOLLADA: Fig.4, [0100], “The text data 406 is processed to generate text features 426 according to a Bidirectional Encoder Representations from Transformers (BERT) model 416.”);
(ii) an acoustic model trained to receive speech input and to generate speech embeddings (see KOLLADA: Fig.4, [0100], “The text data 406 is processed to generate text features 426 according to a Bidirectional Encoder Representations from Transformers (BERT) model 416.”);and
(iii) a vision model trained to receive image and video input and to generate visual embeddings (see KOLLADA: Fig.4, [0101], “The audio and video encoding subnetworks 432 and 434 may have a neural network architecture.”);
wherein, as a result of using the knowledge units that were previously stored to enhance the knowledge embeddings, the multi-modal main model is configured to adapt to new domains more easily as compared to when the knowledge embeddings are not enhanced (see KOLLADA: Fig.4, [0104], “he multi-modal product fusion representation 470 can be utilized in a variety of applications, including supervised classification, supervised regression, supervised clustering, etc., Accordingly, the multi-modal product fusion representation 470 is fed into one or more neural networks 480. The neural networks 480 may each be trained to classify one or more mental health conditions or output a regression result for a mental health condition.”)
obtain one or more transformer layers configured to integrate the knowledge
generate the multi-modal machine learning model by compiling the language model, the acoustic model, the vision model, and the knowledge module in parallel (see KOLLADA: Fig.3B, [0094], “, a pre-fusion module 340 may be included between the encoder module 320 and the product fusion layer 360. The pre-fusion module 340 may include a plurality of attention-based subnetworks including a first attention-based subnetwork 342, a second attention-based network 344, and so on up to a Nth attention-based subnetwork 346. In one example, each of the plurality of attention based subnetworks may implement a multihead self-attention based mechanism to generate contextualized unimodal representations that are modified embeddings having context information (i.e. knowledge module)), such that the one or more transformer layers receives the textual embeddings, speech embeddings, visual embeddings (see KOLLADA: Fig.3B, [0083], “the multi-modal representation 370 is generated by combining all of the modalities and in each modality, all of the unimodal representations are considered for the combination. The multi-modal representation captures unimodal contributions, bimodal interactions, as well as higher order interactions (trimodal, quadmodal, etc.) depending on a number of modalities used for mental health evaluation. The multi-modal representation 370 is generated by computing an outer product of all the unimodal representations from each of the modality data. As a non-limiting example, for a mental health evaluation system acquiring audio modality data, video modality data, text modality data, and EEG modality data, a multi-modal product fusion representation (t) is generated by computing an outer product of unimodal embeddings of all the modalities.”) and knowledge embeddings as input and generates comprising a combined output based on integrating the knowledge embeddings into the textual embeddings, speech embeddings, and visual embeddings (see KOLLADA: Fig.3B, [0083], “the multi-modal representation 370 is generated by combining all of the modalities and in each modality, all of the unimodal representations are considered for the combination. The multi-modal representation captures unimodal contributions, bimodal interactions, as well as higher order interactions (trimodal, quadmodal, etc.) depending on a number of modalities used for mental health evaluation. The multi-modal representation 370 is generated by computing an outer product of all the unimodal representations from each of the modality data. As a non-limiting example, for a mental health evaluation system acquiring audio modality data, video modality data, text modality data, and EEG modality data, a multi-modal product fusion representation (t) is generated by computing an outer product of unimodal embeddings of all the modalities.”)
KOLLADA does not teach the system wherein:
access a knowledge module wherein the knowledge model:
(i) obtains trained to receive multi-modal knowledge inputs from different external sources and generates knowledge embeddings for the multi-modal knowledge inputs,
(ii) represents the multi-modal knowledge inputs as structured knowledge units that are formatted into one or more representations, and
(iii) updates the knowledge module with new knowledge without having to re-train the plurality of single-modality models of the multi-modal main model in response to receipt of new knowledge, language model, acoustic model, or vision model,
wherein external attention is applied between the multi-modal main model and the knowledge module, resulting in the knowledge module being configured to enhance the knowledge embeddings with knowledge units that were previously stored.
However, FUERST teach the system wherein:
the computing system obtaining a knowledge (see FUERST: Fig.1, [0066], “the knowledge fusion model 110 is deployed (e.g., in a container, or as part of an IoT cloud-edge framework) and processes new data 202.”), wherein the knowledge model:
(i) obtains multi-modal knowledge inputs from different external sources (see FUERST: Fig.1, [0054], “e knowledge infusion device 108 includes multiple knowledge functions (e.g., a first knowledge function (KF.sub.1), a second knowledge function (KF.sub.2), and so on), a knowledge function abstraction layer, multiple ensembles (e.g., majority ensemble, generative ensemble, tutor for reinforced learning (Tutor4RL), and so on), and a knowledge model abstraction layer. As shown, external knowledge sources 102 may be an application programming interface (API) such as a weather API that is directly accessed by the knowledge functions of the knowledge infusion device 108. Additionally, and/or alternatively, the external knowledge sources 102 may be a knowledge graph that integrates/links into a common knowledge base (e.g., a knowledge graph that integrates various aspects of a smart city).”) and generates knowledge embeddings for the multi-modal knowledge inputs (see FUERST: Fig.6, [0110], “, the computing device generates a knowledge model based on the plurality of strong functions and the plurality of weak functions. For example, to generate the knowledge model, the computing device may map the strong and weak functions to one or more ensemble methods (e.g. a majority ensemble method, a generative ensemble method, or a Tutor for Reinforced Learning (Tutor4RL) ensemble method)”)
(ii) represents the multi-modal knowledge inputs as structured knowledge units that are formatted into one or more representations (see FUERST: Fig.1, [0074], “the moving mean value of the uncertainty value is less than the threshold, this may indicate that the ML model and reality (e.g., data distribution, environment properties) still match. In such examples, the knowledge base 104 may be updated with the new output from the knowledge fusion model 110. The knowledge model within the knowledge fusion model 110 may be updated using the new knowledge from the knowledge base 104 (e.g., for a different location context).”)
(iii) updates without having to re-train the plurality of single-modality models of the multi-modal main model in response to receipt of new knowledge (see FUERST: Fig.1, [0086], “As a BMS is usually a tightly-integrated and hard to extend system, the present invention directly updates the knowledge graph 104 with new sensing values and the outputs of the knowledge fusion model 110. Other applications and users might update the knowledge graph 104 as well. For example, there can be an interface to update building regulations, schedules and other relevant knowledge. Through these updates (and the updates in sensing and output values), the present invention is able to adapt the model automatically and ensure robustness in its output according to safety regulations.”)
external attention is applied between the multi-modal main model and the knowledge module, resulting in the knowledge module being configured to enhance the knowledge embeddings with knowledge units that were previously stored (see FUERST: Fig.1, [0074], “If the moving mean value of the uncertainty value is greater than the threshold, this may indicate that reality (e.g., data distribution, environment properties) may have shifted away from the ML model and the ML model might not be able to create outputs with a good performance (e.g., with sufficient accuracy). In such examples, a re-training of the ML model (for the RL agent, the RL policy may be reset) may be triggered. The re-training may occur through the continuously updated knowledge model that reflects changes in external 102 and internal knowledge bases 104 through its adaptive knowledge functions.”)
Because both KOLLADA and REVEAL are in the same/similar field of Multi-modal input processing, accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of KOLLADA to include the system that is configured to update the knowledge module with new knowledge without having to re-train the language model, acoustic model, or vision model as taught by REVEAL. After modification of KOLLADA, the multimodal framework with text, audio and vision model that is integrated using transform base fusion can also incorporate a knowledge adaptor that can be updated independently and integrate with the other input modalities as taught by REVEAL. One would have been motivated to make such a combination in enhance extensibility and avoid retraining.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
PGPUB
NUMBER:
INVENTOR-INFORMATION:
TITLE / DESCRIPTION
US 20210287103 A1
Bangalore; Pavan Kapanipathi
Title: INFUSING KNOWLEDGE INTO NATURAL LANGUAGE PROCESSING TASKS USING GRAPH STRUCTURES
Description: The field of embodiments of the invention generally relate to natural language processing (NLP).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZELALEM W SHALU whose telephone number is (571)272-3003. The examiner can normally be reached M- F 0800am- 0500pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Zelalem Shalu/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights