DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to correspondence filed 15 January 2026 in reference to application 18/428,208. Claims 1-20 are pending and have been examined.
Response to Amendment
The amendment filed 15 January 2026 has been accepted and considered in this office action. Claims 1, 11, and 20 have been amended.
Response to Arguments
Applicant argues, see Remarks pages 7-8, that Tai fails to teach the added limitations of “computing a training loss based on outputs from the domain language model using the domain-specific data and adjusting at least one activation function associated with the domain language model.” Examiner notes however, that what is being claimed is backpropagation which is the standard method of training neural network models, where the output of a model is compared to the ground truth, and the weights and biases of the activation functions are adjusted to minimize a training loss. As is the case in many scholarly papers, Tai omits disclosing these steps, assuming that the reader of such papers on Neural Machine Learning would understand these steps are implicit unless otherwise disclosed. However, for the sake of clarity, Examiner is citing another reference to explicitly teach these limitations.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1, 4-8, 10, 11, 14-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tai et al. (exBERT: Extending Pre-trained Models with Domain-specific Vocabular Under Constrained Training Resources) in view of Kim et al. (US PAP 2022/0374993).
Consider claim 1, Tai teaches A method for creating an enhanced domain-specific language learning model (abstract), the method comprising:
training a domain language model using domain-specific data (section 3.1, 3.2, extension module, section 4.1, Adaptive pretraining extension module trained using medical data);
receiving input corpus for one or more downstream tasks (section 3.1, input text);
using the domain language model with the input corpus to generate a first set of embeddings (Section 3.1, and 3.2, extension module used to generate embeddings, also figure 1b);
using a pre-trained large language model (LLM) with the input corpus to generate a second set of embeddings (Section 3.1, and 3.2, using off the shelf BERT model to generate embeddings, also figure 1b);
combining the first and second sets of embeddings to form a combined set of embeddings (3.2 combining extension embeddings and off the shelf embeddings using weights ); and
performing the one or more downstream tasks using the combined set of embeddings (section 4.1, performing downstream tasks such as Named Entity Recognition).
Although suggested implicitly, Tai does not explicitly teach training by computing a training loss based on outputs from the domain language model using the domain-specific data and adjusting at least one activation function associated with the domain language model.
In the same field of BERT based machine learning, Kim teaches raining by computing a training loss based on outputs from the domain language model using the domain-specific data and adjusting at least one activation function associated with the domain language model (0049, training samples, 0049-60, training using loss functions, to adjust weights and biases of activation functions of the model, see, 0050 and 0060 specifically).
It would have been obvious to one of ordinary skill in the art at the time of effective filing to use training losses to adjust activations of models as taught by Kim in the system of Tai in order to make use of the most well-known and understood method of training neural network models.
Consider claim 4, Tai teaches The method of claim 1, wherein the domain language model is domain agnostic, and wherein prior to training the domain language model, the method further comprises: receiving the domain-specific data from domains in at least finance, insurance, medicine, or artificial intelligence (AI) services (section 4.1, using medical corpus).
Consider claim 5, Tai teaches the method of claim 1, wherein the pre-trained LLM is a generative pre-trained transformer (GPT) model or a bidirectional encoder representation from transformers (BERT) model (section 3.1, off the shelf BERT model).
Consider claim 6, Tai teaches The method of claim 1, wherein combining the first and second sets of embeddings comprises:
generating the combined set of embeddings to capture and integrate both general and domain-specific knowledge respectively learned using the pre-trained LLM and the domain language model (3.2 combining extension embeddings and off the shelf embeddings using weights); and
using the combined set of embeddings as input to the one or more downstream tasks (section 4.1-4.3, using embeddings to performed named entity recognition (NER)).
Consider claim 7, Tai teaches the method of claim 6, wherein the downstream tasks include one or more of classification, clustering, named entity recognition (NER), and retrieval augmented generation (RAG) (section 4.1-4.3, using embeddings to performed named entity recognition (NER))).
Consider claim 8, Tai teaches the method of claim 6, wherein the combined set of embeddings are generated based on concatenation or weighted averaging (section 3.2, weighted summation, i.e. average or of the shelf and extension embeddings).
Consider claim 10, Tai teaches The method of claim 1, further comprising preprocessing the input corpus (section 3.1, removing overlapping words that also appear in pre-trained BERT corpus).
Consider claim 11, Tai teaches A system for creating an enhanced domain-specific language learning model, the system comprising:
a processor (section 4.1, GPUS); and
a memory in communication with the processor and comprising instructions (section 4.1, using GPU which include memories) which, when executed by the processor, program the processor to:
train a domain language model using domain-specific data (section 3.1, 3.2, extension module, section 4.1, Adaptive pretraining extension module trained using medical data);
receive input corpus for one or more downstream tasks (section 3.1, input text);
use the domain language model with the input corpus to generate a first set of embeddings (Section 3.1, and 3.2, extension module used to generate embeddings, also figure 1b);
use a pre-trained large language model (LLM) with the input corpus to generate a second set of embeddings (Section 3.1, and 3.2, using off the shelf BERT model to generate embeddings, also figure 1b);
combine the first and second sets of embeddings to form a combined set of embeddings (3.2 combining extension embeddings and off the shelf embeddings using weights ); and
perform the one or more downstream tasks using the combined set of embeddings (section 4.1, performing downstream tasks such as Named Entity Recognition).
Although suggested implicitly, Tai does not explicitly teach training by computing a training loss based on outputs from the domain language model using the domain-specific data and adjusting at least one activation function associated with the domain language model.
In the same field of BERT based machine learning, Kim teaches raining by computing a training loss based on outputs from the domain language model using the domain-specific data and adjusting at least one activation function associated with the domain language model (0049, training samples, 0049-60, training using loss functions, to adjust weights and biases of activation functions of the model, see, 0050 and 0060 specifically).
It would have been obvious to one of ordinary skill in the art at the time of effective filing to use training losses to adjust activations of models as taught by Kim in the system of Tai in order to make use of the most well-known and understood method of training neural network models.
Claim 14 contains similar limitations as claim 4 and therefore is rejected for the same reasons.
Claim 15 contains similar limitations as claim 5 and therefore is rejected for the same reasons.
Claim 16 contains similar limitations as claim 6 and therefore is rejected for the same reasons.
Claim 17 contains similar limitations as claim 7 and therefore is rejected for the same reasons.
Claim 18 contains similar limitations as claim 8 and therefore is rejected for the same reasons.
Consider claim 20, Tai teaches A computer program product for creating an enhanced domain-specific language learning model, the computer program product comprising a non-transitory computer-readable medium having computer readable program code stored thereon (section 4.1, using GPU which include memories), the computer readable program code configured to:
train a domain language model using domain-specific data (section 3.1, 3.2, extension module, section 4.1, Adaptive pretraining extension module trained using medical data);
receive input corpus for one or more downstream tasks (section 3.1, input text);
use the domain language model with the input corpus to generate a first set of embeddings (Section 3.1, and 3.2, extension module used to generate embeddings, also figure 1b);
use a pre-trained large language model (LLM) with the input corpus to generate a second set of embeddings (Section 3.1, and 3.2, using off the shelf BERT model to generate embeddings, also figure 1b);
combine the first and second sets of embeddings to form a combined set of embeddings (3.2 combining extension embeddings and off the shelf embeddings using weights ); and
perform the one or more downstream tasks using the combined set of embeddings (section 4.1, performing downstream tasks such as Named Entity Recognition).
Although suggested implicitly, Tai does not explicitly teach training by computing a training loss based on outputs from the domain language model using the domain-specific data and adjusting at least one activation function associated with the domain language model.
In the same field of BERT based machine learning, Kim teaches raining by computing a training loss based on outputs from the domain language model using the domain-specific data and adjusting at least one activation function associated with the domain language model (0049, training samples, 0049-60, training using loss functions, to adjust weights and biases of activation functions of the model, see, 0050 and 0060 specifically).
It would have been obvious to one of ordinary skill in the art at the time of effective filing to use training losses to adjust activations of models as taught by Kim in the system of Tai in order to make use of the most well-known and understood method of training neural network models.
Claim(s) 2, 3, 12, 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tai and Kim as applied to claims 1 and 11 above, and further in view of Shekhar et al. (US PAP 2025/0232134).
Consider claim 2, Tai and Kim teach the method of claim 1, wherein the domain language model is trained to recognize and capture linguistic patterns, structures, and semantics of one or more specialized domains based on the domain-specific data (Tai section 3.1 and 3.2, extension module captures meaning and structures of domain specific words and phrases), but do not specifically teach wherein the domain language model is a domain-specific causal language model (CLM).
In the same field of language modeling, Shekhar teaches wherein the domain language model is a domain-specific causal language model (CLM) (0056, LM may be Causal Language Model).
It would have been obvious to one of ordinary skill in the art at the time of effective filing to use a CLM as taught by Shekhar in the system of Tai and Kim in order to allow for learning context of the words and phrases, (Shekhar 0056).
Consider claim 3, Shekhar teaches the method of claim 2, further comprising performing statistical evaluation of the CLM by examining at least one of perplexity scores, training loss, and validation loss (0026, using losses for training, i.e. training losses to train the model.).
Claim 12 contains similar limitations as claim 2 and therefore is rejected for the same reasons.
Claim 13 contains similar limitations as claim 3 and therefore is rejected for the same reasons.
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tai and Kim as applied to claims 6 and 16 above and further in view of Yan et al. (US PAP 2025/0094718).
Consider claim 9, Tai and Kim teach The method of claim 6, but do not specifically teach applying dimensionality reduction to the combined embeddings.
In the same field of combining text embeddings, Yan teaches applying dimensionality reduction to the combined embeddings (0081, performing dimensionality reduction on the combined embedding).
It would have been obvious to one of ordinary skill in the art at the time of effective filing to use dimensionality reduction as taught by Yan in the system of Tai and Kim in order to reduce processing requirements needed to process the embeddings.
Claim 19 contains similar limitations as claim 9 and therefore is rejected for the same reasons.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DOUGLAS C GODBOLD whose telephone number is (571)270-1451. The examiner can normally be reached 6:30am-5pm Monday-Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DOUGLAS GODBOLD
Examiner
Art Unit 2655
/DOUGLAS GODBOLD/Primary Examiner, Art Unit 2655