Prosecution Insights
Last updated: November 20, 2026
Application No. 18/385,669

PRE-TRAINED LANGUAGE MODELS INCORPORATING SYNTACTIC KNOWLEDGE USING OPTIMIZATION FOR OVERCOMING CATASTROPHIC FORGETTING

Final Rejection §103
Filed
Oct 31, 2023
Examiner
BEJCEK II, ROBERT H
Art Unit
2148
Tech Center
2100 — Computer Architecture & Software
Assignee
International Business Machines Corporation
OA Round
2 (Final)
63%
Grant Probability
Moderate
3-4
OA Rounds
9m
Est. Remaining
85%
With Interview

Examiner Intelligence

Grants 63% of resolved cases
63%
Career Allowance Rate
162 granted / 258 resolved
+7.8% vs TC avg
Strong +22% interview lift
Without
With
+22.5%
Interview Lift
resolved cases with interview
Typical timeline
3y 9m
Avg Prosecution
23 currently pending
Career history
292
Total Applications
across all art units

Statute-Specific Performance

§101
22.6%
-17.4% vs TC avg
§103
41.1%
+1.1% vs TC avg
§102
10.5%
-29.5% vs TC avg
§112
22.4%
-17.6% vs TC avg
Black line = Tech Center average estimate • Based on career data from 258 resolved cases

Office Action

§103
DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Information Disclosure Statement The information disclosure statement filed on 4/29/2026 fails to comply with the provisions of 37 CFR 1.97, 1.98 and MPEP § 609 because a copy of the NPL reference #2 was not provided. Instead, it appears two copies of reference #3 were submitted. It has been placed in the application file, but the information referred to therein concerning NPL reference #2 has not been considered as to the merits. Applicant is advised that the date of any re-submission of any item of information contained in this information disclosure statement or the submission of any missing element(s) will be the date of submission for purposes of determining compliance with the requirements based on the time of filing the statement, including all certification requirements for statements under 37 CFR 1.97(e). See MPEP § 609.05(a). Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 1-3, 12-15, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai et al. (hereinafter Bai), Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees in view of Ede et al. (hereinafter Ede), Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI. Regarding Claim 1, Bai discloses a computer-implemented method for incorporating syntactic knowledge into a pre- trained language model [“Pre-trained language models like BERT” Abstract], the method comprising: selecting, using at least one hardware processor, at least one syntactic pre-training task from among a set of syntactic pre-training tasks [“syntactic information has been proved to be crucial for the success of NLP applications.” Abstract; “incorporates syntactic knowledge” §1 ¶2; Fig. 1; Examiner Note: Figure 1 shows finding different parts of speech]; retraining, for the selected at least one syntactic pre-training task and using the at least one hardware processor, the pre-trained language model by using an optimization function which prevents catastrophic forgetting during the retraining, and wherein the retraining adds syntactic [“syntactic information has been proved to be crucial for the success of NLP applications” Abstract] knowledge to the pre-trained language model while preserving original semantic [“Pre-trained language models like BERT” Abstract] knowledge of the pre-trained language model, and without changing a format of the pre-trained language model; and performing, using the at least one hardware processor, inferencing using the retrained language model [“we evaluate the performance of Syntax-BERT on two natural language inference datasets: SNLI and MNLI.” §4 ¶1]. However, Bai fails to explicitly disclose retraining, for the selected at least one syntactic pre-training task and using the at least one hardware processor, the pre-trained language model by using an optimization function which prevents catastrophic forgetting during the retraining, and wherein the retraining adds syntactic knowledge to the pre-trained language model while preserving original semantic knowledge of the pre-trained language model, and without changing a format of the pre-trained language model. Ede discloses retraining, for the selected at least one syntactic pre-training task and using the at least one hardware processor, the pre-trained language model by using an optimization function which prevents catastrophic forgetting during the retraining, and wherein the retraining adds syntactic knowledge to the pre-trained language model while preserving original semantic knowledge of the pre-trained language model, and without changing a format of the pre-trained language model [“Relevance-based Neural Freezing (RNF), a novel approach to alleviate catastrophic forgetting that builds upon the aforementioned pruning technique. Instead of compressing the network, the information about unit importance is used to freeze the knowledge represented by the learned network parameters by inhibiting or completely stopping the training process for those parts of the model that are relevant for a specific task, while the remaining units are free to learn further tasks.” §1 ¶3; Fig. 1]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai and Ede before him before the effective filing date of the claimed invention, to modify the method of Bai using NLP and syntactic information to incorporate the knowledge retention with additional training of Ede. Given the advantage of preventing catastrophic forgetting by learning new tasks without forgetting previous tasks and which increases the utility of a model, one having ordinary skill in the art would have been motivated to make this obvious modification. Regarding Claim 2, Bai and Ede disclose the method of claim 1. Bai further discloses wherein the set of syntactic pre-training tasks are selected from a group consisting of a deprel prediction task, a phrase detection task, and a main/subordinate detection task [“All phrases are labeled to one of five fine-grained categories of sentiment polarity.” §4.1 ¶1; Fig. 1]. Regarding Claim 3, Bai and Ede disclose the method of claim 1. Bai further discloses wherein the retraining further comprises learning at least one syntactic relationship between two tokens [“reflecting different relationships and distances of tokens in a syntax tree” §3.1 ¶2; Fig. 1]. Regarding Claim 12, Bai and Ede disclose the method of claim 1. Bai further discloses further comprising applying the retrained language model to a downstream task to perform inferencing for language understanding [“we evaluate the performance of Syntax-BERT on two natural language inference datasets: SNLI and MNLI.” §4 ¶1]. Claim 13 is rejected on the same grounds as claim 1. Claim 14 is rejected on the same grounds as claim 1. Claim 15 is rejected on the same grounds as claim 3. Claim 20 is rejected on the same grounds as claim 12. Claim(s) 4-5, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Cormack, What is coordination?. Regarding Claim 4, Bai and Ede disclose the method of claim 1. However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a coordination detection task for predicting a label of a parallel structure, wherein the parallel structure is one of a token, a phrase, and a sentence. Cormack discloses wherein the set of syntactic pre-training tasks includes a coordination detection task for predicting a label of a parallel structure, wherein the parallel structure is one of a token, a phrase, and a sentence [“Coordination is a particular syntactic manifestation of conjunction (or disjunction), which is symmetric with respect to the conjuncts/disjuncts (i.e. the conjuncts/disjuncts have the same syntactic status).” §1 ¶3; Examples 4 on page 398]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Cormack before him before the effective filing date of the claimed invention, to modify the combination to incorporate the coordination analysis of Cormack. Given the advantage of using a different method of language analysis which permits finding potentially missed relationships, one having ordinary skill in the art would have been motivated to make this obvious modification. Regarding Claim 5, Bai, Ede, and Cormack disclose the method of claim 4. However, Bai fails to explicitly disclose wherein the label is a label of a head token, a child token, or a parallel conjunction of the corresponding parallel structure. Cormack discloses wherein the label is a label of a head token, a child token, or a parallel conjunction of the corresponding parallel structure [“Coordination is a particular syntactic manifestation of conjunction” §1 ¶3; “the grammar contains combinators. These occupy the head positions” §1 ¶9]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Cormack before him before the effective filing date of the claimed invention, to modify the combination to incorporate the coordination analysis of Cormack. Given the advantage of using a different method of language analysis which permits finding potentially missed relationships, one having ordinary skill in the art would have been motivated to make this obvious modification. Claim 16 is rejected on the same grounds as claim 4. Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Nivre et al. (hereinafter Nivre), Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection. Regarding Claim 6, Bai and Ede disclose the method of claim 1. However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a deprel prediction task for predicting dependency labels. Nivre discloses wherein the set of syntactic pre-training tasks includes a deprel prediction task for predicting dependency labels [“UD provides a taxonomy of 37 universal relation types to classify syntactic relations” §2.3 ¶2; Table 1]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Nivre before him before the effective filing date of the claimed invention, to modify the combination to incorporate the use of deprel through universal dependencies. Given the advantage of a structured and efficient method to aid in the model’s accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification. Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Basirat et al. (hereinafter Basirat), Syntactic Nuclei in Dependency Parsing– A Multilingual Exploration. Regarding Claim 7, Bai and Ede disclose the method of claim 1. However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a phrase detection task for predicting a relationship between phrases. Basirat discloses wherein the set of syntactic pre-training tasks includes a phrase detection task for predicting a relationship between phrases [“the elementary syntactic units of a dependency tree are nuclei rather than word forms, then the English and Finnish sentences will have the same dependency trees” §1 ¶2; Fig. 2]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Basirat before him before the effective filing date of the claimed invention, to modify the combination to incorporate the nuclei concept which uses phrases for syntax of Basirat. Given the advantage of accuracy forming phrase comparisons for syntax analysis, one having ordinary skill in the art would have been motivated to make this obvious modification. Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Nikolaev et al. (hereinafter Nikolaev), Word-order typology in Multilingual BERT: A case study in subordinate-clause detection. Regarding Claim 8, Bai and Ede disclose the method of claim 1. However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a main/subordinate detection task for predicting two labels to classify clauses into main and subordinate. Nikolaev discloses wherein the set of syntactic pre-training tasks includes a main/subordinate detection task for predicting two labels to classify clauses into main and subordinate [“capture the distinction between main and subordinate clauses” §1 ¶2]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Nikolaev before him before the effective filing date of the claimed invention, to modify the combination to incorporate main and subordinate distinction in language of Nikolaev. Given the advantage of capturing the distinction between main and subordinate clauses to better distinguish language for processing, one having ordinary skill in the art would have been motivated to make this obvious modification. Claim(s) 9 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Tian et al. (hereinafter Tian), Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking. Regarding Claim 9, Bai and Ede disclose the method of claim 1. However, Bai fails to explicitly disclose further comprising learning, in parallel, a dependency masking (DM), which predicts whether there is a dependency between two words, and a masked dependency prediction (MDP), which predicts a type of dependency relationship, and wherein input for the masked dependency prediction is replaced with one of the four syntactic pre-training tasks. Tian discloses further comprising learning, in parallel, a dependency masking (DM), which predicts whether there is a dependency between two words, and a masked dependency prediction (MDP), which predicts a type of dependency relationship [“a weakly supervised learning task, namely, dependency masking (DM) with masked dependency prediction (MDP), to enhance text encoder pre-training, where they are paired processes that DM masks all connections and dependency types” §2.2 ¶1], and wherein input for the masked dependency prediction is replaced with one of the four syntactic pre-training tasks [“leverage auto-parsed dependency information through learning word-word associations (i.e., dependency connections) and their dependency types” §2.2 ¶1]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Tian before him before the effective filing date of the claimed invention, to modify the combination to incorporate the DM and MDP of Tian. Given the advantage of increasing performance for relation extraction, one having ordinary skill in the art would have been motivated to make this obvious modification. Claim 17 is rejected on the same grounds as claim 9. Claim(s) 10 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Yu et al. (hereinafter Yu), Gradient Surgery for Multi-Task Learning. Regarding Claim 10, Bai and Ede disclose the method according to claim 1. However, Bai fails to explicitly disclose wherein the optimization function is configured for solving gradient conflicts in multi-task learning and computes gradients for each task involved in multi-task learning, discards adversarial elements of the gradients that conflict with each other, and sums resultant gradients to obtain a single gradient vector. Yu discloses wherein the optimization function is configured for solving gradient conflicts in multi-task learning [“one of the main optimization issues in multi-task learning arises from gradients from different tasks conflicting with one another in a way that is detrimental to making progress” §1 ¶2] and computes gradients for each task involved in multi-task learning, discards adversarial elements of the gradients that conflict with each other, and sums resultant gradients to obtain a single gradient vector [“To deconflict gradients during optimization, PCGrad adopts a simple procedure: if the gradients between two tasks are in conflict, i.e. their cosine similarity is negative, we project the gradient of each task onto the normal plane of the gradient of the other task. This amounts to removing the conflicting component of the gradient for the task, thereby reducing the amount of destructive gradient interference between tasks.” §2.3 ¶3; “g = g1 + g2” Definition 4]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Yu before him before the effective filing date of the claimed invention, to modify the combination to incorporate gradient surgery of Yu. Given the advantage of a substantial improvements in terms of data efficiency, optimization speed, and final performance, one having ordinary skill in the art would have been motivated to make this obvious modification. Claim 18 is rejected on the same grounds as claim 10. Claim(s) 11 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Kirkpatrick et al. (hereinafter Kirkpatrick), Overcoming catastrophic forgetting in neural networks. Regarding Claim 11, Bai and Ede disclose the method according to claim 1. However, Bai fails to explicitly disclose wherein the optimization function is configured for continuous learning optimization in which tasks are learned in sequence and, given two tasks, A and B, first searches for an optimal solution for task A, then searches for parameters that perform well in both tasks A and B and, when the model is fine-tuned for B after A, updates more important parameters of the task A with smaller weights while updating less important parameters of the task A with larger weights. Kirkpatrick discloses wherein the optimization function is configured for continuous learning optimization in which tasks are learned in sequence and, given two tasks, A and B, first searches for an optimal solution for task A, then searches for parameters that perform well in both tasks A and B and, when the model is fine-tuned for B after A, updates more important parameters of the task A with smaller weights while updating less important parameters of the task A with larger weights [“Continual learning poses particular challenges for artificial neural networks due to the tendency for knowledge of previously learnt task(s) (e.g. task A) to be abruptly lost as information relevant to the current task (e.g. task B) is incorporated.” §1 ¶2; “over-parameterization makes it likely that there is a solution for task B, θ*B, that is close to the previously found solution for task A, θ*A. While learning task B, EWC therefore protects the performance in task A by constraining the parameters to stay in a region of low error for task A centered around θ*A, as shown schematically in Figure 1. This constraint is implemented as a quadratic penalty, and can therefore be imagined as a spring anchoring the parameters to the previous solution, hence the name elastic. Importantly, the stiffness of this spring should not be the same for all parameters; rather, it should be greater for those parameters that matter most to the performance during task A.” §2 ¶2]. It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Kirkpatrick before him before the effective filing date of the claimed invention, to modify the combination to incorporate elastic weight consolidation of Kirkpatrick. Given the advantage of learning new tasks without forgetting previous tasks which increases the utility of a model, one having ordinary skill in the art would have been motivated to make this obvious modification. Claim 19 is rejected on the same grounds as claim 11. Examiner’s Note The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well. Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification. Response to Arguments Regarding the prior art rejections, Applicant's arguments with respect to the claims have been considered but are moot because the arguments do not apply to the references being used in the current rejection of the limitations. Conclusion Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /R.B./ Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148
Read full office action

Prosecution Timeline

Oct 31, 2023
Application Filed
Apr 30, 2026
Non-Final Rejection mailed — §103
Jun 23, 2026
Applicant Interview (Telephonic)
Jun 24, 2026
Examiner Interview Summary
Jul 29, 2026
Response Filed
Nov 05, 2026
Final Rejection mailed — §103 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12737689
Scale-Permuted Machine Learning Architecture
2y 1m to grant Granted Sep 15, 2026
Patent 12737592
PARTIAL ACTIVATION OF MULTIPLE PATHWAYS IN NEURAL NETWORKS
1y 10m to grant Granted Sep 15, 2026
Patent 12651203
SYSTEM AND METHOD FOR TRAINING TEACHER MODEL BASED ON STRUCTURE OF STUDENT MODEL
4y 9m to grant Granted Jun 09, 2026
Patent 12639600
DATABASE UTILIZING SPATIAL PROBABILITY MODELS FOR DATA COMPRESSION
3y 7m to grant Granted May 26, 2026
Patent 12619909
STUDENT PERFORMANCE EVALUATION METHOD AND SYSTEM BASED ON AI IDENTIFICATION DATA
4y 2m to grant Granted May 05, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

3-4
Expected OA Rounds
63%
Grant Probability
85%
With Interview (+22.5%)
3y 9m (~9m remaining)
Median Time to Grant
Moderate
PTA Risk
Based on 258 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month