DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement filed on 4/29/2026 fails to comply with the provisions of 37 CFR 1.97, 1.98 and MPEP § 609 because a copy of the NPL reference #2 was not provided. Instead, it appears two copies of reference #3 were submitted. It has been placed in the application file, but the information referred to therein concerning NPL reference #2 has not been considered as to the merits. Applicant is advised that the date of any re-submission of any item of information contained in this information disclosure statement or the submission of any missing element(s) will be the date of submission for purposes of determining compliance with the requirements based on the time of filing the statement, including all certification requirements for statements under 37 CFR 1.97(e). See MPEP § 609.05(a).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 12-15, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai et al. (hereinafter Bai), Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees in view of Ede et al. (hereinafter Ede), Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI.
Regarding Claim 1, Bai discloses a computer-implemented method for incorporating syntactic knowledge into a pre- trained language model [“Pre-trained language models like BERT” Abstract], the method comprising:
selecting, using at least one hardware processor, at least one syntactic pre-training task from among a set of syntactic pre-training tasks [“syntactic information has been proved to be crucial for the success of NLP applications.” Abstract; “incorporates syntactic knowledge” §1 ¶2; Fig. 1; Examiner Note: Figure 1 shows finding different parts of speech];
retraining, for the selected at least one syntactic pre-training task and using the at least one hardware processor, the pre-trained language model by using an optimization function which prevents catastrophic forgetting during the retraining, and wherein the retraining adds syntactic [“syntactic information has been proved to be crucial for the success of NLP applications” Abstract] knowledge to the pre-trained language model while preserving original semantic [“Pre-trained language models like BERT” Abstract] knowledge of the pre-trained language model, and without changing a format of the pre-trained language model; and
performing, using the at least one hardware processor, inferencing using the retrained language model [“we evaluate the performance of Syntax-BERT on two natural language inference datasets: SNLI and MNLI.” §4 ¶1].
However, Bai fails to explicitly disclose retraining, for the selected at least one syntactic pre-training task and using the at least one hardware processor, the pre-trained language model by using an optimization function which prevents catastrophic forgetting during the retraining, and wherein the retraining adds syntactic knowledge to the pre-trained language model while preserving original semantic knowledge of the pre-trained language model, and without changing a format of the pre-trained language model.
Ede discloses retraining, for the selected at least one syntactic pre-training task and using the at least one hardware processor, the pre-trained language model by using an optimization function which prevents catastrophic forgetting during the retraining, and wherein the retraining adds syntactic knowledge to the pre-trained language model while preserving original semantic knowledge of the pre-trained language model, and without changing a format of the pre-trained language model [“Relevance-based Neural Freezing (RNF), a novel approach to alleviate catastrophic forgetting that builds upon the aforementioned pruning technique. Instead of compressing the network, the information about unit importance is used to freeze the knowledge represented by the learned network parameters by inhibiting or completely stopping the training process for those parts of the model that are relevant for a specific task, while the remaining units are free to learn further tasks.” §1 ¶3; Fig. 1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai and Ede before him before the effective filing date of the claimed invention, to modify the method of Bai using NLP and syntactic information to incorporate the knowledge retention with additional training of Ede.
Given the advantage of preventing catastrophic forgetting by learning new tasks without forgetting previous tasks and which increases the utility of a model, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 2, Bai and Ede disclose the method of claim 1. Bai further discloses wherein the set of syntactic pre-training tasks are selected from a group consisting of a deprel prediction task, a phrase detection task, and a main/subordinate detection task [“All phrases are labeled to one of five fine-grained categories of sentiment polarity.” §4.1 ¶1; Fig. 1].
Regarding Claim 3, Bai and Ede disclose the method of claim 1. Bai further discloses wherein the retraining further comprises learning at least one syntactic relationship between two tokens [“reflecting different relationships and distances of tokens in a syntax tree” §3.1 ¶2; Fig. 1].
Regarding Claim 12, Bai and Ede disclose the method of claim 1. Bai further discloses further comprising applying the retrained language model to a downstream task to perform inferencing for language understanding [“we evaluate the performance of Syntax-BERT on two natural language inference datasets: SNLI and MNLI.” §4 ¶1].
Claim 13 is rejected on the same grounds as claim 1.
Claim 14 is rejected on the same grounds as claim 1.
Claim 15 is rejected on the same grounds as claim 3.
Claim 20 is rejected on the same grounds as claim 12.
Claim(s) 4-5, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Cormack, What is coordination?.
Regarding Claim 4, Bai and Ede disclose the method of claim 1.
However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a coordination detection task for predicting a label of a parallel structure, wherein the parallel structure is one of a token, a phrase, and a sentence.
Cormack discloses wherein the set of syntactic pre-training tasks includes a coordination detection task for predicting a label of a parallel structure, wherein the parallel structure is one of a token, a phrase, and a sentence [“Coordination is a particular syntactic manifestation of conjunction (or disjunction), which is symmetric with respect to the conjuncts/disjuncts (i.e. the conjuncts/disjuncts have the same syntactic status).” §1 ¶3; Examples 4 on page 398].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Cormack before him before the effective filing date of the claimed invention, to modify the combination to incorporate the coordination analysis of Cormack.
Given the advantage of using a different method of language analysis which permits finding potentially missed relationships, one having ordinary skill in the art would have been motivated to make this obvious modification.
Regarding Claim 5, Bai, Ede, and Cormack disclose the method of claim 4.
However, Bai fails to explicitly disclose wherein the label is a label of a head token, a child token, or a parallel conjunction of the corresponding parallel structure.
Cormack discloses wherein the label is a label of a head token, a child token, or a parallel conjunction of the corresponding parallel structure [“Coordination is a particular syntactic manifestation of conjunction” §1 ¶3; “the grammar contains combinators. These occupy the head positions” §1 ¶9].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Cormack before him before the effective filing date of the claimed invention, to modify the combination to incorporate the coordination analysis of Cormack.
Given the advantage of using a different method of language analysis which permits finding potentially missed relationships, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 16 is rejected on the same grounds as claim 4.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Nivre et al. (hereinafter Nivre), Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection.
Regarding Claim 6, Bai and Ede disclose the method of claim 1.
However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a deprel prediction task for predicting dependency labels.
Nivre discloses wherein the set of syntactic pre-training tasks includes a deprel prediction task for predicting dependency labels [“UD provides a taxonomy of 37 universal relation types to classify syntactic relations” §2.3 ¶2; Table 1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Nivre before him before the effective filing date of the claimed invention, to modify the combination to incorporate the use of deprel through universal dependencies.
Given the advantage of a structured and efficient method to aid in the model’s accuracy, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Basirat et al. (hereinafter Basirat), Syntactic Nuclei in Dependency Parsing– A Multilingual Exploration.
Regarding Claim 7, Bai and Ede disclose the method of claim 1.
However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a phrase detection task for predicting a relationship between phrases.
Basirat discloses wherein the set of syntactic pre-training tasks includes a phrase detection task for predicting a relationship between phrases [“the elementary syntactic units of a dependency tree are nuclei rather than word forms, then the English and Finnish sentences will have the same dependency trees” §1 ¶2; Fig. 2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Basirat before him before the effective filing date of the claimed invention, to modify the combination to incorporate the nuclei concept which uses phrases for syntax of Basirat.
Given the advantage of accuracy forming phrase comparisons for syntax analysis, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Nikolaev et al. (hereinafter Nikolaev), Word-order typology in Multilingual BERT: A case study in subordinate-clause detection.
Regarding Claim 8, Bai and Ede disclose the method of claim 1.
However, Bai fails to explicitly disclose wherein the set of syntactic pre-training tasks includes a main/subordinate detection task for predicting two labels to classify clauses into main and subordinate.
Nikolaev discloses wherein the set of syntactic pre-training tasks includes a main/subordinate detection task for predicting two labels to classify clauses into main and subordinate [“capture the distinction between main and subordinate clauses” §1 ¶2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Nikolaev before him before the effective filing date of the claimed invention, to modify the combination to incorporate main and subordinate distinction in language of Nikolaev.
Given the advantage of capturing the distinction between main and subordinate clauses to better distinguish language for processing, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 9 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Tian et al. (hereinafter Tian), Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking.
Regarding Claim 9, Bai and Ede disclose the method of claim 1.
However, Bai fails to explicitly disclose further comprising learning, in parallel, a dependency masking (DM), which predicts whether there is a dependency between two words, and a masked dependency prediction (MDP), which predicts a type of dependency relationship, and wherein input for the masked dependency prediction is replaced with one of the four syntactic pre-training tasks.
Tian discloses further comprising learning, in parallel, a dependency masking (DM), which predicts whether there is a dependency between two words, and a masked dependency prediction (MDP), which predicts a type of dependency relationship [“a weakly supervised learning task, namely, dependency masking (DM) with masked dependency prediction (MDP), to enhance text encoder pre-training, where they are paired processes that DM masks all connections and dependency types” §2.2 ¶1], and wherein input for the masked dependency prediction is replaced with one of the four syntactic pre-training tasks [“leverage auto-parsed dependency information through learning word-word associations (i.e., dependency connections) and their dependency types” §2.2 ¶1].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Tian before him before the effective filing date of the claimed invention, to modify the combination to incorporate the DM and MDP of Tian.
Given the advantage of increasing performance for relation extraction, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 17 is rejected on the same grounds as claim 9.
Claim(s) 10 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Yu et al. (hereinafter Yu), Gradient Surgery for Multi-Task Learning.
Regarding Claim 10, Bai and Ede disclose the method according to claim 1.
However, Bai fails to explicitly disclose wherein the optimization function is configured for solving gradient conflicts in multi-task learning and computes gradients for each task involved in multi-task learning, discards adversarial elements of the gradients that conflict with each other, and sums resultant gradients to obtain a single gradient vector.
Yu discloses wherein the optimization function is configured for solving gradient conflicts in multi-task learning [“one of the main optimization issues in multi-task learning arises from gradients from different tasks conflicting with one another in a way that is detrimental to making progress” §1 ¶2] and computes gradients for each task involved in multi-task learning, discards adversarial elements of the gradients that conflict with each other, and sums resultant gradients to obtain a single gradient vector [“To deconflict gradients during optimization, PCGrad adopts a simple procedure: if the gradients between two tasks are in conflict, i.e. their cosine similarity is negative, we project the gradient of each task onto the normal plane of the gradient of the other task. This amounts to removing the conflicting component of the gradient for the task, thereby reducing the amount of destructive gradient interference between tasks.” §2.3 ¶3; “g = g1 + g2” Definition 4].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Yu before him before the effective filing date of the claimed invention, to modify the combination to incorporate gradient surgery of Yu.
Given the advantage of a substantial improvements in terms of data efficiency, optimization speed, and final performance, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 18 is rejected on the same grounds as claim 10.
Claim(s) 11 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bai and Ede in view of Kirkpatrick et al. (hereinafter Kirkpatrick), Overcoming catastrophic forgetting in neural networks.
Regarding Claim 11, Bai and Ede disclose the method according to claim 1.
However, Bai fails to explicitly disclose wherein the optimization function is configured for continuous learning optimization in which tasks are learned in sequence and, given two tasks, A and B, first searches for an optimal solution for task A, then searches for parameters that perform well in both tasks A and B and, when the model is fine-tuned for B after A, updates more important parameters of the task A with smaller weights while updating less important parameters of the task A with larger weights.
Kirkpatrick discloses wherein the optimization function is configured for continuous learning optimization in which tasks are learned in sequence and, given two tasks, A and B, first searches for an optimal solution for task A, then searches for parameters that perform well in both tasks A and B and, when the model is fine-tuned for B after A, updates more important parameters of the task A with smaller weights while updating less important parameters of the task A with larger weights [“Continual learning poses particular challenges for artificial neural networks due to the tendency for knowledge of previously learnt task(s) (e.g. task A) to be abruptly lost as information relevant to the current task (e.g. task B) is incorporated.” §1 ¶2; “over-parameterization makes it likely that there is a solution for task B, θ*B, that is close to the previously found solution for task A, θ*A. While learning task B, EWC therefore protects the performance in task A by constraining the parameters to stay in a region of low error for task A centered around θ*A, as shown schematically in Figure 1. This constraint is implemented as a quadratic penalty, and can therefore be imagined as a spring anchoring the parameters to the previous solution, hence the name elastic. Importantly, the stiffness of this spring should not be the same for all parameters; rather, it should be greater for those parameters that matter most to the performance during task A.” §2 ¶2].
It would have been obvious to one having ordinary skill in the art, having the teachings of Bai, Ede, and Kirkpatrick before him before the effective filing date of the claimed invention, to modify the combination to incorporate elastic weight consolidation of Kirkpatrick.
Given the advantage of learning new tasks without forgetting previous tasks which increases the utility of a model, one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim 19 is rejected on the same grounds as claim 11.
Examiner’s Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification.
Response to Arguments
Regarding the prior art rejections, Applicant's arguments with respect to the claims have been considered but are moot because the arguments do not apply to the references being used in the current rejection of the limitations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.B./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148