DETAILED ACTION
The Response filed on 11/26/2025 has been correspondingly accepted and considered in the office action. Claims 1-20 are pending. Claims 1, 14 and 15 are independent and amended. Claims 16-20 are new.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The rejection to Claim 15 under 35 U.S.C. § 101 because the claimed invention is directed to non-statutory subject matter has been withdrawn in view of Applicant’s amendments to the claims.
The rejections to Claims 1-15 under 35 U.S.C. § 101 as being directed to Abstract Idea without significantly more have been withdrawn in view of Applicant’s amendments to the claims and persuasive arguments.
Response to Arguments
Claims 1-15 stand rejected under 35 U.S.C. § 103. Applicant’s arguments with respect to Claims 1-15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
In order to expedite prosecution, and as to the material from the Specifications that are not in the Claim and are argued by the Applicant, please note He et al. ("BERT-MK: Integrating graph contextualized knowledge into pre-trained language models." Findings of the Association for Computational Linguistics: EMNLP 2020, 2020).
For at least the supra provided reasons, Applicant's arguments have been fully considered but they are not persuasive.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 and 11-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al., (US Pub No. 2022/0075945, hereinafter, Zhang) in view of Xie et al., ("A contextual alignment enhanced cross graph attention network for cross-lingual entity alignment." Proceedings of the 28th International Conference on Computational Linguistics. 2020, hereinafter, Xie) further in view of He et al., ("BERT-MK: Integrating graph contextualized knowledge into pre-trained language models." Findings of the Association for Computational Linguistics: EMNLP 2020, 2020).
Regarding Claim 1,
Zhang discloses a method for representation learning of cross-language texts (Zhang, par [004], "…a computer-implemented method is provided for cross-lingual transfer..."), comprising:
obtaining a source language text and a target language text (Zhang, Fig.2, paras [032-034], source corpus 201 and target corpus 202);
generating an initial joint representation of the source language text and the target language text (Zhang, paras [018, 060-064], "…At block 640, input the source dependency parsing tree and the target dependency parsing tree into a graph encoder pretrained on a translation language modeling task to extract common language information for transfer....");
Zhang does not explicitly discloses the limitations, “identifying relations among a plurality of words in the source language text and the target language text; generating a joint representation of the source language text and the target language text based on the initial joint representation and the relations; projecting the joint representation to at least a target language representation corresponding to the target language text.”
However, Xie, in the analogous field of endeavor, discloses identifying relations among a plurality of words in the source language text and the target language text (Xie, Fig 2, 2.2 Cross-KG Aggregation, pg. 5920, "…The cross-KG aggregation is used to transfer graph information across different KGs through seed alignments. By taking full advantage of pre-aligned entity pairs, we can make full use of pre-aligned neighbors as contextual information to predict new alignments....");
generating a joint representation of the source language text and the target language text based on the initial joint representation and the relations (Xie, Fig.2, 2.3 Attention-based Cross-KG Propagation, pg. 5921-5922, "…we use an attention-based cross-KG propagation layer, which is inspired by graph attention network (GAT) (Velickovic et al., 2017), to aggregate neighborhood features...By stacking L cross-KG aggregation and attention-based cross-KG propagation layers, we can obtain new entity embeddings, which contain both cross-KG and multi-hop neighborhood information...");
projecting the joint representation to at least a target language representation corresponding to the target language text (Xie, Algorithm 1, 2.4 Optimization and Prediction, pg. 5922-5923 , "…13: Use the learned CAECGAT model to predict the final entity embeddings E1 and E2 (E1 and E2 are entries in knowledge graphs, G1 and G2, in different languages, respectively)...").
Therefore, It would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified the Graph Convolution Network (GCN)-based cross-lingual transfer via semantic and synthetic representation learning of Zhang with a contextual alignment enhanced cross graph attention network (CAECGAT) with a reasonable expectation of success to jointly learn the embeddings in different knowledge graphs to improve existing GNN-based methods, which model different KGs separately and ignore the useful pre-aligned links between two KGs (Xie, Abstract and Introduction).
Zhang further discloses pre-training a representation obtaining model by:
masking one or more nodes in a node set of a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample in a graph constructed based on the relations (Zhang, par [018], "…Graph Encoder Pretraining: dependency parsing trees in Step 1 are fed to a graph encoder, which is pretrained on the Translation Language Modeling (TLM) task..."; i.e., TLM is an extension of the masked language modeling (MLM) and is used to predict the masked tokens in translation task ; par [035], "…Regarding mark and tokenization block/step 203, the source corpus 201 and target corpus 202 are randomly masked, tokenized by pretrained NLP models, and are transformed to representations of dependency parsing trees..." );
for each masked node, recovering the node with at least a representation of at least one neighbor node of the node (Zhang, Fig.2, par [037], "…mask language prediction block/step 205...
H
s
2
and
H
t
2
(i.e., source and target representations in the graph network layer) are concatenated and denoted as X=[
H
s
2
H
t
2
]. The TLM task is trained in this step: it is a filling-in-blank task where the mask tokens are predicted by the model according to their surrounding context of words...");
Zhang discloses the graph encoder (i.e., for representation) pretraining based on the masked language modeling (MLM), but neither Zhang nor Xie discloses the following limitations. However, He, in the analogous field of endeavor discloses computing a recovery loss based on the recovery of the masked node: and updating parameters of the representation obtaining model by minimizing the recovery loss (He, Fig.3, 1 Introduction, "…we propose an approach to learn knowledge from subgraphs, and inject graph contextualized knowledge into the pretrained language model..."; 2.1 Learning Graph Contextualized Knowledge, "…we propose a Transformer-based (Vaswani et al., 2017)..."; 2.1.2 GCKE, "…the graph contextualized knowledge embedding learning module, called GCKE, as shown in Figure 3...The inputs are fed into a Transformer-based model to encode the node information…The Masking function in Equation 3 restraints the contextualized dependency among the input nodes...The node embeddings are learned by minimizing a margin-based loss function on the training data:
PNG
media_image1.png
48
312
media_image1.png
Greyscale
Γ > 0 is a margin hyperparameter, f(t) is an entity replacement operation that the head entity or the tail entity in a triple is replaced and the replaced triple is an invalid triple in the KG...").
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a contextually aligned cross-lingual representation via a graph attention network taught by Zhang in view of Xie with a transformer-based model to encode the node information of He with a reasonable expectation of success to exploit graph-level knowledges and integrate the learned knowledge with a pre-trained language model to do the knowledge generalization (He, Abstract).
Regarding Claim 2,
Zhang in view of Xie further in view of He discloses the method of claim 1.
Zhang further discloses wherein the source language text includes a set of source language words, the target language text includes a set of target language words (Zhang, Fig.2, paras [033-034], the source corpus 201 and target corpus 202 are a multilingual texts written in natural languages. The target corpus is a translation of the source corpus), and the identifying relations comprises:
Xie further discloses identifying alignment relations between the set of source language words and the set of target language words (Xie, Fig.2, 2.1 Overview, "…Figure 2 illustrates the structure of the proposed CAECGAT model...given two different KGs G1 and G2, and a collection of pre-aligned entity pairs, we first use the cross-KG aggregation layer to transfer the entity information across the two KG..."); and/or
Zhang further discloses identifying dependency relations among the set of source language words and dependency relations among the set of target language words (Zhang, Fig.6, par [063], "…At block 630, transform the source tokens and the target tokens into a source dependency parsing tree and a target dependency parsing tree...").
Regarding Claim 3,
Zhang in view of Xie further in view of He discloses the method of claim 1, further comprising:
Zhang further discloses constructing a graph corresponding to the source language text and the target language text based on the relations (Zhang, Fig.6, par [064], "…At block 640, input the source dependency parsing tree and the target dependency parsing tree into a graph encoder pretrained on a translation language modeling task..."), and
Xie further discloses wherein the generating a joint representation comprises: updating the initial joint representation to the joint representation based on the graph (Xie, 2.3 Attention-based Cross-KG Propagation, "…We have encoded cross-KG information in the entity embeddings by applying the cross-KG aggregation layer, we can further propagate the cross-KG information using GNNs...use an attention-based cross-KG propagation layer to aggregate neighborhood features...").
Regarding Claim 4,
Zhang in view of Xie further in view of He discloses the method of claim 3, wherein the constructing a graph comprises:
Zhang further discloses setting a set of source language words in the source language text and a set of target language words in the target language text as a plurality of nodes (Zhang, Fig.2, par [035], "…the source corpus 201 and target corpus 202 are randomly masked, tokenized by pretrained NLP models, and are
transformed to representations of dependency parsing trees..."; "…(1) token representation by the NLP pretrained models; (2) Part-Of-Speech (POS) tagging...");
determining a set of edges among the plurality of nodes based on the relations (Zhang, Fig.2, par [035], "…(3) Universal Dependency (UD) relation..."); and
combining the plurality of nodes and the set of edges into the graph (Zhang, Fig.2, par [036], "…a GCN is designed as a graph encoder for knowledge transfer…").
Regarding Claim 5,
Zhang in view of Xie further in view of He discloses the method of claim 3, wherein the initial joint representation is updated through iteratively performing an update operation, the update operation comprising:
Xie further discloses obtaining current attention information based on the graph and a previous joint representation (Xie, 2.3 Attention-based Cross-KG Propagation, pg. 5921-5922, "…we use an attention-based cross-KG propagation layer to aggregate neighborhood features..."; "…Given the output entity embeddings of cross-KG aggregation layer, e.g., H1 and H2 for two KGs G1 and G2..."); and
updating the previous joint representation to a current joint representation based on the current attention information (Xie, 2.3 Attention-based Cross-KG Propagation, pg. 5922, "…we use a graph attention mechanism to update the entity embeddings by gathering neighborhood information..."; See Eq.5
PNG
media_image2.png
81
288
media_image2.png
Greyscale
where crossAtt is the attention-based cross-KG propagation function).
Regarding Claim 11,
Zhang in view of Xie further in view of He discloses the method of claim 1, wherein the joint representation is generated through a representation obtaining model, pre-training of the representation obtaining model comprising at least:
Zhang further discloses masking one or more nodes in only one node set of a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample (Zhang, par [018], "…Graph Encoder Pretraining: dependency parsing trees in Step 1 are fed to a graph encoder, which is pretrained on the Translation Language Modeling (TLM) task..."; i.e., TLM is an extension of the masked language modeling (MLM) and is used to predict the masked tokens in translation task ; par [035], "…Regarding mark and tokenization block/step 203, the source corpus 201 and target corpus 202 are randomly masked, tokenized by pretrained NLP models, and are transformed to representations of dependency parsing trees..." ); and
for each node of the one or more nodes, recovering the node with at least a representation of at least one alignment neighbor node of the node (Zhang, Fig.2, par [037], "…mask language prediction block/step 205...
H
s
2
and
H
t
2
(i.e., source and target representations in the graph network layer) are concatenated and denoted as X=[
H
s
2
H
t
2
]. The TLM task is trained in this step: it is a filling-in-blank task where the mask tokens are predicted by the model according to their surrounding context of words...").
Regarding Claim 12,
Zhang in view of Xie further in view of He discloses the method of claim 1, wherein the joint representation is generated through a representation obtaining model, pre-training of the representation obtaining model comprising at least:
Zhang further discloses masking one or more node pairs (Zhang, par [018], "…Graph Encoder Pretraining: dependency parsing trees in Step 1 are fed to a graph encoder, which is pretrained on the Translation Language Modeling (TLM) task..."; i.e., TLM is an extension of the masked language modeling (MLM) and is used to predict the masked tokens in translation task ; par [035], "…Regarding mark and tokenization block/step 203, the source corpus 201 and target corpus 202 are randomly masked, tokenized by pretrained NLP models, and are transformed to representations of dependency parsing trees..." ), and Xie further discloses one or more node pairs having alignment edges in a source language node set corresponding to a source language text sample and a target language node set corresponding to a target language text sample (Xie, Fig.2, 2.1 Overview, "…Figure 2 illustrates the structure of the proposed CAECGAT model...given two different KGs G1 and G2, and a collection of pre-aligned entity pairs, we first use the cross-KG aggregation layer to transfer the entity information across the two KG..."); and
Zhang further discloses for each node of the one or more node pairs, recovering the node with a representation of at least one dependency neighbor node of the node (Zhang, Fig.2, par [037], "…mask language prediction block/step 205...
H
s
2
and
H
t
2
(i.e., source and target representations in the graph network layer) are concatenated and denoted as X=[
H
s
2
H
t
2
]. The TLM task is trained in this step: it is a filling-in-blank task where the mask tokens are predicted by the model according to their surrounding context of words...").
Regarding Claim 13,
Zhang in view of Xie further in view of He discloses the method of claim 11, wherein the pre-training of the representation obtaining model further comprises:
Zhang further discloses recovering the node with a representation of the node (Zhang, Fig.2, par [037], "…mask language prediction block/step 205...
H
s
2
and
H
t
2
(i.e., source and target representations in the graph network layer) are concatenated and denoted as X=[
H
s
2
H
t
2
]. The TLM task is trained in this step: it is a filling-in-blank task where the mask tokens are predicted by the model according to their surrounding context of words...").
Claim 14 is an apparatus claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally,
Zhang discloses an apparatus for representation learning of cross-language texts, comprising: at least one processor (Fig.1, par [021], "…processor..."); and a memory storing computer-executable instructions that, when executed, cause the at least one processor to (Fig.1, par [023], "…memory 130 may store various data and software used during operation of the computing device 100, such as operating systems, applications, programs, libraries, and drivers..."):
…
Rationale for combination is similar to that provided for Claim 1.
Claim 15 is a non-transitory machine-readable medium claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally,
Zhang discloses at least one non-transitory machine-readable medium comprising instruction for representation learning of cross-language texts that, when executed by at least one processor (Zhang, par [067], "… a computer readable storage medium ( or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention..."):
…
Rationale for combination is similar to that provided for Claim 1.
Claim 16 is a non-transitory machine-readable medium claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 17 is a non-transitory machine-readable medium claim with limitations similar to the limitations of Claim 3 and is rejected under similar rationale.
Claim 18 is a non-transitory machine-readable medium claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale.
Claim 19 is a non-transitory machine-readable medium claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Claims 6-10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Xie further in view of He further in view of Hajarnis et al., (US Pub No. 2022/0051080, hereinafter, Hajarnis).
Regarding Claim 6,
Zhang in view of Xie further in view of He discloses the method of claim 5, but Zhang, Xie, nor He explicitly discloses the obtaining current attention information.
However, Hajarnis, in the analogous field of endeavor, discloses calculating a current attention score corresponding to every two words in the source language text and the target language text based on the graph, to obtain a set of current attention scores (Hajarnis, Fig.2, par [031], "…Attention layer 206 can be a self-attention layer that may calculate vectors of attention scores for input data, where a vector of attention scores associated with an input data point may represent the relevancies between the present input data point with other input data points..."; par [032], "…each decoder in stacked-up decoders may also include an attention layer 210 and a feed forward layer 212...the source language may be fed into the stacked-up encoders, and the documents containing the target language may be fed into the stacked-up decoders..."); and
combining the set of current attention scores into the current attention information (Hajarnis, Fig.2, par [031], "…The vectors of attention scores may be fed into the feed forward layer 208. Feed forward layer 208 may perform linear calculations that may transform the vectors of attention scores into a form that may be fed into a next encoder or decoder..."; par [032], "…stacked-up encoders 202 and stacked-up decoders 204 are fully connected, meaning each decoder may receive not only the final but all intermediate encoded representation outputs from stacked-up encoders 202…").
Therefore, It would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a contextually aligned cross-lingual representation via a graph attention network taught by Zhang in view of Xie further in view of He with transformer neural network models for processing text-based document of Hajarnis with a reasonable expectation of success to overcome the deficiencies of RNN and CNN architectures, thus achieving the determination of word dependencies among all words in a sentence with fast implementations using TPUs and GPUs. (Hajarnis, paras [011-018]).
Regarding Claim 7,
Zhang in view of Xie further in view of He further in view of Hajarnis discloses the method of claim 6, wherein the calculating a current attention score comprises:
Hajarnis further discloses obtaining a current word representation of each of the two words based at least on the graph (Hajarnis, Fig.2, par [029], "…A first encoder in the stacked-up encoders 202 may receive a segment of word embeddings 114A-114D, and then process to generate a first encoded representation output..."); and
calculating the current attention score based on two current word representations corresponding to the two words (Hajarnis, Fig.2, par [031], "… each encoder may include an attention layer 206 and a feed forward layer 208. Attention layer 206 can be a self-attention layer that may calculate vectors of attention scores for input
Data...").
Regarding Claim 8,
Zhang in view of Xie further in view of He further in view of Hajarnis discloses the method of claim 7, wherein the obtaining a current word representation comprises:
Xie further discloses obtaining a previous word representation corresponding to the word from the previous joint representation (Xie, Fig.2, 2.1 Overview, "…Figure 2 illustrates the structure of the proposed CAECGAT model, which consists of multiple CGAT layers. A CGAT layer contains a cross-KG aggregation layer and an attention-based cross-KG propagation layer...By stacking multiple CGAT layers (i.e., using output from a previous word representation)");
identifying at least one neighbor node of a node corresponding to the word from the graph (Xie, 2.2 Cross-KG Aggregation, "...give two different KGs G1 = (E1;R1; T1) and G2 = (E2;R2; T2), and a set of seed alignments A = {(e1; e2) | e1 Î E1; e2 Î E2}…"); e1 and e2 are entities (i.e., nodes) in knowledge graph G1 and G2, respectively); and
updating the previous word representation to the current word representation based at least on a representation of the at least one neighbor node (Xie, Fig.2, 2.3 Attention-based Cross-KG Propagation, pg. 5921-5922, "…we use an attention-based cross-KG propagation layer, which is inspired by graph attention network (GAT) to aggregate neighborhood features...Our attention-based cross-KG propagation layer aims to select the most important common neighbors shared by different KGs to enhance the entity embeddings...").
Regarding Claim 9,
Zhang in view of Xie further in view of He further in view of Hajarnis discloses the method of claim 8.
Xie further discloses wherein the at least one neighbor node includes at least one alignment neighbor node having an alignment edge with the node (Xie, Fig.1, See an example of cross-lingual entity with contextual alignment), and
the updating the previous word representation is further based on a semantic difference between the at least one alignment neighbor node and the node (Xie, Fig.2, 2.3 Attention-based Cross-KG Propagation, pg. 5921-5922, "…we use an attention-based cross-KG propagation layer, which is inspired by graph attention network (GAT) to aggregate neighborhood features...Our attention-based cross-KG propagation layer aims to select the most important common neighbors shared by different KGs to enhance the entity embeddings..."; "…the semantic gap can be alleviated by focusing on the common neighbors with cross-KG information and reducing noising neighbors…." )
Regarding Claim10,
Zhang in view of Xie further in view of He further in view of Hajarnis discloses the method of claim 8.
Zhang further discloses wherein the at least one neighbor node includes at least one dependency neighbor node having a dependency edge with the node (Zhang, Fig.5, par [057], "…POS tag 501 and dependency relation 502 are extracted from the dependency parsing tree of each natural language sentence...
The dependency relation shows the relation between two connected tokens..."), and
the updating the previous word representation is further based on an importance of the at least one dependency neighbor node relative to the node ((Zhang, Fig.5, par [057] "…We use a pre-trained cross-lingual language model to represent the token semantics as node embedding 506…the linguistic-informed embedding 508 is used as the input of fully connected neural network layers 510 to generate the output...").
Claim 20 is a non-transitory machine-readable medium claim with limitations similar to the limitations of Claim 6 and is rejected under similar rationale. Rationale for combination is similar to that provided for Claim 6.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Conneau, et al., "Cross-lingual language model pretraining." Advances in neural information processing systems 32, 2019, hereinafter, Conneau), discloses two methods to learn cross-lingual language models (XLMs): one unsupervised
that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective (Conneau, Abstract).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/Examiner, Art Unit 2656 /BHAVESH M MEHTA/Supervisory Patent Examiner, Art Unit 2656