Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
2. This action is in response to the original filing on 05/17/2024. Claims 12-21 are pending and have been considered below.
Information Disclosure Statement
3. The information disclosure statement (IDS(s)) submitted on 05/17/2024 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections – 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 12, 16, and 18-21 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (U.S. Patent Application Pub. No. US 20220230625 A1) in view of Yang et al. (Aligning Cross-Lingual Entities with Multi-Aspect Information; arXiv, published 2019, pages 1-11).
Claim 12: Zhu teaches a computer-implemented method for entity alignment (i.e. After obtaining the first and second knowledge graphs, the first knowledge graph and second knowledge graphs are aligned such that a first subset of entities and relations from the first knowledge graph correspond to a second subset of entities and relations from the second knowledge graph; para. [0178]), the method comprising the following steps:
obtaining a first plurality of initial embeddings of entities of a first graph and a second plurality of initial embeddings of entities of a second graph (i.e. The initial entity embeddings and relation embeddings are generated from the first language module 320 … The first language module 320 also generates initial entity and relation embeddings for the knowledge module (e.g., graph convolution network 340) based on the entity description text 310 and entity relation text … The language module of the integrated knowledge-language module is further configured to produce contextual representations as initials embeddings for knowledge graph entities and relations; para. [0078, 0084, 0127]) based on a pre-trained model (i.e. the first language module 320 and the second language module are formed as the first several transformer layers and the rest layers of a pre-trained language model (e.g., as initialized from BERT or RoBERTA); para. [0073, 0081]), wherein the first plurality of embeddings and the second plurality of embeddings are in a unified space (i.e. The integrating of the language module and knowledge module comprises projecting entity and relations output embeddings and language text embeddings into a shared semantic space; para. [0128]); and
learning entity alignment between the first graph and the second graph over the first plurality of initial embeddings and the second plurality of initial embeddings (i.e. a multi-layer language model (e.g., a first language module 320 and a second language module 360) where the first language module provides embeddings (see embeddings at locations 1 and 2) for both the second language module 360 and a knowledge module configured as a graph convolution network 340. The entity embeddings (e.g., entity representation 342) from the knowledge module are also fed into the second language module (see information fusion 344), which produces the final representation (e.g., context representation 362); para. [0072, 0077,0078]) to using a relative similarity metric (i.e. the model predicts the entity whose embedding in the ECEM 322 is closest to the mentioned entity. Since the number of entities is very large in some instances, the model uses the entity's neighbors and other randomly sampled entities as negative samples. The representational loss is minimized between the predicted embedding and the original embedding using a loss function which is a cross entropy based on the inner product between an embeddings and each candidate entity's embedding; para. [0082, 0083]), wherein negative sampling for non-aligned entities of an entity of one of the first graph or the second graph is performed on the one of the first graph or the second graph during the learning (i.e. the model predicts the entity whose embedding in the ECEM 322 is closest to the mentioned entity. Since the number of entities is very large in some instances, the model uses the entity's neighbors and other randomly sampled entities as negative samples. The representational loss is minimized between the predicted embedding and the original embedding using a loss function which is a cross entropy based on the inner product between an embeddings and each candidate entity's embedding; para. [0079, 0082, 0083]).
Zhu does not explicitly teach wherein the first plurality of initial embeddings and the second plurality of initial embeddings are in a unified space; and learning entity alignment between the first graph and the second graph over the first plurality of initial embeddings and the second plurality of initial embeddings by at least one encoder to push non-aligned entities far away using a relative similarity metric.
However, Yang teaches wherein the first plurality of initial embeddings and the second plurality of initial embeddings are in a unified space (i.e. the goal is to embed cross-lingual entities into the same low-dimensional vector space where equivalent entities are close to each other … The spirit of BERT in the multilingual scenario is to project words or sentences from different languages into the same semantic space; Section 1, 3.1, 3.2, 3.3, pages 1-6); and
learning entity alignment between the first graph and the second graph over the first plurality of initial embeddings and the second plurality of initial embeddings by at least one encoder (i.e. two GCN-based models, namely MAN and HMAN, that learn en tity embeddings from the graph structures. Second, we discuss two uses of a multilingual pre trained BERT model to learn cross-lingual embeddings of entity descriptions: POINTWISEBERT and PAIRWISEBERT. Finally, we investigate two strategies to integrate the GCN-based and the BERT-based modules; Section 3, 3.1, pages 2-3) to push non-aligned entities far away using a relative similarity metric (i.e. Given two knowledge graphs, G1 and G2, and a set of pre-aligned entity pairs I (G1,G2) as training data, our model is trained in a supervised fashion. During the training phase, the goal is to embed cross-lingual entities into the same low-dimensional vector space where equivalent entities are close to each other, our margin-based ranking loss function is defined as: Equation 4, where [x]+ = max{0,x}, I denotes the set of negative entity alignment pairs constructed by corrupting the gold pair (e1,e2) ∈ I. Specifically, we replace e1 or e2 with a randomly-chosen entity in E1 or E2. ρ(x,y) is the 1 distance function, and β > 0 is the margin hyperparameter separating positive and negative pairs; Section 3.1, pages 3-4), wherein negative sampling for non-aligned entities of an entity of one of the first graph or the second graph is performed on the one of the first graph or the second graph during the learning (i.e. Given two knowledge graphs, G1 and G2, and a set of pre-aligned entity pairs I (G1,G2) as training data, our model is trained in a supervised fashion. During the training phase, the goal is to embed cross-lingual entities into the same low-dimensional vector space where equivalent entities are close to each other, our margin-based ranking loss function is defined as: Equation 4, where [x]+ = max{0,x}, I denotes the set of negative entity alignment pairs constructed by corrupting the gold pair (e1,e2) ∈ I. Specifically, we replace e1 or e2 with a randomly-chosen entity in E1 or E2. ρ(x,y) is the 1 distance function, and β > 0 is the margin hyperparameter separating positive and negative pairs; Section 3.1, pages 3-4).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Zhu to include the feature of Yang. One would have been motivated to make this modification because it provides an accurate way to distinguish corresponding entities from non-corresponding candidates and to discover additional alignments between graphs.
Claim 16: Zhu and Yang teach the computer-implemented method of claim 12. Zhu further teaches wherein the at least one encoder is shared between the first graph and the second graph (i.e. a multi-layer language model (e.g., a first language module 320 and a second language module 360) where the first language module provides embeddings (see embeddings at locations 1 and 2) for both the second language module 360 and a knowledge module configured as a graph convolution network 340; para. [0072, 0077, 0177-0181]).
Claim 18: Zhu and Yang teach the computer-implemented method of claim 12. Zhu further teaches wherein the obtaining of the first plurality of initial embeddings and the second plurality of initial embeddings further includes aggregating information of neighbor entities (i.e. The goal of the knowledge module is to model the knowledge graph 330 to generate knowledge-based entity representation (e.g., entity representation 342). To compute entity node embeddings, a graph attention network is employed which uses the self-attention mechanism to specify different weights for different neighboring nodes … a dual-layer graph neural network is used which aggregates 2-hop neighbors; para. [0077, 0079, 0089]).
Claim 19: Zhu teaches a computer-implemented method for entity alignment of graphs (i.e. After obtaining the first and second knowledge graphs, the first knowledge graph and second knowledge graphs are aligned such that a first subset of entities and relations from the first knowledge graph correspond to a second subset of entities and relations from the second knowledge graph; para. [0178]) with natural languages (i.e. The first language knowledge graph 1320 and second language knowledge graph 1322 are optionally language or textual-based knowledge graphs corresponding to the same or different languages; para. [0182]), comprising the following steps:
obtaining a first plurality of initial embeddings of entities of a first graph and a second plurality of initial embeddings of entities of a second graph (i.e. The initial entity embeddings and relation embeddings are generated from the first language module 320 … The first language module 320 also generates initial entity and relation embeddings for the knowledge module (e.g., graph convolution network 340) based on the entity description text 310 and entity relation text … The language module of the integrated knowledge-language module is further configured to produce contextual representations as initials embeddings for knowledge graph entities and relations; para. [0078, 0084, 0127]) based on a pre-trained language model (i.e. the first language module 320 and the second language module are formed as the first several transformer layers and the rest layers of a pre-trained language model (e.g., as initialized from BERT or RoBERTA); para. [0073, 0081]), wherein the first graph and the second graph include the same or different languages (i.e. The first language knowledge graph 1320 and second language knowledge graph 1322 are optionally language or textual-based knowledge graphs corresponding to the same or different languages; para. [0182]), wherein the first plurality of embeddings and the second plurality of embeddings are in a unified space (i.e. The integrating of the language module and knowledge module comprises projecting entity and relations output embeddings and language text embeddings into a shared semantic space; para. [0128]); and
learning entity alignment between the first graph and the second graph over the first plurality of initial embeddings and the second plurality of initial embeddings (i.e. a multi-layer language model (e.g., a first language module 320 and a second language module 360) where the first language module provides embeddings (see embeddings at locations 1 and 2) for both the second language module 360 and a knowledge module configured as a graph convolution network 340. The entity embeddings (e.g., entity representation 342) from the knowledge module are also fed into the second language module (see information fusion 344), which produces the final representation (e.g., context representation 362); para. [0072, 0077,0078]) to using a relative similarity metric (i.e. the model predicts the entity whose embedding in the ECEM 322 is closest to the mentioned entity. Since the number of entities is very large in some instances, the model uses the entity's neighbors and other randomly sampled entities as negative samples. The representational loss is minimized between the predicted embedding and the original embedding using a loss function which is a cross entropy based on the inner product between an embeddings and each candidate entity's embedding; para. [0082, 0083]), wherein negative sampling for non-aligned entities of an entity of one of the first graph or the second graph is performed on the one of the first graph or the second graph during the learning (i.e. the model predicts the entity whose embedding in the ECEM 322 is closest to the mentioned entity. Since the number of entities is very large in some instances, the model uses the entity's neighbors and other randomly sampled entities as negative samples. The representational loss is minimized between the predicted embedding and the original embedding using a loss function which is a cross entropy based on the inner product between an embeddings and each candidate entity's embedding; para. [0079, 0082, 0083]).
Zhu does not explicitly teach wherein the first plurality of initial embeddings and the second plurality of initial embeddings are in a unified space; and learning entity alignment between the first graph and the second graph over the first plurality of initial embeddings and the second plurality of initial embeddings by at least one encoder to push non-aligned entities far away using a relative similarity metric.
However, Yang teaches wherein the first plurality of initial embeddings and the second plurality of initial embeddings are in a unified space (i.e. the goal is to embed cross-lingual entities into the same low-dimensional vector space where equivalent entities are close to each other … The spirit of BERT in the multilingual scenario is to project words or sentences from different languages into the same semantic space; Section 1, 3.1, 3.2, 3.3, pages 1-6); and
learning entity alignment between the first graph and the second graph over the first plurality of initial embeddings and the second plurality of initial embeddings by at least one encoder (i.e. two GCN-based models, namely MAN and HMAN, that learn en tity embeddings from the graph structures. Second, we discuss two uses of a multilingual pre trained BERT model to learn cross-lingual embeddings of entity descriptions: POINTWISEBERT and PAIRWISEBERT. Finally, we investigate two strategies to integrate the GCN-based and the BERT-based modules; Section 3, 3.1, pages 2-3) to push non-aligned entities far away using a relative similarity metric (i.e. Given two knowledge graphs, G1 and G2, and a set of pre-aligned entity pairs I (G1,G2) as training data, our model is trained in a supervised fashion. During the training phase, the goal is to embed cross-lingual entities into the same low-dimensional vector space where equivalent entities are close to each other, our margin-based ranking loss function is defined as: Equation 4, where [x]+ = max{0,x}, I denotes the set of negative entity alignment pairs constructed by corrupting the gold pair (e1,e2) ∈ I. Specifically, we replace e1 or e2 with a randomly-chosen entity in E1 or E2. ρ(x,y) is the 1 distance function, and β > 0 is the margin hyperparameter separating positive and negative pairs; Section 3.1, pages 3-4), wherein negative sampling for non-aligned entities of an entity of one of the first graph or the second graph is performed on the one of the first graph or the second graph during the learning (i.e. Given two knowledge graphs, G1 and G2, and a set of pre-aligned entity pairs I (G1,G2) as training data, our model is trained in a supervised fashion. During the training phase, the goal is to embed cross-lingual entities into the same low-dimensional vector space where equivalent entities are close to each other, our margin-based ranking loss function is defined as: Equation 4, where [x]+ = max{0,x}, I denotes the set of negative entity alignment pairs constructed by corrupting the gold pair (e1,e2) ∈ I. Specifically, we replace e1 or e2 with a randomly-chosen entity in E1 or E2. ρ(x,y) is the 1 distance function, and β > 0 is the margin hyperparameter separating positive and negative pairs; Section 3.1, pages 3-4).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Zhu to include the feature of Yang. One would have been motivated to make this modification because it provides an accurate way to distinguish corresponding entities from non-corresponding candidates and to discover additional alignments between graphs.
Claim 20 is similar in scope to Claim 12 and is rejected under a similar rationale.
Zhu teaches an apparatus for entity alignment, comprising: a memory; and at least one processor coupled to the memory and configured for entity alignment, the at least one processor configured to (i.e. fig. 1, the computing system 110, for example, includes one or more processor(s) 112 (such as one or more hardware processor(s)) and a storage (i.e., hardware storage device(s) 140) storing computer-executable instructions 118 wherein one or more of the hardware storage device(s) 140 is able to house any number of data types and any number of computer-executable instructions 118 by which the computing system 110 is configured to implement one or more aspects of the disclosed embodiments when the computer-executable instructions 118 are executed by the one or more processor(s) 112; para. [0031]).
Claim 21 is similar in scope to Claim 12 and is rejected under a similar rationale.
Zhu teaches a non-transitory computer readable medium on which is stored computer code for entity alignment, the computer code when executed by a processor, causing the processor to perform the following steps (i.e. fig. 1, the computing system 110, for example, includes one or more processor(s) 112 (such as one or more hardware processor(s)) and a storage (i.e., hardware storage device(s) 140) storing computer-executable instructions 118 wherein one or more of the hardware storage device(s) 140 is able to house any number of data types and any number of computer-executable instructions 118 by which the computing system 110 is configured to implement one or more aspects of the disclosed embodiments when the computer-executable instructions 118 are executed by the one or more processor(s) 112; para. [0031, 0214]).
6. Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu in view of Yang, and further in view of Zhang et al. (SCE: Scalable Network Embedding from Sparsest Cut; arXiv, published 2020, pages 1-9).
Claim 13: Zhu and Yang teach the computer-implemented method of claim 12. Zhu does not explicitly teach excludes aligned entities or positive pairs.
However, Zhang teaches wherein the relative similarity metric excludes aligned entities or positive pairs (i.e. we propose SCE for unsupervised network embedding only using negative samples for training. Our method is based on a new contrastive objective inspired by the well-known sparsest cut problem. To solve the underlying optimization problem, we introduce a Laplacian smoothing trick, which uses graph convolutional operators as low-pass filters for smoothing node representations. The resulting model consists of a GCN-type structure as the encoder and a simple loss function. Notably, our model does not use positive samples but only negative samples for training; abs, 1.1, 2.3, pages 1-3).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Zhu and Yang to include the feature of Zhang. One would have been motivated to make this modification because it makes the implementation and tuning much easier, but also reduces the training time significantly.
7. Claims 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu in view of Yang, and further in view of Yuan et al. (U.S. Patent Application Pub. No. US 20220284321 A1)
Claim 14: Zhu and Yang teach the computer-implemented method of claim 12. Zhu does not explicitly teach maintaining respective negative queues for each of the first graph and the second graph during the learning, wherein the respective negative queues include previously encoded batches as negative samples.
However, Yuan teaches maintaining respective negative queues for each of the first graph and the second graph during the learning, wherein the respective negative queues include previously encoded batches as negative samples (i.e. Self-supervised methods utilize contrastive objectives, for instance, comparison to facilitate image representation learning. For example, use of a memory bank which stores pre-computed representations and the noise-contrastive estimation (NCE) for a large number of instance classes. Storing representations from momentum encoders in a dynamic dictionary with a queue enhances the scheme … A dynamic set of key features (for example, length) is maintained by iterative dequeue and enqueue operations; para. [0082, 0098, 0099]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Zhu and Yang to include the feature of Yuan. One would have been motivated to make this modification because it would increase the number of reusable negatives without encoding all graph entities during every minibatch.
Claim 15: Zhu, Yang, and Yuan teach the computer-implemented method of claim 14. Zhu does not explicitly teach wherein the at least one encoder includes an online encoder and a target encoder that is used for encoding a current batch, and wherein the online encoder is updated directly with backpropagation and the target encoder is updated with a momentum.
However, Yuan further teaches wherein the at least one encoder includes an online encoder and a target encoder that is used for encoding a current batch (i.e. Image encoders and momentum encoders embed augmented examples Ij †, Ij † from the same input image Ij in a minibatch, to query and key features; para. [0075, 0090, 0101]), and wherein the online encoder is updated directly with backpropagation (i.e. An inter-modal training scheme is used to enhance the image features by embracing the cross-modal interactions. With carefully designed contrastive losses, features in multiple modalities are adjusted using backpropagation in multiple training paths; para. [0080, 0083]) and the target encoder is updated with a momentum (i.e. Image encoders and momentum encoders embed augmented examples Ij †, Ij † from the same input image Ij in a minibatch, to query and key features; para. [0075, 0090, 0101]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Zhu and Yang to include the feature of Yuan. One would have been motivated to make this modification because momentum encoder representations used with a dynamic queue improve the contrastive learning arrangement.
8. Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu in view of Yang, and further in view of Huynh et al. (Adaptive Network Alignment with Unsupervised and Multi-order Convolutional Networks; IEEE, published 2020, pages 85-96).
Claim 17: Zhu and Yang teach the computer-implemented method of claim 12. Zhu further teaches comprising automatically aligning entities.
Zhu does not explicitly teach without a training label or supervision.
However, Huynh teaches comprising automatically aligning entities without a training label or supervision (i.e. we propose a fully unsupervised network alignment framework based on a multi-order embedding model … We train the same GCN-based model for both source and target networks by a weight-sharing mechanism. This allows their embeddings to be in a common space without any labelled data; abs, 85, 87, 90).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Zhu and Yang to include the feature of Huynh. One would have been motivated to make this modification because ground-truth information is rarely available in real-world networks due to high labelling costs. For instance, checking users’ background and pairing their accounts manually in social networks is very time-consuming. Hence, unsupervised network alignment is desirable.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Bose et al. (Pub. No. US 20190130221 A1), Data from different modalities could exhibit different statistics but have shared underlying meaning. Cross-modal embedding models capture the shared semantics by learning mappings to a joint latent space on which semantic similarity across modality can be measured. Such model can be used for tasks that require understanding of the interplay between modalities. For example, image-caption retrieval needs object and scene recognition, and match the understanding to natural language description.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAN TRAN whose telephone number is (303)297-4266. The examiner can normally be reached on Monday - Thursday - 8:00 am - 5:00 pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached on 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAN H TRAN/Primary Examiner, Art Unit 2141
Prosecution Insights