DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on November 30, 2023 has been considered by the examiner.
Claim Objections
Claims 1-6 and 8 are objected to because of the following informalities. Appropriate correction is required.
In particular, claim 1 recites “training data comprising a first labeled triplet related to an entity” and later recites “a second neural network that has been trained using a first labeled triplet of the training data.” It is unclear as to whether these first labeled triplets are in reference to the same triplet. To avoid any confusion, the Examiner recommends changing the latter phrase to instead recite “a second neural network that has been trained using the first labeled triplet of the training data.”
Claims 2-6 depend from claim 1 and thereby include all of the limitations of claim 1. Accordingly, claims 2-6 are objected to for the same reasons as noted above for claim 1.
Claim 8 recites “the triplet labeled to the training data” and “the triplet labeled to the validation data.” Claim 8 depends from claim 7, which recites “a first triplet” and “a second labeled triplet.” The “triplet labeled to the training data” in claim 8 ostensibly refers to the “first triplet” in claim 7. Such terminology should be consistent to avoid any potential confusion. Similarly, “the triplet labeled to the validation” in claim 8 ostensibly refers to the “second triplet” in claim 7, but such terminology should be consistent.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4-7, 9 and 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over the article entitled, “Relation Extraction using Multi-Encoder LSTM Network on a Distant Supervised Dataset” by Banerjee et al. (“Banerjee”), over the article entitled, “Populating Web-Scale Knowledge Graphs Using Distantly Supervised Relation Extraction and Validation” by Dash et al. (“Dash”), and also over the article entitled, “Building and Querying an Enterprise Knowledge Graph” by Song et al. (“Song”).
Regarding claims 1 and 7, Banerjee generally describes “a novel multi encoder bidirectional Long Short Term Memory (LSTM) model to identify relations in a given sentence.” (Abstract). Like claimed, Banerjee particularly teaches:
generating, based on a knowledge graph and original text data: training data comprising a first labeled triplet related to an entity and a relation of a text, validation data comprising a second labeled triplet related to an entity and a relation of a text, and unlabeled text data (Banerjee generally teaches using distant supervision to automatically construct training data for relation extraction learning tasks; Banerjee suggests that distant supervision entails aligning triples from a knowledge graph, such as DBPedia, with sentences from a document corpus such as Wikipedia articles:
Relation extraction, a text classification task, is used to identify the relation between a pair of entities mentioned in a span of text. Relation extraction has been found to be very effective in expanding knowledge-bases automatically as well as in Question Answering (QA). Manually generating training examples (sentences and corresponding relation labels) for the ever-increasing number of relations in Knowledge-Bases (KB’s) such as Wikidata or Freebase is time-consuming and expensive. As a result, distant supervision has emerged as a popular technique to automatically construct training data for relation extraction learning tasks. The underlying assumption of all distant supervision techniques is the following: If two entities are connected by a relation, a sentence that contains both the entities must describe the same relation. However, such techniques suffer from the noisy data problem as the aligned pairs of relation triples from KB’s and sentences often contain erroneous alignments.
…
In this work, we address the above mentioned concerns. First, we build a distant supervised dataset for relation extraction by mapping triples from DBPedia [5] and sentences from Wikipedia. To reduce the effect of noisy labels, we compute confidence values for the samples based on co-occurrence statistics of dependency paths in the sentences and relations. We use the confidence values as sample weights during model training allowing the models to automatically adjust parameters depending on importances of individual data instances. Second, we propose MEM (Multi-Encoder Model) that uses three simultaneous Long-Short Term Memory (LSTM) [6] units to encode information using features from words, POS tags and dependency paths of the sentences. The hidden states from each of the encoders are concatenated and used to predict the relation using softmax activation [7].
(Section I. Introduction. Footnotes omitted and emphasis added).
Distant supervision has been successfully used for many relation extraction tasks. In DeepQA [9], researchers used distant supervision successfully by aligning relations from DBPedia [5] with sentences from Wikipedia articles. DBPedia (or any other Knowledge Base (KB)) consists of triples mentioning relations between pairs of entities. Any triple can be represented as
e
1
,
e
2
,
r
e
l
where
e
1
and
e
2
are two entities related by a relation
r
e
l
. For example, the following DBPedia triple: (Square Shells, Kurt Vile, artist) represents the information that the entities Square Shells and Kurt Vile are related by an artist relationship. DBPedia triples are retrieved from Wikipedia infoboxes. Naturally, there is a high chance of the relation between the entities being explicitly stated in the content of the Wikipedia article. For example, the first sentence of the Wikipedia article on Square Shells is as follows:
Square Shells is a limited edition EP by American indie rock musician Kurt Vile, released on May 24, 2010 on Matador Records.
The sentence contains both the entities in the triple. Therefore, it can be aligned to the specific triple from DBPedia. We create a distant supervised dataset by aligning sentences and corresponding relation triples….
(Section II.A. Automatic Training Data Construction. Footnotes omitted and emphasis added).
Banerjee further teaches that the set of aligned triples and sentences is randomly split into training, validation and test sets:
We align triples from DBPedia and Wikipedia to obtain a total of 758,662 alignments. We normalized all entities retaining entity types (PERSON, ORGANIZATION, etc.) and proper nouns (as NNP tokens) in the dataset. The alignments resulted in a total of 483 unique relations. In this paper, we only consider the 100 most frequent relations. All the remaining relations are grouped into a separate “NA” class. The top 100 relations contribute 95.9% records of the dataset. We randomly split the dataset into 70, 20, 10 percent to create the training, validation and test sets respectively.
(Section III.A. Dataset Characteristics. Footnotes omitted and emphasis added).
Such an alignment, within the training set, of a triple from the knowledge graph and sentence from the document corpus is considered a training data comprising a first labeled triplet related to an entity and a relation of text. An alignment, within the validation set, of a triple from the knowledge graph and sentence from the document corpus is considered validation data comprising a second labeled triplet related to an entity and a relation of a text. The document corpus is considered unlabeled text data.);
training, with the training data, a first neural network to extract triplets (Banerjee suggests that the training data is used to train a neural network to predict relations from text:
Our proposed neural network architecture, MEM, to predict relations from text is shown in Figure 2. As can be seen from the figure, MEM uses three encoders to encode information from three feature sequences. An RNN with Long-Short Term Memory units (LSTMs) [6] is applied to consecutively process the sequential inputs. More specifically, we used the bidirectional LSTM [10] cell as it performed better than the regular LSTM cell. We encode the following sequences:
Dependency features: The dependency element obtained earlier using the parse is fed as a sequence to the encoder. We create the sequence by concatenating the path to
e
1
, and the path to
e
2
in order as shown earlier.
Word features: We include sequence of words between the entities in the sentence as inputs for processing in the second LSTM encoder. We use an embedding layer over this to convert the raw word indices to distributed representations.
POS features: We include sequence of POS tags of the words between the entities in the sentence as inputs for processing in the third LSTM encoder.
The final cell states of the LSTM encoders are combined (using concatenation) to create a combined cell state. Therefore, if
c
1
,
c
2
and
c
3
are the three cell states from the first, second and third encoder respectively, the combined state
C
is obtained by concatenating the three representations together in the following manner:
C
=
c
1
,
c
2
,
c
3
(2)
We restrict each sequence to a maximum of 10 elements. Sequence elements that are closer to the entities in the sentence have been found to be more important for relation determination [1]. Therefore, for longer sequences (>10 elements), the first and the last 5 elements were used to train the model. Softmax activation function [7] was used in the final dense layer to predict the relations. We also use the confidence values computed during the training data generation phase by feeding them as sample weights when training the model using a categorical cross-entropy loss function [11]. Furthermore, due to the imbalanced nature of our dataset, we apply balanced class weighting to avoid bias when training the model.
(Section II.B. MEM: Multi-Encoder Model. Emphasis added.)
The neural network is thus trained to predict a relation between two entities indicated in a sentence, and is thus considered to be trained to extract triplets comprising the two entities and the predicted relations.);
extracting a new triplet by inputting the text data to the trained first neural network (Banerjee suggests that, once trained, the neural network can be used to predict a relation between entities indicated in a sentence by inputting the sentence from the test set to the neural network:
The results of relation classification on the test instances are shown in Table II. We show the accuracy at two levels (1 and 5). Accuracy@1 implies the percentage of test instances where the model predicts the relation correctly. Accuracy@5 implies the percentage of test instances where the model predicts the correct relation among the top 5 most probable classes. As can be seen from the table, MEM performs better than the other models.
(Section III.B. Comparison between models.)
Accordingly, Banerjee teaches extracting a new triplet, i.e. a relation between two entities, by inputting the text data to the trained first neural network.); and
measuring a first confidence of the new triplet using the trained first neural network (As noted above, Banerjee suggests that the neural network can be used to predict a relation between entities indicated in a sentence by inputting the sentence from the test set to the neural network. Banerjee further discloses that a “[s]oftmax activation function [7] was used in the final dense layer to predict the relations.” Section II.B. MEM: Multi-Encoder Model. As known in the art, a softmax output layer produces a probability distribution, where each output of the layer represents the probability of the output. Accordingly, it is apparent that the softmax layer of the neural network described by Banerjee outputs a probability for each of the plurality of types of relations. This is further suggested in Section III.B. Comparison between models, which recites:
Accuracy@5 implies the percentage of test instances where the model predicts the correct relation among the top 5 most probable classes. As can be seen from the table, MEM performs better than the other models. MEM-ATT (with attention) performs slightly worse than MEM. However, accuracy@5 is the same for both MEM and MEM-ATT implying that both the models can figure out the correct relation among their top 5 predictions in ~94% of the cases.
The probability of the relation output by the neural network is considered a confidence of the new triplet.).
Accordingly, Banerjee teaches a method similar to that of claim 7. The above-described tasks are understandably implemented via computer program instructions stored in memory for execution by one or more processors. A computer comprising such memory and one or more processors to implemented the above-described tasks taught by Banerjee is considered an electronic device similar to that of claim 1. However, Banerjee does not explicitly teach: (i) measuring a second confidence of the new triplet using a second neural network that has been trained using a first labeled triplet of the training data and the second labeled triplet of the validation data; and (ii) expanding the knowledge graph based on the first confidence and the second confidence, as is required by each of claims 1 and 7.
Dash generally describes “a fully automated system to extend knowledge graphs using external information from web-scale corpora.” (Abstract). Similar to Banerjee, Dash particularly teaches:
generating, based on a knowledge graph and original text data: training data comprising information related to an entity and a relation of a text, and unlabeled text data (Dash discloses that the system comprises a relation extraction component and a relation validation component, and that their input is a partially populated knowledge graph (KG) and a document corpus:
In this paper, we present an approach that overcomes the aforementioned problem while offering a scalable solution to extend large knowledge graphs from web-scale corpora. It consists of two main components: relation extraction, a deep-learning-based distantly supervised system to detect relations from text, and relation validation, a deep-learning based knowledge base validation component able to spot inconsistencies in the acquired graphs and improve the global quality. In order to operate these components, the only required input is a partially populated KG and a large scale document corpus. In our experiments, we used DBpedia and Freebase for the KG and Common Crawl web text and New York Times news articles for the document corpora.
(Section 1. Introduction. Emphasis added.).
Dash further discloses that the relation extraction component is trained via distant supervision using training data based on the knowledge graph, wherein the training data particularly comprises sentences from the corpus that mention entities from the knowledge graph:
We use knowledge-level supervision, sometimes called distant supervision, to generate the training needed for deep-learning-based RE systems from a KG and an unannotated corpus. To this aim, we first match all entities in
K
B
t
r
a
i
n
to gather their context sets. That context set provides all the sentences that contain two entity mentions. If those two entities are related by some relation in the input KG, they become positive examples for that binary relation. We then use all the context sets collected from the corpus to train a deep-learning-based RE classifier. We use the system of [3] based on the PCNN model from NRE [8].
(Section 3.1. Relation Extraction. Emphasis added.).
Accordingly, Dash teaches generating training data based on the knowledge graph and original text data, i.e. the document corpus, wherein the training data comprises information related to an entity and a relation of text, i.e. the sentences mentioning that entity. The document corpus is considered unlabeled text data.);
training, with the training data, a first neural network to extract triplets (As noted above, Dash discloses that the system comprises a relation extraction component that is trained using training data based on the knowledge graph. Dash teaches that the relation extraction (RE) component is particularly implemented with a neural network:
To implement the RE component, we applied a state-of-the-art distantly supervised relation extraction system that is capable of recognizing relations among pre-identified entities using a deep neural network approach [3].
(Section 1. Introduction).
We then use all the context sets collected from the corpus to train a deep-learning-based RE classifier. We use the system of [3] based on the PCNN model from NRE [8].
(Section 3.1. Relation Extraction).
Dash further discloses that the relation extraction component is trained to extract quads from a corpus of text documents, wherein each quad comprises a triplet with an associated confidence and the triplet indicates a particular relation between two entities:
In this section, we describe the architecture of our solution for knowledge base population (KBP). KBP is the task of identifying entities and relations from a corpus, according to a predefined schema. It is illustrated by Figure 2, representing the architecture of our final KBP solution. It is composed by a distantly supervised information extraction system that takes a pre-existing KB and a corpus as an input and generates a list of quads representing induced relations with their associated confidence scores. Its output is then merged with the triples in the pre-existing KG and fed into a KBC deep net to train a KBV system whose goal is to re-assess the generated assertions, providing new confidence scores for each of them. Finally, the scores are aggregated by a logistic regression layer that provides the final confidence score for each triple. For all these steps, the same KB is always used for training.
More formally, the information extraction component of KBP generates a set of quads (triples with confidence)
Q
I
E
=
q
1
,
q
2
,
…
,
q
n
'
from a corpora of text documents
C
=
c
1
,
c
2
,
…
,
c
m
. Here, each text document
c
is represented in the form of a sequence of words
c
=
w
a
,
…
,
e
1
,
w
b
,
e
2
,
…
w
z
containing two entity mentions
e
1
and
e
2
. Quads have the form
q
=
e
1
,
r
,
e
2
,
s
, where
e
i
∈
ℇ
are entities found in the corpus,
r
∈
R
is a finite set of relations, and
s
∈
[
0,1
]
is a confidence score. We define the function
τ
e
1
,
r
,
e
2
,
s
=
e
1
,
r
,
e
2
to ignore the confidence of a quad, forming a triple. Since
K
B
is typically the Abox of a handcrafted ontology, we assume all the confidence scores of quads in
K
B
being equal to 1.
For each context
c
∈
C
, the entity detection and linking (EDL) function
ψ
c
=
e
1
,
e
2
returns the two entities contained in it. In our current implementation, EDL is implemented by a simple string match with regard to the entities in the KB; however, it could also be replaced with more advanced EDL solutions if available. For each entity
e
∈
V
, the function
ψ
e
returns all possible contexts where the entity
e
appears in the corpus, and
ψ
e
1
,
e
2
returns all contexts containing both. The RE process consists of applying a deep net to the context returned by
ψ
e
1
,
e
2
for every pair of entities that co-occur in the corpus. The result of the application of RE to a context is a list of quads
q
=
e
1
,
r
i
,
e
2
,
s
i
for all
r
i
∈
R
, where
s
i
represents the confidence of the system on the detection of the relation
r
i
in one or more contexts in
ψ
e
1
,
e
2
, where the two entities co-occur in the corpus. Obviously, most of the relations will have very low scores since all the relations are explored and returned for each pair.
(Section 3. Distantly Supervised Relation Extraction and Validation. Emphasis added.).
Accordingly, Dash teaches using the training data to train a first neural network, i.e. the relation extraction component, to extract triplets.);
extracting a new triplet by inputting the text data to the trained first neural network (As noted above, Dash discloses that the relation extraction component, i.e. first neural network, is trained to extract quads from a corpus of text documents, wherein each quad comprises a triplet with an associated confidence. Dash particularly discloses that, once trained, the relation extraction component is applied to the corpus to generate new triplets with associated confidence scores:
After the system is trained, it is applied to all context sets for every pair of entities in the corpus
C
and generates a set of quads
Q
I
E
, where for each pair of entities
e
1
and
e
2
, up to
R
triples are generated and associated with their confidence score. Minimum confidence is set for extracted quads to control the size and quality of the output.
(Section 3.1. Relation Extraction).
Accordingly, Dash teaches extracting a new triplet by inputting the text data to the trained first neural network.); and
measuring a first confidence of the new triplet using the trained first neural network (as noted above, Dash teaches that, once trained, the relation extraction component is applied to the corpus to generate new triplets with associated confidence scores. The relation extraction component, i.e. first neural network, thus measures a first confidence of each new triplet.).
Moreover, regarding the claimed invention, Dash further teaches:
measuring a second confidence of the new triplet using a second neural network that has been trained using triplets from the knowledge graph (Like noted above, Dash discloses that the system also comprises a relation validation component:
In this paper, we present an approach that overcomes the aforementioned problem while offering a scalable solution to extend large knowledge graphs from web-scale corpora. It consists of two main components: relation extraction, a deep-learning-based distantly supervised system to detect relations from text, and relation validation, a deep-learning based knowledge base validation component able to spot inconsistencies in the acquired graphs and improve the global quality. In order to operate these components, the only required input is a partially populated KG and a large scale document corpus. In our experiments, we used DBpedia and Freebase for the KG and Common Crawl web text and New York Times news articles for the document corpora.
(Section 1. Introduction. Emphasis added.).
Dash further teaches that the relation validation component returns additional confidence scores for each triplet produced by the relation extraction, and that the relation validation component particularly comprises a second neural network, KBV, that is trained using triplets from the knowledge base:
The RE step takes into account mostly information coming from the corpus for each entity pair to predict the relations, if any, between them. It does not take into account global information provided by the structure of the KG. The relation validation component is designed to overcome this problem. It is formally described as a function
K
B
V
∶
E
×
R
×
E
⟼
R
. For any triple produced by IE (
τ
(
q
)
∶
q
∈
Q
I
E
)
, KBV returns a confidence score.
The KBV system is to be trained from a knowledge graph
K
B
consisting of a set of quads. In this paper, we experimented with two different ways of training, producing two-component systems: (a)
K
B
V
, using the ground truth from the knowledge graph
K
B
t
r
a
i
n
, and (b)
K
B
V
I
E
, using the output of information extraction
Q
I
E
. The result is two different functions returning different confidence scores when applied to the same triple.
The three confidence scores generated from IE and by applying
K
B
V
and
K
B
V
I
E
to every triple from
Q
I
E
are then aggregated using a confidence re-estimation layer trained on a validation set to provide a final confidence score, generating the final output
Q
f
i
n
a
l
. In the following subsection, we will describe the distantly supervised RE approach and the knowledge base validation step in detail.
(Section 3. Distantly Supervised Relation Extraction and Validation).
This confidence output by KBV is considered a second confidence of the new triplet, which is measured using a second neural network that has been trained using the triplets from the knowledge graph.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee and Dash before the effective filing date of the claimed invention, to modify the electronic device and method taught by Banerjee so as to include a relation validation component like taught by Dash, which measures a second confidence of the new triplet and is implemented via a second neural network that has been trained using the triplets of the knowledge graph. As noted above, Banerjee teaches that the triplets of the knowledge graph (and aligned sentences) are split into training data and validation data, wherein the training data comprises a first labeled triplet and the validation data comprising a second labeled triplet. Because the second neural network taught by Dash is trained on the entire knowledge graph, it follows that it would be trained using, inter alia, the first labeled triplet of the training data and the second labeled triplet of the validation data, which are both in the knowledge graph. It would have been advantageous to one of ordinary skill to utilize such a validation component because it can boost the performance of relation extraction, as is taught by Dash (see e.g. Section 1. Introduction, which recites, “[o]ur experiments show that the validation step boosts the performance of RE by a wide margin, reporting error reductions of 50%, sometimes resulting in a relative improvement of up to 100%.”). Accordingly, Banerjee and Dash are considered to teach “measuring a second confidence of the new triplet using a second neural network that has been trained using a first labeled triplet of the training data and the second labeled triplet of the validation data,” as is recited in claim 7 and expressed similarly in claim 1.
Dash further teaches that the first and second confidences can be combined (i.e. via logistic regression) to a produce a final confidence for the new triple (see e.g. Section 3.3. Confidence Re-Estimation). Moreover, Dash suggests that the purpose of identifying relations between entities (i.e. generating a new triplet) is to further populate a pre-existing knowledge graph (see e.g. section 1. Introduction, which recites “In this paper, we focus on the problem of identifying relations among entities found in a large corpus with the goal of populating a pre-existing KG [1,2].”). Banerjee and Dash, however, do not explicitly teach expanding the knowledge graph based on the first and second confidence, as is required by claims 1 and 7.
Similar to Banerjee, Song teaches using machine learning to provide a classifier that extracts triplets (i.e. a particular relationship between two entities) from an input sentence, and that measures a confidence (i.e. a probability) of the extracted triplets (see e.g. Section 3.2 Named Entity Recognition and Relation Extraction, which recites “[t]he core of this approach is a machine learning classifier that predicts the probability of a possible relationship for a given pair of identified entities in a given sentence.”). Song further teaches expanding a knowledge graph based on the confidence of the triplet (see e.g. Section 3.2 Named Entity Recognition and Relation Extraction, which recites “[f]or each pair of entities, our system may extract multiple relationships; only those relationships with a confidence score above a pre-defined threshold are then added to our knowledge graph.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee, Dash and Song before the effective filing date of the claimed invention, to modify the electronic device and method taught by Banerjee and Dash so as to expand the knowledge graph based on the confidence of the extracted triplet, as is taught by Song. As noted above, Dash teaches that the final confidence of the triplet is based on the combined first and second confidences. It thus follows that the knowledge graph would be expanded based on the first confidence and the second confidence like claimed. It would have been advantageous to one of ordinary skill to utilize such a combination because it can add new information to the knowledge graph, but prevent too much false information from being added to the graph, as is suggested by Song (see e.g. Section 3.2 Named Entity Recognition and Relation Extraction, which recites “[t]he algorithm is precision oriented in order to avoid introducing too many false positives into our knowledge graph.”). Accordingly, Banerjee, Dash and Song are considered to teach, to one of ordinary skill in the art, an electronic device like that of claim 1 and a method like that of claim 7.
As per claims 4 and 11, it would have been obvious, as is described above, to modify the electronic device and method taught by Banerjee so as to include a relation validation component like taught by Dash, which measures a second confidence of the new triplet and is implemented via a second neural network that has been trained using the triplets of the knowledge graph. Dash suggests that the second neural network (i.e. KBV) is trained to output whether the new triplet (i.e. the triplet produced by the relation extractor/first neural network) is valid:
The RE step takes into account mostly information coming from the corpus for each entity pair to predict the relations, if any, between them. It does not take into account global information provided by the structure of the KG. The relation validation component is designed to overcome this problem. It is formally described as a function
K
B
V
∶
E
×
R
×
E
⟼
R
. For any triple produced by IE (
τ
(
q
)
∶
q
∈
Q
I
E
)
, KBV returns a confidence score.
The KBV system is to be trained from a knowledge graph
K
B
consisting of a set of quads. In this paper, we experimented with two different ways of training, producing two-component systems: (a)
K
B
V
, using the ground truth from the knowledge graph
K
B
t
r
a
i
n
, and (b)
K
B
V
I
E
, using the output of information extraction
Q
I
E
. The result is two different functions returning different confidence scores when applied to the same triple.
(Section 3. Distantly Supervised Relation Extraction and Validation).
After the network is trained, it can be used for both link prediction (i.e., generating the object from a subject and relation input) or validation (i.e., assessing the validity of a new triple composed of known entities and relations). In this paper, we explore the second option.
(Section 3.2. Relation Validation).
Accordingly, the above-described combination of Banerjee, Dash and Song is further considered to teach an electronic device like that of claim 4 and a method like that of claim 11.
As per claim 5, it would have been obvious, as is described above, to modify the electronic device taught by Banerjee so as to include a relation validation component like taught by Dash, which measures a second confidence of the new triplet and is implemented via a second neural network that has been trained using the triplets of the knowledge graph. Dash further suggests applying a first weight of the first confidence and a second weight of the second confidence (Dash teaches using logistic regression to produce a final confidence based on the first confidence produced by the relation extraction component, i.e. the first neural network, and the second confidence produced by the KBV component, i.e. the second neural network, inter alia:
The confidence scores
s
ξ
from the three systems
ξ
∈
I
E
,
K
B
V
I
E
,
K
B
V
are combined to produce a final confidence for each triple
τ
q
∶
q
∈
Q
I
E
, yielding
Q
f
i
n
a
l
. This step uses a simple logistic regression, typically trained on a validation set separate from the training set.
(Section 3.3. Confidence Re-Estimation).
As known the art, logistic regression computes a final output based in part on learned weights applied to the inputs. Accordingly, using logistic regression to compute a final confidence based on the first and second confidences, inter alia, like taught by Dash would understandably entail applying a first weight of the first confidence and a second weight of the second confidence.). Accordingly, the above-described combination of Banerjee, Dash and Song is further considered to teach an electronic device like that of claim 5.
As per claim 6, Dash further suggests that the first weight and the second weight are determined based on a quality of the knowledge graph when the knowledge graph is expanded based on the first confidence and when the knowledge graph is expanded based on the second confidence (as noted above, Dash teaches using logistic regression to produce a final confidence based on the first confidence produced by the relation extraction component, i.e. the first neural network, and the second confidence produced by the KBV component, i.e. the second neural network, inter alia. As further described above, using logistic regression to compute the final confidence of the triplet would understandably entail applying a first weight of the first confidence and a second weight of the second confidence. The weights are indicative of the relative importance of the respective inputs and are obtained via supervised learning, as is entailed by logistic regression, by optimizing the weights based on the quality of the outputs when given predetermined inputs. Accordingly, the first weight and the second weight are considered to be determined based on the quality of the knowledge graph when the knowledge graph is expanded based on the first confidence and when the knowledge graph is expanded based on the second confidence.). Accordingly, the above-described combination of Banerjee, Dash and Song is further considered to teach an electronic device like that of claim 6.
As per claim 9, it would have been obvious, as is described above, to modify the method taught by Banerjee and Dash so as to expand the knowledge graph based on the confidence of the extracted triplet, as is taught by Song. Song particularly teaches that expanding the knowledge graph comprises adding the new triplet to the knowledge graph (see e.g. Section 3.2 Named Entity Recognition and Relation Extraction, which recites “[f]or each pair of entities, our system may extract multiple relationships; only those relationships with a confidence score above a pre-defined threshold are then added to our knowledge graph.”). Accordingly, the above-described combination of Banerjee, Dash and Song is further considered to teach a method like that of claim 9.
As per claim 12, it would have been obvious, as is described above, to modify the method taught by Banerjee and Dash so as to expand the knowledge graph based on the confidence of the extracted triplet, as is taught by Song. As further noted above, Dash teaches that the confidence of the triplet is based on the combined first and second confidences, and so it follows that the knowledge graph would be expanded based on the first confidence and the second confidence like claimed. Dash further suggests that the first and second confidences are combined by applying a first weight of the first confidence and a second weight of the second confidence, inter alia (Dash teaches using logistic regression to produce a final confidence based on the first confidence produced by the relation extraction component, i.e. the first neural network, and the second confidence produced by the KBV component, i.e. the second neural network, inter alia:
The confidence scores
s
ξ
from the three systems
ξ
∈
I
E
,
K
B
V
I
E
,
K
B
V
are combined to produce a final confidence for each triple
τ
q
∶
q
∈
Q
I
E
, yielding
Q
f
i
n
a
l
. This step uses a simple logistic regression, typically trained on a validation set separate from the training set.
(Section 3.3. Confidence Re-Estimation).
As known the art, logistic regression computes a final output based in part on learned weights applied to the inputs. Accordingly, using logistic regression to compute a final confidence based on the first and second confidences, inter alia, like taught by Dash would understandably entail applying a first weight of the first confidence and a second weight of the second confidence.). Accordingly, it follows that expanding the knowledge graph based on the combined first and second confidences like taught by Banerjee, Dash and Song would entail applying a first weight of the first confidence and a second weight of the second confidence. The above-described combination of Banerjee, Dash and Song is thus further considered to teach a method like that of claim 12.
As per claim 13, Dash further suggests that the first weight and the second weight are determined based on a quality of the knowledge graph when the knowledge graph is expanded based on the first confidence and when the knowledge graph is expanded based on the second confidence (as noted above, Dash teaches using logistic regression to produce a final confidence based on the first confidence produced by the relation extraction component, i.e. the first neural network, and the second confidence produced by the KBV component, i.e. the second neural network, inter alia. As further described above, using logistic regression to compute the final confidence of the triplet would understandably entail applying a first weight of the first confidence and a second weight of the second confidence. The weights are indicative of the relative importance of the respective inputs and are obtained via supervised learning, as is entailed by logistic regression, by optimizing the weights based on the quality of the outputs when given predetermined inputs. Accordingly, the first weight and the second weight are considered to be determined based on the quality of the knowledge graph when the knowledge graph is expanded based on the first confidence and when the knowledge graph is expanded based on the second confidence.). Consequently, the above-described combination of Banerjee, Dash and Song is further considered to teach a method like that of claim 13.
Claims 2 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over the above-described combination of Banerjee, Dash and Song, and also over the article entitled, “Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion” by Dong et al. (“Dong”).
Regarding claims 2 and 8, Banerjee, Dash and Song teach an electronic device like that of claim 1 and a method like that of claim 7, as is described above, and which entail generating training data comprising a first labeled triplet and validation data comprising a second labeled triplet. Banerjee, Dash and Song, however, do not explicitly teach generating the training data and the validation data so that the first labeled triplet (i.e. the triplet labeled to the training data) is different from the second labeled triplet (i.e. the triplet labeled to the validation data), as is required by claims 2 and 8.
Dong generally describes “Knowledge Vault, a Web-scale probabilistic knowledge base that combines extractions from Web content…with prior knowledge derived from existing knowledge repositories.” (Abstract). Regarding the claimed invention, Dong particularly teaches splitting training data and validation (i.e. test) data so that a triplet labeled to the training data is different than a triplet labeled to the validation data (see e.g. section 2.1 Evaluation protocol, which recites “[t]o evaluate the quality of our methods, we randomly split this data into a training set (80% of the data) and a test set (20% of the data); we infer labels for these triples using the method described below,” and “[i]f the test set contains the triple (s,p,o), then the training set is guaranteed not to contain the same triple.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee, Dash, Song and Dong before the effective filing date of the claimed invention, to modify the electronic device and method taught by Banerjee, Dash and Song so as to generate the training data and the validation data such that the triplet labeled to the training data (i.e. the first labeled triplet) is different from the triplet labeled to the validation data (i.e. the second labeled triplet), as is taught by Dong. It would have been advantageous to one of ordinary skill to utilize such a combination, because it can provide for more effective validation of the model (i.e. by using unseen triplets), as is evident from Dong (see e.g. section 2.1 Evaluation protocol). Accordingly, Banerjee, Dash, Song and Dong are considered to teach, to one of ordinary skill in the art, an electronic device like that of claim 2 and a method like that of claim 8.
Claims 3 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over the above-described combination of Banerjee, Dash and Song, and also over WIPO Publication No. WO 2022/047011 A1 to Fagnan et al. (“Fagnan”).
Regarding claims 3 and 10, Banerjee, Dash and Song teach an electronic device like that of claim 1 and a method like that of claim 7, as is described above, and which entail training a first neural network to extract values. Banerjee, Dash and Song, however, do not explicitly teach comparing a precision of the first neural network to a first threshold value, and comparing a recall of the first neural network to a second threshold value, as is required by claims 3 and 10.
Fagnan nevertheless teaches comparing a precision of a first machine-learning model to a first threshold value, and comparing a recall of the first machine-learning model to a second threshold value (see e.g. paragraphs 0072 and 0074).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee, Dash, Song and Fagnan before the effective filing date of the claimed invention, to modify the electronic device and method taught by Banerjee, Dash and Song so as to compare a precision of the first machine-learning model (i.e. the first neural network) to a first threshold value, and compare a recall of the first machine-learning model to a second threshold value, as is taught by Fagnan. It would have been advantageous to one of ordinary skill to utilize such a comparison because it can indicate whether the model is ready for deployment or if it needs retrained, as is demonstrated by Fagnan (see e.g. paragraph 0074). Accordingly, Banerjee, Dash, Song and Fagnan are considered to teach, to one of ordinary skill in the art, an electronic device like that of claim 3 and a method like that of claim 10.
Claims 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over the article entitled, “Relation Extraction using Multi-Encoder LSTM Network on a Distant Supervised Dataset” by Banerjee et al. (“Banerjee”), over WIPO Publication No. WO 2022/047011 A1 to Fagnan et al. (“Fagnan”), over the article entitled, “Populating Web-Scale Knowledge Graphs Using Distantly Supervised Relation Extraction and Validation” by Dash et al. (“Dash”), over the article entitled, “Building and Querying an Enterprise Knowledge Graph” by Song et al. (“Song”), and also over the article entitled, “Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion” by Dong et al. (“Dong”).
Regarding claim 14, Banerjee generally describes “a novel multi encoder bidirectional Long Short Term Memory (LSTM) model to identify relations in a given sentence.” (Abstract). Like claimed, Banerjee particularly teaches:
training a first neural network, using training data comprising a first labeled triplet related to an entity and related to a relation of a text from original text data based on a knowledge graph, to extract triplets (As noted above, Banerjee describes “a novel multi encoder bidirectional Long Short Term Memory (LSTM) model to identify relations in a given sentence.” Abstract. The model described by Banerjee is considered a first neural network like claimed, and is trained to extract triplets, i.e. a relation between two entities in a given sentence. Banerjee further teaches that the model is trained with training data generated during a training data generation phase:
We also use the confidence values computed during the training data generation phase by feeding them as sample weights when training the model using a categorical cross-entropy loss function [11]. Furthermore, due to the imbalanced nature of our dataset, we apply balanced class weighting to avoid bias when training the model.
(Section II.B. MEM: Multi-Encoder Model. Emphasis added.)
Banerjee suggests that the training data comprises triples from a knowledge graph, such as DBPedia, and corresponding sentences from a document corpus such as Wikipedia articles, wherein the triples each comprise two entities and a relation therebetween:
Relation extraction, a text classification task, is used to identify the relation between a pair of entities mentioned in a span of text. Relation extraction has been found to be very effective in expanding knowledge-bases automatically as well as in Question Answering (QA). Manually generating training examples (sentences and corresponding relation labels) for the ever-increasing number of relations in Knowledge-Bases (KB’s) such as Wikidata or Freebase is time-consuming and expensive. As a result, distant supervision has emerged as a popular technique to automatically construct training data for relation extraction learning tasks. The underlying assumption of all distant supervision techniques is the following: If two entities are connected by a relation, a sentence that contains both the entities must describe the same relation. However, such techniques suffer from the noisy data problem as the aligned pairs of relation triples from KB’s and sentences often contain erroneous alignments.
…
In this work, we address the above mentioned concerns. First, we build a distant supervised dataset for relation extraction by mapping triples from DBPedia [5] and sentences from Wikipedia. To reduce the effect of noisy labels, we compute confidence values for the samples based on co-occurrence statistics of dependency paths in the sentences and relations. We use the confidence values as sample weights during model training allowing the models to automatically adjust parameters depending on importances of individual data instances. Second, we propose MEM (Multi-Encoder Model) that uses three simultaneous Long-Short Term Memory (LSTM) [6] units to encode information using features from words, POS tags and dependency paths of the sentences. The hidden states from each of the encoders are concatenated and used to predict the relation using softmax activation [7].
(Section I. Introduction. Footnotes omitted and emphasis added).
Distant supervision has been successfully used for many relation extraction tasks. In DeepQA [9], researchers used distant supervision successfully by aligning relations from DBPedia [5] with sentences from Wikipedia articles. DBPedia (or any other Knowledge Base (KB)) consists of triples mentioning relations between pairs of entities. Any triple can be represented as
e
1
,
e
2
,
r
e
l
where
e
1
and
e
2
are two entities related by a relation
r
e
l
. For example, the following DBPedia triple: (Square Shells, Kurt Vile, artist) represents the information that the entities Square Shells and Kurt Vile are related by an artist relationship. DBPedia triples are retrieved from Wikipedia infoboxes. Naturally, there is a high chance of the relation between the entities being explicitly stated in the content of the Wikipedia article. For example, the first sentence of the Wikipedia article on Square Shells is as follows:
Square Shells is a limited edition EP by American indie rock musician Kurt Vile, released on May 24, 2010 on Matador Records.
The sentence contains both the entities in the triple. Therefore, it can be aligned to the specific triple from DBPedia. We create a distant supervised dataset by aligning sentences and corresponding relation triples….
(Section II.A. Automatic Training Data Construction. Footnotes omitted and emphasis added.).
Accordingly, the training data used to train the model described by Banerjee understandably comprises, inter alia, a first labeled triplet related to an entity and related to a relation of a text, i.e. to a sentence, from original text data based on a knowledge graph.);
extracting a new triplet by inputting text data generated from the original text data to the trained first neural network (Banerjee suggests that, once trained, the neural network is used on test instances to predict a relation between entities indicated in a sentence:
The results of relation classification on the test instances are shown in Table II. We show the accuracy at two levels (1 and 5). Accuracy@1 implies the percentage of test instances where the model predicts the relation correctly. Accuracy@5 implies the percentage of test instances where the model predicts the correct relation among the top 5 most probable classes. As can be seen from the table, MEM performs better than the other models.
(Section III.B. Comparison between models. Emphasis added.).
Banerjee further suggests that the test instances comprise sentences generated from the original text data, i.e. from the document corpus, and their corresponding triplets:
We align triples from DBPedia and Wikipedia to obtain a total of 758,662 alignments. We normalized all entities retaining entity types (PERSON, ORGANIZATION, etc.) and proper nouns (as NNP tokens) in the dataset. The alignments resulted in a total of 483 unique relations. In this paper, we only consider the 100 most frequent relations. All the remaining relations are grouped into a separate “NA” class. The top 100 relations contribute 95.9% records of the dataset. We randomly split the dataset into 70, 20, 10 percent to create the training, validation and test sets respectively.
(Section III.A. Dataset Characteristics. Footnotes omitted and emphasis added).
Accordingly, it is apparent that text data generated from the original text data, i.e. from the document corpus, is used as the test data that is input to the trained first neural network to extract a new triplet, i.e. a relation between two entities.); and
calculating a first confidence of the new triplet using the trained first neural network (As noted above, Banerjee suggests that the neural network can be used to predict a relation between entities indicated in a sentence by inputting the sentence from the test set to the neural network. Banerjee further discloses that a “[s]oftmax activation function [7] was used in the final dense layer to predict the relations.” Section II.B. MEM: Multi-Encoder Model. As known in the art, a softmax output layer produces a probability distribution, where each output of the layer represents the probability of the output. Accordingly, it is apparent that the softmax layer of the neural network described by Banerjee outputs a probability for each of the plurality of types of relations. This is further suggested in Section III.B. Comparison between models, which recites:
Accuracy@5 implies the percentage of test instances where the model predicts the correct relation among the top 5 most probable classes. As can be seen from the table, MEM performs better than the other models. MEM-ATT (with attention) performs slightly worse than MEM. However, accuracy@5 is the same for both MEM and MEM-ATT implying that both the models can figure out the correct relation among their top 5 predictions in ~94% of the cases.
The probability of the relation output by the neural network is considered a confidence of the new triplet.).
Accordingly, Banerjee teaches a method similar to that of claim 14. However, while Banerjee suggests creating validation data comprising a second labeled triplet from original text data (see e.g. Section III.A. Dataset characteristics, which recites “[w]e randomly split the dataset into 70, 20, 10 percent to create the training, validation and test sets, respectively”), Banerjee does not explicitly disclose comparing a quality of the trained first neural network to a threshold value using the validation data, as is required by claim 14. Moreover, Banerjee also does not teach: (i) calculating a second confidence of the new triplet using a second neural network trained using the first labeled triplet of the training data and the second labeled triplet of the validation data; (ii) calculating an accuracy of a link prediction model when the knowledge graph is expanded based on the first confidence and the second confidence; and (iii) expanding the knowledge graph with the new triplet based on the accuracy of the link prediction model, as is further required by claim 14.
Fagnan nevertheless generally teaches comparing a quality (e.g. a precision or recall) of a trained model to a threshold value using validation data (see e.g. paragraph 0074).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee and Fagnan before the effective filing date of the claimed invention, to modify the method taught by Banerjee so as to compare a quality of the trained model (i.e. the first neural network) to a threshold value using validation data (i.e. the validation data comprising a second labeled triplet from the original text data), as is taught by Fagnan. It would have been advantageous to one of ordinary skill to utilize such a comparison because it can indicate whether the model is ready for deployment or if it needs retrained, as is demonstrated by Fagnan (see e.g. paragraph 0074).
Dash generally describes “a fully automated system to extend knowledge graphs using external information from web-scale corpora.” (Abstract). Similar to Banerjee, Dash particularly teaches:
training a first neural network, with training data, to extract triplets (Dash discloses that the system comprises a relation extraction component and a relation validation component, and that their input is a partially populated knowledge graph (KG) and a document corpus:
In this paper, we present an approach that overcomes the aforementioned problem while offering a scalable solution to extend large knowledge graphs from web-scale corpora. It consists of two main components: relation extraction, a deep-learning-based distantly supervised system to detect relations from text, and relation validation, a deep-learning based knowledge base validation component able to spot inconsistencies in the acquired graphs and improve the global quality. In order to operate these components, the only required input is a partially populated KG and a large scale document corpus. In our experiments, we used DBpedia and Freebase for the KG and Common Crawl web text and New York Times news articles for the document corpora.
(Section 1. Introduction. Emphasis added.).
Dash teaches that the relation extraction (RE) component is particularly implemented with a neural network:
To implement the RE component, we applied a state-of-the-art distantly supervised relation extraction system that is capable of recognizing relations among pre-identified entities using a deep neural network approach [3].
(Section 1. Introduction).
We then use all the context sets collected from the corpus to train a deep-learning-based RE classifier. We use the system of [3] based on the PCNN model from NRE [8].
(Section 3.1. Relation Extraction).
Dash further discloses that the relation extraction component is trained to extract quads from a corpus of text documents, wherein each quad comprises a triplet with an associated confidence and the triplet indicates a particular relation between two entities:
In this section, we describe the architecture of our solution for knowledge base population (KBP). KBP is the task of identifying entities and relations from a corpus, according to a predefined schema. It is illustrated by Figure 2, representing the architecture of our final KBP solution. It is composed by a distantly supervised information extraction system that takes a pre-existing KB and a corpus as an input and generates a list of quads representing induced relations with their associated confidence scores. Its output is then merged with the triples in the pre-existing KG and fed into a KBC deep net to train a KBV system whose goal is to re-assess the generated assertions, providing new confidence scores for each of them. Finally, the scores are aggregated by a logistic regression layer that provides the final confidence score for each triple. For all these steps, the same KB is always used for training.
More formally, the information extraction component of KBP generates a set of quads (triples with confidence)
Q
I
E
=
q
1
,
q
2
,
…
,
q
n
'
from a corpora of text documents
C
=
c
1
,
c
2
,
…
,
c
m
. Here, each text document
c
is represented in the form of a sequence of words
c
=
w
a
,
…
,
e
1
,
w
b
,
e
2
,
…
w
z
containing two entity mentions
e
1
and
e
2
. Quads have the form
q
=
e
1
,
r
,
e
2
,
s
, where
e
i
∈
ℇ
are entities found in the corpus,
r
∈
R
is a finite set of relations, and
s
∈
[
0,1
]
is a confidence score. We define the function
τ
e
1
,
r
,
e
2
,
s
=
e
1
,
r
,
e
2
to ignore the confidence of a quad, forming a triple. Since
K
B
is typically the Abox of a handcrafted ontology, we assume all the confidence scores of quads in
K
B
being equal to 1.
For each context
c
∈
C
, the entity detection and linking (EDL) function
ψ
c
=
e
1
,
e
2
returns the two entities contained in it. In our current implementation, EDL is implemented by a simple string match with regard to the entities in the KB; however, it could also be replaced with more advanced EDL solutions if available. For each entity
e
∈
V
, the function
ψ
e
returns all possible contexts where the entity
e
appears in the corpus, and
ψ
e
1
,
e
2
returns all contexts containing both. The RE process consists of applying a deep net to the context returned by
ψ
e
1
,
e
2
for every pair of entities that co-occur in the corpus. The result of the application of RE to a context is a list of quads
q
=
e
1
,
r
i
,
e
2
,
s
i
for all
r
i
∈
R
, where
s
i
represents the confidence of the system on the detection of the relation
r
i
in one or more contexts in
ψ
e
1
,
e
2
, where the two entities co-occur in the corpus. Obviously, most of the relations will have very low scores since all the relations are explored and returned for each pair.
(Section 3. Distantly Supervised Relation Extraction and Validation. Emphasis added.).
Moreover, Dash discloses that the relation extraction component is trained via distant supervision using training data based on the knowledge graph, wherein the training data particularly comprises sentences from the corpus that mention entities from the knowledge graph:
We use knowledge-level supervision, sometimes called distant supervision, to generate the training needed for deep-learning-based RE systems from a KG and an unannotated corpus. To this aim, we first match all entities in
K
B
t
r
a
i
n
to gather their context sets. That context set provides all the sentences that contain two entity mentions. If those two entities are related by some relation in the input KG, they become positive examples for that binary relation. We then use all the context sets collected from the corpus to train a deep-learning-based RE classifier. We use the system of [3] based on the PCNN model from NRE [8].
(Section 3.1. Relation Extraction. Emphasis added.).
Accordingly, Dash teaches using training data to train a first neural network, i.e. the relation extraction component, to extract triplets.);
extracting a new triplet by inputting text data to the trained first neural network (As noted above, Dash discloses that the relation extraction component, i.e. first neural network, is trained to extract quads from a corpus of text documents, wherein each quad comprises a triplet with an associated confidence. Dash particularly discloses that, once trained, the relation extraction component is applied to the corpus to generate new triplets with associated confidence scores:
After the system is trained, it is applied to all context sets for every pair of entities in the corpus
C
and generates a set of quads
Q
I
E
, where for each pair of entities
e
1
and
e
2
, up to
R
triples are generated and associated with their confidence score. Minimum confidence is set for extracted quads to control the size and quality of the output.
(Section 3.1. Relation Extraction).
Accordingly, Dash teaches extracting a new triplet by inputting text data to the trained first neural network.); and
calculating a first confidence of the new triplet using the trained first neural network (as noted above, Dash teaches that, once trained, the relation extraction component is applied to the corpus to generate new triplets with associated confidence scores. The relation extraction component, i.e. first neural network, thus calculates a first confidence of each new triplet.).
Moreover, regarding the claimed invention, Dash further teaches:
calculating a second confidence of the new triplet using a second neural network trained using triplets from the knowledge graph (Like noted above, Dash discloses that the system also comprises a relation validation component:
In this paper, we present an approach that overcomes the aforementioned problem while offering a scalable solution to extend large knowledge graphs from web-scale corpora. It consists of two main components: relation extraction, a deep-learning-based distantly supervised system to detect relations from text, and relation validation, a deep-learning based knowledge base validation component able to spot inconsistencies in the acquired graphs and improve the global quality. In order to operate these components, the only required input is a partially populated KG and a large scale document corpus. In our experiments, we used DBpedia and Freebase for the KG and Common Crawl web text and New York Times news articles for the document corpora.
(Section 1. Introduction. Emphasis added.).
Dash further teaches that the relation validation component returns additional confidence scores for each triplet produced by the relation extraction, and that the relation validation component particularly comprises a second neural network, KBV, that is trained using triplets from the knowledge base:
The RE step takes into account mostly information coming from the corpus for each entity pair to predict the relations, if any, between them. It does not take into account global information provided by the structure of the KG. The relation validation component is designed to overcome this problem. It is formally described as a function
K
B
V
∶
E
×
R
×
E
⟼
R
. For any triple produced by IE (
τ
(
q
)
∶
q
∈
Q
I
E
)
, KBV returns a confidence score.
The KBV system is to be trained from a knowledge graph
K
B
consisting of a set of quads. In this paper, we experimented with two different ways of training, producing two-component systems: (a)
K
B
V
, using the ground truth from the knowledge graph
K
B
t
r
a
i
n
, and (b)
K
B
V
I
E
, using the output of information extraction
Q
I
E
. The result is two different functions returning different confidence scores when applied to the same triple.
The three confidence scores generated from IE and by applying
K
B
V
and
K
B
V
I
E
to every triple from
Q
I
E
are then aggregated using a confidence re-estimation layer trained on a validation set to provide a final confidence score, generating the final output
Q
f
i
n
a
l
. In the following subsection, we will describe the distantly supervised RE approach and the knowledge base validation step in detail.
(Section 3. Distantly Supervised Relation Extraction and Validation).
This confidence output by KBV is considered a second confidence of the new triplet, which is measured using a second neural network that has been trained using the triplets from the knowledge graph.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee, Fagnan and Dash before the effective filing date of the claimed invention, to modify the method taught by Banerjee and Fagnan so as to include a relation validation component like taught by Dash, which calculates a second confidence of the new triplet and is implemented via a second neural network that has been trained using the triplets of the knowledge graph. As noted above, Banerjee teaches that the triplets of the knowledge graph (and aligned sentences) are split into training data and validation data, wherein the training data comprises a first labeled triplet and the validation data comprising a second labeled triplet. Because the second neural network taught by Dash is trained on the knowledge graph, it follows that it would be trained using, inter alia, the first labeled triplet of the training data and the second labeled triplet of the validation data, which are both in the knowledge graph. It would have been advantageous to one of ordinary skill to utilize such a validation component because it can boost the performance of relation extraction, as is taught by Dash (see e.g. Section 1. Introduction, which recites, “[o]ur experiments show that the validation step boosts the performance of RE by a wide margin, reporting error reductions of 50%, sometimes resulting in a relative improvement of up to 100%.”). Accordingly, Banerjee, Fagnan and Dash are considered to teach “calculating a second confidence of the new triplet using a second neural network trained using the first labeled triplet of the training data and the second labeled triplet of the validation data,” as is recited in claim 14.
Dash further teaches that the first and second confidences can be combined (i.e. via logistic regression) to a produce a final confidence for the new triple (see e.g. Section 3.3. Confidence Re-Estimation). Moreover, Dash suggests that the purpose of identifying relations between entities (i.e. generating a new triplet) is to further populate a pre-existing knowledge graph (see e.g. section 1. Introduction, which recites “In this paper, we focus on the problem of identifying relations among entities found in a large corpus with the goal of populating a pre-existing KG [1,2].”). Banerjee, Fagnan and Dash, however, do not explicitly teach that the knowledge graph is expanded based on the first and second confidence, as is required by claim 14.
Similar to Banerjee, Song teaches using machine learning to provide a classifier that extracts triplets (i.e. a particular relationship between two entities) from an input sentence, and that measures a confidence (i.e. a probability) of the extracted triplets (see e.g. Section 3.2 Named Entity Recognition and Relation Extraction, which recites “[t]he core of this approach is a machine learning classifier that predicts the probability of a possible relationship for a given pair of identified entities in a given sentence.”). Song further teaches expanding a knowledge graph based on the confidence of the triplet (see e.g. Section 3.2 Named Entity Recognition and Relation Extraction, which recites “[f]or each pair of entities, our system may extract multiple relationships; only those relationships with a confidence score above a pre-defined threshold are then added to our knowledge graph.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee, Fagnan, Dash and Song before the effective filing date of the claimed invention, to modify the method taught by Banerjee, Fagnan and Dash so as to expand the knowledge graph based on the confidence of the extracted triplet, as is taught by Song. As noted above, Dash teaches that the final confidence of the triplet is based on the combined first and second confidences. It thus follows that the knowledge graph would be expanded based on the first confidence and the second confidence like claimed. It would have been advantageous to one of ordinary skill to utilize such a combination because it can add new information to the knowledge graph, but prevent too much false information from being added to the graph, as is suggested by Song (see e.g. Section 3.2 Named Entity Recognition and Relation Extraction, which recites “[t]he algorithm is precision oriented in order to avoid introducing too many false positives into our knowledge graph.”).
Dong generally describes “Knowledge Vault, a Web-scale probabilistic knowledge base that combines extractions from Web content…with prior knowledge derived from existing knowledge repositories.” (Abstract). Similar to Banerjee, Fagnan, Dash and Song, Dong particularly teaches:
training a first model (i.e. an extractor) to extract triplets (see e.g. section 2. OVERVIEW, which recites “Extractors: these systems extract triples from a huge number of web sources,” and section 3.1.1 Text documents (TXT), which recites “we train relation extractors using distant supervision [29].”);
extracting a new triplet by inputting text data generated from original text data to the trained first model (it is apparent that once trained, the extractor can be applied to original text data to extract a new triplet);
calculating a first confidence of the new triplet using the trained first model ((see e.g. section 2. OVERVIEW, which recites “[e]ach extractor assigns a confidence score to an extracted triple, representing uncertainty about the identity of the relation and its corresponding arguments.”); and
calculating a second confidence (i.e. probability) of the new triplet using a second model (i.e. a neural network) (see e.g. section 2. OVERVIEW, which recites “Graph-based priors: these systems learn the prior probability of each possible triple, based on triples stored in an existing KB,” and section 4.2 Neural network model (MLP), which describes a neural network for implementing such a graph-based prior system.).
Moreover, regarding the claimed invention, Dong further teaches calculating an accuracy (i.e. probability) of a link prediction model if the knowledge graph were expanded based on the new triple (see e.g. section 4.1 Path ranking algorithm, which describes another graph-based prior system, particularly a path ranking algorithm, for determining the probability of a given triple; the path ranking algorithm is considered a type of link prediction model that calculates a probability of the given triple, which is considered an accuracy if the knowledge graph were expanded based on the triple.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Banerjee, Fagnan, Dash, Song and Dong before the effective filing date of the claimed invention, to modify the method taught by Banerjee, Fagnan, Dash and Song so as to further utilize a link prediction model like taught by Dong to calculate an accuracy if the knowledge graph were expanded with the new triplet (i.e. to calculate a probability of the new triple). It would have been advantageous to one of ordinary skill to utilize such a combination because it can increase the number of high confidence facts identified, as is suggested by Dong (see e.g. section 5. FUSING EXTRACTORS AND PRIORS). Dong teaches that this accuracy is combined with the confidences produced by the first and second models, inter alia, to determine a total confidence for the triplet (see e.g. section 5. FUSING EXTRACTORS AND PRIORS). As noted above, it would have been obvious to modify the method taught by Banerjee, Fagnan and Dash so as to expand the knowledge graph based on the confidence of the extracted triplet, as is taught by Song. It would have therefore been apparent to expand the knowledge graph with the new triplet based, in part, on the accuracy of the link prediction model. Banerjee, Fagnan, Dash, Song and Dong are thus further considered to teach calculating an accuracy of a link prediction model when the knowledge graph is expanded based on the first confidence and the second confidence, and expanding the knowledge graph with the new triplet based on the accuracy of the link prediction model. Consequently, Banerjee, Fagnan, Dash, Song and Dong teach a method like that of claim 14.
As per claim 15, it would have been obvious, as is described above, to modify the method taught by Banerjee and Fagnan so as to include a relation validation component like taught by Dash, which measures a second confidence of the new triplet and is implemented via a second neural network that has been trained using the triplets of the knowledge graph. Dash suggests that the second neural network (i.e. KBV) is trained to output whether the new triplet (i.e. the triplet produced by the relation extractor/first neural network) is valid:
The RE step takes into account mostly information coming from the corpus for each entity pair to predict the relations, if any, between them. It does not take into account global information provided by the structure of the KG. The relation validation component is designed to overcome this problem. It is formally described as a function
K
B
V
∶
E
×
R
×
E
⟼
R
. For any triple produced by IE (
τ
(
q
)
∶
q
∈
Q
I
E
)
, KBV returns a confidence score.
The KBV system is to be trained from a knowledge graph
K
B
consisting of a set of quads. In this paper, we experimented with two different ways of training, producing two-component systems: (a)
K
B
V
, using the ground truth from the knowledge graph
K
B
t
r
a
i
n
, and (b)
K
B
V
I
E
, using the output of information extraction
Q
I
E
. The result is two different functions returning different confidence scores when applied to the same triple.
(Section 3. Distantly Supervised Relation Extraction and Validation).
After the network is trained, it can be used for both link prediction (i.e., generating the object from a subject and relation input) or validation (i.e., assessing the validity of a new triple composed of known entities and relations). In this paper, we explore the second option.
(Section 3.2. Relation Validation).
Accordingly, the above-described combination of Banerjee, Fagnan, Dash, Song and Dong is further considered to teach a method like that of claim 15.
As per claim 16, it would have been obvious, as is described above, to modify the electronic device taught by Banerjee and Fagnan so as to include a relation validation component like taught by Dash, which measures a second confidence of the new triplet and is implemented via a second neural network that has been trained using the triplets of the knowledge graph. Dash further suggests identifying a confidence of the new triplet by applying a first weight of the first confidence and a second weight of the second confidence (Dash teaches using logistic regression to produce a final confidence based on the first confidence produced by the relation extraction component, i.e. the first neural network, and the second confidence produced by the KBV component, i.e. the second neural network, inter alia:
The confidence scores
s
ξ
from the three systems
ξ
∈
I
E
,
K
B
V
I
E
,
K
B
V
are combined to produce a final confidence for each triple
τ
q
∶
q
∈
Q
I
E
, yielding
Q
f
i
n
a
l
. This step uses a simple logistic regression, typically trained on a validation set separate from the training set.
(Section 3.3. Confidence Re-Estimation).
As known the art, logistic regression computes a final output based in part on learned weights applied to the inputs. Accordingly, using logistic regression to compute a final confidence based on the first and second confidences, inter alia, like taught by Dash would understandably entail applying a first weight of the first confidence and a second weight of the second confidence.). As further described above, it would have been obvious to modify the method taught by Banerjee, Fagnan and Dash so as to expand the knowledge graph based on the confidence of the extracted triplet. Accordingly, it follows that the knowledge graph is expanded based on the confidence of the first triplet like taught by Song, and in which the confidence is determined by applying a first weight of the first confidence and a second weight of the second confidence like taught by Dash. Expanding the knowledge graph thus comprises applying a first weight of the first confidence and a second weight of the second confidence. Consequently, the above-described combination of Banerjee, Fagnan, Dash, Song and Dong is further considered to teach a method like that of claim 16.
As per claim 17, Dash further suggests that the first weight and the second weight are determined based on a quality of the knowledge graph when the knowledge graph is expanded based on the first confidence and when the knowledge graph is expanded based on the second confidence (as noted above, Dash teaches using logistic regression to produce a final confidence based on the first confidence produced by the relation extraction component, i.e. the first neural network, and the second confidence produced by the KBV component, i.e. the second neural network, inter alia. As further described above, using logistic regression to compute the final confidence of the triplet would understandably entail applying a first weight of the first confidence and a second weight of the second confidence. The weights are indicative of the relative importance of the respective inputs and are obtained via supervised learning, as is entailed by logistic regression, by optimizing the weights based on the quality of the outputs when given predetermined inputs. Accordingly, the first weight and the second weight are considered to be determined based on the quality of the knowledge graph when the knowledge graph is expanded based on the first confidence and when the knowledge graph is expanded based on the second confidence.). Accordingly, the above-described combination of Banerjee, Fagnan, Dash, Song and Dong is further considered to teach a method like that of claim 17.
Conclusion
The prior art made of record on form PTO-892 and not relied upon is considered pertinent to applicant’s disclosure. The applicant is required under 37 C.F.R. §1.111(C) to consider these references fully when responding to this action. In particular, the U.S. Patent Application Publication to Ganhotra et al. cited therein describes a method for generating a context-aware knowledge base, wherein the method comprises, inter alia, detecting and extracting resource description framework (RDF) triplets, ranking the extracted triplets, validating one or more RDF triplets associated with the ranked RDF triplets, and connecting the validated RDF triplets to a knowledge graph. The U.S. Patent Application Publication to Sharma et al. cited therein describes a machine learning approach using models built from large unlabeled datasets to extract relationships from documents, wherein the approach comprises collecting training data from a collection of unlabeled documents by matching ground truths for a known entity from existing fact databases with text in the documents describing the known entity.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
6/26/2026
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights