DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on March 31st, 2026 has been entered.
Claims 1-8, 11-18, and 20-23 are pending in the current application, with claims 9, 10, and 19 canceled, claims 1, 11, and 20 being currently amended, and claims 21-23 being new.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4-6, 11, 14-16, 20, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Nallapati (U.S Patent No. US 11604794 B1) in view of Xiaofei Ma (Contrastive Fine-tuning Improves Robustness for Neural Rankers) in further view of Cicero Nogueira dos Santos. (Herein referred to as Cicero) (Beyond [CLS] through Ranking by Generation) in further view of Yu Gu. (Herein referred to as Gu) (Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases) in further view of Jui-Ting Huang et al. (Herein referred to as Huang) (Embedding-based Retrieval in Facebook Search) and in further view of Yu Chen et al. (Herein referred to as Chen) (Bidirectional Attentive Memory Networks for Question Answering over Knowledge Bases)
Regarding claim 1, Nallapati teaches a method of knowledge base question answering, the method comprising: receiving, via a communication interface, a training dataset comprising a question and a corresponding logical form, (“Candidate entity linkages may be determined between an entity recognized in the natural language query and columns in data sets… template queries may be generated and prompted as auto completion for natural language queries, which may guide users to submitting a natural language query that can be understood with a high confidence using the machine learning models from whose training data the template queries have been derived… Entity recognition model 410 may take as input a natural language query, as well as any user-specified entity linkages as discussed above”, pg. 1, Abstract; column 4, lines 12-17; column 16, lines 21-23) (The “template queries” correspond to training data, with a query, comprising a question, and entity linkages between a query and column in data sets corresponding to logical forms.) generating, by accessing a knowledge base, a set of candidate logical forms, (“Linkable entities (e.g., unlinked and/or conflicting entities) identified in natural language query 306 determined by entity detection model 410 may be provided to entity linkage candidate index search 420, in some embodiments. Entity linkage candidate index search 420 may retrieve a list of linking candidates for each entity. For example, for entities of type “column name”, candidates may be retrieved from a column name index. The column name index, which may be generated from metadata provided (as discussed) above for the various available data sets, may be used to identify column names using known column names, column aliases or column “friendly names” associated with a column.”, columns 16 and 17, lines 65-68 for column 16 and lines 1-9 for column 17) (The next quote is what was discussed above in regards to metadata.) (“manually specified metadata can augment database schemas with metadata that can be used by algorithms (e.g., various machine learning models natural language query processing pipeline stages discussed below) to improve question answering… Data set metadata management 340 may take the candidate data sets 404 and obtain those data sets metadata (e.g., from automated and manually specified sources) to generate metadata snapshot 406. Metadata snapshot 406 may be used by various stages of natural language query processing 216 to inform analysis.”, column 8, lines 55-60; column 10, lines 7-12) (The entity linkages correspond to candidate logical forms.) determining, for a first entity mentioned in the question, a first set of candidate entities in the knowledge base that match the first entity, ("Linkable entities (e.g., unlinked and/or conflicting entities) identified in natural language query 306 determined by entity detection model 410 may be provided to entity linkage candidate index search 420, in some embodiments. Entity linkage candidate index search 420 may retrieve a list of linking candidates for each entity.”, columns 16 and 17, lines 65-68 for column 16, and lines 1-3 for column 17) determining linking relations between a second entity mentioned in the question and the first set of candidate entities. (“Entity linkage candidate index search 420 may allow for fuzzy queries based on the Levenshtein distance or other metric for measuring the difference between two sequences… Candidate entity linkages (e.g., a mapping between an entity and column or cell value) returned by entity linkage candidate index search 420 may be limited, in some embodiments, to a top X number of results per entity. Query input bundle 408 may include the natural language query (402), user-specified entity linkages 404, and per-entity linkage candidates determined for recognized entities (e.g., limited to the top X number of linkage candidates).”, column 17, lines 15-17, 26-34) forming a positive input pair based on the question and a corresponding linking relation, ("cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates. The positive candidate may come from the annotated ground truth, and the negative candidates may come from index search results… During training of data set selection model 520, negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates... As indicted at 506, linkages with selected data sets 506 may be provided to intermediate representation generation model 530 along with data set metadata snapshot 406 to provide an intermediate representation 508 of the natural language query" column 18, lines 18-22; column 19, lines 31-35 and 43-47; See also FIG. 5, "data set selection model" 520, "linkage with selected data sets" 506, " intermediate representation generation model" 530) (A positive linking candidate comprises a entity in the natural language query paired with an annotated ground truth, which corresponds to a corresponding linking relation.) forming a plurality of negative input pairs based on the question and the determined linking relations, ("Available data set index search 403 may use text of natural language query 306 to search the metadata index and identify possible column matches of the available data sets for the natural language query… During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates. The positive candidate may come from the annotated ground truth, and the negative candidates may come from index search results… negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates...As indicted at 506, linkages with selected data sets 506 may be provided to intermediate representation generation model 530 along with data set metadata snapshot 406 to provide an intermediate representation 508 of the natural language query.", column 9, lines 48-52; column 19, lines 18-22; column 19, lines 31-35 and 43-47; See also FIG. 5, "data set selection model" 520, "linkage with selected data sets" 506, " intermediate representation generation model" 530) (The negative linking candidates are based on an entity within the natural language query paired with a column value found via an index search, which corresponds to a determined linking relation.) and generating, by the trained ranking model, a ranked list of candidate logical forms from the set of candidate logical forms based on similarity scores between the question and the set of candidate logical forms, respectively. (“a machine learning model may be used to rank the candidate entity linkages according to respectively determined confidence scores, in some embodiments.”, column 23, lines 25-28) (The confidence score correspond to similarity scores)
However, Nallapati does not explicitly teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base; nor training a ranking model based on a contrastive loss by: using the corresponding logical form as a positive sample and negative samples from the generated set of candidate logical forms, nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms, nor training a generation model based on a loss objective using the generated ranked list as training data.
Xiaofei Ma teaches training a ranking model based on a contrastive loss by using the corresponding logical form as a positive sample and negative samples from the generated set of candidate logical forms. (“Contrastive fine-tuning for neural rankers. During fine-tuning, a batch of positive and negative samples from different queries is fed into a neural encoder. The embeddings of query-document pairs from the same query are used to generate ranking scores, which are employed to compute the ranking loss. In parallel, the embeddings of all pairs are used to compute the contrastive loss.”, Figure 1 (See the figure below))
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering, as disclosed by Nallapati, with the training method of a ranking model, as disclosed by Xiaofei Ma. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as fine-tuning a model with contrastive loss improves the robustness and efficiency of a model, as disclosed by Xiaofei Ma, (“As shown in Table 5, adding contrastive loss improves model robustness against all types of perturbations we tested.” pg. 6, right column, under Section 5.3, “Robustness to Query Perturbations”)
However, the combination does not teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms, nor training a generation model based on a loss objective using the generated ranked list as training data.
Cicero teaches training a generation model based on a loss objective using the generated ranked list as training data. (“In this task, given an input question and a set of candidate passages, the goal is to rank the candidate passages so that passages containing the correct answer appear at the top of the ranked list… We fine-tune the LM using the following loss function… A good side effect of using generative models to perform ranking is that we can use the trained model to generate new questions given a passage and vice-versa (depending on the conditioning context used for fine-tuning). This type of synthetically generated data could be used as additional training data to improve discriminative models.”, pg. 1, right column, second paragraph; pg. 2, right column, bottom of the page; pg. 5, directly under section 3.5 “Question and Passage Generation”)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering, as disclosed by Nallapati, with the training method of a ranking model, as disclosed by Xiaofei Ma, and the utilization of a ranked list for training generation models, as disclosed by Cicero. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as using a ranked list as training data for a generative model allows for generated outputs to be using as additional training data, improving models, as disclosed by Cicero. (“A good side effect of using generative models to perform ranking is that we can use the trained model to generate new questions given a passage and vice-versa (depending on the conditioning context used for fine-tuning). This type of synthetically generated data could be used as additional training data to improve discriminative models such as BERT-PR.” pg. 5, directly under section 3.5 “Question and Passage Generation”)
However, the combination does not explicitly teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Gu teaches functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base; (“We employ a simple yet effective strategy to generate candidate logical forms: We enumerate all logical forms, optionally with a count function, within 2 hops starting from each entity identified in the question”, pg. 8, left column, under “4.4 Inference”)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering of Nallapati, as modified by Xiaofei Ma, and Cicero, with the strategy to generate candidate logical forms, as disclosed by Gu. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as the strategy aids in inference for a model, as disclosed by Gu. (“We enumerate all logical forms, optionally with a count function, within 2 hops starting from each entity identified in the question. The recall is 80% on GrailQA. Questions with superlatives and comparatives often do not have a topic entity and are therefore not covered. Ranking can prune the search space… Our Ranking model achieves an F1 of 25.0%, which significantly outperforms the prior art SPARQA [37] by 3.5 percent.”, pg. 8, left column, under “4.4 Inference”)
However, the combination does not teach re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Huang teaches re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs. ("Each batch comprises n positive pairs… We also had observations that the optimal setting is at most two hard negatives per positive… Offline hard negative mining. Offline hard negative mining has the following procedure: (1) generate top K results for each query. (2) select hard negatives based on hard selection strategy. (3) retrain embedding model using the newly generated triplets. (4) the procedure can be iterative.”, pg. 7, right column, second and fourth paragraphs) (The embedding model of Huang is part of a ranking model, with the two or more hard negatives corresponding to a plurality of negative input pairs.)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to the filing date of this application to combine the knowledge base answering system and linking relations of Nallapati, as modified by Xiaofei Ma, Cicero, and Gu, with the retraining of a model of Huang. One of ordinary skill in the art would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for the generation of harder negatives, as disclosed by Huang. (“One limitation of online HNM is that the probability of having any hard negative from random samples could be low and therefore cannot produce hard enough negatives. Next, we look at how to generate harder negatives based on the entire result pool, also known as offline Hard Negative Mining. ", pg. 7, third paragraph)
However, the combination does not teach the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Chen teaches the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms. (“Given a question Q, the goal of a topic entity predictor is to find the best topic entity ˆc from the candidate set {Ci}|C| i=1 returned by external topic entity linking tools… For candidate topic entity Ci, we encode three types of KB aspects, namely, the entity name, entity type and surrounding relations where both entity name and type are represented as a sequence of words while surrounding relations are represented as a bag of sequences of words… We then apply linear projections on the above vectors as follows: Pk i = fk([Cn i ;Ct i;Cr1 i ;Cr2 i ]) Pv i = fv([Cn i ;Ct i;Cr1 i ;Cr2 i ]) (9) where Pk i and Pv i are d-dimensional key and value representations of candidate Ci, respectively. Furthermore, we compute the updated question vector ˆ e using the generalization module mentioned earlier. Next, we use a dot product to compute the similarity score between Q and Ci”, pg. 6, left column, under “3.6 Topic entity prediction”; pg. 6, right column, first and second paragraph) (The vector which the linear projections are applied to correspond to representations of the question and set of candidate logical forms
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to the filing date of this application to combine the knowledge base answering system and linking relations of Nallapati, as modified by Xiaofei Ma, Cicero, Gu, and Huang, with the computation of similarity scores of Chen. One of ordinary skill in the art would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for a similarity score that helps select the best candidate, as described in Chen. (“Next, we use a dot product to compute the similarity score between Q and Ci. A triplet based loss function is used as formulated by o = g(e,Pk i) + g(ˆ e,Pk i) where g(.) is the aforementioned hinge loss function. When training the predictor, along with the candidates returned from ex ternal entity linking tools, we do negative sampling (using string matching) to get more supervision. In the testing phase, the candidate with the highest score is returned as the best topic entity and no negative sampling is applied.”, pg. 6, right column, second paragraph)
Regarding claim 11, Nallapati teaches a system of knowledge base question answering, the system comprising: a communication interface receiving a training dataset comprising a question and a corresponding logical form, (In Nallapati’s disclosure, linkages/linkable entities are the logical forms, as there are in a form that has logic associated with it.) (“template queries may be generated and prompted as auto completion for natural language queries, which may guide users to submitting a natural language query that can be understood with a high confidence using the machine learning models from whose training data the template queries have been derived… Entity recognition model 410 may take as input a natural language query, as well as any user-specified entity linkages as discussed above”, column 4, lines 12-17; column 16, lines 21-23) a memory storing a plurality of processor-executable instruction; (“System memory 1020 may store program instructions and/or data accessible by processor 1010.”, column 26, lines 26-27) a processor reading and executing the plurality of processor-executable instructions to perform operations, (“System memory 1020 may store program instructions and/or data accessible by processor 1010.”, column 26, lines 26-27) generating, by accessing a knowledge base, a set of candidate logical forms, (“Linkable entities (e.g., unlinked and/or conflicting entities) identified in natural language query 306 determined by entity detection model 410 may be provided to entity linkage candidate index search 420, in some embodiments. Entity linkage candidate index search 420 may retrieve a list of linking candidates for each entity. For example, for entities of type “column name”, candidates may be retrieved from a column name index. The column name index, which may be generated from metadata provided (as discussed) above for the various available data sets, may be used to identify column names using known column names, column aliases or column “friendly names” associated with a column.”, columns 16 and 17, lines 65-68 for column 16 and lines 1-9 for column 17) (The next quote is what was discussed above in regards to metadata.) (“manually specified metadata can augment database schemas with metadata that can be used by algorithms (e.g., various machine learning models natural language query processing pipeline stages discussed below) to improve question answering… Data set metadata management 340 may take the candidate data sets 404 and obtain those data sets metadata (e.g., from automated and manually specified sources) to generate metadata snapshot 406. Metadata snapshot 406 may be used by various stages of natural language query processing 216 to inform analysis.”, column 8, lines 55-60; column 10, lines 7-12) (The entity linkages correspond to candidate logical forms.) determining, for a first entity mentioned in the question, a first set of candidate entities in the knowledge base that match the first entity, ("Linkable entities (e.g., unlinked and/or conflicting entities) identified in natural language query 306 determined by entity detection model 410 may be provided to entity linkage candidate index search 420, in some embodiments. Entity linkage candidate index search 420 may retrieve a list of linking candidates for each entity.”, columns 16 and 17, lines 65-68 for column 16, and lines 1-3 for column 17) determining linking relations between a second entity mentioned in the question and the first set of candidate entities. (“Entity linkage candidate index search 420 may allow for fuzzy queries based on the Levenshtein distance or other metric for measuring the difference between two sequences… Candidate entity linkages (e.g., a mapping between an entity and column or cell value) returned by entity linkage candidate index search 420 may be limited, in some embodiments, to a top X number of results per entity. Query input bundle 408 may include the natural language query (402), user-specified entity linkages 404, and per-entity linkage candidates determined for recognized entities (e.g., limited to the top X number of linkage candidates).”, column 17, lines 15-17, 26-34) forming a positive input pair based on the question and a corresponding linking relation, ("cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates. The positive candidate may come from the annotated ground truth, and the negative candidates may come from index search results… During training of data set selection model 520, negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates... As indicted at 506, linkages with selected data sets 506 may be provided to intermediate representation generation model 530 along with data set metadata snapshot 406 to provide an intermediate representation 508 of the natural language query" column 18, lines 18-22; column 19, lines 31-35 and 43-47; See also FIG. 5, "data set selection model" 520, "linkage with selected data sets" 506, " intermediate representation generation model" 530) (A positive linking candidate comprises a entity in the natural language query paired with an annotated ground truth, which corresponds to a corresponding linking relation.) forming a plurality of negative input pairs based on the question and the determined linking relations, ("Available data set index search 403 may use text of natural language query 306 to search the metadata index and identify possible column matches of the available data sets for the natural language query… During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates. The positive candidate may come from the annotated ground truth, and the negative candidates may come from index search results… negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates...As indicted at 506, linkages with selected data sets 506 may be provided to intermediate representation generation model 530 along with data set metadata snapshot 406 to provide an intermediate representation 508 of the natural language query.", column 9, lines 48-52; column 19, lines 18-22; column 19, lines 31-35 and 43-47; See also FIG. 5, "data set selection model" 520, "linkage with selected data sets" 506, " intermediate representation generation model" 530) (The negative linking candidates are based on an entity within the natural language query paired with a column value found via an index search, which corresponds to a determined linking relation.) and generating, by the trained ranking model, a ranked list of candidate logical forms from the set of candidate logical forms based on similarity scores between the question and the set of candidate logical forms, respectively. (“a machine learning model may be used to rank the candidate entity linkages according to respectively determined confidence scores, in some embodiments.”, column 23, lines 25-28) (The confidence score correspond to similarity scores)
However, Nallapati does not explicitly teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base; nor training a ranking model based on a contrastive loss by: using the corresponding logical form as a positive sample and negative samples from the generated set of candidate logical forms, nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms, nor training a generation model based on a loss objective using the generated ranked list as training data.
Xiaofei Ma teaches training a ranking model based on a contrastive loss by using the corresponding logical form as a positive sample and negative samples from the generated set of candidate logical forms. (“Contrastive fine-tuning for neural rankers. During fine-tuning, a batch of positive and negative samples from different queries is fed into a neural encoder. The embeddings of query-document pairs from the same query are used to generate ranking scores, which are employed to compute the ranking loss. In parallel, the embeddings of all pairs are used to compute the contrastive loss.”, Figure 1 (See the figure below))
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering, as disclosed by Nallapati, with the training method of a ranking model, as disclosed by Xiaofei Ma. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as fine-tuning a model with contrastive loss improves the robustness and efficiency of a model, as disclosed by Xiaofei Ma, (“As shown in Table 5, adding contrastive loss improves model robustness against all types of perturbations we tested.” pg. 6, right column, under Section 5.3, “Robustness to Query Perturbations”)
However, the combination does not teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms, nor training a generation model based on a loss objective using the generated ranked list as training data.
Cicero teaches training a generation model based on a loss objective using the generated ranked list as training data. (“In this task, given an input question and a set of candidate passages, the goal is to rank the candidate passages so that passages containing the correct answer appear at the top of the ranked list… We fine-tune the LM using the following loss function… A good side effect of using generative models to perform ranking is that we can use the trained model to generate new questions given a passage and vice-versa (depending on the conditioning context used for fine-tuning). This type of synthetically generated data could be used as additional training data to improve discriminative models.”, pg. 1, right column, second paragraph; pg. 2, right column, bottom of the page; pg. 5, directly under section 3.5 “Question and Passage Generation”)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering, as disclosed by Nallapati, with the training method of a ranking model, as disclosed by Xiaofei Ma, and the utilization of a ranked list for training generation models, as disclosed by Cicero. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as using a ranked list as training data for a generative model allows for generated outputs to be using as additional training data, improving models, as disclosed by Cicero. (“A good side effect of using generative models to perform ranking is that we can use the trained model to generate new questions given a passage and vice-versa (depending on the conditioning context used for fine-tuning). This type of synthetically generated data could be used as additional training data to improve discriminative models such as BERT-PR.” pg. 5, directly under section 3.5 “Question and Passage Generation”)
However, the combination does not explicitly teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Gu teaches functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base; (“We employ a simple yet effective strategy to generate candidate logical forms: We enumerate all logical forms, optionally with a count function, within 2 hops starting from each entity identified in the question”, pg. 8, left column, under “4.4 Inference”)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering of Nallapati, as modified by Xiaofei Ma, and Cicero, with the strategy to generate candidate logical forms, as disclosed by Gu. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as the strategy aids in inference for a model, as disclosed by Gu. (“We enumerate all logical forms, optionally with a count function, within 2 hops starting from each entity identified in the question. The recall is 80% on GrailQA. Questions with superlatives and comparatives often do not have a topic entity and are therefore not covered. Ranking can prune the search space… Our Ranking model achieves an F1 of 25.0%, which significantly outperforms the prior art SPARQA [37] by 3.5 percent.”, pg. 8, left column, under “4.4 Inference”)
However, the combination does not teach re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Huang teaches re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs. ("Each batch comprises n positive pairs… We also had observations that the optimal setting is at most two hard negatives per positive… Offline hard negative mining. Offline hard negative mining has the following procedure: (1) generate top K results for each query. (2) select hard negatives based on hard selection strategy. (3) retrain embedding model using the newly generated triplets. (4) the procedure can be iterative.”, pg. 7, right column, second and fourth paragraphs) (The embedding model of Huang corresponds to a ranking model, with the two or more hard negatives corresponding to a plurality of negative input pairs.)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to the filing date of this application to combine the knowledge base answering system and linking relations of Nallapati, as modified by Xiaofei Ma, Cicero, and Gu, with the retraining of a model of Huang. One of ordinary skill in the art would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for the generation of harder negatives, as disclosed by Huang. (“One limitation of online HNM is that the probability of having any hard negative from random samples could be low and therefore cannot produce hard enough negatives. Next, we look at how to generate harder negatives based on the entire result pool, also known as offline Hard Negative Mining. ", pg. 7, third paragraph)
However, the combination does not teach the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Chen teaches the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms. (“Given a question Q, the goal of a topic entity predictor is to find the best topic entity ˆc from the candidate set {Ci}|C| i=1 returned by external topic entity linking tools… For candidate topic entity Ci, we encode three types of KB aspects, namely, the entity name, entity type and surrounding relations where both entity name and type are represented as a sequence of words while surrounding relations are represented as a bag of sequences of words… We then apply linear projections on the above vectors as follows: Pk i = fk([Cn i ;Ct i;Cr1 i ;Cr2 i ]) Pv i = fv([Cn i ;Ct i;Cr1 i ;Cr2 i ]) (9) where Pk i and Pv i are d-dimensional key and value representations of candidate Ci, respectively. Furthermore, we compute the updated question vector ˆ e using the generalization module mentioned earlier. Next, we use a dot product to compute the similarity score between Q and Ci”, pg. 6, left column, under “3.6 Topic entity prediction”; pg. 6, right column, first and second paragraph) ( The vector which the linear projections are applied to correspond to representations of the question and set of candidate logical forms
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to the filing date of this application to combine the knowledge base answering system and linking relations of Nallapati, as modified by Xiaofei Ma, Cicero, Gu, and Huang, with the computation of similarity scores of Chen. One of ordinary skill in the art would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for a similarity score that helps select the best candidate, as described in Chen. (“Next, we use a dot product to compute the similarity score between Q and Ci. A triplet based loss function is used as formulated by o = g(e,Pk i) + g(ˆ e,Pk i) where g(.) is the aforementioned hinge loss function. When training the predictor, along with the candidates returned from ex ternal entity linking tools, we do negative sampling (using string matching) to get more supervision. In the testing phase, the candidate with the highest score is returned as the best topic entity and no negative sampling is applied.”, pg. 6, right column, second paragraph)
Regarding claim 20, Nallapati teaches a processor-readable non-transitory storage medium storing a plurality of processor- executable instructions for knowledge base question answering, the instructions being executed by one or more processors to perform operations, (“In some embodiments, instructions stored on a non-transitory, computer-accessible medium separate from computer system 1000 may be transmitted to computer system 1000 via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link.”, column 28, lines 6-12) receiving, via a communication interface, a training dataset comprising a question and a corresponding logical form, (In Nallapati’s disclosure, linkages/linkable entities are the logical forms, as there are in a form that has logic associated with it.) (“template queries may be generated and prompted as auto completion for natural language queries, which may guide users to submitting a natural language query that can be understood with a high confidence using the machine learning models from whose training data the template queries have been derived… Entity recognition model 410 may take as input a natural language query, as well as any user-specified entity linkages as discussed above”, column 4, lines 12-17; column 16, lines 21-23) generating, by accessing a knowledge base, a set of candidate logical forms, (“Linkable entities (e.g., unlinked and/or conflicting entities) identified in natural language query 306 determined by entity detection model 410 may be provided to entity linkage candidate index search 420, in some embodiments. Entity linkage candidate index search 420 may retrieve a list of linking candidates for each entity. For example, for entities of type “column name”, candidates may be retrieved from a column name index. The column name index, which may be generated from metadata provided (as discussed) above for the various available data sets, may be used to identify column names using known column names, column aliases or column “friendly names” associated with a column.”, columns 16 and 17, lines 65-68 for column 16 and lines 1-9 for column 17) (The next quote is what was discussed above in regards to metadata.) (“manually specified metadata can augment database schemas with metadata that can be used by algorithms (e.g., various machine learning models natural language query processing pipeline stages discussed below) to improve question answering… Data set metadata management 340 may take the candidate data sets 404 and obtain those data sets metadata (e.g., from automated and manually specified sources) to generate metadata snapshot 406. Metadata snapshot 406 may be used by various stages of natural language query processing 216 to inform analysis.”, column 8, lines 55-60; column 10, lines 7-12) (The entity linkages correspond to candidate logical forms.) determining, for a first entity mentioned in the question, a first set of candidate entities in the knowledge base that match the first entity, ("Linkable entities (e.g., unlinked and/or conflicting entities) identified in natural language query 306 determined by entity detection model 410 may be provided to entity linkage candidate index search 420, in some embodiments. Entity linkage candidate index search 420 may retrieve a list of linking candidates for each entity.”, columns 16 and 17, lines 65-68 for column 16, and lines 1-3 for column 17) determining linking relations between a second entity mentioned in the question and the first set of candidate entities. (“Entity linkage candidate index search 420 may allow for fuzzy queries based on the Levenshtein distance or other metric for measuring the difference between two sequences… Candidate entity linkages (e.g., a mapping between an entity and column or cell value) returned by entity linkage candidate index search 420 may be limited, in some embodiments, to a top X number of results per entity. Query input bundle 408 may include the natural language query (402), user-specified entity linkages 404, and per-entity linkage candidates determined for recognized entities (e.g., limited to the top X number of linkage candidates).”, column 17, lines 15-17, 26-34) forming a positive input pair based on the question and a corresponding linking relation, ("cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates. The positive candidate may come from the annotated ground truth, and the negative candidates may come from index search results… During training of data set selection model 520, negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates... As indicted at 506, linkages with selected data sets 506 may be provided to intermediate representation generation model 530 along with data set metadata snapshot 406 to provide an intermediate representation 508 of the natural language query" column 18, lines 18-22; column 19, lines 31-35 and 43-47; See also FIG. 5, "data set selection model" 520, "linkage with selected data sets" 506, " intermediate representation generation model" 530) (A positive linking candidate comprises a entity in the natural language query paired with an annotated ground truth, which corresponds to a corresponding linking relation.) forming a plurality of negative input pairs based on the question and the determined linking relations, ("Available data set index search 403 may use text of natural language query 306 to search the metadata index and identify possible column matches of the available data sets for the natural language query… During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates. The positive candidate may come from the annotated ground truth, and the negative candidates may come from index search results… negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates...As indicted at 506, linkages with selected data sets 506 may be provided to intermediate representation generation model 530 along with data set metadata snapshot 406 to provide an intermediate representation 508 of the natural language query.", column 9, lines 48-52; column 19, lines 18-22; column 19, lines 31-35 and 43-47; See also FIG. 5, "data set selection model" 520, "linkage with selected data sets" 506, " intermediate representation generation model" 530) (The negative linking candidates are based on an entity within the natural language query paired with a column value found via an index search, which corresponds to a determined linking relation.) and generating, by the trained ranking model, a ranked list of candidate logical forms from the set of candidate logical forms based on similarity scores between the question and the set of candidate logical forms, respectively. (“a machine learning model may be used to rank the candidate entity linkages according to respectively determined confidence scores, in some embodiments.”, column 23, lines 25-28) (The confidence score correspond to similarity scores)
However, Nallapati does not explicitly teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base; nor training a ranking model based on a contrastive loss by: using the corresponding logical form as a positive sample and negative samples from the generated set of candidate logical forms, nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms, nor training a generation model based on a loss objective using the generated ranked list as training data.
Xiaofei Ma teaches training a ranking model based on a contrastive loss by using the corresponding logical form as a positive sample and negative samples from the generated set of candidate logical forms. (“Contrastive fine-tuning for neural rankers. During fine-tuning, a batch of positive and negative samples from different queries is fed into a neural encoder. The embeddings of query-document pairs from the same query are used to generate ranking scores, which are employed to compute the ranking loss. In parallel, the embeddings of all pairs are used to compute the contrastive loss.”, Figure 1 (See the figure below))
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering, as disclosed by Nallapati, with the training method of a ranking model, as disclosed by Xiaofei Ma. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as fine-tuning a model with contrastive loss improves the robustness and efficiency of a model, as disclosed by Xiaofei Ma, (“As shown in Table 5, adding contrastive loss improves model robustness against all types of perturbations we tested.” pg. 6, right column, under Section 5.3, “Robustness to Query Perturbations”)
However, the combination does not teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms, nor training a generation model based on a loss objective using the generated ranked list as training data.
Cicero teaches training a generation model based on a loss objective using the generated ranked list as training data. (“In this task, given an input question and a set of candidate passages, the goal is to rank the candidate passages so that passages containing the correct answer appear at the top of the ranked list… We fine-tune the LM using the following loss function… A good side effect of using generative models to perform ranking is that we can use the trained model to generate new questions given a passage and vice-versa (depending on the conditioning context used for fine-tuning). This type of synthetically generated data could be used as additional training data to improve discriminative models.”, pg. 1, right column, second paragraph; pg. 2, right column, bottom of the page; pg. 5, directly under section 3.5 “Question and Passage Generation”)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering, as disclosed by Nallapati, with the training method of a ranking model, as disclosed by Xiaofei Ma, and the utilization of a ranked list for training generation models, as disclosed by Cicero. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as using a ranked list as training data for a generative model allows for generated outputs to be using as additional training data, improving models, as disclosed by Cicero. (“A good side effect of using generative models to perform ranking is that we can use the trained model to generate new questions given a passage and vice-versa (depending on the conditioning context used for fine-tuning). This type of synthetically generated data could be used as additional training data to improve discriminative models such as BERT-PR.” pg. 5, directly under section 3.5 “Question and Passage Generation”)
However, the combination does not explicitly teach functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base nor re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs; nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Gu teaches functions operating on set-based semantics, by starting from each entity of the set of entities and searching for paths reachable within two hops of each entity in a knowledge base; (“We employ a simple yet effective strategy to generate candidate logical forms: We enumerate all logical forms, optionally with a count function, within 2 hops starting from each entity identified in the question”, pg. 8, left column, under “4.4 Inference”)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of knowledge base question answering of Nallapati, as modified by Xiaofei Ma, and Cicero, with the strategy to generate candidate logical forms, as disclosed by Gu. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as the strategy aids in inference for a model, as disclosed by Gu. (“We enumerate all logical forms, optionally with a count function, within 2 hops starting from each entity identified in the question. The recall is 80% on GrailQA. Questions with superlatives and comparatives often do not have a topic entity and are therefore not covered. Ranking can prune the search space… Our Ranking model achieves an F1 of 25.0%, which significantly outperforms the prior art SPARQA [37] by 3.5 percent.”, pg. 8, left column, under “4.4 Inference”)
However, the combination does not teach re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs nor the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Huang teaches re-training the trained ranking model based on the positive input pair and the plurality of negative input pairs. ("Each batch comprises n positive pairs… We also had observations that the optimal setting is at most two hard negatives per positive… Offline hard negative mining. Offline hard negative mining has the following procedure: (1) generate top K results for each query. (2) select hard negatives based on hard selection strategy. (3) retrain embedding model using the newly generated triplets. (4) the procedure can be iterative.”, pg. 7, right column, second and fourth paragraphs) (The embedding model of Huang corresponds to a ranking model, with the two or more hard negatives corresponding to a plurality of negative input pairs.)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to the filing date of this application to combine the knowledge base answering system and linking relations of Nallapati, as modified by Xiaofei Ma, Cicero, and Gu, with the retraining of a model of Huang. One of ordinary skill in the art would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for the generation of harder negatives, as disclosed by Huang. (“One limitation of online HNM is that the probability of having any hard negative from random samples could be low and therefore cannot produce hard enough negatives. Next, we look at how to generate harder negatives based on the entire result pool, also known as offline Hard Negative Mining. ", pg. 7, third paragraph)
However, the combination does not teach the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms.
Chen teaches the similarity scores are computed by applying linear projections to concatenations of representations of the question and the set of candidate logical forms. (“Given a question Q, the goal of a topic entity predictor is to find the best topic entity ˆc from the candidate set {Ci}|C| i=1 returned by external topic entity linking tools… For candidate topic entity Ci, we encode three types of KB aspects, namely, the entity name, entity type and surrounding relations where both entity name and type are represented as a sequence of words while surrounding relations are represented as a bag of sequences of words… We then apply linear projections on the above vectors as follows: Pk i = fk([Cn i ;Ct i;Cr1 i ;Cr2 i ]) Pv i = fv([Cn i ;Ct i;Cr1 i ;Cr2 i ]) (9) where Pk i and Pv i are d-dimensional key and value representations of candidate Ci, respectively. Furthermore, we compute the updated question vector ˆ e using the generalization module mentioned earlier. Next, we use a dot product to compute the similarity score between Q and Ci”, pg. 6, left column, under “3.6 Topic entity prediction”; pg. 6, right column, first and second paragraph) ( The vector which the linear projections are applied to correspond to representations of the question and set of candidate logical forms
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to the filing date of this application to combine the knowledge base answering system and linking relations of Nallapati, as modified by Xiaofei Ma, Cicero, Gu, and Huang, with the computation of similarity scores of Chen. One of ordinary skill in the art would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for a similarity score that helps select the best candidate, as described in Chen. (“Next, we use a dot product to compute the similarity score between Q and Ci. A triplet based loss function is used as formulated by o = g(e,Pk i) + g(ˆ e,Pk i) where g(.) is the aforementioned hinge loss function. When training the predictor, along with the candidates returned from ex ternal entity linking tools, we do negative sampling (using string matching) to get more supervision. In the testing phase, the candidate with the highest score is returned as the best topic entity and no negative sampling is applied.”, pg. 6, right column, second paragraph)
Regarding claims 4, 14, and 23, Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, and Chen teaches the cross-entropy loss is computed by: generating, by the ranking model, a first logit representing a first similarity score between the question and the positive sample ("Entity linkage model 512 may be implemented as a deep learning model, utilizing a neural network trained to identify and rank entity linkages in a given query text string and other input data... A linear layer may then be applied on the classification token [CLS] to produce a logit score (BERT score). During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates.”, column 18, lines 1-4 and 16-20 (Nallapati)) generating a plurality of logits representing similarity scores between the question and the plurality of negative samples, respectively, ("Entity linkage model 512 may be implemented as a deep learning model, utilizing a neural network trained to identify and rank entity linkages in a given query text string and other input data... A linear layer may then be applied on the classification token [CLS] to produce a logit score (BERT score). During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates.”, column 18, lines 1-4 and 16-20 (Nallapati)) and generating a plurality of logits representing similarity scores between the question and the plurality of negative samples, respectively ("Entity linkage model 512 may be implemented as a deep learning model, utilizing a neural network trained to identify and rank entity linkages in a given query text string and other input data... A linear layer may then be applied on the classification token [CLS] to produce a logit score (BERT score)… Finally, a softmax may be used on the logit scores of all datasets, and compute scores for each candidate dataset.”, column 18, lines 1-4 and 16-20; column 19, lines 29-31 (Nallapati);
However, the combination, as applied in claims 1, 11, and 20, does not teach contrastive loss nor computing the contrastive loss based on the first logit and the plurality of logits.
Xiaofei Ma teach contrastive loss, (“The contrastive loss can be applied to the representations from a variety of encoders… In this work, we explore contrastive fine-tuning for both BERT (Devlin et al., 2018) and BART (Lewis et al.) models”, pg.3, right column, under “2.2 Contrastive Loss”) and with the contrastive loss applying to BERT models (according to Xiaofei Ma), one could combine Xiaofei Ma’s contrastive loss with Nallapati’s logits (which use a BERT score) to compute the contrastive loss based on the first logit and the plurality of logits.
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the teachings of Nallapati, as modified by Xiaofei Ma, Cicero, Gu, Huang, and Chen, with Xiaofei Ma’s contrastive loss fine-tuning. One of ordinary skill in the art would have been motivated to combine the teachings, prior to the filing date of the current application, as fine-tuning a model with contrastive loss improves the robustness and efficiency of a model, as disclosed by Xiaofei Ma, (“As shown in Table 5, adding contrastive loss improves model robustness against all types of perturbations we tested.” pg. 6, right column, under Section 5.3, “Robustness to Query Perturbations”)
Regarding claims 5 and 15, Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, and Chen, teaches the method and system of claims 1 and 11 respectively, as well as training the generation model comprises: generating, by the generation model, a first target logical form from the generated ranked list of candidate logical forms at a current training step. (“During training of data set selection model 520, negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates …the target embeddings consists of: (1) Grammar representation, (2) Column representation, and (3) Entity representation. These target embeddings corresponds to action vocabularies, and they represent the possible actions that the decoder can take at each step.”, column 19, lines 30-35; column 20, lines 25-29 (Nallapati))
Regarding claims 6 and 16, Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, and Chen, teaches the method and system of claims 5 and 15 respectively, as well as generating, by the generation model, a second target logical form from the generated ranked list of candidate logical forms at a next training step, (“During training of data set selection model 520, negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates …the target embeddings consists of: (1) Grammar representation, (2) Column representation, and (3) Entity representation. These target embeddings correspond to action vocabularies, and they represent the possible actions that the decoder can take at each step.”, column 19, lines 30-35; column 20, lines 25-29 (Nallapati)) and computing a cross-entropy loss between the second target logical form and the first target logical form as ground truth. (“A linear layer may then be applied on the classification token [CLS] to produce a logit score (BERT score). During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate and at most Y (e.g., 63) negative candidates. The positive candidate may come from the annotated ground truth, and the negative candidates may come from index search results.”, column 18, lines 16-22 (Nallapati))
Claim(s) 2, 7, 8, 12, 17, 18, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Nallapati, in view of Xiaofei Ma, in further view of Cicero, in further view of Gu, in further view of Huang, in further view of Chen, and in further view of G. Maheshwari et al. (Herein referred to as Maheshwari) (“Learning to Rank Query Graphs for Complex Question Answering over Knowledge Graphs” from “the Semantic Web” (NPL))
Regarding claims 2, 12, and 21, Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, and Chen, teaches the ranking model trained by: randomly sampling a subset of negative samples from the set of candidate logical forms, (“negative random samples”, column 19, line 32 (Nallapati)) forming a positive input of the question and the corresponding logical form, ("During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate… negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates.", column 18, lines 16-20; column 19, lines 32-35 (Nallapati)) forming a plurality of negative inputs from the question and the subset of negative samples, (“Entity linkage model 512 may be implemented as a deep learning model, utilizing a neural network trained to identify and rank entity linkages in a given query text string and other input data... A linear layer may then be applied on the classification token [CLS] to produce a logit score (BERT score). During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate at most Y (e.g., 63) negative candidates.”, column 18, lines 1-4 and 16-20 (Nallapati))
However, the combination does not teach training the ranking model using the positive input and the plurality of negative inputs for a number of epochs at a beginning of training.
Maheshwari teaches training the ranking model using the positive input and the plurality of negative inputs for a number of epochs at a beginning of training. ("Our models are trained with negative sampling, where we sample 100 negative core chains per question, along with the correct one… We train our models for a maximum of 300 epochs, using a 70-10-20 split as train, validation and test data…" pg. 497, Paragraph labeled "Training")
Therefore, it would have been obvious to someone with ordinary skill in the art to combine the knowledge base question answering method of Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, and Chen, with the training of a model with epochs, as taught by Maheshwari. One would be motivated to combine the teachings, prior to the filing date of the current application, as training a model for a number of epochs allows a model to reach its best performance by minimizing error. (“When using training algorithms, the number of epochs can reach thousands, and the process is programmed to continue until the model error is suitably minimized. Typically, tutorials and examples employ figures such as 10, 100, 1000, or even higher.”, Under “What is an epoch in ML?” (Deepchecks))
Regarding claims 7 and 17, Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, and Chen, teaches the method and system of claims 1 and 11 respectively, but does not teach receiving a testing question, nor generating, by the trained ranking model and the trained generation model, a target logical form for the testing question.
Maheshwari teaches receiving a testing question, (“QALD-7 [Question Answering over Linked Data model version 7] is a gold-standard dataset having 220 and 43 training and test questions respectively along with their corresponding SPARQL queries.” pg. 496, last paragraph) and generating, by the trained ranking model and the trained generation model, a target logical form for the testing question. (“We treat KGQA as the task of generating and ranking query graph candidates w.r.t. [with respect to] a given NLQ.", pg. 492, first paragraph; "We now have all the information required to construct the query graph, and the corresponding executable query.”, pg. 496, paragraph 3)
Therefore, it would have been considered obvious to one of ordinary skill in the art, prior to this application’s filing date, to combine the method of Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, and Chen, with the testing data of Maheshwari. One would be motivated to combine the teachings, prior to the filing date of the current application, as, according to Maheshwari, testing data is useful for error analysis. and one might the combination effective for the purpose of providing an improved technique. ("Error Analysis: We now illustrate the effect of different characteristics of the test data on the model performance…An error analysis shows that the model performs especially well on smaller candidate sets and for longer questions which highlights its high potential for answering complicated questions.", pg. 498 under Error Analysis; pg. 502, under “Conclusion and Future Work” (Maheshwari))
Regarding claim 8 and 18, Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, Chen, and Maheshwari, teaches the method and system of claims 7 and 17 respectively, further comprising: generating an answer to the testing question by applying the target logical form on the knowledge base. (“We treat KGQA as the task of generating and ranking query graph candidates w.r.t. [with respect to] a given NLQ...An error analysis shows that the model performs especially well on smaller candidate sets and for longer questions which highlights its high potential for answering complicated questions.", pg. 492, First Paragraph; pg. 502, under Conclusion and Future Work (Maheshwari))
Claim(s) 3, 13, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Nallapati, in view of Xiaofei Ma, in further view of Cicero, in further view of Gu, in further view of Huang, in further view of Chen, in further view of Maheshwari, and in further view of Yixuan Su et al. (Herein referred to as Su) (“Dialogue Response Selection with Hierarchical Curriculum Learning”, relevant pages: pg. 1-3 (NPL))
Regarding claims 3, 13, and 22, Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, Chen, and Maheshwari, teaches forming a set of negative inputs by pairing the question and the one or more negative samples; (“Entity linkage model 512 may be implemented as a deep learning model, utilizing a neural network trained to identify and rank entity linkages in a given query text string and other input data... A linear layer may then be applied on the classification token [CLS] to produce a logit score (BERT score). During training, cross-entropy loss may be calculated on all the linking candidates, including one positive candidate", column 18, lines 1-4 and 16-19 (Nallapati)) and training the ranking model using the positive input and the set of negative inputs at a later stage of training. ("During training of data set selection model 520, negative random samples (e.g., four) may be gathered for each positive training example, and cross-entropy loss may be used to boost the scores of the positive candidates.", column 19, lines 31-35 (Nallapati))
However, the combination does not teach selecting one or more negative samples that are similar to the positive sample from the subset of negative samples
Su teaches selecting one or more negative samples that are similar to the positive sample from the subset of negative samples (“we select the positive context-response pairs according to the corpus-level pacing function pcc(t). Then, for each instance in the selected batch, we sample its associated negative examples according to the instance”, pg. 4, right column, under “3.4 Hierarchical Curriculum Learning “; See also pg. 3, under “3.2 Corpus-Level Curriculum”)
Therefore, it would have been considered obvious to one of ordinary skill of the art, prior to the filing date of this application, to combine the teachings of Nallapati, as modified by Cicero, Xiaofei Ma, Gu, Huang, Chen, and Maheshwari, with the difficulty function of Yixuan Su. One could be motivated to combine the teachings, prior to the filing date of the current application, as the difficulty function allows for the ability to measure the difficulty of each training pair. (“To measure the difficulty of each training context-response pair, we adopt a pre-trained ranking model to calculate its relevance score…”, pg. 3, under “3.2 Corpus-Level Curriculum (Yixuan Su))
Response to Arguments
Applicant's arguments filed on March 12th, 2026 have been fully considered but they are not persuasive. The applicant argues in substance:
Argument 1: The current references do not teach the added limitations in the amended claim 1.
Applicant’s arguments with respect to claim 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Tyler E Iles whose telephone number is (571)272-5442. The examiner can normally be reached 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.E.I./Patent Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights