DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/12/2026 has been entered.
Response to Arguments
1. Regarding the rejection under 35 U.S.C. § 103, Applicant's arguments filed 03/12/2026 have been fully considered but they are not persuasive.
Applicant argues that the cited references does not disclose or make obvious the claimed limitation of “…performing a semantic search to identify a digital file from a set of digital files that is the most semantically similar to a validated answer in a validated question-answer pair, wherein the digital file is not used for generating the validated answer”, specifically that the reference link to another question-answer pair in Ben-Simhon does not read on the claimed digital file in claim 1. The Examiner respectfully disagrees with this argument. Under the broadest reasonable interpretation of “a digital file”, the claims requires that a particular collection of data stored digital on a computer be linked. Since the claim does not specify which kind of digital file is being linked, this digital file can be any form of data stored digitally on a computer, including another question-answer pair. Ben-Simhon teaches this concept. Specifically, Ben-Simhon teaches linking to question-answer pairs (for example, see Fig. 3), and that the QA pairs are stored digitally on a computer (para. 0087). Therefore, Ben-Simhon teaches the concept of linking to a digital file.
Hence, Applicant’s arguments are not persuasive.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claims 1, 3-5, and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Garg et al. (US 2024/0070434, hereinafter Garg) in view of Khosla et al. (US 2025/0005058, hereinafter Khosla) and further in view of Ben-Simhon et al. (US 2021/0124783 A1, hereinafter Ben-Simhon).
Regarding claim 1, Garg discloses A method performed by one or more processing devices (Fig. 4), comprising: accessing a set of digital content available at a set of designated network locations (para. 0035 “…collecting any combination of domain-relevant structured, semi-structured and unstructured information sources (e.g., private, public, or authentication-based websites and content, internal and external documents in any formats, whether static or live, audio files, video files, historical conversations between customers and agents, and postings in community forums). Examples of static documents include text, policy and FAQ documents (e.g., in .pdf, .docx, .xlsx, and .csv formats) and structured data files (e.g., in .xlsx and .xls formats). Examples of live documents include documents that are continuously being developed by multiple authors or entities in a collaborative manner (e.g., Google Docs, such as Google Sheet files), application program interfaces (e.g., to data in databases) and access means to on-line documents (e.g., customized URLs, such as links to customer service management system resources).”); further training a pre-trained large language model (LLM) using the set of digital content to obtain a customized LLM (para. 0041 “For example, in one embodiment, the conversational knowledge base uses a transformer-based deep learning pre-trained-t5-small model, which is fine-tuned on the popular Squad dataset for end-to-end question generation. The transformer-based deep learning model may also be further fine-tuned on client-specific datasets for improved results. Fine-tuning may include extracting a list of consecutive pairs of sentences from the text and passing the extracted sentences into the model for use as context for the questions generated. Relevant context may also be extracted by tokenizing the available text.”) executing the customized LLM to generate a set of question-answer pairs based on the set of digital content using the customized LLM (para. 0040 “At steps 103a and 103b, the conversational knowledge base articles synthesize a diverse and exhaustive set of question-answer pairs. For example, based on the extracted information, question-answer pairs are generated in customer system 500 of FIG. 4 in question-answer generation module 513.”); augmenting the … answer in the … question-answer pair to include a digital asset, wherein the digital asset comprises a link to the digital file…(para. 0056 “One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format). The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.”; para. 0057 “Other options include activating logging of the question posed and the answer delivered, sending out web links along with the answer, and offering additional relevant answers and options of which the customer may be able to take advantage.”).
Garg does not specifically disclose executing a textual entailment model to validate a plurality of question-answer pairs from the set of question-answer pairs, thereby generating a set of validation question-answer pairs, wherein validating a question-answer pair comprises determining that an answer in the question-answer pair is derived from the set of digital content;
performing a semantic search to identify a digital file form a set of digital files that is the most semantically similar to a validated answer in a validated question-answer pair…;
Khosla teaches executing a textual entailment model to validate a plurality of question-answer pairs from the set of question-answer pairs, thereby generating a set of validation question-answer pairs, wherein validating a question-answer pair comprises determining that an answer in the question-answer pair is derived from the set of digital content (para. 0053 “The memory 210 may also include a textual NLI module 236 for determining whether the answer generated from the LLM component 106 contradicts the retrieved passages from the aggregator component 104 by using a premise and hypothesis. The textual NLI interpretation module 236 may use NLI models to determine if there is a contraction. The textual NLI interpretation module 236 may utilize NLI models to take two text sequences as input (e.g., a hypothesis and a premise), and determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise.);
performing a semantic search to identify a digital file from a set of digital files that is the most semantically similar to a validated answer in a validated question-answer pair… (semantic search (retrieving k-nearest embeddings using dense index) performed for a set of files (similar questions stored in QA pairs in search system) to find most semantic similar (e.g. closest to the natural language question embedding); for k=1, only most semantically similar to validated answer (input from verifier component 108) is selected; para. 0019 “For example, the natural language question answering service 102 can have access to different search systems (e.g., the system systems 124) where the search systems may stored passages and question and answer pairs (QA) pairs regarding a particular network-based service…”; para. 0067 “At (10), the attribution component 109 may provide references to the retrieved passages, inline citations to sentences of retrieved passages used in the answer, or provide similar questions to the natural language question. For example, the attribution component 109 may provide reference links and titles to the retrieved passages used by the LLM component 106 (e.g., retrieved passages used as context to generate the answer), which may allow the submitter of the question to get more details on the referenced passages. …As another example, the attribution component 109 may provide questions related to the natural language question by referencing a dense index (e.g., dense index used to create a fixed-dimensional representation of prior questions) and retrieving the k closest questions to the natural language question. The references, inline citations, and similar questions may be provided, but not limited to, within the answer or after the answer generated by the LLM component 106.”).
Garg and Khosla are considered to be analogous to the claimed invention as they both are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Garg to incorporate the teachings of Khosla in order to perform a semantic search to identify a digital file from a set of digital files that is the most semantically similar to a validated answer in a validated question-answer pair. Doing so would be beneficial, as this would effectively identify relevant content related to the user question which may prevent the user needing further assistance, thus improving user experience (Ben-Simhon et al. (US 2021/0124783), para. 0003). Furthermore, it would have been obvious to execute a textual entailment model to validate the question-answer pairs to ensure that the answer is derived from the digital content. Doing so would be beneficial, as this would ensure than the answers generated using the LLM have meaning that can be logically inferred from the digital content, preventing hallucinated output from the LLM (Khosla, para. 0010), improving user experience.
Garg in view of Khosla does not specifically disclose [a digital file…] wherein the digital file is not used for generating the validated answer.
Ben-Simhon teaches a digital asset, wherein the digital asset the digital file is not used for generating the validated answer (para. 0047 “For example, as shown in example 300 of FIG. 3, Q/A pair 210A includes the question “What features are included in the Premium edition?” Q/A pair 210A includes an answer to the question that lists some of the features of the premium edition of the application as well as a reference to Q/A pair 210E for “information on how to upgrade to Premium.” Reference link 220A links to Q/A pair 210E, which includes the question “How do I upgrade to Premium?” and an associated answer.”; Fig. 3, link to a file (related Q/A pair (220A), which reads on the BRI of a “digital file”), provided as digital asset to augment answer, with text (210E) which was not used for generating the answer).
Garg, Khosla, and Ben-Simhon are considered to be analogous to the claimed invention as they are all in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Garg in view of Khosla to incorporate the teachings of Ben-Simhon in order to have the digital file not be used for generating the validated answer. Doing so would be beneficial, as effectively identifying relevant content in response to a question may prevent the user needing further assistance, improving user experience (Ben-Simhon, para. 0002).
Regarding claim 3, Garg in view of Khosla and Ben-Simhon discloses determining an entailment score for the answer in the question-answer pair using the textual entailment model, wherein the set of digital content is a premise and the answer in the question-answer pair is a hypothesis (Khosla, para. 0053 “The memory 210 may also include a textual NLI module 236 for determining whether the answer generated from the LLM component 106 contradicts the retrieved passages from the aggregator component 104 by using a premise and hypothesis. The textual NLI interpretation module 236 may use NLI models to determine if there is a contraction. The textual NLI interpretation module 236 may utilize NLI models to take two text sequences as input (e.g., a hypothesis and a premise), and determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise. The textual NLI interpretation module 236 may use the NLI models to determine if the answer text (hypothesis) from the LLM component 106 is in contradiction with any of the passages retrieved (premises) from the aggregator component 104, if there is a contradiction the verifier component 108 may refrain from verifying the answer to show to the customer of the natural language question”); and including the question-answer pair in the set of validated question answer pairs if the entailment score is equal to or greater than a threshold entailment score (Garg: set of validated question-answer pairs; Khosla: check if entailment score meets threshold: para. 0053 “…determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise…The natural language question answering service 102 may use the LLM component 106 to generate another answer if a prior answer was in contradiction and use the verifier component 108 to verify the new answer.”).
Garg, Khosla, and Ben-Simhon are considered to be analogous to the claimed invention as they both are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Khosla in order to determine an entailment score, and determine to include the question-answer pair as validated if the entailment score is greater than a threshold entailment score. Doing so would be beneficial, as this would ensure than the answers generated using the LLM have meaning that can be logically inferred from the digital content, preventing hallucinated output from the LLM (Khosla, para. 0010), improving user experience.
Regarding claim 4, Garg in view of Khosla and Ben-Simhon discloses wherein performing the semantic search to identify the digital file from the set of digital files that is most semantically similar to the validated answer in the validated question-answer pair comprises: generating an embedding vector for each digital file of a set of digital files using an embedding model (Khosla, para. 0035 “The dense retriever module 219 may be created using dense embedding (encoder) models that can aim to capture the most salient semantic parts of each document retrieved (e.g., from search systems 124) and convert them into a fixed-dimensional dense representation that can be used to create a matrix of fixed-dimensional vectors.”); generating an embedding vector for the validated question-answer pair using the embedding model (Khosla, para. 0035 “To perform retrieval, the dense retriever module 219 may transform the natural language question into a similar dense embedding…”); estimating a similarity score to measure a similarity between the embedding vector for the validated question-answer pair and the embedding vector for each digital files (Khosla, para. 0035, K-nearest neighbors, distance between embeddings reflects similarity score); ranking the set of digital files based on respective similarity scores of the set of digital files to generate a ranked list of digital files (Khosla, para. 0035 “To perform retrieval, the dense retriever module 219 may transform the natural language question into a similar dense embedding and find the K nearest neighbors from an index matrix…”); and selecting the digital asset from the ranked list of digital files based on the digital file having a greater similarity score than other digital files (Khosla, para. 0035 “To perform retrieval, the dense retriever module 219 may transform the natural language question into a similar dense embedding and find the K nearest neighbors from an index matrix…”; Choice of k=1 for selecting most similar embedding; para. 0075 “As another example, the attribution component 109 may provide questions related to the natural language question by referencing a dense index (e.g., dense index used to create a fixed-dimensional representation of prior questions) and retrieving the k closest questions to the natural language question. The references, inline citations, and similar questions may be provided, but not limited to, within the answer or after the answer generated by the LLM component 106.”).
Garg, Khosla, and Ben-Simhon are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Khosla in order to select the digital file based on a similarity score measuring a similarity between an embedding vector for the question-answer pair and the embedding vector for each digital file. Doing so would be beneficial, as this would effectively identify relevant content related to the user question which may prevent the user needing further assistance, thus improving user experience (Ben-Simhon et al. (US 2021/0124783), para. 0003).
Regarding claim 5, Garg in view of Khosla and Ben-Simhon discloses transmitting the set of validated question-answer pairs to a reviewer client device (Garg, para. 0046 “The set of question-answer pairs and corresponding topics are sent to the moderator for quality review, so that customers may receive high-quality and moderated answers.”; Fig. 4, 510); and receiving feedback from the reviewer client device to generate an updated set of validated question-answer pairs (Garg, para. 0048 “At step 105, the question-answer pairs are accessed for review, for supervised and unsupervised training, and for moderation by one or more administrators to ensure high-quality. In customer service system 500 of FIG. 4, for example, the administrator may conduct this process through application program 509 and an integrated API of data access layer 508. The approved question-answer pairs are saved into conversational knowledge base 106 along with any other relevant meta information available (e.g., titles, article links, entities, and additional similar questions).”).
Garg, Khosla, and Ben-Simhon are considered to be analogous to the claimed invention as they both are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Khosla in order to transmit the validated question-answer pairs to a reviewer client device to receive feedback regarding the pairs to update the set of question-answer pairs. Doing so would be beneficial, as human-based review of the question-answer pairs would result in customers receiving high quality and moderated answers (para. 0046), improving user experience.
Regarding claim 15, claim 15 is a non-transitory computer-readable medium claim with limitations similar to method claim 1, and is thus rejected under similar rationale.
Additionally, Khosla teaches A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising (para. 0031, Fig. 2: 214).
Regarding claim 16, Garg in view of Khosla and Ben-Simhon discloses wherein the set of designated network locations comprises one or more uniform resource locators (URLs) (Garg, para. 0035 “Examples of live documents include documents that are continuously being developed by multiple authors or entities in a collaborative manner (e.g., Google Docs, such as Google Sheet files), application program interfaces (e.g., to data in databases) and access means to on-line documents (e.g., customized URLs, such as links to customer service management system resources).”).
Regarding claim 17, Garg in view of Khosla and Ben-Simhon discloses determining an entailment score for the answer in the question-answer pair using the textual entailment model, wherein the set of digital content is a premise and the answer in the question-answer pair is a hypothesis (Khosla, para. 0053 “The memory 210 may also include a textual NLI module 236 for determining whether the answer generated from the LLM component 106 contradicts the retrieved passages from the aggregator component 104 by using a premise and hypothesis. The textual NLI interpretation module 236 may use NLI models to determine if there is a contraction. The textual NLI interpretation module 236 may utilize NLI models to take two text sequences as input (e.g., a hypothesis and a premise), and determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise. The textual NLI interpretation module 236 may use the NLI models to determine if the answer text (hypothesis) from the LLM component 106 is in contradiction with any of the passages retrieved (premises) from the aggregator component 104, if there is a contradiction the verifier component 108 may refrain from verifying the answer to show to the customer of the natural language question”); determining whether the entailment score is equal to or greater than a threshold entailment score (Garg: set of validated question-answer pairs; Khosla: check if entailment score meets threshold: para. 0053 “…determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise….”); and including the question-answer pair in the set of validated question-answer pairs in response to determining that the entailment score is equal to or greater than a threshold entailment score (Garg: set of validated question-answer pairs; Khosla: check if entailment score meets threshold: para. 0053 “…determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise…The natural language question answering service 102 may use the LLM component 106 to generate another answer if a prior answer was in contradiction and use the verifier component 108 to verify the new answer.”).
Garg, Khosla, and Ben-Simhon are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Khosla in order to determine an entailment score, and determine to include the question-answer pair as validated if the entailment score is greater than a threshold entailment score. Doing so would be beneficial, as this would ensure than the answers generated using the LLM have meaning that can be logically inferred from the digital content, preventing hallucinated output from the LLM (Khosla, para. 0010), improving user experience.
Regarding claim 18, Garg in view of Khosla and Ben-Simhon discloses excluding the question-answer pair from the set of validated question-answer pairs in response to determining that the entailment score is less than a threshold entailment score (Khosla, para. 0053 “…determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise…The natural language question answering service 102 may use the LLM component 106 to generate another answer if a prior answer was in contradiction and use the verifier component 108 to verify the new answer.”).
Garg, Khosla, and Ben-Simhon are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Khosla in order to exclude the question-answer pair as validated if the entailment score is less than a threshold entailment score. Doing so would be beneficial, as this would ensure that validated Q/A pairs are not hallucinated output from the LLM (Khosla, para. 0010), improving user experience.
3. Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Garg in view of Khosla and Ben-Simhon, and further in view of Demeure (NPL Question Extractor, Github repository).
Regarding claim 2, Garg in view of Khosla and Ben-Simhon discloses generating question-answer pairs using the customized LLM (Garg, para. 0040) but does not specifically disclose prior to generating the set of question-answer pairs, receiving a prompt; and generating the set of question-answer pairs based on the prompt and the set of digital content using the customized LLM.
Demeure teaches prior to generating the set of question-answer pairs, receiving a prompt (pg. 1 “Inner-workings”; prompt “write a numbered list of questions…”); and generating the set of question-answer pairs based on the prompt and the set of digital content using the customized LLM (pg. 1 “Inner-workings”: “…the code loops on all files, for each file it extracts a list of questions using the following prompt followed by a chunk of text…it then loops on the questions, producing an answer by passing the following prompt followed by a chunk of text and a question…”).
Garg, Khosla, Ben-Simhon, and Demeure are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Demeure in order to generate the set of question-answer pairs based on the prompt and the set of digital content using the customized LLM. Doing so would be beneficial, as this would allow for automatic extraction of question-answer pairs from textual data, eliminating all manual work (pg. 1, 1st para).
4. Claims 6-14 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Garg in view of Khosla and Ben-Simhon, and further in view of Kislal et al. (WO 2025/042737, hereinafter Kislal).
Regarding claim 6, Garg in view of Khosla and Ben-Simhon discloses receiving a user question from a user computing device (Garg, para. 0050 “When a customer query is received at step 107, NLP processor 515, using NLP techniques, calls upon user intent classification module 516 to determine the user's intent (e.g., to ask about tracking a delivery).”; Fig. 4, 504-505); generating an embedding vector for the user question (Garg, para. 0050 “In one embodiment, knowledge base search algorithms transform the query into a machine representation (“embedding”), which is used as a template for retrieving the relevant question and answer pairs.”); and causing the answer in the validated question-answer pair to be displayed on the user computing device along with the digital asset (Garg, Fig. 4, 504-505; para. 0056 “One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format). The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.”).
Garg in view of Khosla and Ben-Simhon does not specifically disclose generating an embedding vector for each question in the set of validated question- answer pairs; estimating a similarity score to measure a similarity between the embedding vector for the user question and the embedding vector for each question in the set of validated question-answer pairs; determining that a question in a validated question-answer pair has a highest similarity score of all questions in the set of validated question-answer pairs; determining whether the highest similarity score is equal to or greater than a predetermined threshold similarity score; and in response to determining that the highest similarity score is equal to or greater than the predetermined threshold similarity score, [causing the answer in the validated question-answer pair to be displayed on the user computing device along with the digital asset].
Kislal teaches generating an embedding vector for each question in the set of validated question- answer pairs (para. 025 “Optionally, a question embedding 128, which may be considered to represent the meaning of the question, is also stored for each of the trusted questions, for example, computed using a technique such as described in Cer et al., “Universal Sentence Encoder”, arXiv 1803.11175, 2018.”); estimating a similarity score to measure a similarity between the embedding vector for the user question and the embedding vector for each question in the set of validated question-answer pairs (para. 026 “A question embedding is computed for the user’s question 104, and compared to question embeddings 128 for the question of each of the trusted Q/A pairs to determine a similarity (i.e., a “semantic match”) for each of the Q/A pairs (e.g., as a Euclidean distance between the vectors, and inner product or angle between the vectors, etc.).”); determining that a question in a validated question-answer pair has a highest similarity score of all questions in the set of validated question-answer pairs (para. 026 “The most similar of the trusted Q/A pairs is then determined.”); determining whether the highest similarity score is equal to or greater than a predetermined threshold similarity score (para. 027 “The similarity may be used to determine if the trusted Q/A pair should be used, and if so, optionally how it should be presented. For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold)…”); and in response to determining that the highest similarity score is equal to or greater than the predetermined threshold similarity score (para. 028 “If the similarity is high enough (e.g., the distance between vectors is low…”).
Garg, Khosla, Ben-Simhon, and Kislal are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kislal in order display an answer in a validated question-answer pair based on determining that a highest matching question-answer pair embedding is greater than a predetermined threshold. Doing so would be beneficial, as returning a validated question answer pair for a user question provides responses which have been previously vetted as being good, and thus has a high likelihood that the trusted answer matched to the trusted question would be a good answer to the new question (Kislal, para. 034), improving user experience.
Regarding claim 7, Garg in view of Khosla, Ben-Simhon, and Kislal discloses wherein the digital asset comprises a link to additional content relevant to the answer in the validated question answer pair (Garg, para. 0050 “As illustrated in FIG. 1, knowledge base search algorithms at step 108 retrieve from conversational knowledge basis 106 relevant question and answer pairs. In one embodiment, knowledge base search algorithms transform the query into a machine representation (“embedding”), which is used as a template for retrieving the relevant question and answer pairs.”; para. 0056 “One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format). The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.”; para. 0057 “Other options include activating logging of the question posed and the answer delivered, sending out web links along with the answer, and offering additional relevant answers and options of which the customer may be able to take advantage.”).
Regarding claim 8, Garg in view of Khosla, Ben-Simhon, and Kislal discloses in response to determining that the highest similarity score is less than the predetermined threshold similarity score, causing a message to be displayed on the user computing device indicating that a responsive answer to the user question is not found (Kislal, para. 027 “For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold), then no answer from the trusted Q/A pairs may be presented to the user and a response to the user may be something of the sort “I am sorry, I don’t have an answer to your question.””).
Garg, Khosla, Ben-Simhon, and Kislal are considered to be analogous to the claimed invention as they both are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kislal in order to display a message that the user question is not found if a threshold similarity score is not met. Doing so would be beneficial, as this would prevent unreliable answers from being given (Kislal, para. 005-006), improving user experience.
Regarding claim 9, Garg discloses receiving a user question via an online platform from a user computing device (para. 0050 “When a customer query is received at step 107, NLP processor 515, using NLP techniques, calls upon user intent classification module 516 to determine the user's intent (e.g., to ask about tracking a delivery); causing a predefined answer paired with a predefined question…to be displayed on the user computing device (Fig. 4, 404-405; para. 0056 “One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format). The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.”) wherein the predefined question and the predefined answer are generating using a customized large language model (LLM) … (para. 0040 “At steps 103a and 103b, the conversational knowledge base articles synthesize a diverse and exhaustive set of question-answer pairs. For example, based on the extracted information, question-answer pairs are generated in customer system 500 of FIG. 4 in question-answer generation module 513.”) wherein a pre-trained LLM is further trained using a set of digital content associated with the online platform to obtain the customized LLM (para. 0041 “For example, in one embodiment, the conversational knowledge base uses a transformer-based deep learning pre-trained-t5-small model, which is fine-tuned on the popular Squad dataset for end-to-end question generation. The transformer-based deep learning model may also be further fine-tuned on client-specific datasets for improved results. Fine-tuning may include extracting a list of consecutive pairs of sentences from the text and passing the extracted sentences into the model for use as context for the questions generated. Relevant context may also be extracted by tokenizing the available text.”); augmenting the predefined answer paired with the predefined question…to include a digital asset, wherein the digital asset comprises a link to the digital file … (para. 0056 “…The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.”; para. 0057 “Other options include activating logging of the question posed and the answer delivered, sending out web links along with the answer, and offering additional relevant answers and options of which the customer may be able to take advantage.”); and causing the digital asset to be displayed with the answer on the user computing device (para. 0056 “One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format) …”).
Garg does not specifically disclose:
A system, comprising: a memory component; a processing device coupled to the memory component, the processing decide to perform operations comprising:
[wherein the predefined question and the predefined answer are…] validated using a textual entailment model…
performing a semantic search to identify a digital file from a set of digital files that is most semantically similar to a predefined answer in a predefined question-answer pair…
Khosla teaches A system (Fig. 2A), comprising: a memory component (Fig. 2A, 214); a processing device coupled to the memory component, the processing decide to perform operations comprising (Fig. 2A, 206; para. 0031):
[wherein the predefined question and the predefined answer are…] validated using a textual entailment model…( para. 0053 “The memory 210 may also include a textual NLI module 236 for determining whether the answer generated from the LLM component 106 contradicts the retrieved passages from the aggregator component 104 by using a premise and hypothesis. The textual NLI interpretation module 236 may use NLI models to determine if there is a contraction. The textual NLI interpretation module 236 may utilize NLI models to take two text sequences as input (e.g., a hypothesis and a premise), and determine whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise.).
performing a semantic search to identify a digital file from a set of digital files that is most semantically similar to a predefined answer in a predefined question-answer pair… (semantic search (retrieving k-nearest embeddings using dense index) performed for a set of files (similar questions stored in QA pairs in search system) to find most semantic similar (e.g. closest to the natural language question embedding); for k=1, only most semantically similar to validated answer (input from verifier component 108) is selected; para. 0019 “For example, the natural language question answering service 102 can have access to different search systems (e.g., the system systems 124) where the search systems may stored passages and question and answer pairs (QA) pairs regarding a particular network-based service…”; para. 0067 “At (10), the attribution component 109 may provide references to the retrieved passages, inline citations to sentences of retrieved passages used in the answer, or provide similar questions to the natural language question. For example, the attribution component 109 may provide reference links and titles to the retrieved passages used by the LLM component 106 (e.g., retrieved passages used as context to generate the answer), which may allow the submitter of the question to get more details on the referenced passages. …As another example, the attribution component 109 may provide questions related to the natural language question by referencing a dense index (e.g., dense index used to create a fixed-dimensional representation of prior questions) and retrieving the k closest questions to the natural language question. The references, inline citations, and similar questions may be provided, but not limited to, within the answer or after the answer generated by the LLM component 106.”).
Garg and Khosla are considered to be analogous to the claimed invention as they are all in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Garg to incorporate the teachings of Khosla in order select a digital asset using a semantic similarity algorithm. Doing so would be beneficial, as this would effectively identify relevant content related to the user question which may prevent the user needing further assistance, thus improving user experience (Ben-Simhon et al. (US 2021/0124783), para. 0003). Furthermore, it would have been obvious to incorporate the teachings of Khosla in order to validate to predefined question and predefined answer using a textual entailment model. Doing so would be beneficial, as this would ensure than the answers generated using the LLM have meaning that can be logically inferred from the digital content, preventing hallucinated output from the LLM (Khosla, para. 0010), improving user experience
Garg in view of Khosla does not specifically disclose:
wherein the digital file is not used for generating the predefined answer.
Ben-Simhon teaches wherein the digital file is not used for generating the predefined answer. (para. 0047 “For example, as shown in example 300 of FIG. 3, Q/A pair 210A includes the question “What features are included in the Premium edition?” Q/A pair 210A includes an answer to the question that lists some of the features of the premium edition of the application as well as a reference to Q/A pair 210E for “information on how to upgrade to Premium.” Reference link 220A links to Q/A pair 210E, which includes the question “How do I upgrade to Premium?” and an associated answer.”; Fig. 3, link to related Q/A pair (220A), provided as digital asset to augment answer, with text (210E) which was not used for generating the answer).
Garg, Khosla, and Ben-Simhon are considered to be analogous to the claimed invention as they are all in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Garg in view of Khosla to incorporate the teachings of Ben-Simhon in order to have the digital asset specifically be a link to a digital file that is not used for generating the answer. Doing so would be beneficial, as effectively identifying relevant content in response to a question may prevent the user needing further assistance, improving user experience (Ben-Simhon, para. 0002).
Garg in view of Khosla and further in view of Ben-Simhon does not specifically teach:
estimating a semantic similarity between the user question and each predefined question in a set of predefined question-answer pairs to generate a set of semantic similarity scores;
determining whether a highest semantic similarity score in the set of semantic similarity scores is equal to or greater than a threshold value;
causing a predefined answer paired with a predefined question] to the highest semantic similarity score [to be displayed on the user computing device].
Kislal teaches estimating a semantic similarity between the user question and each predefined question in a set of predefined question-answer pairs to generate a set of semantic similarity scores (para. 026 “A question embedding is computed for the user’s question 104, and compared to question embeddings 128 for the question of each of the trusted Q/A pairs to determine a similarity (i.e., a “semantic match”) for each of the Q/A pairs (e.g., as a Euclidean distance between the vectors, and inner product or angle between the vectors, etc.).”);
determining whether a highest semantic similarity score in the set of semantic similarity scores is equal to or greater than a threshold value (para. 026 “The most similar of the trusted Q/A pairs is then determined.”; para. 027 “The similarity may be used to determine if the trusted Q/A pair should be used, and if so, optionally how it should be presented. For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold)…”); para. 028 “If the similarity is high enough (e.g., the distance between vectors is low…”);
causing a predefined answer paired with a predefined question] to the highest semantic similarity score] to be displayed on the user computing device (para. 026 “The most similar of the trusted Q/A pairs is then determined.”; para. 028 “If the similarity is high enough (e.g., the distance between vectors is low), the answer of the trusted Q/A pair may be provided as the answer to the user’s question without any indication that another question (i.e., the question of the trusted Q/A pair) was used to determine that answer. The response may be of the sort “The answer to your question is <trusted answer>”.”).
Garg, Khosla, Ben-Simhon, and Kislal are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Garg in view of Khosla and Ben-Simhon to incorporate the teachings of Kislal in order display an answer in a validated question-answer pair based on determining that a highest matching question-answer pair embedding is greater than a predetermined threshold. Doing so would be beneficial, as returning a validated question answer pair for a user question provides responses which have been previously vetted as being good, and thus has a high likelihood that the trusted answer matched to the trusted question would be a good answer to the new question (Kislal, para. 034), improving user experience.
Regarding claim 10, Garg in view of Khosla, Ben-Simhon, and Kislal discloses wherein the set of predefined question and answers are generated based on a set of digital content specific to the online platform (Kislal, para. 010 “There are a variety of ways that a system can form the trusted Q/A pairs. For example, the trusted Q/A pairs may result from prior user questions received in operation where the scores of the answers are found to be high or where there is explicit user feedback (“thumbs up”), implicit user feedback (e.g., clicking through an answer link, not searching further with a semantically similar question, etc.), or explicit third-party labeling (after the fact) based on manual review of the questions and answers.”).
Garg, Khosla, Ben-Simhon, and Kislal are considered to be analogous to the claimed invention as they are all in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Khosla in order to have the predefined question and answers generated based on digital content specific to the online platform. Doing so would be beneficial, as using Q/A pairs from prior user questions with positive user feedback (para. 010) would lead to higher chances of displaying answers which are satisfactory to the user, improving user experience.
Regarding claim 11, Garg in view of Khosla, Ben-Simhon, and Kislal discloses generating an embedding vector for the user question (Garg, para. 0050 “In one embodiment, knowledge base search algorithms transform the query into a machine representation (“embedding”), which is used as a template for retrieving the relevant question and answer pairs.”); generating an embedded vector for each predefined question in the set of predefined question-answer pairs (Kislal, para. 025 “Optionally, a question embedding 128, which may be considered to represent the meaning of the question, is also stored for each of the trusted questions, for example, computed using a technique such as described in Cer et al., “Universal Sentence Encoder”, arXiv 1803.11175, 2018.”); and estimating a semantic similarity score to measure a similarity between the embedding vector for the user question and the embedding vector for each predefined question in the set of predefined question-answer pairs (Kislal, para. 026 “A question embedding is computed for the user’s question 104, and compared to question embeddings 128 for the question of each of the trusted Q/A pairs to determine a similarity (i.e., a “semantic match”) for each of the Q/A pairs (e.g., as a Euclidean distance between the vectors, and inner product or angle between the vectors, etc.).”).
Garg, Kislal, Ben-Simhon and Khosla are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kislal in order to estimate semantic similarity score between the embedding vector for the user question and the embedding vector for each predefined question. Doing so would be beneficial, as returning a validated question answer pair for a user question provides responses which have been previously vetted as being good, and thus has a high likelihood that the trusted answer matched to the trusted question would be a good answer to the new question (Kislal, para. 034), improving user experience.
Regarding claim 12, Garg in view of Khosla, Ben-Simhon, and Kislal discloses determining the highest semantic similarity score is less than the threshold value (Kislal, para. 027 “For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold),”); and causing a message to be displayed on the user computing device indicating that a responsive answer to the user question is not found (Kislal, para. 027 “For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold), then no answer from the trusted Q/A pairs may be presented to the user and a response to the user may be something of the sort “I am sorry, I don’t have an answer to your question.””).
Garg, Kislal, Ben-Simhon, and Khosla are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kislal in order to display a message that the user question is not found if a threshold similarity score is not met. Doing so would be beneficial, as this would prevent unreliable answers from being given (Kislal, para. 005-006), improving user experience.
Regarding claim 13, Garg in view of Khosla, Ben-Simhon, and Kislal discloses wherein performing the semantic search to identify the digital file from the set of digital files that is the most semantically similar to the predefined answer in the predefined question-answer pair comprises: generating an embedding vector for each digital file of the set of digital files using an embedding mode (Khosla, para. 0035 “The dense retriever module 219 may be created using dense embedding (encoder) models that can aim to capture the most salient semantic parts of each document retrieved (e.g., from search systems 124) and convert them into a fixed-dimensional dense representation that can be used to create a matrix of fixed-dimensional vectors.”); generating an embedding vector for a predefined question-answer pair comprising the predefined question with the highest semantic similarity score and corresponding predefined answer using the embedding model (Khosla, para. 0035 “To perform retrieval, the dense retriever module 219 may transform the natural language question into a similar dense embedding…”; Kislal: highest semantic similarity score: para. 026 “The most similar of the trusted Q/A pairs is then determined.””); estimating a similarity score to measure a similarity between the embedding vector for the predefined question-answer pair and the embedding vector for each digital file using the semantic search algorithm (Khosla, para. 0035, K-nearest neighbors, distance between embeddings reflects similarity score); ranking the set of digital files based on respective similarity scores of the set of digital files to generate a ranked list of digital files (Khosla, para. 0035 “To perform retrieval, the dense retriever module 219 may transform the natural language question into a similar dense embedding and find the K nearest neighbors from an index matrix…”); and selecting the digital file from the ranked list of digital files based on the digital file having a greater similarity score than other digital file (Khosla, para. 0035 “To perform retrieval, the dense retriever module 219 may transform the natural language question into a similar dense embedding and find the K nearest neighbors from an index matrix…”; Choice of k=1 for selecting most similar embedding; para. 0075 “As another example, the attribution component 109 may provide questions related to the natural language question by referencing a dense index (e.g., dense index used to create a fixed-dimensional representation of prior questions) and retrieving the k closest questions to the natural language question. The references, inline citations, and similar questions may be provided, but not limited to, within the answer or after the answer generated by the LLM component 106.”).
Garg, Kislal, Ben-Simhon, and Khosla are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Khosla in order to select the digital asset based on a similarity score measuring a similarity between an embedding vector for the question-answer pair and the embedding vector for each digital asset. Doing so would be beneficial, as this would effectively identify relevant content related to the user question which may prevent the user needing further assistance, thus improving user experience (Ben-Simhon et al. (US 2021/0124783), para. 0003).
Regarding claim 14, Garg in view of Khosla, Ben-Simhon, and Kislal discloses wherein the digital asset comprises a uniform resource locator (URL) link to additional content (Garg, para. 0050 “As illustrated in FIG. 1, knowledge base search algorithms at step 108 retrieve from conversational knowledge basis 106 relevant question and answer pairs. In one embodiment, knowledge base search algorithms transform the query into a machine representation (“embedding”), which is used as a template for retrieving the relevant question and answer pairs.”; para. 0056 “One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format). The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.”; para. 0057 “Other options include activating logging of the question posed and the answer delivered, sending out web links along with the answer, and offering additional relevant answers and options of which the customer may be able to take advantage.”).
Regarding claim 19, Garg in view of Khosla and Ben-Simhon discloses receiving a user question from a computing device (Garg, para. 0050 “When a customer query is received at step 107, NLP processor 515, using NLP techniques, calls upon user intent classification module 516 to determine the user's intent (e.g., to ask about tracking a delivery), and causing the answer in the validated question-answer pair to be displayed on a user computing device along with the digital asset (Garg, Fig. 4, 404-405; para. 0056 “One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format). The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.”).
Garg in view of Khosla and Ben-Simhon does not specifically disclose:
estimating a semantic similarity between the user question and each question in the set of validated question-answer pairs to generate a set of semantic similarity scores;
determining that a question in a validated question-answer pair has a highest similarity score of all questions in the set of validated question-answer pairs;
determining whether the highest similarity score is equal to or greater than a predetermined threshold similarity score; and
in response to determining that the highest similarity score is equal to or greater than the predetermined threshold similarity score, [causing the answer in the validated question-answer pair to be displayed on a user computing device along with the digital asset].
Kislal teaches estimating a similarity between the user question and each question in the set of validated question-answer pairs to generate a set of semantic similarity scores (para. 026 “A question embedding is computed for the user’s question 104, and compared to question embeddings 128 for the question of each of the trusted Q/A pairs to determine a similarity (i.e., a “semantic match”) for each of the Q/A pairs (e.g., as a Euclidean distance between the vectors, and inner product or angle between the vectors, etc.).”); determining that a question in a validated question-answer pair has a highest similarity score of all questions in the set of validated question-answer pairs (para. 026 “The most similar of the trusted Q/A pairs is then determined.”); determining whether the highest similarity score is equal to or greater than a predetermined threshold similarity score (para. 027 “The similarity may be used to determine if the trusted Q/A pair should be used, and if so, optionally how it should be presented. For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold)…”); and in response to determining that the highest similarity score is equal to or greater than the predetermined threshold similarity score (para. 028 “If the similarity is high enough (e.g., the distance between vectors is low…”).
Garg, Khosla, Ben-Simhon, and Kislal are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kislal in order display an answer in a validated question-answer pair based on determining that a highest matching question-answer pair embedding is greater than a predetermined threshold. Doing so would be beneficial, as returning a validated question answer pair for a user question provides responses which have been previously vetted as being good, and thus has a high likelihood that the trusted answer matched to the trusted question would be a good answer to the new question (Kislal, para. 034), improving user experience.
Regarding claim 20, Garg in view of Khosla, Ben-Simhon, and Kislal discloses in response to determining that the highest similarity score is less than the predetermined threshold similarity score, causing a message to be displayed on the user computing device indicating that a responsive answer to the user question is not found (Kislal, para. 027 “For instance, if the best similarity is not sufficiently similar (e.g., the distance between the vectors is greater than a predetermined threshold), then no answer from the trusted Q/A pairs may be presented to the user and a response to the user may be something of the sort “I am sorry, I don’t have an answer to your question.””).
Garg, Kislal, Ben-Simhon, and Khosla are considered to be analogous to the claimed invention as they are in the same field of question answering. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Kislal in order to display a message that the user question is not found if a threshold similarity score is not met. Doing so would be beneficial, as this would prevent unreliable answers from being given (Kislal, para. 005-006), improving user experience.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Alexander (US 2022/0318501): determine similar scores for utterance and questions from a frequently asked questions bank, selecting a highest similarity score satisfying a similarity criterion (Fig. 3B)
Wang et al. (US 2021/0149964 A1): for each user question, compare to questions in question-answer pair repository, identify questions that are semantically similar, and retrieve corresponding answers that are semantically similar to the question
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CODY DOUGLAS HUTCHESON whose telephone number is (703)756-1601. The examiner can normally be reached M-F 8:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CODY DOUGLAS HUTCHESON/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659