Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office Action is in response to the application filed on 05/28/2024.
Claims 1-20 are pending.
Information Disclosure Statement
The information disclosure statement (IDS) filed on 05/28/2024 has been considered (see form-1449, MPEP 609).
Drawings
The drawings filed on 05/28/2024 are accepted.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Rennie et al. (US PGPUB 2024/0265041, hereinafter Rennie), in view of Khosla et al. (US PGPUB 2025/0005057, hereinafter Khosla).
As per as claim 1, Rennie discloses:
A method, comprising:
receiving, by a device, a plurality of documents and a plurality of questions associated with the plurality of documents (Rennie, e.g., [abstract], [005], [007], [0010], “…machine-learning question-answering (“Q-A”) platform with improved document processing and information retrieval operations achieved by improved intake processing (ingestion) of user data and by improved training processes of machine learning information retrieval models used by the Q-A platform…”);
determining, by the device, a plurality of ground truth answers corresponding to the plurality of questions (Rennie, e.g., [0126], [0129], [0139], [0161] and [0275], “… question-answer ground truths that may be publicly available, or may have been internally/privately developed by the customer using a document processing system…”);
normalizing, by the device, the plurality of questions to generate a normalized plurality of questions (Rennie, e.g., [0149], [0172], [0237], [0256], “…transforming and/or normalizing the submitting query to modify the question submitted using, for example, a trained learning engine. In some embodiments, answer data determined for the submitted query (e.g., based on content retrieved from the DOM repository 140 via the query processing module 136) may be processed (by a separate module) to formulate further questions from the answer. Such derived questions can then be re-submitted to the query processing module to retrieve follow-up answers…”);
selecting, by the device, a set of most frequent questions from the normalized plurality of questions (Rennie, e.g., [0157], “…performing question tersification/degradation processing on well-formed questions may be based on a term frequency-inverse document frequency (TF-IDF) approach. The TF-IDF approach provides statistical numerical values that represent the importance of terms, words, or expressions in a document in view of a collection of documents. A TF-IDF procedure may be configured to derive TF-IDF values for words and phrases of a document (e.g., based on the frequency of words/terms within the collection of documents)…” and [0172], “…the question forwarded to the question tersifier 330 may be simplified using a term frequency-inverse document frequency (TF-IDF) approach. Under this approach, the question tersifier 320 computes statistical numerical values for words that comprise the question being processed…”);
utilizing, by the device, regular expressions and natural language processing to generate, from the plurality of ground truth answers, a set of answers to the set of most frequent questions (Rennie, e.g., [0126], [0129], [0139], [0157], [0167], [0172], [0208], and [0275], “… question-answer ground truths that may be publicly available, or may have been internally/privately developed by the customer using a document processing system…”);
dynamically selecting, by the device, prompts for large language models (LLMs) based on the set of most frequent questions and based on context provided to the LLMs for generating the set of answers (Rennie, e.g., [0060], [0063-0067], “… Dynamically adjusting the operational characteristics of the Q-A system may include adjusting number of determined results of the coarse Q-A search, for which the subsequent fine-detail search is to be performed…” and “… [0148], [0157], [0172], “…answers/contents corresponding to frequently asked questions…answers can track the frequency at which specific questions and answers have been submitted and/or retrieved…”); and
optimizing, by the device and based on the set of most frequent questions, the set of answers, the prompts, and parameters of configurations for the LLMs, accuracies of the LLMs to generate optimized LLMs (Rennie, e.g., [0120], [0276], “…optimization performed according to MMI criteria…Q-A systems, the language model may be based on a transformer-based model (e.g., BERT, GPT3, T5, BART, etc.), trained on a large volume of data (e.g., the Stanford Question Answering Database, or Squad, and/or other question-answer repositories and privately collected and annotated data…” and see [0148], [0157], “…answers/contents corresponding to frequently asked questions…” and [0280], “…adjust (optimize) the machine learning parameters of the model in such a way that it optimally pushes the correct answer…”)).
To make records clearer regarding to “the set of most frequent questions, the set of answers are generated” (although as stated above, Rennie functional disclose the features of set of most frequent questions and the set of answers are generated).
However Khosla, in an analogous art, discloses “the set of most frequent questions, the set of answers are generated” (Khosla, e.g., [0010], “…(AI) models (e.g., large language models (LLM), question answering assistants, chatbots, etc.) while, at times may answer a natural language question accurately (e.g., gives an answer to a question…”, [0021-0022], “…search system 124 may be a data store which contains frequently asked questions (FAQ) (and associated answers) …”). Thus, it would have been obvious to one of ordinary skill in the art BEFORE the effective filling date of the claimed invention to combine the teaching of Khosla and Rennie to aggregate of the natural language question answering service can retrieve passages from search systems based on the question and generate a prompt, wherein a large language model (LLM) of the natural language question answering service may receive the prompt and provide an answer and the answer may be verified by a verifier of the natural language question answering service and retrieved passages to produce references, inline citations, and similar questions (Khosla, e.g., [abstract]).
As per as claim 2, the combination of Khosla and Rennie disclose:
The method of claim 1, further comprising:
implementing at least one of the optimized LLMs in an LLM based application (Khosla, e.g., figs. 2A and 2B, associating with texts description, “…retrieve passages and question and answer (QA) pairs from search systems based on a natural language question in accordance with aspects of the present application…”).
As per as claim 3, the combination of Khosla and Rennie disclose:
The method of claim 1, wherein normalizing the plurality of questions to generate
the normalized plurality of questions comprises:
performing a semantic analysis on the plurality of questions to identify single representations for the plurality of questions that have a same meaning,
wherein the single representations correspond to the normalized plurality of questions (Khosla, e.g., [0012-0015], “…the LLM may be a trained machine learning model utilizing Retrieval Augmented Generation (RAG) techniques to generate answers using semantics (e.g., in addition to or alternatively to lexical techniques) to answer the question…” and [0029], “…determine a semantic meaning of the natural language question. The aggregator component 104 may take that meaning of the question and utilize it to determine which search systems 124 to retrieve passages from…”).
As per as claim 4, the combination of Khosla and Rennie disclose:
The method of claim 1, wherein selecting the set of most frequent questions from the normalized plurality of questions comprises:
selecting, as the set of most frequent questions, a normalized plurality of questions that make up a particular percentage of all questions asked (Rennie, e.g., [0060], [0063-0067], “… Dynamically adjusting the operational characteristics of the Q-A system may include adjusting number of determined results of the coarse Q-A search, for which the subsequent fine-detail search is to be performed…” and “… [0148], [0157], [0172], “…answers/contents corresponding to frequently asked questions…answers can track the frequency at which specific questions and answers have been submitted and/or retrieved…”) and (Khosla, e.g., [0010], “…(AI) models (e.g., large language models (LLM), question answering assistants, chatbots, etc.) while, at times may answer a natural language question accurately (e.g., gives an answer to a question…”, [0021-0022], “…search system 124 may be a data store which contains frequently asked questions (FAQ) (and associated answers) …”).
As per as claim 5, the combination of Khosla and Rennie disclose:
The method of claim 1, wherein utilizing the regular expressions and the natural language processing to generate, from the plurality of ground truth answers, the set of answers to the set of most frequent questions comprises:
utilizing the regular expressions and the natural language processing to convert the plurality of ground truth answers to minimum acceptable formats (Rennie, e.g., [0126], [0129], [0139], [0157], [0167], [0172], [0208], and [0275], “… question-answer ground truths that may be publicly available, or may have been internally/privately developed by the customer using a document processing system…”) and (Khosla, e.g., [0010], “… (AI) models (e.g., large language model (LLM), question answering assistants, chatbots, etc.) while, at times may answer a natural language question accurately (e.g., gives an answer to a question…”, [0021-0022], “…search system 124 may be a data store which contains frequently asked questions (FAQ) (and associated answers) …” and [0061], “…The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.); and
generating the set of answers to the set of most frequent questions based on the minimum acceptable formats (Rennie, e.g., [0060], [0063-0067], “… Dynamically adjusting the operational characteristics of the Q-A system may include adjusting number of determined results of the coarse Q-A search, for which the subsequent fine-detail search is to be performed…” and “… [0148], [0157], [0172], “…answers/contents corresponding to frequently asked questions…answers can track the frequency at which specific questions and answers have been submitted and/or retrieved…”) and (Khosla, e.g., [0010], “… (AI) models (e.g., large language model (LLM), question answering assistants, chatbots, etc.) while, at times may answer a natural language question accurately (e.g., gives an answer to a question…”, [0021-0022], “…search system 124 may be a data store which contains frequently asked questions (FAQ) (and associated answers) …” and [0061], “…The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.).
As per as claim 6, the combination of Khosla and Rennie disclose:
The method of claim 1, wherein dynamically selecting the prompts for the LLMs based on the set of most frequent questions and based on the context provided to the LLMs for generating the set of answers comprises:
dynamically selecting the prompts for LLMs that generate the set of answers to the set of most frequent questions in a specific format (Rennie, e.g., [0060], [0063-0067], “… Dynamically adjusting the operational characteristics of the Q-A system may include adjusting number of determined results of the coarse Q-A search, for which the subsequent fine-detail search is to be performed…” and “… [0148], [0157], [0172], “…answers/contents corresponding to frequently asked questions…answers can track the frequency at which specific questions and answers have been submitted and/or retrieved…”) and (Khosla, e.g., [0010], “… (AI) models (e.g., large language model (LLM), question answering assistants, chatbots, etc.) while, at times may answer a natural language question accurately (e.g., gives an answer to a question…”, [0021-0022], “…search system 124 may be a data store which contains frequently asked questions (FAQ) (and associated answers) …” and [0061], “…The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.).
As per as claim 7, the combination of Khosla and Rennie disclose:
The method of claim 1, wherein the prompts instruct the LLMs on expected formats for the set of answers to the set of most frequent questions (Rennie, e.g., [0060], [0063-0067], “… Dynamically adjusting the operational characteristics of the Q-A system may include adjusting number of determined results of the coarse Q-A search, for which the subsequent fine-detail search is to be performed…” and “… [0148], [0157], [0172], “…answers/contents corresponding to frequently asked questions…answers can track the frequency at which specific questions and answers have been submitted and/or retrieved…”) and (Khosla, e.g., [0010], “… (AI) models (e.g., large language model (LLM), question answering assistants, chatbots, etc.) while, at times may answer a natural language question accurately (e.g., gives an answer to a question…”, [0021-0022], “…search system 124 may be a data store which contains frequently asked questions (FAQ) (and associated answers) …” and [0061], “…The LLM component 106 may utilize APIs to receive the prompt and the user context from the user context component 105 where the prompt may be formatted in a certain manner to be sent or communicated over an API.).
Claims 8-14 are essentially the same as claims 1-7 except that they set forth the claimed invention as a device rather a method, respectively and correspondingly, therefore is rejected under the same reasons set forth in rejections of claims 1-7.
Claims 15-20 are essentially the same as claims 1-7 except that they set forth the claimed invention as a non-transitory computer readable medium rather a method, respectively and correspondingly, therefore is rejected under the same reasons set forth in rejections of claims 1-7.
Additional Art Considered
The prior art made of record and not relied upon is considered pertinent to the Applicants’ disclosure.
The following patents and papers are cited to further show the state of the art at the time of Applicants’ invention with respect to receive a plurality of documents and a plurality of questions for the plurality of documents, and may determine a plurality of ground truth answers corresponding to the plurality of questions. The device may normalize the plurality of questions to generate a normalized plurality of questions, and may select a set of most frequent questions from the normalized plurality of questions. The device may utilize regular expressions and natural language processing to generate, from the plurality of ground truth answers, a set of answers to the set of most frequent questions, and may dynamically select prompts for LLMs based on the set of most frequent questions and based on context provided to the LLMs.
Sassak, JR. et al. (US PGPUB 2025/0272506, hereafter Sassak); “Methods and Systems For Retrieval-Augmented Generation using synthetic question embeddings” discloses “retrieval-augmented generation are described. Responsive to a user input, an input embedding associated with the user input is obtained. A synthetic question embedding is retrieved from an embeddings database, based on a similarity to the input embedding. The synthetic question embedding is used to obtain a relevant source text based on a stored mapping between the synthetic question embedding and the source text. A prompt is provided to a large language model (LLM) to generate and display a textual response to the user input, based on the user input and the source text”.
Sassak also teaches retrieving a synthetic question embedding from an embeddings database, based on a similarity to the input embedding; obtaining a source text based on a stored mapping between the synthetic question embedding and the source text; using a large language model (LLM), generating a textual response to the user input [0010-0012].
Sassak further teaches “generates prompts to a large language model (LLM), including the user input and an identified relevant source text, and receives output from the LLM to more efficiently generate output that may assist a user” ([00472-0043]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TUAN A PHAM whose telephone number is (571)270-3173. The examiner can normally be reached M-F 7:45 AM - 6:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tony Mahmoudi can be reached on 571-272-4078. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TUAN A PHAM/Primary Examiner, Art Unit 2163
Prosecution Insights