DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C 101 because the claimed invention is mental process without significantly more.
Independent claims 1 and 11 recite a method and computer program product comprising non-transitory computer-readable program code that, as drafted under its broadest reasonable interpretation (BRI), covers A method comprising steps of: responsive to obtaining a plurality of tuples in a Retrieval-Augmented Generation (RAG) system with each tuple including a first value and a second value, generating a plurality of different first values from a corresponding first value where the plurality of different first values are similar to the corresponding first value (a human can obtain a plurality of appropriate tuples from ChatGPT and write them down including having the plurality of different first values); determining top-k, k is an integer greater than or equal to one, matches for the plurality of different first values to the second values in the RAG system (a human can determine the top-k matches by comparing the first values with the second values recorded from the RAG ChatGPT system); determining a confusion matrix based on the top-k matches; and utilizing the confusion matrix to debug the RAG system.
As described above, these limitations can be carried out as a series of mental steps. The judicial exception is not integrated into a practical application because the only additional elements recited are a RAG system which is a generic medium and nothing more that can be accessed on a computer (ChatGPT) and non-transitory computer-readable program code that is conventional components that utilizes the basic functions of a computer.
This claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as described above, the only additional elements recited are a system comprising of a RAG system, which is general purpose software being used as a tool to implement the mental process, and non-transitory computer-readable program code that is conventional components that utilizes the basic functions of a computer.
The remaining dependent claims fail to add patent eligible subject matter to independent claim 1:
Claims 2, 12 simply adds specifics requirements to what makes up the tuples which a human can do with a pen and paper or mentally referencing a paper copy containing part of the “RAG domain” recorded from ChatGPT.
Claims 3, 13 simply adds specifics requirements to what makes up the tuples which a human can do with a pen and paper or mentally referencing a paper copy containing part of the “RAG domain” recorded from ChatGPT.
Claims 4, 14 simply adds specifics requirements to use a LLM to generate different first values which a human can do using a generic medium such as ChatGPT.
Claims 5, 15 simply adds generating a certain number of values and additional instructions related to the corresponding first value which a human can do via asking ChatGPT (ex. Tell ChatGPT generate 3 variations of this question but make sure the core topic does not change).
Claims 6, 16 simply adds limits obtained from the corresponding first value that the plurality of generated values should follow which a human can do via asking ChatGPT (ex. Rewrite the description from the corresponding first value but don’t reuse the word replacing).
Claims 7, 17 simply adds calculating certain metrics from a confusion matrix which a human can do via a pen and paper (drawing confusion matrix, counting true/false positive/negatives, computing scores using mathematical formulas)
Claims 8, 18 simply adds adding a new entry to the data that intentionally points to a wrong value, which a human can do mentally or with a pen and paper by writing a table of existing tuples then adding a new row with a new first value and deliberately wrong second value)
Claims 9, 19 simply adds adding another first value to pair with an existing second value which a human can do mentally or via pen and paper.
Claims 10, 20 simply adds that the generating is performed by a planner in an AI agent system which a person can do by using ChatGPT as a “planner” giving it instructions.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. IN202441016399, filed on March 7, 2024.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Acknowledgement was also made of applicant claim for priority to provisional application 63/619,349 however the specification of the provisional application was found to not contain the full scope of claim 1 and claim 11 as no mention of confusion matrices for the use of debugging was mentioned. Therefore, for this application the foreign priority date of March 7, 2024 was used for claim 1, 11 and their dependent claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-2, 4-7, 10-12, 14-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Alakuijala et al. (hereinafter Alakuijala) (US 20230281193 A1) in view of Khosla et al. (hereinafter Khosla) (US 20250005058 A1) in further view of Chow et al. (hereinafter Chow) (US 20080154807 A1).
Regarding claim 1, Alakuijala teaches:
generating a plurality of different first values from a corresponding first value where the plurality of different first values are similar to the corresponding first value (Alakuijala, P[0008]: ("variants" are the plurality of different first values based on a corresponding first (original/submitted) query);;
determining top-k, k is an integer greater than or equal to one, matches for the plurality of different first values to the second values in the RAG system (Alakuijala, P[0008]: "For example, the output can include the “best” response (e.g., as indicated by response scores provided by the search system), multiple of the “best” responses, and/or a variant and corresponding response(s), (reads on top-k (k-amount of best responses));
Alakuijala does not teach:
A method comprising steps of:
responsive to obtaining a plurality of tuples in a Retrieval-Augmented Generation (RAG) system with each tuple including a first value and a second value,
determining a confusion matrix based on the top-k matches; and
utilizing the confusion matrix to debug the RAG system.
However, Khosla teaches:
responsive to obtaining a plurality of tuples in a Retrieval-Augmented Generation (RAG) system with each tuple including a first value and a second value ((Khosla, P[0013] reads on RAG system, QA question answer pairs reads on first value and second value)
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Alakuijala in view of Khosla. Doing so would have provided the AI-agent planner based generation of Khosla (Khosla, Abstract) with the RAG system framework for retrieval, vector similarity, and generation of Alakuijala (Alakuijala, Abstract) thus leading to RAG systems that retrieve and evaluate leading to improvements in system performance.
The combination of Alakuijala and Khosla do not teach:determining a confusion matrix based on the top-k matches; and
utilizing the confusion matrix to debug the RAG system.
However, Chow teaches:determining a confusion matrix based on the top-k matches (Chow, Abstract, P[0037]: confusion matrix based on top-k matches); and
utilizing the confusion matrix to debug the RAG system (Chow, P[0002]: "A confusion matrix is a visualization and diagnostic tool… is altered to provide optimal performance with future data, the category of which is unknown.", (altered to provide optimal performance with future data reads on debugging)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Alakuijala in view of Khosla to incorporate the teachings of Chow. Doing so would have provided the AI-agent planner based generation of Khosla (Khosla, Abstract) with the techniques for computing and using a confusion matrix of Chow (Chow, Abstract) with the RAG system framework for retrieval, vector similarity, and generation of Alakuijala (Alakuijala, Abstract) thus leading to RAG systems that include a robust evaluation and beugging of retrieval correctness from the insight and analysis of confusion matrices.
Regarding claim 2, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 1.
Khosla, in combination with Alakuijala and Chow, further teaches: wherein the first value is a question and the second value is an answer, based on a domain associated with the RAG system ((Khosla, P[0013] reads on RAG system, QA question answer pairs reads on first value and second value).
Regarding claim 4, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 1.
Alakuijala, in combination with Khosla and Chow, further teaches: wherein the generating is via a Large Language Model (LLM) (Alakuijala, P[0004], P[0005]: "In some implementations where the generative model is a neural network model with memory layers, the generative model is a sequence to sequence model." (neural network generative model with memory layers reads on LLM) which is presented with instructions and the first value (Alakuijala, P[0019]: "applying tokens of the original query as input to a trained generative model; and generating multiple variants of the original query based on application of tokens of the original query." (original query corresponds to claimed first value and it is explicitly presented as input to generative model), P[0006]: "In some of those implementations, the type of query variant to be generated for a given pass of the generative model can be indicated based on a type value input applied to the model in the given pass." ("type value input" constitutes instructions provided to the model controlling the form of the generated outputs)).
Regarding claim 5, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 4.
Alakuijala, in combination with Khosla and Chow, further teaches:wherein the instructions include a number of the plurality of different values to generate (Alakuijala, P[0016]: "such dynamic control can often lead to a relatively large (e.g., more than 5, more than 10, or more than 15) quantity of variants being generated" (explicit disclosure of controlling generation quantity reads directly on specifying a number of outputs to generate), P[0003]: "The disclosed implementations enable a plurality of query variants to be tested automatically.") and limitations on the plurality of different values relative to the corresponding first value (Akakuija, P[0006]: "Types of query variants can include, for example, an equivalent query, a follow-up query, a generalization query" (these variant categories impose semantic limitations relative to the original query)).
Regarding claim 6, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 4.
Alakuijala, in combination with Khosla and Chow, further teaches:
wherein the instructions include limitations on the plurality of different values relative to the corresponding first value (Akakuija, P[0006]: "the type of query variant to be generated for a given pass of the generative model can be indicated based on a type value input" (Type value constrains the semantic relationship between output and original query)), the limitations include a limit on contents from the first value that should be in any of the plurality of different values (Akakuija, P[0019]: "applying tokens of the original query as input to a trained generative model." (tokens of the original query are input and variant type (equivalent, generalization, etc.) constrains output class, P[0006]: (Types of queries listed necessarily impose limits on how much of the original content appears in variants (ex. Generalization type reduces specificity; equivalent maintains the semantic content, etc.)))
Regarding claim 7, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 17.
Chow, in combination with Alakuijala and Khosla, further teaches:
wherein the steps further include: determining one or more of accuracy, precision, recall, and an F-score using the confusion matrix (Chow, Fig 1. shows recall and precision with confusion matrix).
Regarding claim 10, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 1.
Alakuijala, in combination with Khosla and Chow, further teaches:
wherein the generating is performed by a planner (Alakuijala, P[0016]: "In some implementations, a trained control model is utilized to determine, at each of a plurality of time steps, whether a variant is to be generated", (trained control model determines generation decisions, functioning as the planner controlling variant generation), P[0067]-[0068], (critic is evaluating and directing the actor and acts as a planner)) in an Artificial Intelligence (AI) agent system (Alakuijala, P[0004]: "In some implementations where the generative model is a neural network model" (neural network generative model combined with actor-critic control above constitutes AI system performing autonomous decision-making steps).
Regarding claim 11, claim 11 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 1 and is rejected under the same grounds stated above.
Additionally, the combination further discloses or makes obvious: A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to implement steps of (Akuijala, P[0024]):
Regarding claim 12, claim 12 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 2 and is rejected under the same grounds stated above.
Regarding claim 14, claim 14 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 4 and is rejected under the same grounds stated above.
Regarding claim 15, claim 15 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 5 and is rejected under the same grounds stated above.
Regarding claim 16, claim 16 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 6 and is rejected under the same grounds stated above.
Regarding claim 17, claim 17 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 7 and is rejected under the same grounds stated above.
Regarding claim 20, claim 20 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 10 and is rejected under the same grounds stated above.
Claim(s) 3, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Alakuijala et al. (hereinafter Alakuijala) (US 20230281193 A1) in view of Khosla et al. (hereinafter Khosla) (US 20250005058 A1) in further view of Chow et al. (hereinafter Chow) (US 20080154807 A1) in furthest view of Qin (US 20240346256 A1)
Regarding claim 3, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 1.
The combination of Alakuijala, Khosla, and Chow does not teach:
wherein the first value is a description and the second value is an algorithm, based on a domain associated with the RAG system.
However, Qin teaches:
wherein the first value is a description (Qin, P[0045]: "receiving a query; generating a first feature vector based on the query;" (query corresponds to the first value which is a description of what user wants. Feature vector generation encodes description for retrieval)) and the second value is an algorithm (Qin, P[0093]: "providing, to the large language model, an augmented prompt generated based at least on the query and the retrieved pieces of augmentation information; and receiving a response generated by the large language model." (response generated by the LLM functions as the second value (algorithmic output) based on the description)), based on a domain associated with the RAG system (Qin, Abstract: "retrieval augmented artificial intelligence to generate" (reads on RAG)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Alakuijala in view of Khosla to incorporate the teachings of Chow and Qin. Doing so would have provided the explicit, detailed vector embedding retrieval and augmentation for a LLM generation RAG architecture of Qin (Qin, Abstract) with AI-agent planner-based generation of Khosla (Khosla, Abstract) with the techniques for computing and using a confusion matrix of Chow (Chow, Abstract) with the RAG system framework for retrieval, vector similarity, and generation of Alakuijala (Alakuijala, Abstract) thus leading to RAG systems that include a robust evaluation and begging of retrieval correctness from the insight and analysis of confusion matrices statistics from vector retrieval matches.
Regarding claim 13, claim 13 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 3 and is rejected under the same grounds stated above.
Claim(s) 8, 9, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Alakuijala et al. (hereinafter Alakuijala) (US 20230281193 A1) in view of Khosla et al. (hereinafter Khosla) (US 20250005058 A1) in further view of Chow et al. (hereinafter Chow) (US 20080154807 A1) in furthest view of Wang et al (hereinafter Wang) (MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation) (NOTE: Publishing date of 2/25/26 of this NPL is after the provisional filing date, however foreign priority date was used for this rejection (See Priority Section Above))
Regarding claim 8, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 1.
The combination of Alakuijala, Khosla, and Chow do not teach: wherein the utilizing the confusion matrix to debug the RAG system includes:
adding an entry in the plurality of tuples for a different first value that points to a wrong second value.
However, Wang teaches: wherein the utilizing the confusion matrix to debug the RAG system includes:
adding an entry in the plurality of tuples for a different first value that points to a wrong second value (Wang, Page 1, "Augmented generation techniques such as Retrieval-Augmented Generation (RAG) and Cache-Augmented Generation (CAG) have revolutionized the field by enhancing large language model (LLM) outputs with external knowledge and cached information." (MeTMaP explicitly teaching using RAG/CAG to enhance LLM outputs), Wang, Page 6: "…we create both positive and negative vectors…using the base vector as our query vector…conversely, we consider the matching erroneous" (shows multiple entries (positive/negative vectors) exist for the same query and the negative vector points to a wrong value), Wang, Page 5: "recognize and extract the quantifiers in the sentence. This can be accomplished by employing a regular expression… meaning by altering the quantified value." (Demonstrates generating a negative entry in the tuple (different first value pointing to wrong value) via quantifier modification), Wang, Page 6: "the Davinci model for completion as well. More specifically, we construct the prompt words along the Chain-of-Thought [72] style… would have been considered infeasible (Creating negative sentence that breaks inference corresponds to tuple entry pointing to incorrect value), Wang, page 6: "distance from the base sentence to the positive sentence is noted as “positive distance”, while its distance to the negative sentence is “negative distance”. A smaller distance indicates higher similarity, with a distance of 0 signifying identical sentences. Ideally, the positive distance should be a number as small as possible, whereas the negative distance should be larger." (confirms negative entry represents wrong value in tuple and explains how the system evaluates which tuple (positive or negative) is selected based on similarity)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Alakuijala in view of Khosla to incorporate the teachings of Chow and Wang. Doing so would have provided the methods of detecting incorrect retrieval Wang (Wang, Pages 5-8) with AI-agent planner-based generation of Khosla (Khosla, Abstract) with the techniques for computing and using a confusion matrix of Chow (Chow, Abstract) with the RAG system framework for retrieval, vector similarity, and generation of Alakuijala (Alakuijala, Abstract) thus leading to RAG systems that include a robust evaluation and begging of retrieval correctness from the insight and analysis of confusion matrices statistics from vector retrieval matches alongside the ability to identify and update the mappings of false vector matchings.
Regarding claim 9, the combination of Alakuijala, Khosla, and Chow discloses the method according claim 1.
The combination of Alakuijala, Khosla, and Chow do not teach:
wherein the utilizing the confusion matrix to debug the RAG system includes:
modifying an entry for the corresponding first value so that a different first value matches the second value of the corresponding first value
However, Wang teaches:
wherein the utilizing the confusion matrix to debug the RAG system includes: modifying an entry for the corresponding first value so that a different first value matches the second value of the corresponding first value (Wang, Page 6: "We create both positive and negative vectors as the potential candidates. Using the base vector as our query vector, we emulate a scenario of retrieving information from a vector database. The goal is to find the item in the database that has the highest similarity to the query vector. We calculate the distances between the base vector and each of the two candidates separately. and the candidate vector with the smaller distance is then selected as the output of this matching operation. If the result point to the positive sentence, we consider the match correct. Conversely, we consider the matching erroneous." (demonstrates how system selects correct vector corresponding to a first value and supports modifying an entry so the first value points to the intended second value), "Wang Page 6: "construct the prompt words along the Chain-of-Thought [72] style. The purpose… been considered infeasible." (shows method of systematically generative positive/negative candidates so the update aligns with correct semantic meaning) Wang, Page 6: "We measure sentence pair similarity using the distance between feature vectors. The distance from the base sentence to the positive sentence is noted as “positive distance”, while its distance to the negative sentence is “negative distance”. A smaller distance indicates higher similarity, with a distance of 0 signifying identical sentences. Ideally, the positive distance should be a number as small as possible, whereas the negative distance should be larger." (confirms update is verified quantitatively and the system is checking distances to ensure first value points to correct second value)).
Regarding claim 18, claim 18 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 8 and is rejected under the same grounds stated above.
Regarding claim 19, claim 19 recites the non-transitory computer-readable medium containing instructions corresponding to the method presented in claim 9 and is rejected under the same grounds stated above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/ Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655