Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1:
Claim 1 is directed to a process, which is a statutory category of invention. (Step 1: YES)
Step 2A, Prong One:
Claim 1 recites:
A method comprising:
for each question/answer (QA) pair in a curated dataset of QA pairs, wherein each of the QA pairs includes a question, a squashing instruction, and a referenced pattern of information (RPI), the RPIs including correct RPIs (cRPIs) (the recited dataset is not limited in any manner, and the specification does not include any specific definitions that would limit the term “dataset” to a physical embodiment; Rather, the term “dataset” encompasses collections of data in any form; The curated dataset of QA pairs is simply a collection of data, which is iterated over in the claimed process):
assessing the squashing instruction to determine a diversification strategy for diversifying the RPI (the step of determining a diversification strategy is expressly described in the specification as being performed by a human subject matter expert, see paragraphs [0042] and [0098]; this step therefore clearly covers mental processes capable of being performed in the human mind) ; and
performing the diversification strategy on the RPI to generate a variational RPI (vRPI) from the RPI (the step of performing the diversification strategy is not limited to any particular implementation and diversification strategies include strategies such as simply appending additional predefined RPIs, see paragraph [00115]; this step could therefore be performed mentally by a human or with a pen and paper and encompasses a mental process);
storing the vRPIs generated from the RPIs in the curated dataset of QA pairs in a curated dataset of variational QA pairs (the recited dataset is not limited in any manner, and the specification does not include any specific definitions that would limit the term “dataset” to a physical embodiment; Rather, the term “dataset” encompasses collections of data in any form; Therefore, storing the vRPIs could encompass a human simply writing down the associated vRPIs using a pen and paper, which encompasses a mental process); and
performing automated verification in a generative system using the variational QA pairs (this is an additional element discussed further below).
Thus, a broadest reasonable interpretation of the claim as set forth above encompasses a series of mental steps that can practically be performed in the human mind. Claim 1 therefore recites an abstract idea (Step 2A, Prong One: YES).
Step 2A, Prong Two:
Claim 1 recites an additional element of performing automated verification in a generative system using the variational QA pairs. MPEP 2106.05(f) provides the following considerations for determining whether a claim simply recites a judicial exception with the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. In this case, the claim merely instructs to perform automated verification in a generative system. The claim includes no details as to how the automated verification is performed. Rather, this limitation amounts to a generic instruction to “use” the variational QA pairs to perform automated verification. Furthermore, the recited “generative system” is invoked merely as a tool to perform the existing process of verification, such as by collecting human feedback, as discussed in Applicant’s specification (paragraphs [0011-0014]).
Even when viewed in combination, this additional element amounts to mere instructions to “use” the variational QA pairs generated using mental processes in an automated verification process. Claim 1 is therefore directed to the judicial exception. (Step 2A, YES).
Step 2B: As discussed above, even when considered in combination, the additional elements amount to no more than mere instructions to generically apply the judicial exception for automated verification. The additional elements therefore do not provide an inventive concept. Step 2B: No).
Claim 2 recites an additional step of determining a list of unique squashing instruction types and associating at least one diversification strategy with each of the unique squashing instruction types. This also encompasses steps that could be performed mentally with a pen and paper. Claim 2 therefore recites an abstract idea without any additional elements. The analysis applied to claim 1 regarding additional elements applies to claim 2 for the same reasons.
Claim 3 is recites an additional step of determining the most similar squashing instruction type in the list based on a distance measurement. The specification does not indicate what particular distance measurement is used to determine the most similar squashing instruction type (see paragraph [00104]). However, the specification discloses a “distance measurement” may comprise mathematical calculations such as a Euclidean distance or a cosine distance. Therefore, determining the most similar squashing instruction type in the list based on a distance measurement comprising a Euclidean distance or a cosine distance is a mathematical function and claim 3 recites an abstract idea without any additional elements. The analysis applied to claim 1 regarding additional elements applies to claim 3 for the same reasons.
Claims 4 and 5 further define the determined diversification strategies. The diversification strategies include, for example, a hard coded diversification strategy that merely appends a pre-defined list of variational answers to the RPI, which could be performed mentally by a human using a pen and paper. Claims 4 and 5 therefore recite an abstract idea without any additional elements. The analysis applied to claim 1 regarding additional elements applies to claims 4 and 5 for the same reasons.
Claim 6 recites cleaning the variational RPI based on a distance measurement. The specification discloses the distance measurement may comprise a Levenstein Distance, which is a mathematical calculation. Claim 6 therefore recites an abstract idea without any additional elements. The analysis applied to claim 1 regarding additional elements applies to claim 6 for the same reasons.
Claim 7 recites performing a pair-wise comparison to identify similar or redundant RPIs. This could be performed mentally by a human mentally comparing RPIs in pairs. Claim 7 therefore recites an abstract idea without any additional elements. The analysis applied to claim 1 regarding additional elements applies to claim 7 for the same reasons.
Claims 8-10 recite performing a performance check on the clean RPI to generate a final variational RPI by removing ambiguous RPIs identified by connectors that indicate a condition. The specification discloses this is performed by comparing a list of conditional connector words (“but”, “maybe”, etc.). These checks could be performed by a human mentally performing the comparisons and mentally determining whether an RPI was ambiguous. Claims 8-10 therefore recite an abstract idea without any additional elements. The analysis applied to claim 1 regarding additional elements applies to claims 8-10 for the same reasons.
Claim 11 recites steps comprising inputting a variational RPI pair into the generative system, evaluating an answer of the generative system, and generating a cumulative score based on scores generating by comparing the answer generated by the generative system to the vRPI. The claim does not limit how the evaluation step is performed, and does not limit how the scores are generated based on the evaluation. Each of the claimed steps, therefore, would encompass a human mentally evaluating the output of the generative system, mentally determining a score, and mentally determining a cumulative score. Claim 11 therefore recites an abstract idea.
Claim 11 also recites the additional element of the generative system. However, the generative system is utilized as a “black box” with no details as to how the generative system functions. The generative system is used merely as a tool in the otherwise mental process, whereby the human simply provides input to the generative system and mentally evaluates the output of the generative system. The claim does not improve the functioning of the generative system, and therefore does not provide a practical application. The generative system provides insignificant extra-solution activity as the input and observed response are mere data gathering steps. Claim 11 therefore does not include significantly more than the judicial exception.
Claim 12 recites further details of generating the scores that involve “setting” bits and assigning a score based on the set bits. Although “bits” are used to represent information in a computer, the plain and ordinary meaning of the term “bit” is not limited to physical systems. Rather, the term “bit” encompasses any binary representation of logical state. The steps of claim 12, therefore, could be performed mentally by a human using a pen and paper to tally bits and assign scores based on the bits. Claim 12 therefore recites an abstract idea without any additional elements.
Claim 13 recites generating the curated dataset of QA pairs by distilling knowledge of the generative system or from a source into the QA pairs and further defines the elements of the QA pairs. The claim places no limitations on how the “distilling knowledge” is performed. A broadest reasonable interpretation of “distilling knowledge” would include a human mentally generating the QA pairs using a pen and paper by observing the generative system or other source and mentally determining appropriate QA pairs. Claim 13 therefore recites an abstract idea. Although claim 13 includes the additional element of the generative system, similar to claim 11, it merely serves as a source of data for the otherwise mental process of distilling information. The claim does not improve the functionality of the generative system or provide significantly more than the judicial exception.
Claim 14 recites a feedback loop comprising a number of “flows” that are “based on user feedback”. The specification discloses that the recited feedback loop comprises a human expert performing the discard flow, refinement flow, and/or accept flow for each of the QA pairs. Claim 14 therefore clearly recites mental processes performed by a human. Claim 14 further recites the variational QA pairs “are configured to align the generative system to final user preferences”. However, the claim does not recite any steps of performing the alignment or any details as to how the QA pairs would be used to align the generative system. The claim recites only an idea of a solution or outcome (i.e. an aligned generative system) without any indication as to how this solution or outcome is accomplished. Claim 14 therefore does not practically apply the judicial exception or provide significantly more than the judicial exception.
Claims 15-20 are directed to a non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising the methods of claims 1-11. MPEP 2106.04(a)(2) states that both product and process claims may recite mental processes. Claims 15-20 are therefore rejected for the same reasons as claims 1-11.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, and 13-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beller et al. (U.S. Patent No. 9,767,094, hereinafter “Beller”), in view of Wen et al. (Benchmarking Complex Instruction-Following with Multiple Constraints Composition, hereinafter “Wen”).
In regard to claim 1, Beller discloses a method (Fig. 1, 700) comprising:
for each question/answer (QA) pair in a curated dataset of QA pairs, wherein each of the QA pairs includes a question, a squashing variable, and a referenced pattern of information (RPI), the RPIs including correct RPIs (cRPIs) (step 701, a set of question and answer pairs are received, column 19, lines 5-9; where each QA pair includes a question, a “cRPI”, i.e. a correct answer, and a squashing variable, i.e. a semantic type to constrain generated answers to the semantic type, column 4, lines 25-33):
assessing the squashing variable to determine a diversification strategy for diversifying the RPI (a semantic variant generation component determines rule-based expansions to perform based on the semantic type, column 19, lines 9-16; see also column 14, lines 28-34); and
performing the diversification strategy on the RPI to generate a variational RPI (vRPI) from the RPI (the semantic variant generation component performs rule-based expansion to populate type-specific expansion templates, column 19, lines 17-28);
storing the vRPIs generated from the RPIs in the curated dataset of QA pairs in a curated dataset of variational QA pairs (the semantic variants are saved, column 19, lines 33-45); and
performing automated verification in a generative system using the variational QA pairs (the QA pairs are used to evaluate the performance of the question answering system, column 3, lines 14-41).
While Beller discloses the squashing variable constrains the RPIs to a particular semantic type, Beller does not disclose the squashing variable is provided in the form of an “instruction” (i.e., a natural language statement).
Wen discloses an automated verification method wherein answer constraints are provided in the form of a squashing instruction (constraints are provided in the form of natural language statements, section 3.2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to associate the constraints disclosed by Weller with the squashing instructions disclosed by Wen, because in real world use of language models, almost all tasks are formulated as instruction following where natural language instructions impose constraints on the model output, as taught by Wen (section 1, first paragraph ).
In regard to claim 2, Beller discloses associating at least one diversification strategy with each of the unique squashing variables (each semantic type is associated with specific rule-based expansions, column 14, lines 16-34).
Beller does not disclose generating a strategy guideline by generating a list of unique squashing instruction types from the squashing instructions included in the curated dataset of QA pairs.
Wen discloses generating a list of unique squashing instruction types from the squashing instructions included in the curated dataset of QA pairs (every constraint instruction is categorized as one of a plurality of constraint types, section 3.2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to associate unique squashing instruction types from the squashing instructions included in the curated dataset of QA pairs to at least one diversification strategy with each of the unique squashing types, because in real world use of language models, almost all tasks are formulated as instruction following where natural language instructions impose constraints on the model output, as taught by Wen (section 1, first paragraph ).
In regard to claim 13, Beller discloses generating the curated dataset of QA pairs by distilling knowledge of the generative system or from a source into the QA pairs, wherein each of the QA pairs includes a question, a squashing instruction, and an RPI, wherein the squashing instruction is configured to reduce a span of correct answers within an output space (questions are generated by evaluating a corpus of documents, column 6, lines 40-54; where each QA pair includes a question, a “cRPI”, i.e. a correct answer, and a squashing variable, i.e. a semantic type to constrain generated answers to the semantic type, column 4, lines 25-33).
In regard to claim 14, Beller discloses performing a feedback loop on the QA pairs in the curated dataset of QA pairs (see Fig. 5, user interface 500, column 16, lines 48-53), wherein the QA pairs are curated during the feedback loop, wherein the feedback loop includes a discard flow, a refinement flow, and an accept flow that are performed based on user feedback (the user may edit fields 502 and 503, delete entries, or save entries, column 16, line 53 to column 17, line 8), wherein the variational QA pairs in the variational dataset of QA pairs are configured to align the generative system to final user preferences (the questions and answers curated according to user input, column 7, lines 40-61), wherein the generative system comprises a retrieval augmented generation (RAG) system (the QA system answers questions based on a corpus of data, column 7, lines 25-39).
In regard to claim 15, Beller discloses a non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations (column 20, lines 25-33) comprising:
performing for each question/answer (QA) pair in a curated dataset of QA pairs, wherein each of the QA pairs includes a question, a squashing variable, and a referenced pattern of information (RPI), the RPIs including correct RPIs (cRPIs) (step 701, a set of question and answer pairs are received, column 19, lines 5-9; where each QA pair includes a question, a “cRPI”, i.e. a correct answer, and a squashing variable, i.e. a semantic type to constrain generated answers to the semantic type, column 4, lines 25-33):
assessing the squashing variable to determine a diversification strategy for diversifying the RPI (a semantic variant generation component determines rule-based expansions to perform based on the semantic type, column 19, lines 9-16; see also column 14, lines 28-34); and
performing the diversification strategy on the RPI to generate a variational RPI (vRPI) from the RPI (the semantic variant generation component performs rule-based expansion to populate type-specific expansion templates, column 19, lines 17-28);
storing the vRPIs generated from the RPIs in the curated dataset of QA pairs in a curated dataset of variational QA pairs (the semantic variants are saved, column 19, lines 33-45); and
performing automated verification in a generative system using the variational QA pairs (the QA pairs are used to evaluate the performance of the question answering system, column 3, lines 14-41).
While Beller discloses the squashing variable constrains the RPIs to a particular semantic type, Beller does not disclose the squashing variable is provided in the form of an “instruction” (i.e., a natural language statement).
Wen discloses an automated verification method wherein answer constraints are provided in the form of a squashing instruction (constraints are provided in the form of natural language statements, section 3.2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to associate the constraints disclosed by Weller with the squashing instructions disclosed by Wen, because in real world use of language models, almost all tasks are formulated as instruction following where natural language instructions impose constraints on the model output, as taught by Wen (section 1, first paragraph ).
Claim(s) 11 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beller, in view of Wen, and further in view of Allen et al. (U.S. Patent Application Pub. No. 2015/0339574, hereinafter “Allen”).
In regard to claims 11 and 20, Beller is silent as to the details of verifying the generative system.
Allen discloses a method for automated verification of a QA system, wherein the automated verification comprises:
for each variational QA pair (QA pairs utilizing an expanded answer key, paragraphs [0070-0071]):
inputting a variational QA pair into the generative system, the QA pair including a question and a vRPI (an input question along with the extended answer key is received, paragraph [0094]);
evaluating an answer of the generative system in response to the question in the variational QA pair (candidate answers are provided to a validation engine to evaluate the candidate answers, paragraph [0095]);
generating a score for the QA pair based on a comparison of the answer generated by the generative system to the vRPI (validators determine whether the criteria of the selected validators are satisfied or not, paragraph [0087]); and
generating a cumulative score that includes scores for all of the variational QA pairs, wherein the cumulative score represents an alignment of the generative system to final user preferences (the validation status objects are aggregated to determine whether the system meets user preferences, paragraphs [0089-0090]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to perform automated verification using the method disclosed by Allen, because it would ensure the system is generating corrects answers for the correct reasons, as taught by Allen (paragraphs [0025-0026]).
Allowable Subject Matter
Claims 3-10, 12, and 16-19 would be allowable over the prior art of record if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the 35 U.S.C. 101 rejections were overcome.
The following is a statement of reasons for the indication of allowable subject matter:
In regard to claim 3, although Wen discloses acquiring constraint dimensions in reference instructions and assigning them to corresponding annotation tasks according to a minimal editing distance (section 4.1, “Task Allocation”), Beller, Wen, Allen, and the additional prior art of record do not disclose or suggest to assess the squashing instruction to determine a most similar squashing instruction type in the list based on a distance measurement.
Claims 4-10 would be allowable for their dependence on claim 3.
In regard to claim 12, Allen discloses applying the logic of selected validators to determine if the criteria of the selected validators are satisfied or not (i.e. a binary determination equivalent to “setting a bit”), Beller, Wen, Allen, and the additional prior art of record do not disclose or suggest to specifically set an abstain bit when the answer represents abstaining, wherein the cumulative score is not penalized when the abstain bit is set and assigning a maximum penalty to the cumulative score when the abstain bit is not set and no correctness bits are set.
Claim 16 would be allowable for the same reasons as claim 3.
Claim 17 would be allowable for its dependence on claim 16.
In regard to claim 18, similarly to claim 3, Beller, Wen, Allen and the additional prior art of record do not disclose or suggest to clean the variational RPI by excluding similar or redundant RPIs in the variational RPI based on a distance measurement.
Claim 19 would be allowable for its dependence on claim 18.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Joty et al., Gado et al., Khafizov et al., Wang et al., Imani et al., Peng et al., Cordes et al., Bhagwat et al., Allen et al., Chen et al., Shinoda et al., Zhou et al., Adlakha et al., and Wang et al. disclose additional methods for evaluating generative systems.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIAN LOUIS ALBERTALLI whose telephone number is (571)272-7616. The examiner can normally be reached M-F 8AM-3PM, 4PM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
BLA 8/5/26
/BRIAN L ALBERTALLI/Primary Examiner, Art Unit 2656
Prosecution Insights