Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to application 18/968,920, which was filed 12/04/24. Claims 1-20 are pending in the application and have been considered.
Foreign Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 6, and 12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Pace et al. (“West-of-N: Synthetic Preference Generation for Improved Reward Modeling”. arXiv:2401.12086v1 [cs.CL] 22 Jan 2024).
Consider claim 1, Pace discloses an annotation method for a large language model (pseudolabeling preference pairs for language model training of preference modeling using queries/responses, Section 4.1, page 3), comprising:
obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement (sampling N responses to a given query, Fig 1, page 2; the responses having a difference in preference label confidence above a threshold, see Pseudo-Preference Filtering, page 4);
obtaining a plurality of scores corresponding to the plurality of response texts, wherein each score of the plurality of scores indicates a degree to which a response text corresponding to the score in the plurality of response texts matches the request text (scored responses y1-yn, Fig 1, page 2; the reward score predicts how a human would judge how well the responses matches the request in terms of correctness, helpfulness, etc., page 4, Section 5); and
obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores, wherein the annotated text is used to adjust a parameter of the large language model (constructing West-of-N preference pairs to generate a pseudo preference dataset consisting of a query, a best score response, and a worst scored response, Fig 1, page 2, Section 4.1, page 3; these are used to self-train a reward model, i.e. adjust a parameter of the large language model, Section 4.2, page 3-4).
Consider claim 2, Pace discloses the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises: determining a first response text as the annotated text in response to determining that a score, corresponding to the first response text, in the plurality of scores satisfies a threshold score condition (the highest scoring response, y1, Fig 1, page 2).
Consider claim 3, Pace discloses the obtaining a plurality of scores corresponding to the plurality of response texts comprises: obtaining, for each response text of the plurality of response texts, a level selected from a plurality of predetermined ordered levels as a score for the response text, and wherein the score satisfying the threshold score condition corresponds to the highest level of the plurality of ordered levels (the highest scoring response, y1, Fig 1, page 2, considered to be at the top level of the scored responses y1-yN shown in Fig 1).
Consider claim 6, Pace discloses the difference requirement indicates at least one of the following: a word segmentation difference and a reward model-based difference between the response texts (the best and worst responses having a difference in preference label confidence above a threshold, i.e. a reward model-based difference, see Pseudo-Preference Filtering, page 4).
Consider claim 12, Pace discloses the obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement comprises: obtaining a response text set, wherein each response text in the response text set is generated by the large language model for the request text (sampling N responses to a given query, Fig 1, page 2;); and selecting the plurality of response texts from the response text set based on the difference requirement (selecting the best and worst responses having differences in preference label confidence above a threshold as a training pair for the dataset, see Pseudo-Preference Filtering, page 4.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Pace et al. (“West-of-N: Synthetic Preference Generation for Improved Reward Modeling”. arXiv:2401.12086v1 [cs.CL] 22 Jan 2024) in view of Madaan et al. (“SELF-REFINE: Iterative Refinement with Self-Feedback”. 37th Conference on Neural Information Processing Systems, 2023).
Consider claim 4, Pace discloses the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises: determining a response text from the plurality of response texts, in response to determining that none of the plurality of scores satisfy a threshold score condition (determining the best response does not have enough of a difference in preference label confidence with that of the worst response based on a threshold, see Pseudo-Preference Filtering, page 4; this means none of the responses have enough of a difference in preference label confidence with that of the worst response, i.e. that that none of the plurality of scores satisfy a threshold score condition).
Pace does not specifically mention determining a second response text; determining a response text satisfying a modification condition; and obtaining a modified version of the second response text as the annotated text.
Madaan discloses determining a second response text (refined response c, Fig 2, page 3); determining a response text satisfying a modification condition (determining initial response at a needs refinement, Fig 2, page 3); and obtaining a modified version of the second response text as the annotated text (iterating SELF-REFINE generates another revision of the refined response, page 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace by determining a second response text; determining a response text satisfying a modification condition; and obtaining a modified version of the second response text as the annotated text in order to ensure desired response quality is achieved, as suggested by Madaan (Section 1, page 1). Doing so would have led to predictable results of helping the model succeed in tasks with intricate requirements, as suggested by Madaan (Section 1, page 1). The references cited are analogous art in the same field of natural language processing.
Consider claim 5, Pace discloses the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises: obtaining the at least one response text of the plurality of response texts, in response to determining that none of the plurality of scores satisfy a threshold score condition and to determining that the plurality of response texts comprise no response text satisfying a modification condition (determining the best response does not have enough of a difference in preference label confidence with that of the worst response based on a threshold, see Pseudo-Preference Filtering, page 4; this means none of the responses have enough of a difference in preference label confidence with that of the worst response, i.e. that that none of the plurality of scores satisfy a threshold score condition, and that none of the plurality of scores should be added to the training dataset, i.e. modifying it, satisfying a modification condition).
Pace does not specifically mention obtaining an evaluation text for the at least one response text as the annotated text.
Madaan discloses obtaining an evaluation text for the at least one response text as the annotated text (feedback text, Fig 2, page 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace by obtaining an evaluation text for the at least one response text as the annotated text for reasons similar to those for claim 1.
Claims 7-11 are rejected under 35 U.S.C. 103 as being unpatentable over Pace et al. (“West-of-N: Synthetic Preference Generation for Improved Reward Modeling”. arXiv:2401.12086v1 [cs.CL] 22 Jan 2024) in view of Shridhar et al. (“The ART of LLM Refinement: Ask, Refine, and Trust”. arXiv:2311.07961v1 [cs.CL] 14 Nov 2023).
Consider claim 7, Pace discloses: before the obtaining a plurality of scores corresponding to the plurality of response texts, obtaining the plurality of response texts (generating the responses, Fig 1, page 2).
Pace does not specifically mention obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts.
Shridhar discloses obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts (we train an Asker to determine whether a prediction is correct or not and whether refinement is needed, Section 3.2, page 3, displayed on Figure 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace by obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts in order to rectify errors, predictably reducing inaccuracies in output, as suggested by Shridhar (Section 1, page 1). The references cited are analogous art in the same field of natural language processing.
Consider claim 8, Pace does not, but Shridhar discloses the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking a degree to which each response text of the plurality of response texts matches the request text (the asker evaluates the degree to which the initial answer answered the query, Section 3.2, page 3, Fig 1, page 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace such that obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking a degree to which each response text of the plurality of response texts matches the request text for reasons similar to those for claim 7.
Consider claim 9, Pace does not, but Shridhar discloses the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking correctness of a fact recorded in each response text of the plurality of response texts (we train an Asker to determine whether a prediction is correct or not and whether refinement is needed, Section 3.2, page 3, Figure 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace such that the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking correctness of a fact recorded in each response text of the plurality of response texts for reasons similar to those for claim 7.
Consider claim 10, Pace does not, but Shridhar discloses the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking language expression of each response text of the plurality of response texts (checking correctness of the prediction, i.e. whether the language used expresses the correct answer, Section 3.2, page 3, Figure 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace such that the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking language expression of each response text of the plurality of response texts for reasons similar to those for claim 7.
Consider claim 11, Pace does not, but Shridhar discloses the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking correctness of logic of each response text of the plurality of response texts (checking correctness of logic in the prediction, Section 3.2, page 3, checking questions shown in Figure 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace such that the obtaining a plurality of pieces of critique data corresponding to the plurality of response texts comprises: obtaining the plurality of pieces of critique data by checking correctness of logic of each response text of the plurality of response texts for reasons similar to those for claim 7.
Claims 13-15, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Pace et al. (“West-of-N: Synthetic Preference Generation for Improved Reward Modeling”. arXiv:2401.12086v1 [cs.CL] 22 Jan 2024) in view of Dong et al. (US 20250227078).
Consider claim 13, Pace discloses:
obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement (sampling N responses to a given query, Fig 1, page 2; the responses having a difference in preference label confidence above a threshold, see Pseudo-Preference Filtering, page 4);
obtaining a plurality of scores corresponding to the plurality of response texts, wherein each score of the plurality of scores indicates a degree to which a response text corresponding to the score in the plurality of response texts matches the request text (scored responses y1-yn, Fig 1, page 2; the reward score predicts how a human would judge how well the responses matches the request in terms of correctness, helpfulness, etc., page 4, Section 5); and
obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores, wherein the annotated text is used to adjust a parameter of the large language model (constructing West-of-N preference pairs to generate a pseudo preference dataset consisting of a query, a best score response, and a worst scored response, Fig 1, page 2, Section 4.1, page 3; these are used to self-train a reward model, i.e. adjust a parameter of the large language model, Section 4.2, page 3-4).
Pace does not specifically mention an electronic device, comprising: one or more processors; a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions.
Dong discloses an electronic device (computing device, [0093]), comprising: one or more processors (CPUs, [0093]); a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions (memory storing instructions executed by processor, [0093], [0097]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace by including an electronic device, comprising: one or more processors; a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions in order to increase speech and efficiency, predicting supporting realtime capabilities, as suggested by Dong ([0051]). The references cited are analogous art in the same field of natural language processing.
Consider claim 20, Pace discloses:
obtaining a plurality of response texts that are generated by a large language model for a request text and that meet a difference requirement (sampling N responses to a given query, Fig 1, page 2; the responses having a difference in preference label confidence above a threshold, see Pseudo-Preference Filtering, page 4);
obtaining a plurality of scores corresponding to the plurality of response texts, wherein each score of the plurality of scores indicates a degree to which a response text corresponding to the score in the plurality of response texts matches the request text (scored responses y1-yn, Fig 1, page 2; the reward score predicts how a human would judge how well the responses matches the request in terms of correctness, helpfulness, etc., page 4, Section 5); and
obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores, wherein the annotated text is used to adjust a parameter of the large language model (constructing West-of-N preference pairs to generate a pseudo preference dataset consisting of a query, a best score response, and a worst scored response, Fig 1, page 2, Section 4.1, page 3; these are used to self-train a reward model, i.e. adjust a parameter of the large language model, Section 4.2, page 3-4).
Pace does not specifically mention a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform processing.
Dong discloses non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform processing (memory storing instructions executed by processor, [0093], [0097]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace by including non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform processing for reasons similar to those for claim 17.
Consider claim 14, Pace discloses the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises: determining a first response text as the annotated text in response to determining that a score, corresponding to the first response text, in the plurality of scores satisfies a threshold score condition (the highest scoring response, y1, Fig 1, page 2).
Consider claim 15, Pace discloses the obtaining a plurality of scores corresponding to the plurality of response texts comprises: obtaining, for each response text of the plurality of response texts, a level selected from a plurality of predetermined ordered levels as a score for the response text, and wherein the score satisfying the threshold score condition corresponds to the highest level of the plurality of ordered levels (the highest scoring response, y1, Fig 1, page 2, considered to be at the top level of the scored responses y1-yN shown in Fig 1).
Consider claim 18, Pace discloses the difference requirement indicates at least one of the following: a word segmentation difference and a reward model-based difference between the response texts (the best and worst responses having a difference in preference label confidence above a threshold, i.e. a reward model-based difference, see Pseudo-Preference Filtering, page 4).
Claims 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Pace et al. (“West-of-N: Synthetic Preference Generation for Improved Reward Modeling”. arXiv:2401.12086v1 [cs.CL] 22 Jan 2024) in view of Dong et al. (US 20250227078), in further view of Madaan et al. (“SELF-REFINE: Iterative Refinement with Self-Feedback”. 37th Conference on Neural Information Processing Systems, 2023).
Consider claim 16, Pace discloses the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises: determining a response text from the plurality of response texts, in response to determining that none of the plurality of scores satisfy a threshold score condition (determining the best response does not have enough of a difference in preference label confidence with that of the worst response based on a threshold, see Pseudo-Preference Filtering, page 4; this means none of the responses have enough of a difference in preference label confidence with that of the worst response, i.e. that that none of the plurality of scores satisfy a threshold score condition).
Pace and Dong do not specifically mention determining a second response text; determining a response text satisfying a modification condition; and obtaining a modified version of the second response text as the annotated text.
Madaan discloses determining a second response text (refined response c, Fig 2, page 3); determining a response text satisfying a modification condition (determining initial response at a needs refinement, Fig 2, page 3); and obtaining a modified version of the second response text as the annotated text (iterating SELF-REFINE generates another revision of the refined response, page 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace and Dong by determining a second response text; determining a response text satisfying a modification condition; and obtaining a modified version of the second response text as the annotated text in order to ensure desired response quality is achieved, as suggested by Madaan (Section 1, page 1). Doing so would have led to predictable results of helping the model succeed in tasks with intricate requirements, as suggested by Madaan (Section 1, page 1). The references cited are analogous art in the same field of natural language processing.
Consider claim 17, Pace discloses the obtaining an annotated text for at least one response text of the plurality of response texts based on the plurality of scores comprises: obtaining the at least one response text of the plurality of response texts, in response to determining that none of the plurality of scores satisfy a threshold score condition and to determining that the plurality of response texts comprise no response text satisfying a modification condition (determining the best response does not have enough of a difference in preference label confidence with that of the worst response based on a threshold, see Pseudo-Preference Filtering, page 4; this means none of the responses have enough of a difference in preference label confidence with that of the worst response, i.e. that that none of the plurality of scores satisfy a threshold score condition, and that none of the plurality of scores should be added to the training dataset, i.e. modifying it, satisfying a modification condition).
Pace and Dong do not specifically mention obtaining an evaluation text for the at least one response text as the annotated text.
Madaan discloses obtaining an evaluation text for the at least one response text as the annotated text (feedback text, Fig 2, page 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace and Dong by obtaining an evaluation text for the at least one response text as the annotated text for reasons similar to those for claim 16.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Pace et al. (“West-of-N: Synthetic Preference Generation for Improved Reward Modeling”. arXiv:2401.12086v1 [cs.CL] 22 Jan 2024) ) in view of Dong et al. (US 20250227078), in further view of Shridhar et al. (“The ART of LLM Refinement: Ask, Refine, and Trust”. arXiv:2311.07961v1 [cs.CL] 14 Nov 2023).
Consider claim 19, Pace discloses: before the obtaining a plurality of scores corresponding to the plurality of response texts, obtaining the plurality of response texts (generating the responses, Fig 1, page 2).
Pace and Dong do not specifically mention obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts.
Shridhar discloses obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts (we train an Asker to determine whether a prediction is correct or not and whether refinement is needed, Section 3.2, page 3, displayed on Figure 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Pace and Dong by obtaining a plurality of pieces of critique data corresponding to the plurality of response texts, wherein each piece of critique data of the plurality of pieces of critique data indicates an error in a response text corresponding to the piece of critique data, and the plurality of pieces of critique data are used for display in association with the plurality of response texts in order to rectify errors, predictably reducing inaccuracies in output, as suggested by Shridhar (Section 1, page 1). The references cited are analogous art in the same field of natural language processing.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yuan et al. (“Self-Rewarding Language Models”. arXiv:2401.10020v1 [cs.CL] 18 Jan 2024) discloses generating synthetic reward datasets
US 20220293094 Mao discloses machine learning models for conversations by selecting a response out of candidate responses using a distance measure
US 11748577 Aberle discloses generation of natural language text from a corpus
US 20210248996 Itoh discloses generating a matched corpus for language model training
US 20240211686 Davis discloses context-based natural language processing
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jesse Pullias whose telephone number is 571/270-5135. The examiner can normally be reached on M-F 8:00 AM - 4:30 PM. The examiner’s fax number is 571/270-6135.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Andrew Flanders can be reached on 571/272-7516.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jesse S Pullias/
Primary Examiner, Art Unit 2655 07/13/26
Prosecution Insights