Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 3/19/24, 5/8/24, and 10/28/24. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to a computer-readable storage medium and the specification Paragraph [0082] indicates that "computer-readable media" can include signals, however, the specification also mentions that “computer-readable storage media” excludes signals. As the specification does not directly define the claimed "medium" but rather "media", Claim 20 encompasses non-statutory transitory forms of signal transmission and is rejected.
Claim 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter without significantly more.
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
Step 1: The claim does not fall within one of the four statutory categories of invention (process, machine, manufacture, or composition of matter), or,
Step 2: The claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial Page 4 exception, as determined using the following analysis:
Step 2A, Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Step 2A, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
MPEP 2106.04(a)(2)(I) states: "The mathematical concepts grouping is defined as mathematical relationships, mathematical formulas or equations, and mathematical calculations.”
MPEP 2106.04(a)(2)(III) states: "Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgements, and opinions.
Further, the MPEP states: "The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g. pen and paper or a slide run) to perform the claim limitation.
Using the two-step inquiry, it is clear that Claims 1-20 are each directed to non-statutory
subject matter as shown below:
Please note the following:
The following groups of claims are expressed in different statutory categories:
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claims 1-15 are drawn to a method and claims 16-19 are drawn to an apparatus.
As to claim 1:
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Yes, this claim is to a process.
Claim 1 recites “A computer-implemented method comprising: obtaining a state of an environment; inputting the state to a first machine learning model; receiving a first action from the first machine learning model, the first machine learning model selecting the first action based at least on the state; inputting the state and the first action to a refinement model; receiving a refined action from the refinement model, the refinement model selecting the refined action based at least on the state and the first action; providing the state, the first action, and the refined action to a reward model; receiving, from the reward model, respective reward values for the first action and the refined action; based at least on the reward values, selecting the first action or the refined action as a selected action; and outputting the selected action.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
the first machine learning model selecting the first action based at least on the state (mental process – the selection of the first action is based on the state’s condition, selecting an initial action based on immediate situation, a mental judgement which is a mental process); See MPEP § 2106.04(a)(2)(III).
the refinement model selecting the refined action based at least on the state and the first action (mental process – the selection of the refined action is based on the state’s condition and first action, mental judgement of determining the best action which is a mental process); See MPEP § 2106.04(a)(2)(III).
based at least on the reward values, selecting the first action or the refined action as a selected action (mental process – reward values help decide the action to select, mental evaluation); See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “obtaining a state of an environment” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “inputting the state to a first machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “receiving a first action from the first machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “inputting the state and the first action to a refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “receiving a refined action from the refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “providing the state, the first action, and the refined action to a reward model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “a first machine learning model”, “a refinement model”, and “a reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “outputting the selected action” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “obtaining a state of an environment” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significant contribution or explanation as to why obtaining state of an environment is integral, hence this element is adding insignificant extra-solution activity.
No, the limitation “inputting the state to a first machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significant contribution or explanation as to why inputting state to the first machine learning model is integral, hence this element is adding insignificant extra-solution activity.
No, the limitation “receiving a first action from the first machine learning model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore, the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
No, the limitation “inputting the state and the first action to a refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significant contribution or explanation as to why inputting state and first action to a refinement model is integral, hence this element is adding insignificant extra-solution activity.
No, the limitation “receiving a refined action from the refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore, the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
No, the limitation “providing the state, the first action, and the refined action to a reward model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significant contribution or explanation as to why giving the state, the first action, and refined action to a reward model is integral, hence this element is adding insignificant extra-solution activity.
No, the limitation “a first machine learning model”, “a refinement model”, and “a reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitations do not add significance as to why these models are needed and therefore the limitations use these elements to apply abstract ideas on a computer.
No, the limitation “outputting the selected action” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significant contribution or explanation as to why outputting the selected action is integral, hence this element is adding insignificant extra-solution activity.
As to claim 2:
Claim 2 recites “The method of claim 1, wherein the first machine learning model comprises a first generative language model.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the first machine learning model comprises a first generative language model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the first machine learning model comprises a first generative language model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1). The limitation of first machine learning model consisting of a first generative language model does not add any significance and applies the abstract idea as basic instruction, hence the element does not amount to significantly more than the judicial exception.
As to claim 3:
Claim 3 recites “The method of claim 2, wherein the refinement model comprises a second generative language model.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the refinement model comprises a second generative language model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the refinement model comprises a second generative language model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1). The limitation of the refinement model consisting of a second generative language model does not add any significance and applies the abstract idea as basic instruction, hence the element does not amount to significantly more than the judicial exception.
As to claim 4:
Claim 4 recites “The method of claim 3, wherein the state includes a user query.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the state includes a user query” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the state includes a user query” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significance as to why the user query is integral in the state, hence the element amounts to adding insignificant extra-solution activity to the judicial exception.
As to claim 5:
Claim 5 recites “The method of claim 4, wherein the first action and the refined action include one or more application programming interfaces provided by a service.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “one or more application programming interfaces provided by a service” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “the first action and the refined action include one or more application programming interfaces provided by a service” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “one or more application programming interfaces provided by a service” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not specify why the application programming interfaces (APIs) are significant to be provided by a service, hence this element is adding elements that amount to applying abstract ideas into the claim.
No, the limitation “the first action and the refined action include one or more application programming interfaces provided by a service” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significance as to why the first action and the refined action are included in the APIs, hence these elements add an extra step and is insignificant extra-solution activity.
As to claim 6:
Claim 6 recites “The method of claim 5, further comprising: inputting a first prompt to the first generative language model instructing the first generative language model to generate the first action based at least on the user query, available application programming interfaces, and one or more example solutions to example user queries, the example solutions utilizing the available application programming interfaces.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “inputting a first prompt to the first generative language model instructing the first generative language model to generate the first action based at least on the user query” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “available application programming interfaces” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “and one or more example solutions to example user queries, the example solutions utilizing the available application programming interfaces” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “inputting a first prompt to the first generative language model instructing the first generative language model to generate the first action based at least on the user query” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). Furthermore, the additional element is directed to receiving or transmitting data over a network, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
No, the limitation “available application programming interfaces” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). Similar to claim 5, APIs present do not justify their significance, hence this element is adding elements that amount to applying abstract ideas into the claim.
No, the limitation “one or more example solutions to example user queries, the example solutions utilizing the available application programming interfaces” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). This limitation is adding an additional step that does not add any significance, the limitation notes using example solutions from the APIs to see potential user queries does not add any improvement, hence these elements add an extra step and are insignificant extra-solution activity.
As to claim 7:
Claim 7 recites “The method of claim 6, the first generative language model and the second generative language models being decoder-based models.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the first generative language model and the second generative language models being decoder-based models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the first generative language model and the second generative language models being decoder-based models” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add any more significance than the judicial exception since the element does not claim any significance or structure as to why the models must be decoder-based models, hence the element adds this limitation to provide instructions to implement the abstract idea.
As to claim 8:
Claim 8 recites “The method of claim 1, further comprising: performing offline training of the reward model by generating multiple actions for a given state, receiving annotations from a human or machine learning model for the multiple actions, and training the reward model based at least on the annotations.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
receiving annotations from a human or machine learning model for the multiple actions (mental process – annotations are being provided by a human, pen and pencil from a person to provide instructions on which action to take); See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “performing offline training of the reward model by generating multiple actions for a given state” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “reward model” and “machine learning model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1).
No, the limitation “training the reward model based at least on the annotations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “reward model” and “machine learning model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitations do not add significance as to why these models are needed and therefore the limitations use these elements to apply abstract ideas on a computer.
No, the limitation “training the reward model based at least on the annotations” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2). The limitation is doing an existing process, training models is an ordinary task, hence the element does not add any significance and only applies the existing process onto the computer.
As to claim 9:
Claim 9 recites “The method of claim 1, further comprising: performing online training of the reward model based at least on explicit feedback or sentiment analysis using a sentiment model.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
performing online training of the reward model based at least on explicit feedback or sentiment analysis (mental process – the ability to perform training based on explicit feedback, a human-based response to improve on the model, mental evaluation which is a mental process); See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “…sentiment analysis using a sentiment model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “reward model” and “sentiment model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “…sentiment analysis using a sentiment model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2). The limitation of utilizing a sentiment analysis does not amount to any significance in the claim as it does not specify why the analysis is necessary, hence the element amounts to adding apply-it to an established process.
No, the limitation “reward model” and “sentiment model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the models are needed and are merely instructions, hence these instructions are intended to apply the reward model and sentiment model.
As to claim 10:
Claim 10 recites “The method of claim 1, further comprising: performing offline training of the refinement model based at least on reward values from the reward model for refined actions generated by the refinement model when refining actions present in a training data set.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
performing offline training of the refinement model based at least on reward values from the reward model (mental process – performing training based on reward values, determining set of choices based on given values is a mental judgement) See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “for refined actions generated by the refinement model when refining actions present in a training data set” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “refinement model” and “reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “for refined actions generated by the refinement model when refining actions present in a training data set” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation sets a condition that is an insignificant extra step, specifically when there are existing refining actions in training data set, hence due to no explanation or significance in the claim language, the element amounts to adding insignificant extra-solution activity.
No, the limitation “refinement model” and “reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the models are needed and are merely instructions, hence these instructions are intended to apply the refinement model and reward model.
As to claim 11:
Claim 11 recites “The method of claim 1, further comprising: performing online training of the refinement model based at least on reward values from the reward model for refined actions generated by the refinement model when interacting with a user.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
performing online training of the refinement model based at least on reward values from the reward model (mental process – performing training based on reward values, determining set of choices based on given values is a mental judgement) See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “for refined actions generated by the refinement model when interacting with a user” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “refinement model” and “reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “for refined actions generated by the refinement model when interacting with a user” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation sets a condition that is an insignificant extra step, specifically when model is interacting with a user, hence due to no explanation or significance in the claim language, the element amounts to adding insignificant extra-solution activity.
No, the limitation “refinement model” and “reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the models are needed and are merely instructions, hence these instructions are intended to apply the refinement model and reward model.
As to claim 12:
Claim 12 recites “The method of claim 1, wherein the refinement model and the reward model share one or more parameters.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the refinement model and the reward model share one or more parameters” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “refinement model” and “reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the refinement model and the reward model share one or more parameters” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation stating the sharing parameter is an extra step as they do not explicitly explain why the models need to share a parameter, hence the element is adding insignificant extra-solution activity.
No, the limitation “refinement model” and “reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the models are needed and are merely instructions, hence these instructions are intended to apply the refinement model and reward model.
As to claim 13:
Claim 13 recites “The method of claim 12, wherein the reward model computes rewards over particular embeddings output by the refinement model.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the reward model computes rewards over particular embeddings output by the refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “the reward model” and “the refinement model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the reward model computes rewards over particular embeddings output by the refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation states the computing of rewards over particular embeddings which is an additional step that does not include the significance of the particular embedding in the claim language; hence the element amounts to adding insignificant extra-solution activity.
No, the limitation “the reward model” and “the refinement model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the models are needed and are merely instructions, hence these instructions are intended to apply the refinement model and reward model.
As to claim 14:
Claim 14 recites “The method of claim 13, the particular embeddings being embeddings of final tokens output by the refinement model.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the particular embeddings being embeddings of final tokens output by the refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “the refinement model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the particular embeddings being embeddings of final tokens output by the refinement model” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation further extends the embedding to be embedding of final tokens output but does not add significance as to why the output is important, hence the element is adding insignificant extra-solution activity.
No, the limitation “the refinement model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the model is needed and is merely instructions, hence these instructions are intended to apply to the refinement model.
As to claim 15:
Claim 15 recites “The method of claim 1, the first machine learning model having fixed parameters.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “the first machine learning model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
No, the limitation “the first machine learning model having fixed parameters” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “the first machine learning model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the model is needed and is merely instructions, hence these instructions are intended to apply the first machine learning model.
No, the limitation “the first machine learning model having fixed parameters” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significance in the claim language as to what the first machine model would improve if they have fixed parameters, hence this element only adds insignificant extra-solution activity.
Due to claim 16 sharing similar syntax to claim 1, claim 16 will be rejected similarly to claim 1.
As to claim 17:
Claim 17 recites “The system of claim 16, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: train the reward model using training data generated with a generative language model.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “when executed by the hardware processing unit, cause the system to” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “train the reward model using training data generated with a generative language model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “the reward model” and “the refinement model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “when executed by the hardware processing unit, cause the system to” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). This element is excessive and does not add any significance as to why this condition is required, hence the element adds insignificant extra-solution activity.
No, the limitation “train the reward model using training data generated with a generative language model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2). The limitation states training of a learning model which is already an existing process for the model as they require datasets and training to determine decisions, hence the element is performing an existing process in its ordinary capacity and does not add any significance to the judicial exception.
No, the limitation “the reward model” and “a generative language model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the models are needed and are merely instructions, hence these instructions are intended to apply the refinement model and a generative language model.
As to claim 18:
Claim 18 recites “The system of claim 17, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: train the refinement model using rewards generated by the reward model.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “when executed by the hardware processing unit, cause the system to” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “train the refinement model using rewards generated by the reward model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §§ 2106.04(d), 2106.05(f)(2).
No, the limitation “the refinement model” and “the reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “when executed by the hardware processing unit, cause the system to” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). This element is excessive and does not add any significance as to why this condition is required, hence the element adds insignificant extra-solution activity.
No, the limitation “train the refinement model using rewards generated by the reward model” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP § 2106.05(f)(2). The limitation states training of a learning model which is already an existing process for the model as they require datasets and training to determine decisions, hence the element is performing an existing process in its ordinary capacity and does not add any significance to the judicial exception.
No, the limitation “the refinement model” and “the reward model” are additional elements that amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not add significance to why the models are needed and are merely instructions, hence these instructions are intended to apply the refinement model and the reward model.
As to claim 19:
Claim 19 recites “The system of claim 16, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: extract one or more application programming interfaces from the selected action; and invoke the one or more application programming interfaces on a service.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the analysis of the parent claim is incorporated.
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, the limitation “when executed by the hardware processing unit, cause the system to” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “extract one or more application programming interfaces from the selected action” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “invoke the one or more application programming interfaces on a service” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP §§ 2106.04(d), 2106.05(g).
No, the limitation “one or more application programming interfaces” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP §§ 2106.04(d), 2106.05(f)(1).
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
No, the limitation “when executed by the hardware processing unit, cause the system to” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). This element is excessive and does not add any significance as to why this condition is required, hence the element adds insignificant extra-solution activity.
No, the limitation “extract one or more application programming interfaces from the selected action” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significance as to why extracting APIs from selected action is important, hence the element is adding insignificant extra-solution activity.
No, the limitation “invoke the one or more application programming interfaces on a service” is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception. See MPEP § 2106.05(g). The limitation does not add any significance as to why invoking APIs from a service is important, hence the element is adding insignificant extra-solution activity.
No, the limitation “one or more application programming interfaces” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer. See MPEP § 2106.05(f)(1). The limitation does not specify why the APIs are significant, hence this element is adding elements that amount to applying abstract ideas into the claim.
As to claim 20:
Claim 20 recites “A computer-readable storage medium storing computer-readable instructions which, when executed by a processing unit, cause the processing unit to perform acts comprising: obtaining a state of an environment; inputting the state to a first machine learning model; receiving a first action from the first machine learning model, the first machine learning model selecting the first action based at least on the state; inputting the state and the first action to a refinement model; receiving a refined action from the refinement model, the refinement model selecting the refined action based at least on the state and the first action; providing the state, the first action, and the refined action to a reward model; receiving, from the reward model, respective reward values for the first action and the refined action; based at least on the reward values, selecting the first action or the refined action as a selected action; and outputting the selected action.”
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
No, this claim is signal per se which does not fall in a statutory category as the claim mentions a computer-readable storage medium and the specification is silent on the storage medium.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-8, 10-11, 15-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Yan, Tianqiang, and Tiansheng Xu. “Refining the Responses of LLMs by Themselves.” arXiv.Org, 6 May 2023. Yan was filed on May 6, 2023, and this date is before the effective filing date of this application, i.e., February 20, 2024. Therefore, Yan constitutes prior art under 35 U.S.C. 102(a)(1).
With respect to Claim 1:
Yan mentions Section 2.1 Paragraph 1 and Section 2.2 Paragraph 4:
A computer-implemented method comprising: obtaining a state of an environment;
obtain state of an environment (2.1 Step 1) where chat history (environment) receives user query (state).
inputting the state to a first machine learning model; input state to first machine learning model (2.1 Step 1).
receiving a first action from the first machine learning model, the first machine learning model selecting the first action based at least on the state; receives first action based on state (2.1 Step 1) where query is sent to remote LLM and LLM responds with a first action based on user’s query.
inputting the state and the first action to a refinement model; input state and first action to refinement model (2.1 Step 2 and 3) where query and action are returned back to terminal (refinement model).
receiving a refined action from the refinement model, the refinement model selecting the refined action based at least on the state and the first action; receives refined action from refinement model dictated based on first action and state (2.1 Step 3 and 4) where terminal analyzes query and first answer to generate optimized response and receives the response.
providing the state, the first action, and the refined action to a reward model; provides state, first action, and refined action to reward model (2.1 Step 5), under BRI and due to the reward model being silent in the specification, reward model can be interpreted as a system that favors elements that are of value or meets the criteria more, so the terminal runs a continuous automatic process (reward model) that is self-selected by a voting prompt to verify actions given state to select final response.
receiving, from the reward model, respective reward values for the first action and the refined action; receives reward values for actions from reward model (2.1 Step 6 and 2.2 Paragraph 4), as mentioned prior, voting system (reward model) provides value based on limiting instructions and reward values.
based at least on the reward values, selecting the first action or the refined action as a selected action; based on reward values (2.1 Step 6 and 2.2 Paragraph 4), employs a greedy strategy (immediate high reward values) to select first or refined action.
and outputting the selected action. (2.1 Step 5 and 6), the response (selected action) is generated to the user in their conversation with the chatbot (output).
With respect to Claim 2:
Yan teaches:
“The method of claim 1, wherein the first machine learning model comprises a first generative language model.”
Yan mentions in Section 2.1 Step 1 of first machine learning model being a LLM, which is a generative language model.
With respect to Claim 3:
Yan teaches:
“The method of claim 2, wherein the refinement model comprises a second generative language model.”
Yan mentions in Section 2.1 Step 3 that the terminal that has a LLM (refinement model) is a generative language model.
With respect to Claim 4:
Yan teaches:
“The method of claim 3, wherein the state includes a user query.” Yan mentions in Section 2.1 Step 1 of state having user query.
With respect to Claim 5:
Yan teaches:
“The method of claim 4, wherein the first action and the refined action include one or more application programming interfaces provided by a service.”
Yan mentions in Section 2.1 Step 1 and 4 of first action and refined action include application programming interfaces (APIs) specifically using LLM API to generate the actions.
With respect to Claim 6:
Yan teaches:
“The method of claim 5, further comprising:
inputting a first prompt to the first generative language model instructing the first generative language model to generate the first action based at least on
the user query,
available application programming interfaces,
and one or more example solutions to example user queries,
the example solutions utilizing the available application programming interfaces.”
Yan mentions in Section 2.1 Step 2, 3, and 6 that remote LLM responding to query to generate a model-based answer and API sends response back to terminal. The LLM is instructed to analyze the user query and provide solutions using the LLM API to return response in terminal.
With respect to Claim 7:
Yan teaches:
“The method of claim 6, the first generative language model and the second generative language models being decoder-based models.”
Yan mentions in Section 1.1 Paragraph 2 that the LLMs being used are GPT 3.5 based models which are models that encode and decode sequential data (decoder-based models).
With respect to Claim 8:
Yan teaches:
“The method of claim 1, further comprising: performing offline training of the reward model by generating multiple actions for a given state,
receiving annotations from a human or machine learning model for the multiple actions,
and training the reward model based at least on the annotations.”
Yan mentions in Section 1.2 Paragraph 2 that the reward model can have offline training by generating multiple actions for a given state and using human annotation to help train the model.
With respect to Claim 10:
Yan teaches:
“The method of claim 1, further comprising: performing offline training of the refinement model based at least on reward values from the reward model for refined actions generated by the refinement model when refining actions present in a training data set.”
Yan mentions in Section 1.2 Paragraph 2 of the offline training of the dataset to update reward models and value when generating high-quality responses.
With respect to Claim 11:
Yan teaches:
“The method of claim 1, further comprising: performing online training of the refinement model based at least on reward values from the reward model for refined actions generated by the refinement model when interacting with a user.”
Yan mentions in Section 1.2 Paragraph 3 of the training where users can interact with the refined actions and dictate the reward values, specifically rating the responses which is then influencing the reward value of the reward model.
With respect to Claim 15:
Yan teaches:
“The method of claim 1, the first machine learning model having fixed parameters.”
Yan mentions in Section 4 Paragraph 1 that their experimentation does not have access to any parameters (inaccessible, as the specification is fixed).
With respect to Claim 16:
Yan mentions Section 2.1 Paragraph 1 and Section 2.2 Paragraph 4:
A system comprising: a hardware processing unit; and a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the system to:
inputting the state to a first machine learning model; input state to first machine learning model (2.1 Step 1).
receiving a first action from the first machine learning model, the first machine learning model selecting the first action based at least on the state; receives first action based on state (2.1 Step 1) where query is sent to remote LLM and LLM responds with a first action based on user’s query.
inputting the state and the first action to a refinement model; input state and first action to refinement model (2.1 Step 2 and 3) where query and action are returned back to terminal (refinement model).
receiving a refined action from the refinement model, the refinement model selecting the refined action based at least on the state and the first action; receives refined action from refinement model dictated based on first action and state (2.1 Step 3 and 4) where terminal analyzes query and first answer to generate optimized response and receives the response.
providing the state, the first action, and the refined action to a reward model; provides state, first action, and refined action to reward model (2.1 Step 5), under BRI and due to the reward model being silent in the specification, reward model can be interpreted as a system that favors elements that are of value or meets the criteria more, so the terminal runs a continuous automatic process (reward model) that is self-selected by a voting prompt to verify actions given state to select final response.
receiving, from the reward model, respective reward values for the first action and the refined action; receives reward values for actions from reward model (2.1 Step 6 and 2.2 Paragraph 4), as mentioned prior, voting system (reward model) provides value based on limiting instructions and reward values.
based at least on the reward values, selecting the first action or the refined action as a selected action; based on reward values (2.1 Step 6 and 2.2 Paragraph 4), employs a greedy strategy (immediate high reward values) to select first or refined action.
and outputting the selected action. (2.1 Step 5 and 6), the response (selected action) is generated to the user in their conversation with the chatbot (output).
With respect to Claim 17:
Yan teaches:
“The system of claim 16, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: train the reward model using training data generated with a generative language model.”
Yan mentions in Section 1.2 Paragraph 2 that system can train the reward model based on high-quality generated responses after initial dataset of demonstrations with human feedback.
With respect to Claim 18:
Yan teaches:
“The system of claim 17, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to: train the refinement model using rewards generated by the reward model.”
Yan mentions in Section 1.2 Paragraph 2 that the refinement model is trained on rewards of the reward model, based on human feedback and updates both refinement model and reward model on aggregated user feedback.
With respect to Claim 19:
Yan teaches: “The system of claim 16, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the system to:
extract one or more application programming interfaces from the selected action;
and invoke the one or more application programming interfaces on a service.”
Yan mentions in Section 2.1 Paragraph 2 of interacting with model API based on selected action and invoking API to perform action.
With respect to Claim 20:
Yan mentions Section 2.1 Paragraph 1 and Section 2.2 Paragraph 4:
A computer-readable storage medium storing computer-readable instructions which, when executed by a processing unit, cause the processing unit to perform acts comprising:
obtain state of an environment (2.1 Step 1) where chat history (environment) receives user query (state).
inputting the state to a first machine learning model; input state to first machine learning model (2.1 Step 1).
receiving a first action from the first machine learning model, the first machine learning model selecting the first action based at least on the state; receives first action based on state (2.1 Step 1) where query is sent to remote LLM and LLM responds with a first action based on user’s query.
inputting the state and the first action to a refinement model; input state and first action to refinement model (2.1 Step 2 and 3) where query and action are returned back to terminal (refinement model).
receiving a refined action from the refinement model, the refinement model selecting the refined action based at least on the state and the first action; receives refined action from refinement model dictated based on first action and state (2.1 Step 3 and 4) where terminal analyzes query and first answer to generate optimized response and receives the response.
providing the state, the first action, and the refined action to a reward model; provides state, first action, and refined action to reward model (2.1 Step 5), under BRI and due to the reward model being silent in the specification, reward model can be interpreted as a system that favors elements that are of value or meets the criteria more, so the terminal runs a continuous automatic process (reward model) that is self-selected by a voting prompt to verify actions given state to select final response.
receiving, from the reward model, respective reward values for the first action and the refined action; receives reward values for actions from reward model (2.1 Step 6 and 2.2 Paragraph 4), as mentioned prior, voting system (reward model) provides value based on limiting instructions and reward values.
based at least on the reward values, selecting the first action or the refined action as a selected action; based on reward values (2.1 Step 6 and 2.2 Paragraph 4), employs a greedy strategy (immediate high reward values) to select first or refined action.
and outputting the selected action. (2.1 Step 5 and 6), the response (selected action) is generated to the user in their conversation with the chatbot (output).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9, 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Yan, Tianqiang, and Tiansheng Xu (“Refining the Responses of LLMs by Themselves, filed on May 6, 2023) in view of Chan, Alex J., et al. (“Dense Reward for Free in Reinforcement Learning from Human Feedback.” filed on February 1, 2024).
With respect to Claim 9:
Yan does not appear to explicitly disclose:
“The method of claim 1, further comprising: performing online training of the reward model based at least on explicit feedback or sentiment analysis using a sentiment model.”
However, Chan teaches:
“The method of claim 1, further comprising: performing online training of the reward model based at least on explicit feedback or sentiment analysis using a sentiment model.” Chan mentions in Section 5.1 Paragraph 2 that online training of the reward model is based on sentiment analysis using a sentiment model.
Yan and Chan are analogous to the art and in the same field of invention because both mention reward models and training methods for reward models. It would have been obvious to a person having ordinary skill in the art to make this combination of utilizing sentiment analysis for reward model training as it further fine-tunes “the labelled dataset to predict the classification” and help produce stronger responses for the user.
With respect to Claim 12:
Yan does not appear to explicitly disclose:
“The method of claim 1, wherein the refinement model and the reward model share one or more parameters.”
However, Chan teaches:
“The method of claim 1, wherein the refinement model and the reward model share one or more parameters.” Chan mentions in Section 6 Paragraph 1 about refinement model and reward model sharing one or more parameters.
Yan and Chan are both analogous art and in the same field of invention because both teach variations of reward model learning and machine learning optimization. While Yan mentions self-sufficient optimization for the models, Chan explores the methods of potential optimization for both reward and generative models. It is obvious to a person having ordinary art in the skill to combine these elements as it would “ensure that both models consider the same MDP” (Markov decision process) and further follow similar logic structures when sharing one or more parameters.
With respect to Claim 13:
Yan does not appear to explicitly disclose:
“The method of claim 12, wherein the reward model computes rewards over particular embeddings output by the refinement model.”
However, Chan teaches:
“The method of claim 12, wherein the reward model computes rewards over particular embeddings output by the refinement model.” Chan mentions in Section 3.1 of rewards being particular tokens as output.
Both are analogous to the art and in the same field of invention because Yan mentions reward models and Chan specifies more into the reward model methodology, specifically, particular embeddings output from refinement model. It would have been obvious to a person having ordinary skills in the art to combine these elements as having the reward model computer rewards over particular embeddings output by refinement model makes embeddings “denser, more informative, and easier to optimize than, yet requires no significant extra computation.”
With respect to Claim 14:
Yan does not appear to explicitly disclose:
“The method of claim 13, the particular embeddings being embeddings of final tokens output by the refinement model.”
However, Chan teaches:
“The method of claim 13, the particular embeddings being embeddings of final tokens output by the refinement model.” Chan mentions in Section 3.1 of rewards being final tokens output.
Both are analogous to the art and in the same field of invention because both as previous claim focus on reward model methodology, it would have been obvious to a person having ordinary skills in the art to combine these elements because the final tokens tells “how much of the final reward received is relevant to the token”, as weight is easier to predict the reward value.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANTAKRAK ATH-LY whose telephone number is (571)270-5326. The examiner can normally be reached Monday-Friday (7:30AM-5:00PM) EST Off every other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D Reyes can be reached at (571)270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
com/C.A./Examiner, Art Unit 2142
/HAIMEI JIANG/Primary Examiner, Art Unit 2142
