DETAILED ACTION
This action is in response to the claims filed 05/15/2024 for Application number 18/664,896. Claims 1-20 are currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/15/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1 Analysis: Claim 1 is directed to a system, which falls within one of the four statutory categories.
Step 2A Prong 1 Analysis: Claim 1 recites, the limitations of:
generates a score for a sentence generated [by a large language model (LLM)], wherein the score is based on one or more tokens comprised in the sentence and an attribute associated with the at least one self-reward model can be considered to be an evaluation in the human mind
alters a text generation process [employed by the LLM] to generate the sentence, such that respective sampling probabilities of respective tokens comprised in a vocabulary employed by the LLM to generate a new token are updated by the LLM based on the score can be considered to be an evaluation in the human mind
These limitations, as drafted, are processes that, under broadest reasonable interpretation, falls within the “Mental Processes” grouping of abstract ideas.
Accordingly, the claim recites an abstract idea.
Step 2A Prong 2 Analysis: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements –
“a memory that stores computer executable components;”, “a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise…” “at least one self-reward model that:..”, “… by a large language model (LLM)”, and “…employed by the LLM”. Thus, the elements in the claim are recited at a high level of generality (i.e. as a generic processor performing a generic computer function of generating an index) such that they amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP 2106.05(f).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of utilizing a “memory”, “processor”, “self-reward model” and “LLM” to perform the steps of the claimed process amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible.
Regarding claim 2, the rejection of claim 1 is further incorporated, and further, the claim recites: a model generation component that generates the at least one self-reward model, wherein generating the at least one self-reward model comprises.
generating, via the LLM, respective sentence embeddings for respective sentences comprised in an annotated dataset, wherein the annotated dataset further comprises labels assigned to the respective sentences based on the attribute;
training a linear classifier based on the new dataset to model an optimal embedding space based on the attribute, wherein the optimal embedding space comprises a favorable subspace, an unfavorable subspace and a decision boundary, wherein the optimal embedding space represents the at least one self-reward model, and wherein the score represents a margin of the sentence evaluated against the decision boundary. These limitations amount to mere instructions to apply the judicial exception using a generic computer component. Please see MPEP 2106.05(f).
The claim further recites: generating a new dataset comprising the respective sentences, the respective sentence embeddings and the labels; This limitation amounts to additional mental steps in addition to the judicial exception identified in the rejection of claim 1.
The claim does recite the additional elements of “a model generation component that generates” and “a linear classifier”, however they do not amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception, for the reasons set forth in connection with the rejection of claim 1 above. The claim is not patent eligible.
Regarding claim 3, the rejection of claim 1 is further incorporated, and further, the claim recites: wherein the LLM and the at least one self-reward model are comprised in a larger machine learning model. This limitation amounts to more specifics of the judicial exception identified in the rejection of claim 1 above.
The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 4, the rejection of claim 2 is further incorporated, and further, the claim recites: wherein the optimal embedding space is modeled via closed-form expressions. This limitation amounts to more specifics of the judicial exception identified in the rejection of claim 1 above.
The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 5, the rejection of claim 2 is further incorporated, and further, the claim recites: wherein the model generation component generates one or more additional self-reward models directed to different respective attributes by training different respective linear classifiers, and wherein the one or more additional self-reward models generate respective scores for the sentence. This limitation amounts to mere instructions to apply the judicial exception using a generic computer component. Please see MPEP 2106.05(f).
The claim does recite the additional elements of “one or more additional self-reward models” and “different respective linear classifier”, however they do not amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception, for the reasons set forth in connection with the rejection of claim 1 above. The claim is not patent eligible.
Regarding claim 6, the rejection of claim 1 is further incorporated, and further,
The claim does recite the additional element of “wherein the at least one self-reward model generates the score without employing external models.”, however it does not amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception, for the reasons set forth in connection with the rejection of claim 1 above. Please see MPEP 2106.05(f). The claim is not patent eligible.
Regarding claim 7, the rejection of claim 1 is further incorporated, and further, the claim recites: wherein updating the respective sampling probabilities of the respective tokens comprises reweighting a probability distribution over the vocabulary. This claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus recites a judicial exception.
The claim does not include any additional elements that amount to an integration of the judicial exceptions into a practical application, nor to significantly more than the judicial exceptions. The claim is not patent eligible.
Regarding claim 8, the rejection of claim 1 is further incorporated, and further, the claim recites: wherein updating the respective sampling probabilities of the respective tokens based on the score ensures that the new token belongs to a favorable class. This claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus recites a judicial exception.
The claim does not include any additional elements that amount to an integration of the judicial exceptions into a practical application, nor to significantly more than the judicial exceptions. The claim is not patent eligible.
Regarding claim 9, the rejection of claim 1 is further incorporated, and further, the claim recites: wherein the text generation process is altered by evaluating the score upon generation of an ending token of the sentence to reweight a subsequent sentence generated by the LLM or by evaluating the score upon generation of each token in the sentence to reweight a subsequent token generated by the LLM. This claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus recites a judicial exception.
The claim does not include any additional elements that amount to an integration of the judicial exceptions into a practical application, nor to significantly more than the judicial exceptions. The claim is not patent eligible.
Regarding Claim 10, it recites features similar to claim 1 and is rejected for at least the same reasons therein.
Regarding Claims 11-18, they recite features similar to claims 2-9 and are rejected for at least the same reasons therein.
Claim 19 recites features similar to claim 1 and is rejected for at least the same reasons therein. Claim 19 additionally requires analysis for “A computer program product for autoregressive sampling for LLMs, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to…” however this is an additional element that amounts to mere instructions to apply the judicial exception using a generic computer component. Please see MPEP 2106.05(f).
Regarding Claim 20, it recites features similar to claims 2 and 11 and is rejected for at least the same reasons therein.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3, 6-10, 12, and 15-19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Zhu et al. ("In-context Learning Reward Guided Decoding for Controlled Text Generation", hereinafter "Zhu").
Regarding claim 1, Zhu teaches A system, comprising:
a memory that stores computer executable components; and
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise (Zhu, page 1120, §V. LIMITATIONS, lines 2-6 “Firstly, in-context learning increases computational cost and inference time. We reduced time complexity to 𝑂 (𝑘(𝑝 + 𝑚)) using a unidirectional model, but this adds GPU memory costs for storing and tracking past key values.” Examiner’s Note: GPU memory implies a processor is being utilized to perform the elements in the claim.):
at least one self-reward model that (See Zhu, page 1117, § A. ICL-based Reward Model):
generates a score for a sentence generated by a large language model (LLM), wherein the score is based on one or more tokens comprised in the sentence and an attribute associated with the at least one self-reward model (“The process of reward scoring determines how we transform the predictions of a language model …Specially, we directly map the desired and undesired attributes to tokens in the vocabulary of reward model, and guide the model to complete the sentence in response to the prompt. The conditional probability of the corresponding token is extracted as the attribute score” [pg. 1117, § A. ICL-based Reward Model]); and
alters a text generation process employed by the LLM to generate the sentence, such that respective sampling probabilities of respective tokens comprised in a vocabulary employed by the LLM to generate a new token are updated by the LLM based on the score. (“Specially, we directly map the desired and undesired attributes to tokens in the vocabulary of reward model, and guide the model to complete the sentence in response to the prompt. The conditional probability of the corresponding token is extracted as the attribute score” [pg. 1117, § A. ICL-based Reward Model; note: This process disclosed by Zhu is considered to be a “text generation process”]);
Regarding Claim 3, Zhu teaches, The system of claim 1, wherein the LLM and the at least one self-reward model are comprised in a larger machine learning model. (Zhu, page 1117, See Figure 1. As shown in the figure, the language model and reward model are a part of ML workflow therefore would be comprised in a “larger” machine learning model)
Regarding claim 6, Zhu teaches The system of claim 1, Zhu teaches wherein the at least one self-reward model generates the score without employing external models. (See Figure 1, and pg. 1117, Zhu never mentions any uses of employing external models in association to the generation of the scores by the reward model.)
Regarding claim 7, Zhu teaches The system of claim 1, wherein updating the respective sampling probabilities of the respective tokens comprises reweighting a probability distribution over the vocabulary. (Zhu, page 1118, §II. IN-CONTEXT LEARNING REWARD GUIDED DECODING, paragraph C. Weighted Decoding “During the process of controlling text generation, adjusting probabilities for the entire vocabulary proves computationally inefficient. To ensure both stochasticity and computational efficiency, IRGD utilizes top-k sampling, which involves selecting the top k candidate tokens from the probability distribution output by the model, to truncate the unreliable long tail of the probability distribution [24], [25]. Subsequently, IRGD re-weights the probabilities of the tokens within the smaller subset…”)
Regarding claim 8, Zhu teaches The system of claim 1, Zhu teaches wherein updating the respective sampling probabilities of the respective tokens based on the score ensures that the new token belongs to a favorable class. (Zhu, Abstract, lines 11-14 “Specifically, IRGD utilizes ICL outputs to score the alignment reward between sequences and target attributes, subsequently modifying the sampling probabilities to favor tokens with higher reward scores.”).
Regarding claim 9, Zhu teaches The system of claim 1, wherein the text generation process is altered by evaluating the score upon generation of an ending token of the sentence to reweight a subsequent sentence generated by the LLM or by evaluating the score upon generation of each token in the sentence to reweight a subsequent token generated by the LLM. (Zhu, page 1117-1118, § II. IN-CONTEXT LEARNING REWARD GUIDED DECODING, paragraph A. ICL-based Reward Model, “To alleviate this issue, our main idea is to employ a unidirectional Transformer decoder with causal masking [23], wherein the unidirectionality confines the decoder to depend exclusively on left-contextual information. As a result, at each generation timestep, previously computed representations can be recycled, and the reward model only requires calculating the representation of the newly appended token. By evaluating only the newly added token at each timestep, rather than reprocessing the entire generated sequence, this approach reduces the computational complexity…” note: The claim recites “or” thus the examiner under BRI is only required to map to one of the corresponding elements.).
Regarding claims 10, 12, and 15-18, they are substantially similar to claims 1, 3, and 6-9 respectively, and are rejected in the same manner, the same art, and reasoning applying.
Claim 19 recites features similar to claim 1 and is rejected for at least the same reasons therein. Claim 19 additionally requires A computer program product for autoregressive sampling for LLMs, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to… (Zhu, See pg. 1120, V. Limitation, GPU memory costs implies use of computers and processors)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2, 4-5, 11, 13-14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu in view of Da Raedt et al. ("A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders", hereinafter "Da Raedt").
Regarding claim 2, Zhu teaches The system of claim 1, further comprising:
a model generation component that generates the at least one self-reward model (“In this paper, we further alleviate the costs associated with additional training by introducing a modular, efficient and fine tuning-free approach called In-context learning based Reward Guided Decoding (IRGD) (“model generation component”), which steers the decoding process based on attribute consistency scores computed from the output of a frozen unidirectional reward model.” [pg. 1117, left col, top para]), wherein generating the at least one self-reward model comprises:
generating, via the LLM, respective sentence embeddings for respective sentences comprised in an annotated dataset, wherein the annotated dataset further comprises labels assigned to the respective sentences based on the attribute (“We follow a standard prompt-based ICL paradigm, prompting the language model to generate a predefined text response y ∈ Ylabel (e.g., positive, negative, etc) based on the provided prompt (constructed based on xinput ), thereby accomplishing the task of attribute classification for input texts.” [pg. 1118, § B. Prompt Construction, ¶1]);
generating a new dataset comprising the respective sentences, the respective sentence embeddings and the labels (See pg. 1118, § B. Prompt Construction, ¶1]); and
However fails to explicitly teach training a linear classifier based on the new dataset to model an optimal embedding space based on the attribute, wherein the optimal embedding space comprises a favorable subspace, an unfavorable subspace and a decision boundary, wherein the optimal embedding space represents the at least one self-reward model, and wherein the score represents a margin of the sentence evaluated against the decision boundary.
De Raedt teaches teach training a linear classifier based on the new dataset to model an optimal embedding space based on the attribute (“Our approach assumes that both labels of the considered property are linearly separable in z space. A linear classifier C is trained on examples of the linguistic property” [pg. 2, 3.2., Linear Property Classifier]), wherein the optimal embedding space comprises a favorable subspace, an unfavorable subspace (See pg. 2, Figure 1(c) desired and undesired classifications would correspond to favorable/unfavorable subspaces) and a decision boundary (See §3.2, decision boundary), wherein the optimal embedding space represents the at least one self-reward model (See 3.4, “final reward obtained by using CMAB”), and wherein the score represents a margin of the sentence evaluated against the decision boundary. (See pg. 3, 3.3, “As a result, the proposed geometric transformation comes down to shifting z in the direction perpendicular to H, and beyond it, into the region where C would predict the opposite label of the property”)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Zhu’s teachings with the linear property classifier of De Raedt. One would have been motivated to make this modification as powerful sentence encoders are capable of embedding a wide range of linguistic properties into vector representations. [Abstract, De Raedt]
Regarding claim 4, Zhu/De Raedt teaches The system of claim 2, Zhu further teaches wherein the optimal embedding space is modeled via closed-form expressions (“This makes the reward model deal with O(km(p + m)) tokens. Given that p could be quite large, the computation cost becomes significantly high.” [pg. 1117, §II. IN-CONTEXT LEARNING REWARD GUIDED DECODING, paragraph A. ICL-based Reward Model; note: Note that in the instant application’s specification, notes that an optimal embedding space can be a reward model. Additionally, the formula used to represent the reward model in Zhu is considered a closed-formed expression. A closed form expression is formed with constants, variables, and a set of functions considered basic and connected by arithmetic operators and function composition. ])
Regarding claim 5, Zhu/De Raedt teaches The system of claim 2,
Zhu teaches wherein the one or more additional self-reward models generate respective scores for the sentence. (pg. 1117, §II. IN-CONTEXT LEARNING REWARD GUIDED DECODING, paragraph A. ICL-based Reward Model, “attribute score”)
De Raedt teaches wherein the model generation component generates one or more additional self-reward models directed to different respective attributes by training different respective linear classifiers, (“We implement CMAB using the LinUCB with Disjoint Linear Models algorithm from Li et al. (2010), which assumes that the expected reward obtained from choosing arm λ is linear with respect to its input features (in our case, sentence encoding z). For each discrete allowed value (‘arm’) for λ, LinUCB learns a separate ridge-regression model, with learnable parameters A ∈ Rn×n and b ∈ Rn” [pg. 3, right col, bottom para])
Same motivation to combine the teachings of Zhu/De Raedt as claim 2.
Regarding claims 11, 13, 14 and 20, they are substantially similar to claims 2, 4 and 5 respectively, and are rejected in the same manner, the same art, and reasoning applying.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL H HOANG whose telephone number is (571)272-8491. The examiner can normally be reached Mon-Fri 8:30AM-4:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL H HOANG/PRIMARY EXAMINER, Art Unit 2122