Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: "a text embedding model configured to", "a dimension reduction module configured to", "an optimization module configured to", "an invertible embedding model configured", "a performance evaluation module configured to", and "the system is configured to" in claim 1-7, and 9-10.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claims 1–7 and 9–10 are interpreted under 35 U.S.C. § 112(f) in accordance with the three-prong analysis set forth in MPEP § 2181. Although the specification states that the applicant intends only limitations expressly using “means for” or “step for” to invoke 35 U.S.C. § 112(f), the applicant’s stated intent is not controlling when the claim language otherwise invokes 35 U.S.C. § 112(f). If applicant did not intend for these limitations to be construed under 35 U.S.C. § 112(f), amendment of the claims is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1, 6-7, and 18 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the term “high,” as used in the limitations “high-dimensional embedding space,” “high-dimensional soft text prompt,” and “high-dimensional machine-interpretable vector,” is a relative term whose scope is unclear. Neither claim 1 nor the specification provides an objective standard, numerical range, minimum number of dimensions, or other ascertainable boundary for determining when an embedding space, soft text prompt, or vector is “high-dimensional.” Although claim 1 contrasts the high-dimensional representation with a “lower-dimensional” representation, it is unclear whether “high-dimensional” requires a particular number or range of dimensions or merely means that the representation has more dimensions than the subsequently generated lower-dimensional representation. The specification describes high-dimensional vectors generally but does not establish how many dimensions are required for a vector or embedding space to qualify as “high-dimensional.” Accordingly, a person of ordinary skill in the art would not be reasonably apprised of the metes and bounds of claim 1 or be able to determine with reasonable certainty whether a particular embedding space, soft text prompt, or vector falls within the scope of the term “high-dimensional.”
Claim 6 recites "a performance evaluation module configured to evaluate the performance of the soft text prompt."
Claim 7 recites "calculate performance metrics given the soft text prompt."
Claim 1, from which claims 6 and 7 depend, recites two soft text prompts: "a high-dimensional soft text prompt" and "a lower-dimensional soft text prompt". The Limitation in claim 6 and 7 "the soft text prompt" therefore there is insufficient antecedent basis; it is unclear whether the performance evaluation module evaluates (i) the high-dimensional soft text prompt, (ii) the lower-dimensional soft text prompt, or (iii) the optimized lower-dimensional soft text prompt.
Claim 18 recites the limitation "a performance evaluation model" in line 2 of the claim. There is insufficient antecedent basis for this limitation in the claim. It is unclear whether it is the same “a performance evaluation model” that is in claim 16 or a different one.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP 2106 (III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject matter. The analysis of claims 1-20, in accordance with these steps, follows.
Step 1 Analysis:
Claims 1-10 are directed to a system (machine). Claims 11-20 are directed to method (processes). Therefore, claims 1-20 fall into one of four statutory categories (i.e., process, machine, article of manufacture).
As to claim 1,
Step 2A Prong 1: this claim recites the following abstract ideas:
encode the discrete text prompt into a high-dimensional embedding space, creating a high-dimensional soft text prompt represented as a high-dimensional machine-interpretable vector; (the limitation describes converting text into a numerical vector representation, which is a mental process implemented using a pen and paper.)
transform the high-dimensional soft text prompt to a lower-dimensional embedding space, creating a lower-dimensional soft text prompt represented as a low-dimensional machine-interpretable vector; (the limitation describes reducing a set of numerical values to a smaller set of numerical values, which is a mental process implemented using a pen and paper.)
optimize the lower-dimensional soft text prompt by optimizing the low-dimensional machine-interpretable vector; (the limitation describes adjusting numerical values to improve an objective, which is a mental process implemented using a pen and paper.)
decode the optimized lower-dimensional soft text prompt into an output text prompt by converting the optimized low-dimensional machine-interpretable vector to the output text prompt; (the limitation describes converting a numerical vector representation back into text, which is a mental process implemented using a pen and paper.)
Step 2A Prong 2 and 2B: the claim recited the following additional elements:
A system for optimizing instructions for Large Language Models (LLMs), comprising: a text embedding model configured to; a dimension reduction module configured to; an optimization module configured to; an invertible embedding model configured to; (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
receive a discrete text prompt from a user device; (this limitation describes data collection/receiving, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
which is configured to be input to a blackbox LLM via API calls. (This limitation describes an intended use of the output text prompt, and the claim does not require inputting the output text prompt to the LLM. Further, the limitation describes data transmission, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claim 11,
Step 2A Prong 1: this claim recites the following abstract ideas:
encoding ... the discrete text prompt into a high-dimensional embedding space, creating a high-dimensional soft text prompt represented as a high-dimensional machine-interpretable vector; (the limitation describes converting text into a numerical vector representation, which is a mental process implemented using a pen and paper.)
transforming ... the high-dimensional soft text prompt to a lower-dimensional embedding space, creating a lower-dimensional soft text prompt represented as a low-dimensional machine-interpretable vector; (the limitation describes reducing a set of numerical values to a smaller set of numerical values, which is a mental process implemented using a pen and paper.)
optimizing ... the lower-dimensional soft text prompt; (the limitation describes adjusting numerical values to improve an objective, which is a mental process implemented using a pen and paper.)
decoding ... the optimized lower-dimensional soft text prompt into an output text prompt by converting the optimized low-dimensional machine-interpretable vector to the output text prompt; (the limitation describes converting a numerical vector representation back into text, which is a mental process implemented using a pen and paper.)
Step 2A Prong 2 and 2B: the claim recited the following additional elements:
by a text embedding model; by a dimension reduction module; by an optimization module; by an invertible embedding model; (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
receiving a discrete text prompt from a user device; (this limitation describes data collection/receiving, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
which is configured to be input to blackbox LLM via API calls. (This limitation describes an intended use of the output text prompt, and the claim does not require inputting the output text prompt to the LLM. Further, the limitation describes data transmission, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claims 2 and 12,
Step 2A Prong 1: This claim does not recite an additional abstract idea, but the claims depend on claims 1 and 11.
Step 2A Prong 2 and 2B: those claims recited the following additional elements:
wherein the text embedding model is configured to encode or encoding, by the text embedding model, the discrete text prompt into the high-dimensional embedding space using a pre-trained language model. (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claims 3 and 13,
Step 2A Prong 1: those claims recite the following abstract ideas:
transform or transforming ... the high-dimensional soft text prompt into the lower-dimensional embedding space using uniform projection. (the limitation describes reducing a set of numerical values to a smaller set of numerical values using a projection technique, which is a mental process implemented using a pen and paper.)
Step 2A Prong 2 and 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) 1.), failing step 2A prong 2. The claims are ineligible.
As to claims 4 and 14,
Step 2A Prong 1: those claims recite the following abstract ideas:
optimize or optimizing ... the lower-dimensional soft text prompt using a gradient-free method, considering each soft prompt and a corresponding zero-shot performance as an input-output pair of an optimization objective. (the limitation describes adjusting numerical values according to an optimization objective by treating each prompt and its performance as an input-output pair, which is a mental process implemented using a pen and paper.)
Step 2A Prong 2 and 2B: those claims recited the following additional elements:
wherein the optimization module is a Bayesian Optimization (BO) module or by the optimization module according to Bayesian Optimization (BO); (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claims 5 and 15,
Step 2A Prong 1: those claims recite the following abstract ideas:
decode or decoding ... the optimized lower-dimensional soft text prompt back into a discrete text instruction, which can be used as input to the blackbox LLM. (the limitation describes converting a numerical vector representation back into a text instruction, which is a mental process implemented using a pen and paper. Examiner notes that "which can be used as input..." is directed to intended use and the claims do not require inputting the instruction to the LLM.)
Step 2A Prong 2 and 2B: those claims recited the following additional elements:
wherein the invertible embedding model is a pre-trained invertible language model or by the invertible embedding model, using a pre-trained invertible language model; (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claims 6 and 16,
Step 2A Prong 1: those claims recite the following abstract ideas:
evaluate or performing performance evaluation to evaluate ... the performance of the soft text prompt or the output text prompt using predefined testing data, including ground truth. (the limitation describes evaluating performance by comparing results against known testing data, which is an evaluation and judgment activity that can be performed as a mental process in the human mind.)
Step 2A Prong 2 and 2B: those claims recited the following additional elements:
a performance evaluation module configured to or by a performance evaluation model; (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claims 7 and 17,
Step 2A Prong 1: those claims recite the following abstract ideas:
calculate ... performance metrics given the soft text prompt, with the metrics used to evaluate the performance of the optimized instruction prompt. (the limitation describes calculating metric values from data, which is a mental process implemented using a pen and paper.)
Step 2A Prong 2 and 2B: those claims recited the following additional elements:
wherein the performance evaluation module is configured to or by a performance evaluation module; (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claims 8 and 18,
Step 2A Prong 1: those claims recite the following abstract ideas:
wherein the performance metrics include or utilizing ... at least one of an F1 score or an Area Under Receiver Operating Characteristic Curve (AUROC) as the performance metrics. (the limitation describes calculating a statistical score such as an F1 score or AUROC, which is a mental process implemented using a pen and paper.)
Step 2A Prong 2 and 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) 1.), failing step 2A prong 2. The claims are ineligible.
As to claims 9 and 19,
Step 2A Prong 1: those claims recite the following abstract ideas:
optimize or optimizing ... the lower-dimensional soft prompt using a mix of ... initialization and human handy-craft initialization to provide a diverse set of initial soft prompts for the optimization. (the limitation describes selecting a diverse set of initial starting values for an optimization, which is an evaluation and judgment activity that can be performed as a mental process in the human mind.)
Step 2A Prong 2 and 2B: those claims recited the following additional elements:
Large Language Model (LLM) initialization; (This limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
As to claims 10 and 20,
Step 2A Prong 1: This claim does not recite an additional abstract idea, but the claims depend on claims 1 and 11.
Step 2A Prong 2 and 2B: those claims recited the following additional elements:
wherein the system is configured to optimize or optimizing, by the optimization module, instructions for the LLMs in Natural Language Processing (NLP) tasks, thereby improving performance of the blackbox LLM in both zero-shot and few-shot scenarios. (This limitation describes an intended use and field of use of the optimization, limiting the abstract idea to the NLP environment, which does not integrate the judicial exception into a practical application, see MPEP 2106.05(h). Further, the limitation is directed to mere instruction to apply the abstract idea on a generic computer to process, which is a well-understood, routine, conventional activity, see MPEP 2106.05(d)(II)(i))
The additional element does not integrate the judicial exception into practical application and does not amount to significantly more than the Judicial exception.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-8, 11-12, and 14-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (USE YOUR INSTINCT: INSTRUCTION OPTIMIZATION USING NEURAL BANDITS COUPLED WITH TRANSFORMERS, 2 Oct 2023) in view of Vu et al. (US 20240020546 A1).
As to claim 1, Lin teaches A system for optimizing instructions for Large Language Models (LLMs), comprising: (see Lin Section [1] "Therefore, it is of paramount importance to develop efficient methods to automatically optimize the instructions/prompts to attain the best performance of LLMs. In this work, we refer to this problem as instruction optimization and use instructions/prompts interchangeably.")
and encode the discrete text prompt into a high-dimensional embedding space, creating a high-dimensional soft text prompt represented as a high-dimensional machine-interpretable vector; (see Lin Section [2.1] "Specifically, a soft prompt z ∈ Z ⊂ R^d is a d-dimensional continuous vector and corresponds to the token embeddings of a number Nz of soft tokens (Lester et al., 2021). A soft prompt z is prepended to the token embeddings of a fixed set E of input-output exemplars E = {(xτ, yτ)}κτ=1 for the task. These concatenated embeddings are used as the input to the white-box LLM w ... The soft prompt z is normally high-dimensional (e.g., d = 5120 × Nz when w is Vicuna 13B)")
Examiner note: Converting the discrete text into token embeddings concatenated with the d-dimensional (5120 × Nz) continuous vector is encoding discrete text into a high-dimensional embedding space as a machine-interpretable vector.
a dimension reduction module configured to transform the high-dimensional soft text prompt to a lower-dimensional embedding space, creating a lower-dimensional soft text prompt represented as a low-dimensional machine-interpretable vector; (see Lin Section [2.1] "The soft prompt z is normally high-dimensional (e.g., d = 5120 × Nz when w is Vicuna 13B), which makes it challenging for BO to optimize. So, InstructZero (Chen et al., 2023b) has adopted the technique of random projection to reduce the input dimension. That is, given a matrix A ∈ R^(d×d′) (d′ ≪ d) with randomly sampled elements and a d′-dimensional continuous vector ẑ, the vector z = Aẑ is used as the soft prompt. After substituting the z by Aẑ, the input variable to be optimized in equation 1 is changed to ẑ and hence the input dimension of the optimization problem is reduced to d′. The reduced input dimension d′, i.e., the intrinsic dimension, is chosen as d′ = 10 in InstructZero.")
an optimization module configured to optimize the lower-dimensional soft text prompt by optimizing the low-dimensional machine-interpretable vector; and (see Lin Section [2.1] "the discrete optimization problem of optimizing ρ is converted to the optimization of a continuous soft prompt z ... In every iteration t of BO, the current observation history is used to update the GP model which is then used to calculate an acquisition function αt(z). Then, a soft prompt zt is selected by maximizing αt(z): zt = arg max z∈Z αt(z). ... the input variable to be optimized in equation 1 is changed to ẑ and hence the input dimension of the optimization problem is reduced to d′")
configured to decode the optimized lower-dimensional soft text prompt into an output text prompt by converting the optimized low-dimensional machine-interpretable vector to the output text prompt (see Lin Section [3.3] "After the soft prompt zt is selected (Sec. 3.2), we proceed to evaluate its performance. Specifically, the selected soft prompt zt is prepended to the embeddings of a set E of exemplars (as well as other texts such as "The instruction was to"), and then the concatenated embeddings are used as the input to the white-box LLM w to generate an instruction ρt = w(zt, E)")
Examiner note: The optimized low-dimensional vector ẑt deterministically yields the soft prompt zt = Aẑt through the fixed projection, and the model converts it into the discrete instruction ρt, thereby converting the optimized low-dimensional vector to the output text prompt.
which is configured to be input to a blackbox LLM via API calls. (see Lin Section [3.3] "we prepend the generated instruction ρt to xi and then use them as the input to the black-box LLM f to generate its output sentence ŷi = f(ρt, xi)") and (see Lin Appendix [C.1] "we select 200 data points from the original test dataset as a test dataset to save the cost of evaluation (i.e., the cost of calling ChatGPT API) ... We follow InstructZero to use Vicuna-13B as the default white-box model for instruction generation and GPT-3.5-turbo as the default black-box model for evaluation.")
Lin does not explicitly teach "a text embedding model configured to receive a discrete text prompt from a user device", and "an invertible embedding model"
However, Vu teaches
a text embedding model configured to receive a discrete text prompt from a user device, (see Vu paragraph [0072] "The user computing device 102 can also include one or more user input component 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.", and see Vu paragraph [0086] "In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output.")
an invertible embedding model (see Vu paragraph [0088] "In some implementations, the input to the machine-learned model(s) of the present disclosure can be latent encoding data (e.g., a latent space representation of an input, etc.). ... As another example, the machine-learned model(s) can process the latent encoding data to generate a reconstruction output.", and see Vu paragraph [0091] "In some cases, the machine-learned model(s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding). ... In another example, the task may comprise generating an embedding for input data")
Examiner note: A machine-learned model that generates embeddings and performs the "corresponding decoding", reconstructing the original data from its latent (embedding-space) representation, is an invertible embedding model, i.e., one applying a reversible transformation between data and its embedding.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lin to receive the discrete text prompt from a user device and to implement the embedding and decoding functions with Vu's pre-trained encoder-decoder machine-learned model that generates latent text embeddings and performs the corresponding decoding/reconstruction, as taught by Vu, because Vu expressly teaches that such an arrangement provides "improved computational efficiency and improvements in the functioning of a computing system" since "instead of having to train billions of parameters of a model for each new task, a user can train tens of thousands of parameters of a soft prompt which can then utilize the billions of pre-trained parameters of the pre-trained machine-learned model" (see Vu paragraph [0063]), and because "by leveraging the computational resources of a server and the datasets stored thereon, a user can train prompts on a user computing device with limited computational power and with limited data" (see Vu paragraph [0309]). Both Lin and Vu are directed to the same field of optimizing soft prompts to condition pre-trained large language models, and the combination merely applies Vu's known user-device input and pre-trained encoder-decoder model to Lin's soft-prompt optimization pipeline to yield the predictable result of an accessible, computationally efficient instruction-optimization system.
As to claim 2, Lin as modified by Vu teaches the system of claim 1, wherein the text embedding model is configured to encode the discrete text prompt into the high-dimensional embedding space using a pre-trained language model. (see Lin Section [2.1] "a soft prompt z ∈ Z ⊂ R^d is a d-dimensional continuous vector and corresponds to the token embeddings of a number Nz of soft tokens (Lester et al., 2021). A soft prompt z is prepended to the token embeddings of a fixed set E of input-output exemplars") and (see Lin Appendix [C.1] "We follow InstructZero to use Vicuna-13B as the default white-box model for instruction generation ... We stack an MLP on top of the hidden representations of the pre-trained transformer language model.")
Examiner note: The token embeddings that encode the discrete text are the embedding representations of the pre-trained transformer language model (Vicuna-13B), so the encoding uses a pre-trained language model.
As to claim 4, Lin as modified by Vu teaches the system of claim 1, wherein the optimization module is a Bayesian Optimization (BO) module, (see Lin Section [2.1] "Based on this formulation, InstructZero (Chen et al., 2023b) has adopted Bayesian optimization (BO) (Garnett, 2023) to maximize the objective function h(ρ(z)) (equation 1). To achieve this, a Gaussian process (GP) (Rasmussen & Williams, 2006) is used as a surrogate to model the function h(ρ(z)).")
configured to optimize the lower-dimensional soft text prompt using a gradient-free method, (see Lin Section [2.1] "In every iteration t of BO, the current observation history is used to update the GP model which is then used to calculate an acquisition function αt(z). Then, a soft prompt zt is selected by maximizing αt(z): zt = arg max z∈Z αt(z). Next, the selected zt is used as input to the white-box LLM w to produce an instruction ρt, which is then evaluated using the black-box LLM f to produce a score ht") and (see Lin Section [B.1] "AutoPrompt (Shin et al., 2020) and FluentPrompt (Shi et al., 2022) have adopted gradient-based methods to search for an optimal sequence of discrete tokens that form an instruction. ... However, these methods cannot be used to optimize prompts for black-box LLMs, which are typically more powerful.")
Examiner note: Lin's BO expressly proceeds using only queried (soft prompt, score) observations of the black-box objective, in contrast to the gradient-based methods Lin identifies as unusable for black-box LLMs, which is optimization by a gradient-free method.
considering each soft prompt and a corresponding zero-shot performance as an input-output pair of an optimization objective. (see Lin Section [2.1] "Lastly, the newly collected input-output pair (zt, ht) is added to the observation history to update the GP model, which is then used to select the soft prompt zt+1 in the next iteration.", and see Lin Section [4.2] "Here we show that our INSTINCT algorithm can further improve over this zero-shot CoT instruction across multiple tasks in Table 3.", and see Lin Appendix [D.2] "To directly adopt in-context learning at test time (i.e., for our test-time-only one-shot INSTINCT algorithm), we modify the evaluation template to include one exemplar as a demonstration.")
Examiner note: The pair (zt, ht) is expressly the "input-output pair" of the objective, and the default (unmodified) evaluation template supplies only the instruction and test input with no exemplars, Lin's Table 5 labels this default configuration "zero-shot", so ht is the zero-shot performance corresponding to each soft prompt zt.
As to claim 5, Lin as modified by Vu teaches the system of claim 1,
configured to decode the optimized lower-dimensional soft text prompt back into a discrete text instruction, which can be used as input to the blackbox LLM. (see Lin Section [3.3] "the selected soft prompt zt is prepended to the embeddings of a set E of exemplars (as well as other texts such as "The instruction was to"), and then the concatenated embeddings are used as the input to the white-box LLM w to generate an instruction ρt = w(zt, E) (Step 3). Next, for every input xi in the validation set DV = {(xi, yi)}n i=1, we prepend the generated instruction ρt to xi and then use them as the input to the black-box LLM f")
Lin does not explicitly teach "wherein the invertible embedding model is a pre-trained invertible language model"
However, Vu teaches
wherein the invertible embedding model is a pre-trained invertible language model, (see Vu paragraph [0088] "the machine-learned model(s) can process the latent encoding data to generate a reconstruction output.", and see Vu paragraph [0197] "The pre-trained machine-learned model can include one or more encoder blocks and one or more decoder blocks. For example, the pre-trained machine-learned model can include an encoder-decoder model such as a transformer model.", and see Vu paragraph [0203] "In some implementations, the pre-trained machine-learned model can include a transformer model (e.g., a T5 model or a BERT model). ... In some implementations, the pre-trained machine-learned model can include an encoder-decoder model. The pre-trained machine-learned model can include a large language model pre-trained with mask training.")
Examiner note: A pre-trained large language model having paired encoder and decoder blocks, encoding inputs into latent representations and decoding/reconstructing outputs from latent (embedding) representations, is a pre-trained invertible language model.
As to claim 6, Lin as modified by Vu teaches the system of claim 1, further comprising: a performance evaluation module configured to evaluate the performance of the soft text prompt using predefined testing data, including ground truth. (see Lin Section [2.1] "with the ground truth output sentence yi to return a score s(ŷi, yi). As a result, instruction optimization can be formulated as the problem of finding the optimal instruction ρ* that achieves the highest score averaged over the validation set DV. Note that the performance of the instruction ρ we find using the validation set DV is evaluated using a separate test set DT.")
As to claim 7, Lin as modified by Vu teaches the system of claim 6, wherein the performance evaluation module is configured to calculate performance metrics given the soft text prompt, with the metrics used to evaluate the performance of the optimized instruction prompt. (see Lin Section [3.3] "which is used to calculate a score s(ŷi, yi). The score ht for ρt is therefore calculated by averaging over the validation set: ht = (1/n) Σn i=1 s(ŷi, yi) (Step 5).") and (see Lin Section [4] "For every algorithm, after finding the best instruction using the validation set DV, we evaluate the discovered instruction using the separate test set DT and report the test accuracy as the score.")
As to claim 8, Lin as modified by Vu teaches the system of claim 7, wherein the performance metrics include at least one of an F1 score or an Area Under Receiver Operating Characteristic Curve (AUROC). (see Lin Appendix [C.1] "For instruction induction, we use the F1 score for 'common_concept', 'informal_to_formal' and SAMSum; we use the exact set matching for 'orthography_starts_with' and 'taxonomy_animal'")
As to claim 11, this is directed to a method that corresponds to the system of claim 1, See the rejection for claim 1 above, which also applies to claim 11.
As to claim 12, this is directed to a method that corresponds to the system of claim 2, See the rejection for claim 2 above, which also applies to claim 12.
As to claim 14, this is directed to a method that corresponds to the system of claim 4, See the rejection for claim 4 above, which also applies to claim 14.
As to claim 15, this is directed to a method that corresponds to the system of claim 5, See the rejection for claim 5 above, which also applies to claim 15.
As to claim 16, this is directed to a method that corresponds to the system of claim 6, See the rejection for claim 6 above, which also applies to claim 16.
As to claim 17, this is directed to a method that corresponds to the system of claim 7, See the rejection for claim 7 above, which also applies to claim 17.
As to claim 18, this is directed to a method that corresponds to the system of claim 8, See the rejection for claim 8 above, which also applies to claim 18.
Claim(s) 3, and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (USE YOUR INSTINCT: INSTRUCTION OPTIMIZATION USING NEURAL BANDITS COUPLED WITH TRANSFORMERS, 2 Oct 2023) in view of Vu et al. (US 20240020546 A1) and Zhou et al. (LARGE LANGUAGE MODELS ARE HUMAN-LEVEL PROMPT ENGINEERS, 10 Mar 2023).
As to claim 3, Lin as modified by Vu teaches the system of claim 1,
Lin does not explicitly teach "wherein the dimension reduction module is configured to transform the high-dimensional soft text prompt into the lower-dimensional embedding space using uniform projection."
However, Chen teaches
wherein the dimension reduction module is configured to transform the high-dimensional soft text prompt into the lower-dimensional embedding space using uniform projection. (see Chen Section [4.1] "We draw entries of the random projection matrix A from a uniform distribution between [−1, 1]. The dimensionality d of p is set to 10. In experiments, we apply a mini-batch version of INSTRUCTZERO that explores 25 soft prompts in every iteration. The only major change required is to select the top-25 soft prompts with the largest u(p) instead of maximizing Eq. (7) in Line 8 of Algorithm 1.")
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined invention of Lin and Vu to initialize the soft-prompt optimization with a mix of LLM-generated and human-designed initial candidates, as taught by Zhou, because Zhou expressly teaches that "language models are very good at generating diverse natural language text" such that a pretrained LLM can "propose a good set U of candidate solutions that will guide our search procedure" (see Zhou Section [3.1]), that a search may otherwise fail "because it lacks of diversity or does not contain any candidates with a suitably high score" (see Zhou Section [3.3]), and that starting from human-designed instructions is beneficial where "there may exist more appropriate prompts than the samples above" (see Zhou Section [3.1]). Both Lin and Zhou are directed to instruction optimization for black-box LLMs, and the combination merely applies Zhou's known hybrid initialization technique to seed Lin's initial set of soft prompts, yielding the predictable result of a more diverse, higher-quality starting population for the optimization.
As to claim 13, this is directed to a method that corresponds to the system of claim 3, See the rejection for claim 3 above, which also applies to claim 13.
Claim(s) 9-10, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (USE YOUR INSTINCT: INSTRUCTION OPTIMIZATION USING NEURAL BANDITS COUPLED WITH TRANSFORMERS, 2 Oct 2023) in view of Vu et al. (US 20240020546 A1) and Chen et al. (INSTRUCTZERO: EFFICIENT INSTRUCTION OPTIMIZATION FOR BLACK-BOX LARGE LANGUAGE MODELS, 8 Aug 2023).
As to claim 9, Lin as modified by Vu teaches the system of claim 1, wherein the optimization module is configured to optimize the lower-dimensional soft prompt (see Lin Section [2.1] "After substituting the z by Aẑ, the input variable to be optimized in equation 1 is changed to ẑ and hence the input dimension of the optimization problem is reduced to d′.")
soft prompts for the optimization. (see Lin Section [3.2] "we generate a discrete domain Z of soft prompts (details in the next paragraph) using a scrambled Sobol sequence following the common practice in BO (Eriksson et al., 2019), which ensures that the discrete domain Z has a good coverage of the original continuous domain Z.", and see Lin Section [4] "Following InstructZero, we initialize our algorithm by randomly selecting 40 soft prompts, and then run our INSTINCT to query another 125 soft prompts.")
Lin does not explicitly teach "using a mix of Large Language Model (LLM) initialization and human handy-craft initialization to provide a diverse set of initial"
However, Zhou teaches
using a mix of Large Language Model (LLM) initialization and human handy-craft initialization to provide a diverse set of initial (see Zhou Section [3.1] "Recent progress in NLP has shown language models are very good at generating diverse natural language text. Therefore, we consider leveraging a pretrained LLM to propose a good set U of candidate solutions that will guide our search procedure. ... For example, in our TruthfulQA experiments, we start with the human-designed instructions from the original dataset (Lin et al., 2022) and ask the the "reverse" model to propose initial instruction samples that fit the missing context (Figure 2 (Bottom)).")
Examiner note: Starting from human-designed instructions while a pretrained LLM proposes additional initial instruction samples is a mix of LLM initialization and human handy-craft initialization yielding a diverse initial candidate set.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined invention of Lin and Vu to perform the dimension reduction using a projection matrix drawn from a uniform distribution, as taught by Chen, because Lin expressly adopts Chen's random projection technique (see Lin Section [2.1] "InstructZero (Chen et al., 2023b) has adopted the technique of random projection to reduce the input dimension"), and Chen expressly teaches the reasons for using it: the soft prompt "usually has dimensionality too high (e.g., thousands for Vicuna) to be handled by existing black-box optimization approaches," and "the random projection is distance-preserving according to Johnson-Lindenstrauss Lemma [Kleinberg, 1997], which leads to comparable kernel similarities before and after the random projection, i.e., k(pi, pj) ≈ k(Api, Apj), so BO in the original space and dimension-reduced space are consistent" (see Chen Section [2.1]). The combination merely implements the dimension-reduction step that Lin already borrows from Chen using Chen's disclosed uniform-distribution projection matrix, yielding the predictable result of an efficient, distance-preserving low-dimensional optimization.
As to claim 10, Lin-Vu as modified by Zhou teaches the system of claim 1, wherein the system is configured to optimize instructions for the LLMs in Natural Language Processing (NLP) tasks, thereby improving performance of the blackbox LLM in both zero-shot and few-shot scenarios. (see Zhou Section [2] "In this paper, we view LLMs as black-box computers that execute programs specified by natural language instructions and investigate how to control an LLM's behavior using model-generated instructions.", and see Zhou Section [4.1] "We assess the effectiveness of zero-shot and few-shot in-context learning on 24 instruction induction tasks proposed in Honovich et al. (2022). The tasks span many facets of language understanding, from simple phrase structure to similarity and causality identification. ... Figure 4 shows the zero-shot performance of InstructGPT using human instructions and model generated instructions. Our algorithm outperforms "Greedy" on every task and achieves equal or better than human performance on 24 of 24 tasks.", and see Zhou Section [C.1] "As shown in Figure 8, adding an instruction achieves a comparable or better test performance than the standard in-context learning performance on 21 of 24 tasks.")
As to claim 19, this is directed to a method that corresponds to the system of claim 9, See the rejection for claim 9 above, which also applies to claim 19.
As to claim 20, this is directed to a method that corresponds to the system of claim 10, See the rejection for claim 10 above, which also applies to claim 20.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABDULLAH K ABOUD whose telephone number is (571)272-0025. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen, can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABDULLAH KHALED ABOUD/ Examiner, Art Unit 2121
/Li B. Zhen/ Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights