DETAILED ACTION
This action is in response to the claims filed February 29, 2024. Claims 1-20 are pending. Claims 1, 9, and 16 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 1 is objected to because of the following informalities:
- Claim 1 reads “a memory in communication with the processor, the memory comprising executable instructions…”. This would likely read “a memory in communication with the processor, the memory storing executable instructions”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-7, 9, 12-15, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over “Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise” by Lin et. al (hereinafter Lin), further in view of US 20220035605 A1 (hereinafter “Borstel”).
Regarding claim 1, Lin discloses:
- encoding the natural language prompt to generate a [source text] embedding that embeds the natural language prompt into a continuous space using an encoder (Page 2, “The encoder in GENIE is a 6-layer transformer model which takes the source text s as input with bidirectional self-attention. Specifically, given a source text sequence s = {
x
1
s
,
x
2
s
, . . . ,
x
n
s
} with n tokens, the encoder model computes the vector
h
i
for each token
w
i
[encoding the natural language prompt to generate a [source text] embedding that embeds the natural language prompt into a continuous space using an encoder]”; Page 8, “DiffsuionLM (Li et al., 2022b) maps discrete tokens into continuous latent variable, achieving more complex controllable text generation through continuous diffusion”) [Examiner’s remarks: Natural language prompt (source text) is embedded into a continuous space using an encoder.];
- iteratively denoising a [output text] embedding conditioned on the [source text] embedding using a pre-trained denoiser to generate a denoised code snippet embedding (Page 1, “The diffusion model iterates over multiple time steps, and gradually denoises the output text at each step”; Page 8, “Our model adopts a sequence-to-sequence framework, where a bidirectional encoder encodes the source sequence and a denoising decoder predicts and removes noise from the target sequence in a non auto-regressive fashion. This design allows us to generate diverse text by gradually refining the output from a noisy initial state [iteratively denoising a [output text] embedding conditioned on the [source text] embedding using a pre-trained denoiser to generate a denoised code snippet embedding]”);
- decoding the denoised [output text] embedding using a pre-trained decoder having full self-attention and cross-attention with the [output text] embedding to generate a final predicted [output text] embedding (Page 1, “GENIE follows the encoder-decoder architecture, where the encoder transforms the input text to hidden vectors, and the diffusion model restores the output text from a random Gaussian noise, guided by the encoder hidden vectors”; Page 2, “The encoder in GENIE is a 6-layer transformer model which takes the source text s as input with bidirectional self-attention. Specifically, given a source text sequence s = {
x
1
s
,
x
2
s
, . . . ,
x
n
s
} with n tokens, the encoder model computes the vector
h
i
for each token
w
i
” ; Page 3, “The diffusion model in GENIE is a 6-layer transformer with cross-attention on the source text representation Hs. It learns to predict Gaussian noise zθ (xt, t, Hs) conditioned on the current diffusion step t and the state xt, where xt is the continuous latent representation of the target text” [decoding the denoised [output text] embedding using a pre-trained decoder having full self-attention and cross-attention with the [output text] embedding to generate a final predicted [output text] embedding]);
- projecting the final predicted [output text] embedding to discrete [text] tokens … based on the probability distribution (Page 3 “Inference Phase To generate text from the diffusion model, we start from the final step t = T and sample a state xT from a standard Gaussian distribution. Then we iteratively generate the noise for the previous step using equations 3 and 4, and subtract it from the current state to obtain xt−1. After arriving at t = 0, we apply the clamping trick (Li et al., 2022b) to replace the values of x0 with its closest word embeddings, and then decode the discrete tokens from x0 [projecting the final predicted [output text] embedding to discrete [text] tokens … based on the probability distribution]”) [Examiner’s remarks: Final predicted embedding is turned into discrete text tokens based on the closest word embedding.]
Lin discloses [source text], [output text], and [text]. Lin does not explicitly disclose [source text] being a code description, [output text] being code snippet, or [text] being code. Lin also does not explicitly disclose:
A code generating system comprising:
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor alone or in combination with other processors, cause the code generating system to perform functions of:
- receiving a code description prompt which includes a natural language description of a code snippet for the code generating system to generate via a user interface;
…
- … using a classification head that computes a probability distribution over code tokens and selects code tokens for a predicted code snippet in a target programming language …; and
- returning the predicted code snippet to the user interface.
However, Borstel discloses:
- code description, code snippet, and code (Paragraph [0049], “Following start block 602, the process begins with NL request 110, which is provided as input to Text2Code seq2seq model 109 and Text2AST seq2seq model 111, as illustrated above in FIG. 1 a”; Paragraph [0025], “Multi-model probabilistic source code model 100 is designed to receive NL requests (also referred to as NL input or NL requirements) and convert them into a sequence of source code tokens and a sequence of AST tokens”) [Examiner’s remarks: Borstel discloses acquiring a natural language description of code as input and generating from the description, code as output. One of ordinary skill in the art may replace the NL code description and code of Borstel with the source and output of Lin.].
A code generating system comprising:
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor alone or in combination with other processors, cause the code generating system to perform functions of (Paragraph [0065]:
- receiving a code description prompt which includes a natural language description of a code snippet for the code generating system to generate via a user interface (Paragraph [0049], “Following start block 602, the process begins with NL request 110, which is provided as input to Text2Code seq2seq model 109 and Text2AST seq2seq model 111, as illustrated above in FIG. 1 a”; Paragraph [0025], “Multi-model probabilistic source code model 100 is designed to receive NL requests (also referred to as NL input or NL requirements) and convert them into a sequence of source code tokens and a sequence of AST tokens [receiving a code description prompt which includes a natural language description of a code snippet for the code generating system to generate via a user interface]”);
…
- …using a classification head that computes a probability distribution over code tokens and selects code tokens for a predicted code snippet in a target programming language based on the probability distribution (Paragraph [0026], “Source code Bayesian LSTM decoder 104 outputs a sequence of source code tokens 116, each having a respective probability distribution (PD)”; Paragraph [0029], “In the sequences, each target token is the best candidate based on its probability distribution in the sequence”; Paragraph [0054], “This dataset, which comprises a JSON file that is available at https://conala-corpus.github.io/, contains NL samples derived from Stack Overflow questions and their corresponding Python snippets”) [Examiner’s remarks: Code tokens are generated from NL description through a process which calculates the probability distribution of each code token and selects code tokens based on the probability distribution in a target code language (in this case, python).]; and
- returning the predicted code snippet to the user interface (Paragraph [0051], “After the Fixing rules are applied, the revised token sequence is fed back to block 612 and the operations of decision block 614 and blocks 616 and 618 are repeated until the answer to decision block 614 is YES, at which point the source code is output and the process completes, as depicted by an end block 620”; Paragraph [0065], “In one example, graphics interface 1040 can drive a high definition (HD) display that provides an output to a user [returning the predicted code snippet to the user interface]”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Borstel into the teachings of Lin to include “A code generating system comprising: a processor; and a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor alone or in combination with other processors, cause the code generating system to perform functions of”, “receiving a code description prompt which includes a natural language description of a code snippet for the code generating system to generate via a user interface”, “using a classification head that computes a probability distribution over code tokens and selects code tokens for a predicted code snippet in a target programming language based on the probability distribution”, and “returning the predicted code snippet to the user interface”. As stated in Borstel, “therefore, the use of DL and Probabilistic models is gaining interest to automate source code generation. Accurate and reduced computational cost technologies are needed to provide an advantage to companies that depend on Software/Firmware development to reduce the time to market of their software and hardware products” (Paragraph [0003]). Use of machine learning models for code generation reduces the number of human hours necessary for programming. Therefore, it would be obvious to one of ordinary skill in the art to combine a large language model with code generation.
Regarding claim 4, the rejection of claim 1 is incorporated and Lin further discloses:
PNG
media_image1.png
177
698
media_image1.png
Greyscale
- pre-training the denoiser based on a loss function, the loss function including a first part which corresponds to an error between a predicted noise in a [target] embedding computed by the denoiser and actual noise in the [target] embedding, wherein the error between the predicted noise and the actual noise is minimized to train the denoiser (Page 3, “The training objective is to minimize the squared error between the predicted and true noise, as well as the reconstruction error between x0 and the target embeddings, as expressed in equation 12: [See attached screenshot of equation below]”) [Examiner’s remarks: The training is completed using the above loss function, which includes a squared error between the predicted and true noise (predicted noise in a program embedding and actual noise).].
Lin discloses embedding target. Lin does not explicitly disclose embedding program. However, Borstel discloses program (Paragraph [0049], “Following start block 602, the process begins with NL request 110, which is provided as input to Text2Code seq2seq model 109 and Text2AST seq2seq model 111, as illustrated above in FIG. 1 a”; Paragraph [0025], “Multi-model probabilistic source code model 100 is designed to receive NL requests (also referred to as NL input or NL requirements) and convert them into a sequence of source code tokens and a sequence of AST tokens”) [Examiner’s remarks: Borstel discloses generating program code as an end result of code generation. Hence program code is the target. Lin discloses embedding the target. Since program code is the target to be generated, then one of ordinary skill in the art may replace the generic text of Lin with program text of Borstel.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Borstel into the teachings of Lin to include “program”. As stated in Borstel, “therefore, the use of DL and Probabilistic models is gaining interest to automate source code generation. Accurate and reduced computational cost technologies are needed to provide an advantage to companies that depend on Software/Firmware development to reduce the time to market of their software and hardware products” (Paragraph [0003]). Use of machine learning models for code generation reduces the number of human hours necessary for programming. Therefore, it would be obvious to one of ordinary skill in the art to combine a large language model with code generation.
Regarding claim 5, the rejection of claim 4 is incorporated; and Lin further discloses:
PNG
media_image1.png
177
698
media_image1.png
Greyscale
- pre-training the decoder based on the loss function, the loss function including a second part which corresponds to an error between decoder output and the embedded ground truth [output], and wherein the error is minimized to train the decoder (Page 3, “To train the diffusion model for sequence-to-sequence tasks, we first convert the target sequence
y
=
{
w
1
y
,
w
2
y
,
.
.
.
,
w
n
y
}
into a continuous state
x
0
using the embedding function with a additional Gaussian noise permutation, which can be expressed as…”; Page 3, “The training objective is to minimize the squared error between the predicted and true noise, as well as the reconstruction error between x0 and the target embeddings, as expressed in equation 12: [See attached screenshot of equation below]”) [Examiner’s remarks: The training loss equations includes a second part calculating the reconstruction error between x0 and the target embeddings (decoder output and embedded ground truth.].
Lin discloses an embedded ground truth output. Lin does not explicitly disclose the output being code. However, Borstel discloses the target being code (Paragraph [0049], “Following start block 602, the process begins with NL request 110, which is provided as input to Text2Code seq2seq model 109 and Text2AST seq2seq model 111, as illustrated above in FIG. 1 a”; Paragraph [0025], “Multi-model probabilistic source code model 100 is designed to receive NL requests (also referred to as NL input or NL requirements) and convert them into a sequence of source code tokens and a sequence of AST tokens”) [Examiner’s remarks: Borstel discloses generating program code as an end result of code generation. Hence program code is the target. Lin discloses a ground truth of target. Since program code is the target to be generated, then one of ordinary skill in the art may replace the generic text of Lin with program text of Borstel.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Borstel into the teachings of Lin to include “code”. As stated in Borstel, “therefore, the use of DL and Probabilistic models is gaining interest to automate source code generation. Accurate and reduced computational cost technologies are needed to provide an advantage to companies that depend on Software/Firmware development to reduce the time to market of their software and hardware products” (Paragraph [0003]). Use of machine learning models for code generation reduces the number of human hours necessary for programming. Therefore, it would be obvious to one of ordinary skill in the art to combine a large language model with code generation.
Regarding claim 6, the rejection of claim 1 is incorporated; and Lin further discloses:
- wherein the denoiser is a transformer block with full self-attention over the noisy [target] snippet embedding and cross attention between the noisy [target] snippet embedding and the [source] description embedding (Page 2, “The encoder in GENIE is a 6-layer transformer model which takes the source text s as input with bidirectional self-attention. Specifically, given a source text sequence s = {
x
1
s
,
x
2
s
, . . . ,
x
n
s
} with n tokens, the encoder model computes the vector
h
i
for each token
w
i
”; Page 3, “The diffusion model in GENIE is a 6-layer transformer with cross-attention on the source text representation Hs. It learns to predict Gaussian noise zθ (xt, t, Hs) conditioned on the current diffusion step t and the state xt, where xt is the continuous latent representation of the target text”).
Lin discloses a target and source. Lin does not explicitly disclose the target being code and the source being a code description. However, Borstel discloses the target being code and the source being code description (Paragraph [0049], “Following start block 602, the process begins with NL request 110, which is provided as input to Text2Code seq2seq model 109 and Text2AST seq2seq model 111, as illustrated above in FIG. 1 a”; Paragraph [0025], “Multi-model probabilistic source code model 100 is designed to receive NL requests (also referred to as NL input or NL requirements) and convert them into a sequence of source code tokens and a sequence of AST tokens”)[Examiner’s remarks: Borstel discloses generating program code as an end result of code generation based on initial natural language code descriptions. Hence program code is the target and a description of the code is the source. Lin discloses a target and source description embedding. Since program code is the target to be generated and the code description is the source, then one of ordinary skill in the art may replace the generic text of Lin with program text and description of Borstel.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Borstel into the teachings of Lin to include “code” and “code description. As stated in Borstel, “therefore, the use of DL and Probabilistic models is gaining interest to automate source code generation. Accurate and reduced computational cost technologies are needed to provide an advantage to companies that depend on Software/Firmware development to reduce the time to market of their software and hardware products” (Paragraph [0003]). Use of machine learning models for code generation reduces the number of human hours necessary for programming. Therefore, it would be obvious to one of ordinary skill in the art to combine a large language model with code generation.
Regarding claim 7, the rejection of claim 1 is incorporated; and Lin further discloses:
- wherein the encoder tokenizes the natural language prompt and transforms the tokenized natural language prompt to the [embedding] using an encoding function learned during pre-training (Page 2, “The encoder in GENIE is a 6-layer transformer model which takes the source text s as input with bidirectional self-attention. Specifically, given a source text sequence s = {
x
1
s
,
x
2
s
, . . . ,
x
n
s
} with n tokens, the encoder model computes the vector
h
i
for each token
w
i
. Thus, the source text s can be represented as
H
s
by the encoder model:”) [Examiner’s remarks: the source text is embedded by the encoder model.].
Lin discloses an embedding the natural language prompt. Lin does not explicitly disclose the being natural language prompt embedding to be a code description embedding. However, Borstel discloses the embedding being for a code description (Paragraph [0049], “Following start block 602, the process begins with NL request 110, which is provided as input to Text2Code seq2seq model 109 and Text2AST seq2seq model 111, as illustrated above in FIG. 1 a”; Paragraph [0025], “Multi-model probabilistic source code model 100 is designed to receive NL requests (also referred to as NL input or NL requirements) and convert them into a sequence of source code tokens and a sequence of AST tokens”) [Examiner’s remarks: Borstel describes the input to a model being a natural language description of code. Lin described embedding the given source text. One of ordinary skill in the art understands that generic source text may be replaced with the code description input of Borstel.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Borstel into the teachings of Lin to include “code description”. As stated in Borstel, “therefore, the use of DL and Probabilistic models is gaining interest to automate source code generation. Accurate and reduced computational cost technologies are needed to provide an advantage to companies that depend on Software/Firmware development to reduce the time to market of their software and hardware products” (Paragraph [0003]). Use of machine learning models for code generation reduces the number of human hours necessary for programming. Therefore, it would be obvious to one of ordinary skill in the art to combine a large language model with code generation.
Claims 9 and 12-15 are method claims corresponding to the system claims hereinabove (claims 1 and 4-7 respectively). Therefore, claims 9 and 12-15 are rejected for the same reasons as set forth in the rejection of claims 1 and 4-7, respectively.
Claims 17 and 20 are computer readable storage medium claims corresponding to the system claims hereinabove (claims 1 and 4, respectively). Therefore, claims 17 and 20 are rejected for the same reasons as set forth in the rejection of claims 1 and 4, respectively.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over “Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise” by Lin et. al (hereinafter Lin), further in view of US 20220035605 A1 (hereinafter “Borstel”), and further in view of “Attention Is All You Need” by Vaswani et. al (hereinafter “Vaswani”).
Regarding claim 8, the rejection of claim 1 is incorporated; and Lin further discloses:
- the final predicted [target] snippet embedding corresponds to a sequence of dimensions, and the full self-attention for the decoder enables the decoder to generate (Page 1, “GENIE follows the encoder-decoder architecture, where the encoder transforms the input text to hidden vectors, and the diffusion model restores the output text from a random Gaussian noise, guided by the encoder hidden vectors”; Page 1, “a self-attention based neural network, has emerged as the dominant paradigm for text generation, thanks to its ability to capture global dependencies and leverage largescale pre-trained language models”) [Examiner’s remarks: Lin discloses predicting a target snippet which corresponds to dimensions (output text) to generate some target text.]….
Lin does not explicitly disclose code. However, Borstel discloses the embedding being code (Paragraph [0049], “Following start block 602, the process begins with NL request 110, which is provided as input to Text2Code seq2seq model 109 and Text2AST seq2seq model 111, as illustrated above in FIG. 1 a”; Paragraph [0025], “Multi-model probabilistic source code model 100 is designed to receive NL requests (also referred to as NL input or NL requirements) and convert them into a sequence of source code tokens and a sequence of AST tokens”) [Examiner’s remarks: Borstel discloses the output to a model being code. Lin discloses generating some target text output. One of ordinary skill may output the generic target text of Lin with the code of Borstel.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Borstel into the teachings of Lin to include “code”. As stated in Borstel, “therefore, the use of DL and Probabilistic models is gaining interest to automate source code generation. Accurate and reduced computational cost technologies are needed to provide an advantage to companies that depend on Software/Firmware development to reduce the time to market of their software and hardware products” (Paragraph [0003]). Use of machine learning models for code generation reduces the number of human hours necessary for programming. Therefore, it would be obvious to one of ordinary skill in the art to combine a large language model with code generation.
The combination of Lin and Borstel does not explicitly disclose:
- …each hidden dimension with full information regarding each dimension in the sequence of dimensions.
However, Vaswani discloses:
- …each hidden dimension with full information regarding each dimension in the sequence of dimensions (Page 2, “Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations”; Page 2, “The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively”; Page 5, “Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position”) [Examiner’s remarks: Self-attention allows each dimension (position) to take into account each additional position when generating an output.].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Vaswani into the combined teachings of Lin and Borstel to include “each hidden dimension with full information regarding each dimension in the sequence of dimensions”. As stated in Vaswani, “As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations. In terms of computational complexity, self-attention layers are faster than recurrent layers” (Page 6). Self-attention improves performance, results in more interpretable models, and allows generation with increased context for token generation. Therefore, it would be obvious to one of ordinary skill in the art to combine a large language model with self-attention.
Claim 16 is a method claim corresponding to the system claim hereinabove (claim 8). Therefore, claim 16 is rejected for the same reasons as set forth in the rejection of claim 8.
Allowable Subject Matter
Claims 2-3, 10-11, and 18-19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. - “Diffusion-LM Improves Controllable Text Generation” by Li et. al describes a diffusion based large language model.
- “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension” by Lewis et. al describes a model for language generation with denoising in pre-training.
- “CodeBERT: A Pre-Trained Model for Programming and Natural Languages” by Feng et. al describes a LLM that can be used for code generation among other tasks.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIVIAN WEIJIA DUAN whose telephone number is (703)756-5442. The examiner can normally be reached Monday-Friday 8:30AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Y Mui can be reached at (571) 272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/V.W.D./Examiner, Art Unit 2191 /WEI Y MUI/Supervisory Patent Examiner, Art Unit 2191