Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the communication filed on 03/30/2026.
Claims 1-20 are pending and addressed in the Action.
Response to Arguments
This is in response to the remarks filed on 03/30/2026.
With regards to the claims subjected to 35 USC 103, Applicants directed to the limitations in the claim 1,
generating a plurality of values using a generative language model, each of the values
indicative of a probability of a respective token being a next token of a token sequence
generated by the generative language model; and
applying a mask to the plurality of values, the mask operating on each value that
mask to reduce or zero the probability of the token being the next token;
wherein the generative language model determines the next token based on the plurality
of values after the mask is applied.
and submitted that,
Kommrusch does not disclose applying a mask to values "indicative of a probability of a
respective token being a next token of a token sequence generated by the generative language
model", let alone applying a mask to such values that operates "on each value that corresponds to a token not compliant with a grammar of [a] programming language".
In generally, Applicant appears submitted as that Masked multi-head attention comprises masking this matrix of weights to zero-out all positions corresponding to future tokens in the input sequence and Kommrusch may depict "next token output probabilities" no mask is applied to them. Instead, the mask is applied to attention scores to prevent the decoder of the transformer from "looking ahead".
Examiner respectfully disagreed with Application’s assertions in the remarks.
First of all, the indications in Kommrusch “read” the limitations. In general, Kommrusch discussed using Neural system to aid the programming generation. Figure 2.5 in page 21, address repairing a portion of code vulnerability by showing a transformer. The transformer is looping with the output as the tokens with probability values generated from Softmax, and the token with highest probability will be a next token in a repaired code portion. We do not know where/why the token “b” is with highest probability chosen. The code should be given in Figure 2.5 as an example, and it not necessary to indicate the “mask” is before or after the generation of “next token output probability”, but with “token masking replaces tokens with a special token <MASK> in the ’buggy’ function; token deletion removes tokens; and token infilling replaces multiple consecutive tokens with a single”,
It clearly that “masking” is performed into the generation of “next token output probability”, for selecting “a next token” because without mask, the machine cannot make a determine to choose the highest values in a set of values to become “&& (d > a * b ”. And it would be with the next transform after “b” appeared in “previous output”, there would be appeared in the “next token output probability” with token “)” with highest probability for being compliant with a grammar of programming language. It should be noted that “Masking” in computing is only a filter/ a function for extracting specific values and leaving others.
The combined reference of Lau is lowest or zero probability chosen if user to mask out the “not compliant”, and thus it would be obvious for an ordinary of skills to add Lau as being recognized of the probability values are determined by users as conditions to select desired aspects or values.
All others of Applicant’s arguments would be considered but they are not persuasive with the same Examiner’s submission above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5-11, 12-13, 16-19, 20 are rejected under 35 U.S.C. 103 as being unpatentable over
Kommrusch, “Machine Learning for Computer Aided Programming: From Stochastic Program Repair to Verifiable Program Equivalence”, 2022, Doctor thesis from Department of Computer Science, Colorado State University, 270 pages, in view of
Lau et al., “Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge”, 2016, Cognitive Science, pages 1-40.
As per Claim 1: Kommrusch discloses,
1. A computer-implemented method for generating programming language code, the computer-implemented method comprising:
generating a plurality of values using a generative language model, each of the values indicative of a probability of a respective token being a next token of a token sequence generated by the generative language model;
(Kommrusch: P. 21, Figure 2.5, Code Generation Neural Architecture, tokens with respective probability shown on top of the figure as “Next token output probabilities” which is output from Softmax – See p. 202, referred to para above sec. 7.3.3: “Softmax layer is to create an output that can be interpreted as representing the probability that a given token is correct, given the training the model has been exposed to.”.)
and applying a mask to the plurality of values, the mask operating on each value that corresponds to a token not compliant [with a grammar of the programming language to reduce or zero the probability of the token being the next token];
(Kommrusch: P. 21, Figure 2.5, box ‘Masked Multi-head Attention’, i.e. ‘applying a mask’, this box is applied to previous output to perform masking to the second ‘input embedding’ of the tokens of code with vulnerability, and further see in p. 123, the first line text line “token masking replaces tokens with a special token <MASK> in the ’buggy’ function;” : it indicates that Token Masking is to mask out a piece of code with errors, or "bugs," that cause it to not work as intended. With masking and code with vulnerability: this action in Figure 2.5 reads on “corresponds to a token not compliant with a grammar”]
And see in p. 115, sec. 5.3.7, in line 3, “Multiple copies of multi-head attention layers learn hidden representations of the input data. These representations are then used by a second set of multi-head attention layers to produce a table of probabilities for the most likely token to output.”: In this case, the masking operation, the table of probability for tokens: a 0.1, b 0.8, c 0.1, d 0.0, + 0.0, - 0.0, etc. as in the top of the Figure. These output includes to reduce or zero or highest probability of the token, but the highest is selected as being the next token in selecting to a next token to select for previous output “ && ( d > a * ”)
wherein the generative language model determines the next token based on the plurality of values after the mask is applied.
(Kommrusch: the last three lines of the texts in p. 21 “For example, in Figure 2.5, after the sequence of tokens ’&& ( d > a *’, has been output, the model predicts that the next token should be ’b’ with a probability of 0.8”).
Thus, the in the set of probability tokens a 0.1, b 0.8, c 0.1, d 0.0, + 0.0, - 0.0, , except ‘b’, all others would be not compliant tokens with a grammar of the programming language. The selection would be ‘b’, if for compliance *, or other of a, b, d, +, - for the non-compliant token (* See further in p.202, para above sec. 7.3.3 “As the output is generated, when the tokens with the highest Softmax values are selected we create a rewrite rule which represents the most likely next edge in our path through the program space” ).
In this case , Kommrusch has the model to apply masking, the mask operates with softmax function to produce the values to reduce or zero the probability (‘non-compliant’), and values with highest probability (‘compliant’).
Kommrusch does not apply the reduce or zero probability tokens for being the next token, as “a grammar of the programming language to reduce or zero the probability of the token being the next”.
In the discussion of Probability view of Linguistic Knowledge,
Lau discusses “…a grammar of the programming language to reduce or zero the probability of the token being the next” (See Lau: p. 5 “One straightforward way of deriving grammaticality from probabilities would be to fix some small positive threshold e and to consider as grammatical all those sentences whose probability is above 𝜀 . However this has some undesirable consequences. Most important, since all of the probabilities must sum to one, though there can be infinitely many sentences with non-zero probability, there can be only finitely many sentences with probability above some finite threshold….. It does therefore seem, in principle, possible to predict acceptability on the basis of a probabilistic model. But this requires that we find a way of filtering out those aspects of probability that vary independently of acceptability, and so cannot be used to predict it. We propose that a probabilistic model can generate both probabilities and acceptability judgments if we augment it with an ACCEPTABILITY MEASURE that compensates for other factors, notably lexical frequency and sentence length. These are functions that normalize
the probability value of a sentence through an equation that discounts its length and the
frequency of its lexical items. Some measures also magnify the contribution of other factors
to the acceptability value of the sentence.”.
And in p. 34 “”Conversely, if we take a CFG which generates the set of all grammatical sentences, where grammaticality is understood in a classical binary mode, and use this grammar to construct a PCFG directly and without additional smoothing, then the resulting probability
distribution will assign zero to all ungrammatical sentences. Such models will need to
be smoothed in some way if they are to generate any ungrammatical sentences, and so
account for the fact that humans can process at least some ill-formed sentences, finding
them acceptable to varying degrees.”
Thus, within the Lau’s discussions, it shows that Grammarly-correct or incorrect is assigned with a probability value, and the low or zero probability are conventionally used for ungrammatical sentences. And thus it suggested that the selection for low or zero probability is always associated with non-complaint grammar. The selection for non-compliant grammar token or complaint grammar token is only the user choice. The choice is only a conventionally concept in math, especially in probability.
Therefore, it would be obvious to an ordinary of skills before the effective filing of the application to combine the teaching discussed by Kommrusch for assigning tokens for the most likely token output in a neural network architecture, with discussion in Lau for showing how one can predict acceptability judgments of sentences on the basis of probability with low or high probability for Grammarly-correct or incorrect. The combination would yield predictable results because of probability concept; it applies values in the probability values for a selection, and the selected values are determined by users.
As per Claim 2: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
Kommrusch further discloses,
2. The computer-implemented method of claim 1, further comprising:
determining a set of valid next tokens based on the token sequence already generated by the generative language model and based on one or more rules of the grammar,
(Kommrusch: Figure 2.5, with token masking, and generated by Softmax, the set of tokens {a, b, c, d, +, -} is the determined set for the token sequence “&& ( d > *” [previous output] .
See p. 201, see in para. start with Multi-head attention, “…the model is trained to produce correct rewrite rules and hence all of the learnable functions are learning representations useful for this task. To illustrate, one of the heads may tend to build a representation for addition and subtraction of vectors, while another
head might build a representation for multiplication and division of scalars.” , )
wherein the set of valid next tokens consists of one or more tokens any one of which, when appended to the token sequence, results in a sequence compliant with the grammar of the programming language (Kommrusch: in p. 21, three last lines
“…in Figure 2.5, after the sequence of tokens ’&& ( d > a *’ has been output, the model predicts that the next token should be ’b’ with a probability of 0.8.”. Thus, {b} is the set of next token, and appeared being appended next to sequence ‘&& ( d > a *’ to be && ( d > a *b and being compliant with the grammar);
and
generating the mask by, for each token not in the set of valid next tokens, generating a corresponding masking value that, when applied, reduces or zeros the probability of the token being the next token.
(Kommrusch: In Figure 2.5, the tokens a, c, d, +, - are masked and generated corresponding to Masked Multi-Head Attention and Softmax to have values reduces or zero, i.e. 0.1, 0.1, 0.0, 0.0, and they {a, c, d, +, -}, not in the set of value next tokens)
As per Claim 5: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
Kommrusch further discloses,
5. The computer-implemented method of claim 2,
wherein the generating the plurality of values using the generative language model and the applying the mask is implemented on a first processing unit;
(Kommrusch, see in Figure 2.5, the “Previous Output” is applied masking ‘Masked Multi‐
Head Attention’)
wherein the determining the set of valid next tokens and the generating the mask is implemented on a second processing unit
(Kommrusch, see in Figure 2.5, the set of tokens with assigned probability values “Next token output probabilities”); and
wherein the method further comprises transmitting the mask from the second processing unit to the first processing unit.
(Kommrusch, see in Figure 2.5, connection from ‘Next token output probabilities’ to ‘previous output’)
As per Claim 6: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
6. The computer-implemented method of claim 2, wherein an immediately preceding token of the token sequence is a first portion of a terminal symbol of the grammar (Figure 2.5: the symbol ‘*’ in the sequence ‘&& ( d > a *’ ), and the set of valid next tokens includes a next portion of the terminal symbol (Figure 2.5, value ‘b’, and the value ‘b’ would be next to ‘*’).
As per Claim 7: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
7. The computer-implemented method of claim 2, wherein based on the token sequence already generated by the generative language model and based on the one or more rules of the grammar (Figure 2.5: The set in “Next token output probabilities”), there are multiple possible terminal symbols of the grammar that can be generated by the generative language model that are compliant with the grammar (In the set {a, b, c, d, +, -} seen in “Next token output probabilities”, there are multiple possible value such as a and c since if a, and c chosen as next value, a*a or a*c, are syntax compliance), and wherein the set of valid next tokens includes tokens each of which is a portion of or equal to one of the multiple possible terminal symbols.
(Figure 2.5, since b has highest probability, it is the best to be selected. With a(0.1) or c(0.1) would be the multiple possible terminal symbols appended to the next symbol ‘*’ in the box of previous output)
As per Claim 8: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
8. The computer-implemented method of claim 2, comprising determining, for each token of a plurality of tokens (Figure 2.5, i.e. a, b, c of the tokens with probability >0 ), whether that token is in the set of valid next tokens.
(Kommrusch, in Figure 2.5, except for zero probability value of d, +, - in Next token output probabilities in the box, see in p. 158, lines 4-5: ‘We assert that our random production rule procedure has a non-zero probability of producing any program allowed by the grammar for our datasets.’ Thus in the box values a, b, and c have non-zero probability, they would be whether in the set of valid next tokens)
As per Claim 9: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
9. The computer-implemented method of claim 8, wherein the plurality of tokens is a set of tokens (Kommrusch: in Figure 2.5, with {a, b, c } is plurality of tokens) containing fewer than all possible tokens that can be generated by the generative language model ({a, b, c } is subset of all token in the box Next token output probabilities), and wherein the set of tokens is determined by retrieving all tokens having a prefix equal to a start of a next possible valid token (With {a,b,c}, see in 158, lines 4-5: ‘We assert that our random production rule procedure has a non-zero probability of producing any program allowed by the grammar for our datasets.’ Thus in the box values a, b, and c have non-zero probability, they would be whether in the set of valid next tokens. In this manner for appending ‘&& ( d > a*…’ reads on all tokens having a prefix equal to a start of a next possible valid token).
As per Claim 10: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
Kommrusch further discloses,
10. The computer-implemented method of claim 9, wherein all possible tokens that can be generated by the generative language model are stored in the form of a tree (Kommrusch: See sec. 2.2.3 Abstract Syntax Trees in p. 26), and wherein the set of tokens corresponds to at least one branch of the tree and fewer than all branches of the tree (In Figure 2.5, the code && ( d < a *… is corresponding to Figure 6.4 in 177, where Figure 6.5 present data flow of code in token generator)
As per Claim 11: Kommrusch and combining Lau, where incorporate with plurality of values addressed by Kommrusch,
Kommrusch further discloses,
11. The computer-implemented method of claim 1, wherein the plurality of values is a plurality of normalized probability values output from a softmax function of the generative language model (See Figure 2.5, of p. 21. The probability values 0.1, 0.8, 0.1, 0.0, 0.0, 0.0 are normalized probability values because the sum of values is 1, and output from a softmax ),
and wherein applying the mask comprises setting to zero probability each of the normalized probability values that corresponds to a token not compliant with the grammar of the programming language (In Figure 2.5, In this model, it selects the highest value as being compliant. Other values such as 0.1, 0.0 are not matched to the incomplete code &&(d > a * …. Obviously, the token such as +(0.0) or –(0.0) is non-compliant if placed after ‘*’ of the code.
The mechanism of Figure 2 meets the claimed recitation.
As per claims 12-13, 16-19: The claims are directed to a system; the claims recite the limitations having functionality corresponding to the method claims 1-2, 5-7, 11 above, respectively. The claims are rejected with the same rationales addressed in claims 1-2, 5-7, 11.
As per claim 20: The claim is directed to non-transitory media; it recites the limitations having functionality corresponding to the method claim 1 above. The claim is rejected with the same rationale addressed in claim 1.
Claims 3, 14 are rejected under 35 U.S.C. 103 as being unpatentable over
Kommrusch, “Machine Learning for Computer Aided Programming: From Stochastic Program Repair to Verifiable Program Equivalence”, 2022, Doctor thesis from Department of Computer Science, Colorado State University, 270 pages, in view of Lau et al., “Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge”, 2016, Cognitive Science, pages 1-40, and further in view of The Science of Machine Learning, “Masking”, 2020, retrieved https://www.ml-science.com/masking , 2 pages (Hereinafter: TSoML).
As per Claim 3: Regarding,
3. The computer-implemented method of claim 2,
wherein generating the plurality of values comprises generating a [first tensor] in a neural network of the generative language model, [the first tensor] including the plurality of values;
wherein the mask is [ a second tensor]; and
wherein applying the mask comprises performing [a tensor] product of [the first tensor and the second tensor].
Per claimed limitations above, it describes a mathematical operation ‘product’ between two first-tensor and second-tensor. It should be noted that masking in Neural network is a matrices operation- it is matrices multiplication. The purpose of multiplication is to filter out unneeded value in a given matrix by multiplying with another value chosen in a second matrix. Tensor is a multi-dimensional arrays, in another word, it is multidimensional matrix- various mathematical concepts show a scalar is as an order-zero, a vector is order-1 tensor, a matrix is order-2 tensor, etc. Therefore, masking is a product and each token or value is a tensor.
Kommrusch discloses “generating the plurality of values comprises generating a tokens/values in a neural network of the generative language model (Figure 2.5), [token/values] including the plurality of values (See in Figure 2.5); wherein the mask is [previous output], and wherein applying the mask comprises performing [masking in Figure 2.5] product of [tokens/values in code with vulnerability and tokens/values in previous output: Figure 2].
Kommrusch discloses operations in sec. 6.9.2, with variable types, discusses tensors as model input in p. 43-44, but does not disclose first tensor, second tensor, and mask comprises performing a tensor product of the first tensor and the second tensor.
TSoML, an analogous art, in p. 1,discloses
first tensor (See Mi, ), second tensor (Mk) , and mask comprises performing a tensor product of the first tensor and the second tensor (See M0=K(Mi,Mk).
The Making is an matrix/tensor operation to filter out unwanted elements. It is included in Machine Leaning Model.
Therefore, it would be obvious to a skills of the art before the effective filing of the application to apply the mathematical operation of masking of TSoML into the masking of the Neural model of Kommrusch. The combination would yield results predictable because machine learning is a mathematical model, and in a processing of selection, mathematical concepts are always applied.
As per claim 14: The claim is directed to a system; the claims recite the limitations having functionality corresponding to the method claim 3 above. The claim is rejected with the same rationale addressed in claim 3.
Allowable Subject Matter
Claims 4, 15 are objected to under the prior art as being dependent upon a rejected base claim above. The claims would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ted T Vo whose telephone number is (571)272-3706. The examiner can normally be reached 8am-4:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Y Mui can be reached at (571) 272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
TTV
April 17, 2026
/Ted T. Vo/
Primary Examiner, Art Unit 2191
Prosecution Insights