DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This action is in reply to the claims filed on 25 November 2024.
Claims 1-20 are currently pending and have been examined.
Claim Objections
Claims 1-20 objected to because of the following informalities:
Claim 1, lines 11-12; Claim 11, lines 14-15; and Claim 20, lines 11-12 recites “ a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences are mapped to a same physical block”. This appears to be a typographical error of “ a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences that are mapped to a same physical block”.
Claim(s) 2-10 and 12-19 inherit the deficiencies of claim 1 and 11, respectively. Therefore, claim(s) 2-10 and 12-19 are objected to.
Appropriate correction is required.
Subject Matter Eligibility
The claims are directed towards eligible subject matter for the following reasons:
Independent claims 1, 11, and 20 recite the following additional elements:
in an iteration of the plurality of iterations under a large language model (LLM):
estimating a first text sequence following a current text sequence based on a speculative decoding method;
forming a plurality of candidate sequences based on the current text sequence and subsequences of the first text sequence;
allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units;
mapping the allocated logical blocks to physical blocks based on a first criterion, wherein the first criterion comprises a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences are mapped to a same physical block; and
determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence in the key-value cache, to form a current text sequence for a next iteration of the plurality of iterations.
These additional elements of the LLM and key-value cache uses speculative decoding and a particular way of mapping a logical blocks to physical blocks to actively determine a new text unit using attention information from the key-value cache. These steps could not practically be performed in the human mind. Therefore, the recited steps define a technical process that does not fall within the categories of abstract ideas in step 2A, prong 1. The claims recite eligible subject matter under U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-2, 6-8, 10-12, 16-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jeon (US 20250245430 A1) in view of Kwon (see attached NPL).
Regarding claim 1, Jeon teaches a method for generating an output text for an input text based on performing a plurality of iterations, (see at least Paragraph [0070] “operations 600 that may be performed by a computing device to generate a response to an input prompt using generative artificial intelligence models”; Fig. 6) the method comprising:
in an iteration of the plurality of iterations under a large language model (LLM) (see at least Paragraph [0062] “The tree 400 generally corresponds to the tokens selected for inclusion in each subset of tokens discussed above with respect to FIG. 3. As illustrated, the tree 400 may include a root node 410 corresponding to the input query (or prompt) based on which the remaining nodes, corresponding to different speculatively generated tokens, are generated. As illustrated, each level in the tree data structure, corresponding to different subsets of tokens”; Fig. 4 and 6; Examiner notes Jeon creates a tree which iteratively determines subsets of tokens based on the prior subsets):
estimating a first text sequence following a current text sequence based on a speculative decoding method; (see at least Paragraph [0072] “Based on the input prompt and the first subset of tokens, and using the first machine learning model, a second subset of tokens may be speculatively generated. The second subset of tokens may correspond to a second portion of the response to the input prompt.”; step 610 of Fig. 6)
forming a plurality of candidate sequences based on the current text sequence and subsequences of the first text sequence; (see at least Paragraph [0073] “speculatively generating a draft set of tokens including a respective group of tokens for each respective token in the first subset of tokens. A top group of tokens may be selected. This top group of tokens generally includes the number of tokens having highest probabilities from the draft set of tokens.”; Paragraph [0079] “the set of tokens comprises a tree data structure including a root node corresponding to the input prompt and one or more levels associated with the one or more subsets of tokens. Each respective subset of the one or more subsets of tokens may correspond to a respective level in the tree data structure.”; Fig. 6)
determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence (see at least Paragraph [0080] “the operations 600 further include generating an attention map based on the tree data structure, the attention map identifying sequences of tokens in the tree data structure. The attention map may be output to the second machine learning model for the second machine learning model to use in verifying the sequence of tokens.”; Fig. 6)
Jeon does not teach:
allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units;
mapping the allocated logical blocks to physical blocks based on a first criterion, wherein the first criterion comprises a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences are mapped to a same physical block; and
determining, by the LLM, a newly generated text unit in the iteration by using attention information of each candidate sequence in the key-value cache, to form a current text sequence for a next iteration of the plurality of iterations.
However, Kwon teaches:
allocating logical blocks to text units in the plurality of candidate sequences in a key-value cache, to store attention information of the text units; (see at least logical KV cache blocks in the figure sequence on Pages 9-12 of Kwon)
mapping the allocated logical blocks to physical blocks based on a first criterion, wherein the first criterion comprises a plurality of logical blocks allocated to a same text unit in the plurality of candidate sequences that are mapped to a same physical block; (see at least Page 5 “PagedAttention naturally enables memory sharing through its block table. Similar to how processes share physical pages, different sequences in PagedAttention can share the blocks by mapping their logical blocks to the same physical block.”; Figure sequence on Pages 9-12 of Kwon) and
attention information of each candidate sequence in the key-value cache (see at least Page 5 “PagedAttention naturally enables memory sharing through its block table. Similar to how processes share physical pages, different sequences in PagedAttention can share the blocks by mapping their logical blocks to the same physical block.”; Figure sequence on Pages 9-12 of Kwon)
These steps of Kwon are applicable to the method of Jeon as they both share characteristics and capabilities, namely, they are directed to improving the response generation of large language models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Jeon to incorporate allocating logical blocks to text units in the candidate sequences in a key value cache to store attention information of the text units and mapping the allocated logical blocks to the physical blocks based on the same text units being mapped to the same physical block as taught by Kwon. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Jeon in order to deliver higher throughput without requiring any model architecture changes (see Page 1 of Kwon).
Regarding claim 2, Jeon in view of Kwon teaches the method according to claim 1. Jeon does not teach:
wherein logical storage spaces corresponding to the key-value cache forms a matrix, and a logical block in a row in the matrix stores a text unit in a candidate sequence; and
the allocating logical blocks to text units in the plurality of candidate sequences comprises:
adding a new row to the matrix for each candidate sequence; and
adding a logical block to the row for a text unit in the candidate sequence.
However, Kwon teaches:
wherein logical storage spaces corresponding to the key-value cache forms a matrix, and a logical block in a row in the matrix stores a text unit in a candidate sequence; (see at least Page 5 “PagedAttention naturally enables memory sharing through its block table. Similar to how processes share physical pages, different sequences in PagedAttention can share the blocks by mapping their logical blocks to the same physical block.”; Figure sequence on Pages 9-12 of Kwon) and
the allocating logical blocks to text units in the plurality of candidate sequences comprises:
adding a new row to the matrix for each candidate sequence; (see Figure “5. Seq A and B generated 3rd token” on Page 11 of Kwon) and
adding a logical block to the row for a text unit in the candidate sequence. (see Figure “5. Seq A and B generated 3rd token” on Page 11 of Kwon)
The motivation for making this modification to the teachings of Jeon is the same as that set forth above, in the rejection of claim 1.
Regarding claim 6, Jeon in view of Kwon teaches the method according to claim 1. Jeon further teaches wherein the determining a newly generated text unit in the iteration comprises:
predicting, by the LLM, a next text unit for each candidate sequence by using the attention information of each candidate sequence (see at least Paragraph [0069] “the draft model may generate the set of tokens 510 and the corresponding attention map 520 and output both the set of tokens 510 and the corresponding attention map 520 to the target model for processing. The target model, meanwhile, can use the attention map 520 to generate attention values for each token in the set of tokens 510 according to the paths illustrated in the attention map and can perform rejection sampling”; Fig. 5)
sequentially forming a second text sequence for next text units respectively predicted for the plurality of candidate sequences; (see at least Paragraph [0041] “As illustrated, the draft model generates four proposed tokens X.sub.1 220A, X.sub.2 220B, X.sub.3 220C, and X.sub.4 220D, with associated probabilities p.sub.1, p.sub.2, p.sub.3, p.sub.4 from an original target distribution q. A target model can sequentially examine these tokens to determine whether to accept or reject each token.”)
comparing the second text sequence with the first text sequence; (see at least Paragraph [0086] “At block 720, the operations 700 proceed with comparing a probability distribution associated with each respective subset of tokens in the set of tokens to a corresponding probability distribution generated by a second machine learning model for the respective subset of tokens.”; Fig. 7) and
determining the newly generated text unit based on a comparison result. (see at least Paragraph [0087] “At block 730, the operations 700 proceed with selecting tokens from the set of tokens based on the comparing.”; Fig. 7)
Jeon does not teach:
attention information of each candidate sequence in the key-value cache. (see at least Page 5 “PagedAttention naturally enables memory sharing through its block table. Similar to how processes share physical pages, different sequences in PagedAttention can share the blocks by mapping their logical blocks to the same physical block.”; Figure sequence on Pages 9-12 of Kwon)
The motivation for making this modification to the teachings of Jeon is the same as that set forth above, in the rejection of claim 1.
Regarding claim 7, Jeon in view of Kwon teaches the method according to claim 6. Jeon further teaches:
determining a first incorrect text unit based on the comparison result, wherein the first incorrect text unit exists in the first text sequence and does not exist at a corresponding location in the second text sequence; (see at least Paragraph [0090] “selecting the tokens from the set of tokens includes rejecting a first token at a first level of a tree data structure representing the set of tokens. […] It is determined whether to accept or reject a second token at the first level of the tree data structure based on the adjusted probability distribution.”; Paragraph [0027] “The final token may then be generated at the first rejection position”; Fig. 7)
determining a text unit following the first incorrect text unit in the first text sequence as a second incorrect text unit; (see at least Paragraph [0090] “selecting the tokens from the set of tokens includes rejecting a first token at a first level of a tree data structure representing the set of tokens. […] Children tokens of the first token at levels deeper than the first level of the tree data structure are discarded or ignored from the tree data structure.”; Fig. 7)
for each incorrect text unit in the first incorrect text unit and the second incorrect text unit, returning memory (see at least Paragraph [0090] “selecting the tokens from the set of tokens includes rejecting a first token at a first level of a tree data structure representing the set of tokens. […] Children tokens of the first token at levels deeper than the first level of the tree data structure are discarded or ignored from the tree data structure.”; Fig. 7)
Jeon does not teach:
for each incorrect text unit in the first incorrect text unit and the second incorrect text unit, returning a logical block and a physical block corresponding to attention information of the incorrect text unit in the key-value cache.
However, Kwon teaches:
storing memory with a logical block and a physical block; (see at least Page 5 “PagedAttention naturally enables memory sharing through its block table. Similar to how processes share physical pages, different sequences in PagedAttention can share the blocks by mapping their logical blocks to the same physical block.”; Figure sequence on Pages 9-12 of Kwon) and
attention information of each candidate sequence in the key-value cache (see at least Page 5 “PagedAttention naturally enables memory sharing through its block table. Similar to how processes share physical pages, different sequences in PagedAttention can share the blocks by mapping their logical blocks to the same physical block.”; Figure sequence on Pages 9-12 of Kwon)
The motivation for making this modification to the teachings of Jeon is the same as that set forth above, in the rejection of claim 1.
Regarding claim 8, Jeon in view of Kwon teaches the method according to claim 1. Jeon further teaches:
wherein the input text comprises a plurality of user request texts, (see at least Paragraph [0002] “Generative artificial intelligence models can be used in various environments in order to generate a response to an input prompt (also referred to as a query or an input). For example, generative artificial intelligence models can be used in chatbot applications in which large language models (LLMs) are used to generate an answer, or at least a response, to an input prompt.”) the current text sequence comprises a plurality of sequence segments, a single sequence segment corresponds to an output text generated for a single user request text, and the first text sequence comprises a plurality of continuation sequences respectively estimated for the plurality of sequence segments. (see at least Paragraph [0072] “generating the set of tokens includes generating a first subset of tokens corresponding to a first portion of the response to the input prompt based on the input prompt and using the first machine learning model. Based on the input prompt and the first subset of tokens, and using the first machine learning model, a second subset of tokens may be speculatively generated. The second subset of tokens may correspond to a second portion of the response to the input prompt.”; step 610 of Fig. 6)
Regarding claim 10, Jeon in view of Kwon teaches the method according to claim 8. Jeon further teaches wherein the estimating a first text sequence following a current text sequence based on a speculative decoding method comprises:
performing the speculative decoding when the LLM generates at least one output text unit for each of the plurality of user request texts, wherein the current text sequence comprises a plurality of sequence segments corresponding to the plurality of user request texts. (see at least Paragraph [0072] “Based on the input prompt and the first subset of tokens, and using the first machine learning model, a second subset of tokens may be speculatively generated. The second subset of tokens may correspond to a second portion of the response to the input prompt.”; step 610 of Fig. 6; Fig. 5)
Claim 11:
Claim 11 is directed to an apparatus. Claim(s) 11 recite limitations parallel in nature as those addressed above for claim 1, which are directed towards a method. Claim(s) 11 is therefore rejected for the same reasons as set above for claim 1. Claim 11 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim 12:
Claim 12 is directed to an apparatus. Claim(s) 12 recite limitations parallel in nature as those addressed above for claim 2, which are directed towards a method. Claim(s) 12 is therefore rejected for the same reasons as set above for claim 2. Claim 12 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim 16:
Claim 16 is directed to an apparatus. Claim(s) 16 recite limitations parallel in nature as those addressed above for claim 6, which are directed towards a method. Claim(s) 16 is therefore rejected for the same reasons as set above for claim 6. Claim 16 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim 17:
Claim 17 is directed to an apparatus. Claim(s) 17 recite limitations parallel in nature as those addressed above for claim 7, which are directed towards a method. Claim(s) 17 is therefore rejected for the same reasons as set above for claim 7. Claim 17 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim 18:
Claim 18 is directed to an apparatus. Claim(s) 18 recite limitations parallel in nature as those addressed above for claim 8, which are directed towards a method. Claim(s) 18 is therefore rejected for the same reasons as set above for claim 8. Claim 18 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim 20:Claim 20 is directed to an apparatus. Claim(s) 20 recite limitations parallel in nature as those addressed above for claim 1, which are directed towards a method. Claim(s) 20 is therefore rejected for the same reasons as set above for claim 1. Claim 20 further recites “a non-transitory, computer-readable medium storing one-or more instructions executable by at least one processor to perform operations” (see at least Paragraph [0006] “non-transitory computer-readable media”; Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim(s) 3-5 and 13-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jeon (US 20250245430 A1) in view of Kwon (see attached NPL) in further view of Flor (US 20110320498 A1).
Regarding claim 3, Jeon in view of Kwon teaches the method according to claim 1. Jeon further teaches:
a text generated by the LLM. (see at least Paragraph [0020] “a large language model (LLM) deployed within a chatbot can generate a response to a query”)
Jeon in view of Kwon does not teach:
wherein the estimating a first text sequence following a current text sequence based on a speculative decoding method comprises:
querying a vocab table by using the current text sequence, to obtain the matching first text sequence, wherein the vocab table comprises a plurality of mapping relationships, each of the mapping relationships is between a text sequence and a continuation text sequence following the text sequence, and the vocab table is obtained based on a text generated by the LLM.
However, Flor teaches:
querying a vocab table by using the current text sequence, to obtain the matching first text sequence, wherein the vocab table comprises a plurality of mapping relationships, each of the mapping relationships is between a text sequence and a continuation text sequence following the text sequence, and the vocab table is obtained based on a text. (see at least Paragraph [0060] “When the system is queried about any given literal n-gram, it searches each word separately, as a unigram, and attempts to retrieve code words for each word of the query (except first). […] a code-worded n-gram equivalent (byte-sequence) is assembled and searched for in the tree.”; Paragraph [0064] “The node for the last byte of the candidate byte-sequence holds the offset for loading the block of next successors--that block is there candidates are sought for the next component of the query.”; Paragraph [0076] “Given a query, […] the processing system at step 704 sets the first unit of the query to be the current unit of the query.”; Paragraph [0044] “This design allows easy […] generation/update of n-gram models from text.”; Fig. 7 of Flor and the associated paragraphs show how a tree of vocabulary is traced based on a sequence of text from a query in order to track down a count statistic).
This step of Flor is applicable to the method of Jeon as they both share characteristics and capabilities, namely, they are directed to using natural language to analyze text. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Jeon to incorporate querying a vocab table as taught by Flor. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Jeon in order to provide for efficient memory use and enhanced speed (see paragraph [0003] of Flor).
Regarding claim 4, Jeon in view of Kwon teaches the method according to claim 3. Jeon in view of Kwon does not teach:
wherein the vocab table comprises a global vocab table and a local vocab table, the global vocab table records a mapping relationship in which a co-occurrence frequency of sequences in the text generated by the LLM is greater than a first frequency threshold, the local vocab table records a mapping relationship in which a co-occurrence frequency of sequences in a text generated for the input text is greater than a second frequency threshold, and the local vocab table is added to the global vocab table after the output text is generated.
However, Flor teaches:
wherein the vocab table comprises a global vocab table and a local vocab table, the global vocab table records a mapping relationship in which a co-occurrence frequency of sequences in the text generated by the LLM is greater than a first frequency threshold, the local vocab table records a mapping relationship in which a co-occurrence frequency of sequences in a text generated for the input text is greater than a second frequency threshold, and the local vocab table is added to the global vocab table after the output text is generated. (see at least Paragraph [0095] “During encoding, the implementation automatically compresses data from Active Tree to Secondary Cache and to disk, and it also moves data in the other direction--when updates are needed for stored n-grams that are no longer in the Active Tree.”; Paragraph [0077] “At step 714, the processing system returns an indication that the query has been matched and retrieves the payload field (e.g., count statistic for that n-gram or some other statistical information or other information, provided that there is a payload (the payload could be null or there could be some other indicator that no data is stored).”; Fig. 3 of Flor shows an active tree and a secondary cache [cashe]; The Examiner notes that in order for a n-gram to be stored, the count statistic would have to be greater than zero, otherwise the value would result in null.)
The motivation for making this modification to the teachings of Jeon is the same as that set forth above, in the rejection of claim 3.
Regarding claim 5, Jeon in view of Kwon teaches the method according to claim 3. Jeon in view of Kwon does not teach:
wherein the mapping relationship in the vocab table corresponds to a score value determined based on a co-occurrence frequency; and
the querying a vocab table by using the current text sequence, to obtain the matching first text sequence comprises:
performing query and matching for the current text sequence in the vocab table; and
when the vocab table comprises a plurality of matching text sequences, determining a text sequence with a highest score value as the first text sequence.
However, Flor teaches:
wherein the mapping relationship in the vocab table corresponds to a score value determined based on a co-occurrence frequency; (see at least Paragraph [0077] “At step 714, the processing system returns an indication that the query has been matched and retrieves the payload field (e.g., count statistic for that n-gram or some other statistical information or other information”; Fig. 7 of Flor) and
the querying a vocab table by using the current text sequence, to obtain the matching first text sequence (Fig. 7 of Flor) comprises:
performing query and matching for the current text sequence in the vocab table; (Fig. 7 of Flor; Examiner notes that step 704 shows that a query is made, steps 706-714 and associated paragraphs show how the method traces the text sequence from the query in a tree to find the nodes that match the current unit in sequential order.) and
when the vocab table comprises a plurality of matching text sequences, determining a text sequence with a highest score value as the first text sequence. (Paragraph [0079] “At step 714, the processing system returns an indication that the query has been matched and retrieves the payload field (e.g., count statistic for that n-gram or some other statistical information or other information, provided that there is a payload (the payload could be null or there could be some other indicator that no data is stored).”; Paragraph [0082] “ranks the relevance of the document to the query, e.g., according to how many times that n-gram is contained in the particular document […] at step 714, the system can retrieve one or more documents responsive to the query (with ranking if desired) as well as a count statistic or other suitable statistical information for that n-gram,”; Fig. 7 of Flor)
The motivation for making this modification to the teachings of Jeon is the same as that set forth above, in the rejection of claim 3.
Additionally, in regard to claim 5, the Examiner further notes the recited “when” on line 6 does not move to distinguish the claimed invention from the cited art. This phrase is a conditional/contingent limitation with the noted “determining a text sequence…” step(s) not necessarily performed. The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. Language that suggests or makes optional but does not require steps to be performed or does not limit a claim to a particular structure does not limit the scope of a claim or claim limitation. [See Ex parte Schulhauser, Appeal 2013-007847 (PTAB April 28, 2016) for an analysis of contingent claim limitations in the context of both method claims and system claims.; MPEP §2111.04 II].
Claim 13:
Claim 13 is directed to an apparatus. Claim(s) 13 recite limitations parallel in nature as those addressed above for claim 3, which are directed towards a method. Claim(s) 13 is therefore rejected for the same reasons as set above for claim 3. Claim 13 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim 14:
Claim 14 is directed to an apparatus. Claim(s) 14 recite limitations parallel in nature as those addressed above for claim 4, which are directed towards a method. Claim(s) 14 is therefore rejected for the same reasons as set above for claim 4. Claim 14 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim 15:
Claim 15 is directed to an apparatus. Claim(s) 15 recite limitations parallel in nature as those addressed above for claim 5, which are directed towards a method. Claim(s) 15 is therefore rejected for the same reasons as set above for claim 5. Claim 15 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jeon (US 20250245430 A1) in view of Kwon (see attached NPL) in further view of Yu (see attached NPL).
Regarding claim 9, Jeon in view of Kwon teaches the method according to claim 8. Jeon further teaches:
wherein before the estimating a first text sequence following a current text sequence based on a speculative decoding method, (see at least Paragraph [0035] “the depth of the tree data structure 110 may be defined a priori.”) the method further comprises:
determining a hardware computing power indicator and (see at least Paragraph [0034] “the draft model may be configured to trigger generation of tokens by the target model (and subsequent speculative decoding) based on various criteria. These criteria may include a complexity criterion or performance criterion, […] these complexity and/or performance criteria may set an upper bound on the number of tokens generated by the draft model for verification by the target model. This upper bound may, in some aspects, be based on a number of nodes in the tree data structure […] The worst-case computational load at the last round of speculative token generation may be configured to be bound by memory bandwidth at the device on which the draft model executes.”; Paragraph [0052] “an input prompt 310 may be received”; Paragraph [0072] “generating the set of tokens includes generating a first subset of tokens corresponding to a first portion of the response to the input prompt based on the input prompt”)
determining a length of the first text sequence based on the available computing power indicator. (see at least Paragraph [0034] “the draft model may be configured to trigger generation of tokens by the target model (and subsequent speculative decoding) based on various criteria. These criteria may include a complexity criterion or performance criterion, […] these complexity and/or performance criteria may set an upper bound on the number of tokens generated by the draft model for verification by the target model. This upper bound may, in some aspects, be based on a number of nodes in the tree data structure and may be influenced, for example, by a branching factor defined for different levels of the tree data structure 110 into which sampled tokens are organized, a depth of the tree data structure 110, or the like. The worst-case computational load at the last round of speculative token generation may be configured to be bound by memory bandwidth at the device on which the draft model executes.”; Paragraph [0035] “the depth of the tree data structure 110 may be defined based on a maximum number of tokens (e.g., words) that can be generated using any pass through the draft model.”)
Jeon in view of Kwon does not teach:
determining a hardware computing power indicator and a first quantity of text units in a target request text, wherein the hardware computing power indicator indicates a maximum quantity of text units whose attention information can be simultaneously computed, and the target request text is a user request text for which a first text unit in an output text is not generated when the iteration starts;
determining an available computing power indicator based on a difference between the maximum quantity of text units and the first quantity.
However Yu teaches:
determining a first quantity of text units in a target request text; (Page 9 “When the scheduler considers a request in the initiation phase, […] the scheduler uses the request’s max_tokens attribute
to reserve max_tokens slots of GPU memory for storing the keys and values in advance“ of Yu)
determining an available computing power indicator based on a difference between the maximum quantity of text units [n_slots] and the first quantity [req.max_tokens]. (see at least Page 9 of Yu “n_slots is a parameter tuned by the ORCA system operator indicating the size of memory region (in terms of slots) allocated to the Attention K/V manager”; Algorithm 1: Orca scheduling Algorithm on Page 9 of Yu shows that the request’s maximum tokens is added to the number of reserved spots (line 24: new_n_rsrv <- n_rsrv + req.max_tokens). If this combined value is more than the number of memory slots available, there is not enough available computing power (line 25: if new_n_rsrv > n_slots then break).)
This step of Yu is applicable to the method of Jeon as they both share characteristics and capabilities, namely, they are directed to managing storage in generative models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to have modified the method of Jeon to incorporate determining a first quantity of text units in a target request text and determining an available computing power indicator as taught by Yu. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to modify Jeon in order to ensure dynamically adding new target requests does not exceed the hardware’s maximum computation limits (see Page 9 of Yu).
Claim 19:
Claim 19 is directed to an apparatus. Claim(s) 19 recite limitations parallel in nature as those addressed above for claim 9, which are directed towards a method. Claim(s) 19 is therefore rejected for the same reasons as set above for claim 9. Claim 19 further recites “at least one processor; and one or more memories coupled to the at least one processor and storing program instructions for execution by at least on processor to performed operations” (see at least Paragraph [0095] “a program memory associated with the CPU” of Jeon)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIELLE ELIZABETH ZEVITZ whose telephone number is (703)756-1070. The examiner can normally be reached Mo-Th 10am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lynda Jasmin can be reached at (571) 272-6782. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIELLE ELIZABETH ZEVITZ/Examiner, Art Unit 3629
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights