Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5 and 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “a point where a rate of decrease in the sum of the squared error factor starts to slow down” in claims 5 and 15 is a relative term which renders the claim indefinite. The term “a point where a rate of decrease in the sum of the squared error factor starts to slow down” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
The specification points to a plot of the sum of the squared error factor values in Fig. 4, where point 407 is described as “the point where the sum of the squared error factor starts to slow down significantly”. However, it is unclear how one of ordinary skill in the art would determine the exact point on this plot where the curve “starts to slow down significantly”. The specification does not provide any disclosure that describes how “the point where the sum of the squared error factor starts to slow down significantly” is calculated. It is further unclear where on the illustrated curve the slow down would be considered “significant” or where this significant slow down “starts”.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 7-14, and 17-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Rezaeian et al. (U.S. Patent Application Pub. No. 2025/0371247, hereinafter “Rezaeian”).
In regard to claim 1, Rezaein discloses a method (Fig. 3, 300) for estimating an optimal size for a context window of a large language model (LLM), the method comprising:
receiving input data for performing a specific task, wherein the input data has a size more than a threshold size of the context window (a large corpus of input text is input, paragraph [0047]; with a size larger than a context window of an LLM, paragraph [0002]);
iteratively computing the optimal size, for the context window, such that the optimal size fits within the threshold size of the context window (the process is repeated until the total size of reduced subsets is less than the size of the context window, paragraph [0055]), wherein the iterative computation of the optimal size comprises:
splitting the input data into a plurality of segments, wherein each of the plurality of segments is split into a first size that is less than the threshold size of the context window (step 310, a plurality of portions of the input data is identified, paragraphs [0017] and [0048]);
determining an optimal number of one or more groups of segments, among the plurality of segments, wherein each of the one or more groups of segments is semantically close to each other (step 330, a cluster generator determines a number of clusters depending on the size of the input data, paragraphs [0021-0024] and [0050]);
obtaining, based on clustering of the one or more groups of segments, one or more clusters of segments having a vector similarity within each of the one or more clusters of segments (step 330, the clustering generator clusters embeddings of the portions together, based on a distance measurement between the embedding vectors of the portions, paragraphs [0021-0024] and [0050]);
selecting, from each of the one or more clusters of segments, a representative segment that is closest to a center of each of the one or more clusters of segments to obtain a set of representative segments (step 340, for each cluster, the center-most embeddings are selected, paragraph [0051]);
performing the specific task by passing each representative segment, of the set of representative segments, through the LLM to obtain a first output corresponding to each of the representative segment (step 340, the center-most clusters are provided to the language model to perform the task of summarization, paragraphs [0051-0052]); and
obtaining, based on combining the first output, a second output of a specific size (step 350, a final summary is generated based on the set of cluster summaries using a map-reduce algorithm, paragraph [0053]); and
estimating that the specific size is the optimal size for the context window for performing the specific task using the LLM based on a result of the iterative computation where the specific size of the second output fits within the threshold size of the context window of the LLM after an iteration (the process is repeated until the total size of the reduced subsets is less than the size of the context window, paragraphs [0055-0056]).
In regard to claim 2, Rezaein discloses estimating that the specific size is the optimal size for the context window comprises:
comparing the specific size of the second output with the threshold size of the context window (the total size is compared to the size of the context window, paragraph [0055]);
determining that the specific size of the second output fits within the threshold size of the context window based on a result of the comparison (the total size is determined to be less than the size of the context window, paragraph [0056]); and
estimating the specific size of the second output as the optimal size of the context window (the final size is used as the size of the context window, paragraph [0056]).
In regard to claim 3, Rezaein discloses determining that the specific size of the second output does not fit within the threshold size of the context window based on the result of the comparison (the total size is determined to not be less than the size of the context window, paragraph [0055]), the method further comprising:
repeating the iterative computation of the optimal size for the context window, wherein the second output from a previous iteration is utilized as the input data for a next iteration (the process is repeated until the total size is determined to be less than the size of the context window, paragraph [0055]).
In regard to claim 4, Rezaien discloses extracting a plurality of features from each of the plurality of segments (each chunk is input to an embedding generator to determine a plurality of values, paragraph [0019]);
representing, using vector embedding techniques, the plurality of features in each of the plurality of segments as a plurality of data points in a vector (an embedding vector is generated for each chunk, paragraph [0019]); and
obtaining a plurality of vectors representing the plurality of features in the plurality of segments in a vector space (see Fig. 1, a plurality of embedding vectors 132 are generated and input to the cluster generator, paragraph [0020]).
In regard to claim 7, Rezaien discloses obtaining the one or more clusters of segments having the vector similarity within each of the one or more clusters of segments comprises:
determining the vector similarity between the one or more groups of segments using clustering techniques (the cluster generator assigns each embedding vector to a single cluster, paragraph [0021]);
clustering the one or more groups of segments using clustering techniques (the cluster generator generates clusters based on distance measures between each embedding and one or more other embeddings, paragraph [0021]); and
obtaining the one or more clusters of segments having the vector similarity within each of the one or more clusters of segments based on a result of the determination (the clusters are generated based on embeddings having a minimum distance (maximum similarity), paragraphs [0021-0022]).
In regard to claim 8, Rezaien discloses invoking the LLM, for performing the specific task using the second output, with the optimal size of the context window based on the estimation that the specific size of the second output is the optimal size (once the total size is less than the context window size, the subsets are provided to the LLM, paragraphs [0045] and [0056]); and
performing the specific task by passing the second output into the LLM as a dataset (the LLM performs a final summarization as the specific task, paragraphs [0045] and [0056]).
In regard to claim 9, Rezaien discloses the second output includes segments with non- redundant input data (a similarity measurement is calculated between an embedding of a current summary for a cluster and an embedding of the summary that was most recently generated, so that if consecutive similarities do not change much, then no more chunks are added to the summary for the cluster, thus ensuring any added chunks are non-redundant, paragraphs [0030-0032]).
In regard to claim 10, Rezaien discloses the threshold size of the context window is the size that the LLM is capable of successfully processing the input data (the maximum size is the size of the context window of the LLM, paragraphs [0025-0026]).
In regard to claim 11, Rezaien discloses a system (Fig. 4, 400) for estimating an optimal size for a context window of a large language model (LLM), comprising:
one or more processors (404, paragraph [0058]);
a memory (406, paragraph [0059]); and
one or more programs stored in the memory (instruction to be executed by the processor, paragraph [0059]), the one or more programs when executed by the one or more processors cause the one or more processors to:
receive input data for performing a specific task, wherein the input data has a size more than a threshold size of the context window (a large corpus of input text is input, paragraph [0047]; with a size larger than a context window of an LLM, paragraph [0002]);
iteratively compute the optimal size, for the context window, such that the optimal size fits within the threshold size of the context window (the process is repeated until the total size of reduced subsets is less than the size of the context window, paragraph [0055]), wherein the iterative computation of the optimal size comprises:
split the input data into a plurality of segments, wherein each of the plurality of segments is split into a first size that is less than the threshold size of the context window (step 310, a plurality of portions of the input data is identified, paragraphs [0017] and [0048]);
determine an optimal number of one or more groups of segments, among the plurality of segments, wherein each of the one or more groups of segments is semantically close to each other (step 330, a cluster generator determines a number of clusters depending on the size of the input data, paragraphs [0021-0024] and [0050]);
obtain, based on clustering of the one or more groups of segments, one or more clusters of segments having a vector similarity within each of the one or more clusters of segments (step 330, the clustering generator clusters embeddings of the portions together, based on a distance measurement between the embedding vectors of the portions, paragraphs [0021-0024] and [0050]);
select, from each of the one or more clusters of segments, a representative segment that is closest to a center of each of the one or more clusters of segments to obtain a set of representative segments (step 340, for each cluster, the center-most embeddings are selected, paragraph [0051]);
perform the specific task by passing each representative segment, of the set of representative segments, through the LLM to obtain a first output corresponding to each of the representative segment (step 340, the center-most clusters are provided to the language model to perform the task of summarization, paragraphs [0051-0052]); and
obtain, based on combining the first output, a second output of a specific size (step 350, a final summary is generated based on the set of cluster summaries using a map-reduce algorithm, paragraph [0053]); and
estimate that the specific size is the optimal size for the context window for performing the specific task using the LLM based on a result of the iterative computation where the specific size of the second output fits within the threshold size of the context window of the LLM after an iteration (the process is repeated until the total size of the reduced subsets is less than the size of the context window, paragraphs [0055-0056]).
In regard to claim 12, Rezaein discloses to estimate that the specific size is the optimal size for the context window, the one or more processors are configured to:
compare the specific size of the second output with the threshold size of the context window (the total size is compared to the size of the context window, paragraph [0055]);
determine that the specific size of the second output fits within the threshold size of the context window based on a result of the comparison (the total size is determined to be less than the size of the context window, paragraph [0056]); and
estimate the specific size of the second output as the optimal size of the context window (the final size is used as the size of the context window, paragraph [0056]).
In regard to claim 13, Rezaein to determine that the specific size of the second output does not fit within the threshold size of the context window based on the result of the comparison (the total size is determined to not be less than the size of the context window, paragraph [0055]), the one or more processors are further configured to:
repeat the iterative computation of the optimal size for the context window, wherein the second output from a previous iteration is utilized as the input data for a next iteration (the process is repeated until the total size is determined to be less than the size of the context window, paragraph [0055]).
In regard to claim 14, Rezaien discloses the one or more processors are further configured to:
extract a plurality of features from each of the plurality of segments (each chunk is input to an embedding generator to determine a plurality of values, paragraph [0019]);
represent, using vector embedding techniques, the plurality of features in each of the plurality of segments as a plurality of data points in a vector (an embedding vector is generated for each chunk, paragraph [0019]); and
obtain a plurality of vectors representing the plurality of features in the plurality of segments in a vector space (see Fig. 1, a plurality of embedding vectors 132 are generated and input to the cluster generator, paragraph [0020]).
In regard to claim 17, Rezaien discloses to obtain the one or more clusters of segments having the vector similarity within each of the one or more clusters of segments, the one or more processors are configured to:
determine the vector similarity between the one or more groups of segments using clustering techniques (the cluster generator assigns each embedding vector to a single cluster, paragraph [0021]);
cluster the one or more groups of segments using clustering techniques (the cluster generator generates clusters based on distance measures between each embedding and one or more other embeddings, paragraph [0021]); and
obtain the one or more clusters of segments having the vector similarity within each of the one or more clusters of segments based on a result of the determination (the clusters are generated based on embeddings having a minimum distance (maximum similarity), paragraphs [0021-0022]).
In regard to claim 18, Rezaien discloses the one or more processors are further configured to:
invoke the LLM, for performing the specific task using the second output, with the optimal size of the context window based on the estimation that the specific size of the second output is the optimal size (once the total size is less than the context window size, the subsets are provided to the LLM, paragraphs [0045] and [0056]); and
perform the specific task by passing the second output into the LLM as a dataset (the LLM performs a final summarization as the specific task, paragraphs [0045] and [0056]).
In regard to claim 19, Rezaien discloses the second output includes segments with non- redundant input data (a similarity measurement is calculated between an embedding of a current summary for a cluster and an embedding of the summary that was most recently generated, so that if consecutive similarities do not change much, then no more chunks are added to the summary for the cluster, thus ensuring any added chunks are non-redundant, paragraphs [0030-0032]).
In regard to claim 20, Rezaien discloses a non-transitory computer-readable storage medium storing program instructions for estimating an optimal size for a context window of a large language model (LLM) (paragraph [0063]), the program instructions, when executed, perform the steps of:
receiving input data for performing a specific task, wherein the input data has a size more than a threshold size of the context window (a large corpus of input text is input, paragraph [0047]; with a size larger than a context window of an LLM, paragraph [0002]);
iteratively computing the optimal size, for the context window, such that the optimal size fits within the threshold size of the context window (the process is repeated until the total size of reduced subsets is less than the size of the context window, paragraph [0055]), wherein the iterative computation of the optimal size comprises:
splitting the input data into a plurality of segments, wherein each of the plurality of segments is split into a first size that is less than the threshold size of the context window (step 310, a plurality of portions of the input data is identified, paragraphs [0017] and [0048]);
determining an optimal number of one or more groups of segments, among the plurality of segments, wherein each of the one or more groups of segments is semantically close to each other (step 330, a cluster generator determines a number of clusters depending on the size of the input data, paragraphs [0021-0024] and [0050]);
obtaining, based on clustering of the one or more groups of segments, one or more clusters of segments having a vector similarity within each of the one or more clusters of segments (step 330, the clustering generator clusters embeddings of the portions together, based on a distance measurement between the embedding vectors of the portions, paragraphs [0021-0024] and [0050]);
selecting, from each of the one or more clusters of segments, a representative segment that is closest to a center of each of the one or more clusters of segments to obtain a set of representative segments (step 340, for each cluster, the center-most embeddings are selected, paragraph [0051]);
performing the specific task by passing each representative segment, of the set of representative segments, through the LLM to obtain a first output corresponding to each of the representative segment (step 340, the center-most clusters are provided to the language model to perform the task of summarization, paragraphs [0051-0052]); and
obtaining, based on combining the first output, a second output of a specific size (step 350, a final summary is generated based on the set of cluster summaries using a map-reduce algorithm, paragraph [0053]); and
estimating that the specific size is the optimal size for the context window for performing the specific task using the LLM based on a result of the iterative computation where the specific size of the second output fits within the threshold size of the context window of the LLM after an iteration (the process is repeated until the total size of the reduced subsets is less than the size of the context window, paragraphs [0055-0056]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5-6 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rezaien, in view of Rokach et al. (Clustering Methods, hereinafter “Rokach”).
In regard to claim 5, Rezaien discloses the optimal number of the one or more groups of segments are determined by optimal clusters determination techniques, wherein the optimal clusters determination techniques comprises:
obtaining one or more clusters of data points having semantically close data points based on an application of clustering techniques on the plurality of vectors (cluster generator generates multiple clusters of embeddings where the text of the embeddings are more likely to be related to each other topic-wise or subject-wise, paragraph [0021]).
Rezaien further discloses the number of clusters may vary (paragraph [0023]), but is silent as to further details regarding selecting the optimal number of clusters.
Rokach discloses optimal clusters determination techniques, wherein the optimal clusters determination techniques comprise:
calculating a sum of a squared error factor for each of the one or more clusters of data points (a sum of squared error is calculated to measure the compactness of each cluster, sections 4.1 and 4.1.1); and
determining, from the one or more clusters of data points, the optimal number of the one or more groups of segments having an optimal compactness of the data points within each of the one or more clusters of data points based on a point where a rate of decrease in the sum of the squared error factor starts to slow down (to determine the optimal number of clusters, the within-cluster depression-decay is calculated and the number of clusters K is selected where the within-cluster decay curve flattens, section 7.1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to determine the optimal number of the one or more groups of segments based on a point where a rate of decrease in the sum of the squared error factor starts to slow down, because it would relieve a user from having to select a pre-set number of clusters, which affects the performance of the clustering algorithm significantly, as taught by Rokach (section 7).
In regard to claim 6, Rezaien does not disclose the sum of the squared error factor represents the sum of a squared distance between each of the plurality of data points within a cluster of data points among the one or more clusters of data points and a corresponding centroid of the cluster of data points.
Rokach discloses the sum of the squared error factor represents the sum of a squared distance between each of the plurality of data points within a cluster of data points among the one or more clusters of data points and a corresponding centroid of the cluster of data points (sum of squared distances between the data instances and their cluster centers, section 7.1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the sum of a squared distance between each of the plurality of data points within a cluster of data points among the one or more clusters of data points and a corresponding centroid of the cluster of data points, because it would relieve a user from having to select a pre-set number of clusters, which affects the performance of the clustering algorithm significantly, as taught by Rokach (section 7).
In regard to claim 15, Rezaien discloses the optimal number of the one or more groups of segments are determined by optimal clusters determination techniques, wherein for the optimal clusters determination techniques the one or more processors are configured to:
obtain one or more clusters of data points having semantically close data points based on an application of clustering techniques on the plurality of vectors (cluster generator generates multiple clusters of embeddings where the text of the embeddings are more likely to be related to each other topic-wise or subject-wise, paragraph [0021]).
Rezaien further discloses the number of clusters may vary (paragraph [0023]), but is silent as to further details regarding selecting the optimal number of clusters.
Rokach discloses optimal clusters determination techniques, wherein the optimal clusters determination techniques comprise:
calculating a sum of a squared error factor for each of the one or more clusters of data points (a sum of squared error is calculated to measure the compactness of each cluster, sections 4.1 and 4.1.1); and
determining, from the one or more clusters of data points, the optimal number of the one or more groups of segments having an optimal compactness of the data points within each of the one or more clusters of data points based on a point where a rate of decrease in the sum of the squared error factor starts to slow down (to determine the optimal number of clusters, the within-cluster depression-decay is calculated and the number of clusters K is selected where the within-cluster decay curve flattens, section 7.1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to determine the optimal number of the one or more groups of segments based on a point where a rate of decrease in the sum of the squared error factor starts to slow down, because it would relieve a user from having to select a pre-set number of clusters, which affects the performance of the clustering algorithm significantly, as taught by Rokach (section 7).
In regard to claim 16, Rezaien does not disclose the sum of the squared error factor represents the sum of a squared distance between each of the plurality of data points within a cluster of data points among the one or more clusters of data points and a corresponding centroid of the cluster of data points.
Rokach discloses the sum of the squared error factor represents the sum of a squared distance between each of the plurality of data points within a cluster of data points among the one or more clusters of data points and a corresponding centroid of the cluster of data points (sum of squared distances between the data instances and their cluster centers, section 7.1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the sum of a squared distance between each of the plurality of data points within a cluster of data points among the one or more clusters of data points and a corresponding centroid of the cluster of data points, because it would relieve a user from having to select a pre-set number of clusters, which affects the performance of the clustering algorithm significantly, as taught by Rokach (section 7).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yuan et al., Agarwal et al., Russak et al., Lintz et al., Bignell et al., Ratner et al., Yin et al., Zhang et al., Li et al., and Zhou et al. disclose additional methods of determining optimal context window sizes.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIAN LOUIS ALBERTALLI whose telephone number is (571)272-7616. The examiner can normally be reached M-F 8AM-3PM, 4PM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
BLA 10/9/26
/BRIAN L ALBERTALLI/Primary Examiner, Art Unit 2656