Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/18/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-9 and 11-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
A broad range or limitation together with a narrow range or limitation that falls within the broad range or limitation (in the same claim) may be considered indefinite if the resulting claim does not clearly set forth the metes and bounds of the patent protection desired. See MPEP § 2173.05(c). In the present instance, claim 1 recites the broad recitation "the relationship is one of a plurality of pre-existing relationships, which indicates that the relationship satisfies the determination when it matches any one member of the plurality. Claim 1 subsequently recites returning a result responsive to “the relationship failing to match at least one of the plurality of pre-existing relationships”, which is satisfied whenever the relationship does not match one or more members of the plurality.
Accordingly, when the plurality includes multiple pre-existing relationships, a relationship that matches one member of the plurality but does not match another member would both be “one of” the plurality and “fail[ ] to match at least one” member of the plurality. It is therefore unclear whether the result is returned only when the relationship fails to match every pre-existing relationship, or whether the result is returned whether the relationship fails to match any single member despite matching another member.
Claims 2-9 depend directly or indirectly from claim 1 and incorporate the indefinite limitation. Claim 11 recites a corresponding limitation. Independent claim 12 recites a corresponding triggering condition, and claims 13-20 depend directly or indirectly from claim 12 and incorporate that limitation.
Claim 12 is further indefinite because claim 12 recites “a cluster assignment model executable by the processor to a combination of the encoded test input, the encoded test output, and the plurality of clusters”.
The phrase “executable by the processor to a combination” does not clearly identity the operation performed by the cluster assignment model with respect to the recited combination. In particular, it is unclear whether the cluster assignment model is executable on the combination, executable to receive the combination as an input, executable to apply the cluster assignment model to the combination, or executable to perform some other operation with respect to the combination.
Although claim 12 subsequently recites that the cluster assignment model identifies an input cluster and an output cluster, that subsequent recitation does not resolve how the encoded test input, the encoded test output, and the plurality of clusters are operatively related to the cluster assignment model. Accordingly, the metes and bounds of the recited cluster assignment model limitation cannot be determined with reasonable certainty.
Claims 13-20 depend directly or indirectly from claim 12 and incorporate the indefinite limitation.
Claim 12 is further indefinite because claim 12 recites a data repository storing “an input cluster and an output cluster, each comprising one of the plurality of clusters”.
The term “comprising” ordinarily indicates inclusion or containment. Thus, the recited language indicates that the input cluster and the output cluster each include or contain one of the plurality of clusters. However, claim 12 subsequently recites that the cluster assignment model identifies “the input cluster to which the encoded test input belongs and the output cluster to which the encoded test output belongs”, which appears to treat the input cluster and the output cluster as respective members selected from the plurality of clusters.
It is therefore unclear whether the input cluster and the output cluster are themselves respective members of the plurality of clusters, or whether the input cluster and the output are separate structures that each contain a respective member of the plurality of clusters. These interpretations require materially different structural relationships and result in different claim scope. Accordingly, the relationship between the input cluster, the output cluster, and the plurality of clusters cannot be determined with reasonable certainty.
Claims 13-20 depend directly or indirectly from claim 12 and incorporate this indefinite limitation.
Claims 4 and 15 are further indefinite because each claim recites “adjusting, responsive to the relationship failing to match the at least one of the plurality of pre-existing relationships, adjusted training data”.
The phrase “adjusting … adjusted training data” does not clearly identify the data upon which the adjusting operation is performed or the result produced by the adjusting operation. In particular, it is unclear whether the claims require adjusting original training data to generate adjusted training data, or instead require performing an additional adjustment on training data that has already been adjusted.
The subsequent recitation of retraining the test large language model using the adjusted training data establishes that adjusted training data is used during retraining, but does not resolve whether the adjusted training data is the input to the adjusting operation or the output generated by the adjusted operation. Accordingly, one or ordinary skill in the art cannot determine with reasonable certainty what data is adjusted or what result is produced by the recited adjusting step.
Allowable Subject Matter
Claim 10 is allowed.
The following is a statement for the indication of allowable subject matter:
Claim 10 recites, inter alia:
“applying a stable large language model to a plurality of past inputs to generate a plurality of past outputs”;
“applying an encoding model to the plurality of past inputs and the plurality of past outputs to generate a plurality of past encoded inputs and a plurality of past encoded outputs”;
“applying a clustering model to the plurality of past encoded inputs and the plurality of past encoded outputs to generate a plurality of clusters”;
“applying a mapping model to the plurality of clusters to generate mapping data defining a plurality of mappings among the plurality of clusters”; and
“determining a plurality of pre-existing relationships among the plurality of clusters from the plurality of mappings”.
The closest prior art of record teaches various individual aspects of the claimed model. For example, Lopatecki (US11775871B1) teaches generating embedding representations, clustering embedded training and production data, detecting drift within clusters, and using drifted data to optimize a machine-learning model. Singh (US20210397610A1) teaches obtaining paired queries and responses, converting textual query and response data to vector representations, and clustering response representations and associated query representations. Kuhn et al. (Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation) teaches applying a language model to inputs, generating natural-language outputs, and grouping generated outputs according to semantic meaning. Rabinovich (US12499878B2) teaches vectorizing textual utterances, clustering novel topics, detecting drift, updating training data, and retraining a machine-learning model.
However, the prior art or record, considered individually and in combination, does not teach or suggest the claimed sequence of applying a stable large language model to past inputs to generate corresponding past outputs, encoding both the past inputs and the corresponding past outputs, generating clusters from both sets of encoded data, applying a mapping model to the clusters to generate mapping data defining mappings among the clusters, and determining pre-existing relationships among the clusters from the generated mapping data.
In particular, the prior art does not teach or suggest generating a historical model-behavior baseline comprising mapping relationships between clusters derived from encoded stable-large-language-model inputs and clusters derived from the corresponding encoded stable-large-language-model outputs. Although the prior art teaches embeddings, semantic clustering, model-drift detection, and paired query-response processing separately, the prior art does not provide an articulated teaching or suggestion to combine those features according to the particular mapping and relationship-generation architecture recited in claim 10.
Accordingly, the foregoing combination is considered to distinguish claim 10 patentably over the closest prior art of record.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Paul Coleman whose telephone number is (571)272-4687. The examiner can normally be reached Mon-Fri.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL COLEMAN/ Examiner, Art Unit 2126
/DAVID YI/ Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights