DETAILED ACTION
This action is responsive to the application filed on July 29, 2024.
Claims 1-20 are pending and presented to examination.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Information Disclosure Statement
As required by M.P.E.P. 609, the applicant’s submission of the Information Disclosure Statements dated July 29, 2024 and April 23, 2025 are acknowledged by the examiner and the cited references have been considered in the examination of the claims now pending.
Drawings
The drawings filed on July 29, 2024 are acceptable for examination purposes.
Claim Objections
Claims 1-20 are objected to because of the following informalities: Claim 1 (and similar for claims 8 and 15) recites the limitation “activating a verification test to determine whether interactions between the LLM and the genAI application generate expected results;” in lines 6-7. Please amend the claim language as indicated in bold.
Claim 1 (and similar for claims 8 and 15) recites the limitation “defining a new verification test to determine whether interactions between the LLM and the genAI application generate expected results;” in lines 10-11. Please amend the claim language as indicated in bold.
Claim 7 (and similar for claim 14) recites the limitation “comprising, providing to a key user, a list of genAI functions which are impacted by either adjusting the artifacts of the LLM or updating the genAI function of the software application.”. Please remove the “,” comma after “comprising”.
Claim 9 (and similar for claim 16) recites the limitation “wherein the artifacts of [[a]] an LLM include a prompt, input parameters, and configuration.”. Please replace “a” to –an--.
Dependent claims 2-6, 10-13 and 17-20 do not overcome the deficiency of the base claim and, therefore, are objected for the same reasons as the base claim.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3-4, 9-11 and 16-18 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 (and similar for claim 16) recites “wherein the artifacts of a LLM include a prompt, input parameters, and configuration.”. Claim 8 already limits “a large language model (LLM), so is unclear whether claim 9 is introducing a new LLM. For purpose of examination examiner will interpret it as “the LLM”. Appropriate correction or clarification is required. Examiner notes: claim 2 correctly limits it as “the LLM”.
Claim 3 (and similar for claims 10 and 17) recites “wherein verification test start situations include: 1) a new genAI function is activated and needs to be tested; 2) usage of the software application is extended; 3) user behavior changes; 4) a verification test reports a problem; and 5) the LLM is replaced with a different LLM or a version of the LLM is changed.”. Claims 1, 8 and 15 recite “activating a verification test” but never recite any “start situations” so it is unclear what these situations are, whether they are positively required, and how they limit the parent claim. Within the same clause, item 4 (“a verification test reports a problem”) reintroduces “a verification test” even though the claim already recites both “a verification test” and “a new verification test” leaving it unclear which test is referenced. Appropriate correction is required.
Claim 4 (and similar for claims 11 and 18) recites “wherein adjusting the artifacts of the LLM is performed by a key user.” In the parent claims this step is recited only as one branch of “either adjusting the artifacts of the LLM or updating the genAI function” Because the adjusting step is contingent (it need not occur if the “updating” branch is taken), it is unclear whether this dependent limitation is actually required. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention recites a judicial exception, is directed to that judicial exception, an abstract idea, as it has not been integrated into practical application and the claims further do not recite significantly more than the judicial exception. Examiner has evaluated the claims under the framework provided in the 2019 Patent Eligibility Guidance published in the Federal Register 01/07/2019, as updated by the 2024 Guidance Update on Patent Subject Matter Eligibility Including on Artificial Intelligence, 89 Fed. Reg. 58128 (July 17, 2024), and has provided such analysis below.
Step 1: Claims 1-7 are directed to methods and fall within the statutory category of processes, 8-14; Claims 8-14 are directed to media and fall within the statutory category of manufacture, and Claims 15-20 are directed to systems and fall within the statutory category of machines. Therefore, “Are the claims to a process, machine, manufacture or composition of matter?” Yes.
In order to evaluate the Step 2A inquiry “Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?” we must determine, at Step 2A Prong 1, whether the claim recites a law of nature, a natural phenomenon or an abstract idea and further whether the claim recites additional elements that integrate the judicial exception into a practical application.
Step 2A Prong 1:
Claims 1, 8 and 15 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitations: Claim 1 (and similar for claims 8 and 15)
a) “customizing artifacts ” – Mental Process, See MPEP 2106.04(a)(2), III. For example, a person can mentally devise or revise the instructions, inputs, and settings that are to be provided to a model (i.e., a prompt, input parameters, and a configuration). b) “activating a verification test to determine whether interactions ” – Mental Process, See MPEP 2106.04(a)(2), III. For example, a person can mentally compare a produced output against what is expected and judge whether the output meets the expectation.
c) “defining a new verification test to determine whether interactions ” – Mental Process, See MPEP 2106.04(a)(2), III. For example, a person can mentally formulate a new check, together with the expected outcome against which a future output will be judged. That is, nothing in the claim elements precludes the step from practically being performed in the mind or with a pen and paper, (i.e., “customizing”, “determining”, “defining”) can be performed in the human mind though observation, evaluation, judgment, opinion with the aid of pen and paper. Thus, these limitations fall within the “Mental Processes” grouping of abstract ideas. Therefore, Yes, claims 1, 8 and 15 recite judicial exceptions. The claims have been identified to recite judicial exceptions, Step 2A Prong 2 will evaluate whether the claims are directed to the judicial exception. Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claims recite the following additional elements: “a large language model (LLM)”, a “genAI application”, and a “software application”, and (for claims 8 and 15) “a non-transitory, computer-readable medium”, “a computer system”, “one or more computers”, “one or more computer memory devices”, and “instructions”. The additional elements are merely instructions to implement an abstract idea on a computer, or merely using a generic computer or computer components (including a generic large language model) as a tool to perform the abstract idea (see MPEP 2106.05(f)). There is an additional element in the claim. Additional element 1 – “activating, as a generative artificial intelligence (AI) (genAI) application, a genAI function of a software application that interfaces with the LLM and is instrumented for customer-side management;” – mere instructions to implement the abstract idea using a generic computer / generic AI application as a tool (“apply it”), See MPEP 2106.05(f); and the recitation that the application is “instrumented for customer-side management” further limits the abstract idea to a particular field of use, See MPEP 2106.05(h). Additional element 2 – “based on a result of the verification test, either adjusting the artifacts of the LLM or updating the genAI function of the software application;” – mere instructions to apply the result of the abstract evaluation using the generic LLM / software application as a tool (“apply it”), See MPEP 2106.05(f). To the extent this element recites “adjusting the artifacts,” it adds nothing beyond the mental “customizing” already recited in limitation (A). Additional element 3 – “activating the new verification test.” – mere instruction to execute / run the test on a computer, i.e., applying the abstract idea using a generic computer as a tool, See MPEP 2106.05(f). Additional element 4 – the struck-through recitations “of a large language model (LLM)” and “between the LLM and genAI application” in limitations above – the large language model and the genAI application are generic computer / AI components recited as tools used to perform the abstract idea, See MPEP 2106.05(f). Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea, thus failing to integrate the abstract idea into a practical application. Therefore, “Do the claims recite additional elements that integrate the judicial exception into a practical application? No, these additional elements do not integrate the abstract idea into a practical application and they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea. After having evaluating the inquires set forth in Steps 2A Prong 1 and 2, it has been concluded that claims 1, 8 and 15 not only recites a judicial exception but that the claim is directed to the judicial exception as the judicial exception has not been integrated into practical application. Step 2B: As discussed above with respect to integration of the abstract idea into a practical application, the additional elements “a large language model (LLM)”, a “genAI application”, a “software application”, and (for claims 8 and 15) “a computer system”, “one or more computers”, “one or more computer memory devices”, and “instructions” are generic computer components used as tools to perform the abstract idea. Further, the recited customize → test-against-expected-output → adjust → re-test cycle is well-understood, routine, and conventional in the field, and the specification itself characterizes the underlying AI lifecycle management building blocks (activation/deactivation of AI functions and management of LLM versions) as existing functionality (Specification paragraph [0005]). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. After further evaluation, the claim as a whole does not improve any function of a computer or any other technology or technical field. Thus, the claims are not patent eligible. Therefore, “Do the claims recite additional elements that amount to significantly more than the judicial exception? No, these additional elements, alone or in combination, do not amount to significantly more than the judicial exception. Having concluded analysis within the provided framework, claims 1, 8 and 15 do not recite patent eligible subject matter under 35 U.S.C. § 101. Claim 2 (and similar for claims 9 and 16) adds the limitation that “the artifacts of the LLM include a prompt, input parameters, and configuration.” This limitation merely specifies the particular data (a prompt, input parameters, and a configuration) that is the subject of the “customizing” recited in claim 1. Specifying the data operated upon by the mental process does not add an additional element; the limitation remains within the mental-process grouping and does not integrate the exception into a practical application. Claim 3 (and similar for claims 10 and 17) adds the limitation that “verification test start situations include” five enumerated situations (a new genAI function is activated; usage is extended; user behavior changes; a verification test reports a problem; and the LLM is replaced or its version is changed). This limitation merely recites the conditions or circumstances under which the mental verification/evaluation is performed; reciting the field of use or the triggering conditions for the abstract idea does not integrate the exception into a practical application. See MPEP 2106.05(h). Claim 4 (and similar for claims 11 and 18) adds the limitation that “adjusting the artifacts of the LLM is performed by a key user.” This limitation merely specifies that the step is performed by a particular person (a key user), which reinforces that the step is a mental process performed by a human and adds nothing that integrates the exception into a practical application. Claim 5 (and similar for claims 12 and 19) adds the limitation “analyzing a communication log of communications between the genAI application and the LLM.” Analyzing the log is itself an evaluation/judgment within the mental-process grouping; and obtaining and recording the underlying communications (the log of communications between the application and the LLM) is mere pre-solution data gathering, which is insignificant extra-solution activity. See MPEP 2106.05(g). Claim 6 (and similar for claims 13 and 20) adds the limitation that “defining a new verification test … is based, at least partially, on collected user feedback.” This limitation merely identifies a source of data (collected user feedback) used in the abstract step of defining the test, and the collection of that feedback is pre-solution data gathering that is insignificant extra-solution activity. See MPEP 2106.05(g). Claim 7 (and similar for claim 14) adds the limitation “providing to a key user, a list of genAI functions which are impacted by either adjusting the artifacts of the LLM or updating the genAI function of the software application.” This limitation merely recites outputting or displaying the results of the abstract operation (a list) to a user, which is insignificant post-solution output activity that does not integrate the exception into a practical application. See MPEP 2106.05(g). Therefore, Claims 1-20 do not recite patent eligible subject matter under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4-6, 8-9, 11-13, 15-16 and 18-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Auld et al. (US Pub. No. 2024/0311094 – hereinafter Auld).
With respect to claim 1, Auld teaches a computer-implemented method (Auld discloses a computer-implemented method for developing, testing, and deploying a large language model (LLM) solution, performed by an experimentation service and related systems of an LLM solution platform. See Auld, paragraph [0004] (reciting a “method” having the recited steps), paragraph [0032] (the platform “provides an end-to-end life cycle for developing LLM solutions, testing the LLM solutions, and deploying the LLM solutions”), and paragraph [0056] (method 500 “is performed by the experimentation service”)), comprising: customizing artifacts of a large language model (LLM) (Auld discloses customizing the artifacts that adapt the LLM to a particular use case. Auld teaches that “the code block templates are further modified by a user to achieve a specific task for the large language model solution” (paragraph [0058]) and that “[a]n experiment configuration includes identifiers for the specific code blocks being tested, the specific parameter values that drive the behavior of the code blocks” (paragraph [0024]). Because the code blocks build up the LLM’s input prompt (paragraphs [0027], [0046]) and the configuration carries the parameter values that drive the LLM, a user’s modification of these code blocks, parameter values, and configuration is a customizing of the LLM’s artifacts as claimed.) activating, as a generative artificial intelligence (AI) (genAI) application, a genAI function of a software application that interfaces with the LLM and is instrumented for customer-side management (Auld discloses activating a genAI function/application that interfaces with the LLM and is instrumented for customer-side management. Auld teaches that “LLM solutions use one or more LLMs … to enable an AI system that is goaled for a narrower use case or task than the LLM itself” (paragraph [0031]) and that the solution is deployed such that “deployment includes an inference solution and logic to capture inference calls that are forwarded to the data system” (paragraph [0039]). The LLM solution is the recited genAI application/function; it interfaces with the LLM through a model caller that “makes the call to the LLM” (paragraph [0048]); and it is instrumented for customer-side management because the platform “makes it easy for users to connect production environments” and an SDK supports “collecting production inference calls and corresponding feedback from end-users” (paragraph [0040]). Activating the genAI application is met by promoting/deploying the solution into the production environment (paragraphs [0036], [0044], [0063]).) activating a verification test to determine whether interactions between the LLM and genAI application generate expected results (Auld discloses a verification test that determines whether the LLM/application interaction produces expected results. Auld teaches an “evaluator component 310 … [that] compares the final post-processed model output completions against the ground truth provided in the experiment request,” where the experiment request “specifically includes the expected output of the LLM solution” (paragraph [0050]); the model output is also “compare[d] … to a production threshold” by the evaluator (paragraph [0061]), and the packaged solution “runs testing” (paragraph [0036]). Running such an experiment with the evaluator is the recited verification test, and comparing the LLM output against the expected output/threshold is the determination of whether the interaction generates expected results.) based on a result of the verification test, either adjusting the artifacts of the LLM or updating the genAI function of the software application (Auld discloses that, based on the result of the verification test, the LLM artifacts are adjusted (and/or the genAI function is updated). Auld teaches “creating a next experiment with modified code blocks for the large language model solution in response to determining the production threshold is not achieved with the experiment,” where “the modified code blocks include modifications to code of the data blocks” (paragraph [0062]); Auld further teaches “modifying the large language model solution in response to the performance of the LLM … being below a threshold” (paragraph [0066]). Because the test result (threshold not achieved / performance below threshold) drives modification of the code blocks that constitute the LLM artifacts (and modification of the solution), both recited alternatives are met.) defining a new verification test to determine whether interactions between the LLM and genAI application generate expected results (Auld discloses defining a new verification test directed to expected results. Auld teaches that “the user builds a new evaluation set using the experimentation service … [and] the new evaluation set includes code blocks for a large language model solution” (paragraph [0044]), and that each evaluation set/data model is “built around an experiment request (data available at inference+ground truth)” (paragraph [0053]). Because the new evaluation set pairs inputs with the ground-truth/expected outputs against which the LLM interaction is judged, building the new evaluation set is defining a new verification test for expected results.) and activating the new verification test (Auld discloses activating the newly defined verification test. Auld teaches that, after the user builds the new evaluation set, “the experimentation service performs an experiment with the new evaluation set using the LLM … [and] measures performance of the new evaluation set and verifies no regression across the evaluation set” (paragraph [0044]). Performing/running the experiment with the new evaluation set is the recited activating of the new verification test.) With respect to claim 2, Auld teaches wherein the artifacts of the LLM include a prompt, input parameters, and configuration (Auld discloses each recited artifact type. A prompt is taught where a “preprocessor component … gathers and modifies information used to construct an input prompt that is sent to an LLM” (paragraph [0046]); input parameters are taught as “the specific parameter values that drive the behavior of the code blocks” (paragraph [0024]); and configuration is taught where “each component … is defined by a single entry in a YAML configuration” (paragraph [0052]) within the “experiment configuration” (paragraph [0024]). Accordingly, the LLM artifacts include a prompt, input parameters, and configuration.) With respect to claim 4, Auld teaches wherein adjusting the artifacts of the LLM is performed by a key user (Auld discloses that the adjusting is performed by a key user, i.e., the developer/administrator user of the platform. Auld teaches that “the code block templates are further modified by a user to achieve a specific task” (paragraph [0058]) and that “the user exchanges the code … for the static context preprocessor component” (paragraph [0054]); the platform “enables users to describe solution chains” (paragraph [0028]). The user who modifies the code blocks/artifacts is the recited key user performing the adjusting.) With respect to claim 5, Auld teaches comprising analyzing a communication log of communications between the genAI application and the LLM (Auld discloses analyzing a log of the application-to-LLM communications. Auld teaches that the inference service “collects real-world inference calls and user feedback … [which is] fed back to the data system for storage and use in analysis of the deployed LLM solution” (paragraph [0037]) and that the platform will “capture the inference requests from the production setting and persist them for future experimentation” (paragraph [0030]); the data service further “cleans and labels the user data” (paragraph [0043]). The captured/persisted inference calls—the requests sent to, and the responses returned from, the LLM—constitute the recited communication log, and Auld’s use of that data “in analysis” is the recited analyzing.) With respect to claim 6, Auld teaches wherein defining a new verification test to determine whether interactions between the LLM and genAI application generate expected results is based, at least partially, on collected user feedback (Auld discloses defining the new verification test based at least partially on collected user feedback. Auld teaches a flow in which “the user provides a prompt to the inference and feedback service,” which “captures the new prompt and forwards [it] to the data service,” which “cleans and labels the user data” (paragraph [0043]), after which “the user builds a new evaluation set” (paragraph [0044]); Auld further teaches “receiving feedback from end users using the large language model solution” and using “the feedback to measure a performance” (paragraph [0006], [0037], [0065]). Thus, the new verification test (new evaluation set) is defined at least partially from the collected user feedback/data.)
With respect to claim 8, the claim is directed to a non-transitory, computer-readable medium that corresponds to the method recited in claim 1, respectively (see the rejection of claim 1 above; wherein Auld also teaches such medium (Auld discloses a non-transitory, computer-readable medium storing instructions executable by a computer system. See Auld, paragraph [0007] (a device comprising “memory … and instructions stored in the memory, the instructions being executable by the processor to” perform the operations) and paragraph [0042] (the components “include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices”))
With respect to claims 9 and 11-13, the claims are directed to a non-transitory, computer-readable medium that corresponds to the method recited in claims 2 and 4-6, respectively (see the rejection of claims 2 and 4-6 above).
With respect to claim 15, the claim is directed to a system that corresponds to the method recited in claim 1, respectively (see the rejection of claim 1 above; wherein Auld also teaches such system (Auld discloses a computer-implemented system comprising one or more computers and one or more memory devices storing instructions that are executed by the computers to perform the operations. See Auld, paragraph [0005] (a device comprising “a processor; memory in electronic communication with the processor; and instructions stored in the memory”) and paragraphs [0008], [0041] (the LLM solution platform implemented across “one or more computing devices”, including across multiple cloud server devices).)
With respect to claims 16 and 18-20, the claims are directed to a system that corresponds to the method recited in claims 2 and 4-6, respectively (see the rejection of claims 2 and 4-6 above).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3, 10 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Auld et al. (US Pub. No. 2024/0311094) in view of Wanqin Ma et al., (“(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs”, hereinafter Ma – IDS 04/23/2025). With respect to claim 3, Auld teaches wherein verification test start situations include: 1) a new genAI function is activated and needs to be tested (Auld teaches that “the experiment is created in response to a user describing a result for the large language model solution” (paragraph [0057]) and that the user “builds a new evaluation set” for a newly developed solution (paragraph [0044]); thus a newly activated genAI function is tested upon its creation.) 2) usage of the software application is extended (Auld teaches that the platform will “capture the inference requests from the production setting and persist them for future experimentation” (paragraph [0030]); the range of captured production inputs grows as usage of the application is extended, which is then reflected in renewed experimentation/testing.) 3) user behavior changes (Auld teaches that the inference service “collects real-world inference calls and user feedback” that are “fed back to the data system for storage and use in analysis of the deployed LLM solution” (paragraph [0037]); changing user behavior captured in this manner drives renewed verification testing of the solution.) 4) a verification test reports a problem (Auld teaches “creating a next experiment with modified code blocks … in response to determining the production threshold is not achieved with the experiment” (paragraph [0062]); a verification test that reports a problem (i.e., the threshold is not achieved) thus initiates further testing.)
Auld is silent to disclose, however, in an analogous art, Ma teaches 5) the LLM is replaced with a different LLM or a version of the LLM is changed (Ma teaches that “LLM APIs are often updated silently and scheduled to be deprecated, forcing users to continuously adapt to evolving models” (p.1, Abstract) and that, of the studied models, “four out of these five models are already scheduled to be deprecated in 2024, effectively forcing application developers to switch to one of the newer models” (section 3.1.2), where such replacement/version change causes measurable performance regression that the developer must re-test (Fig. 1; section 3.2). Ma thus teaches initiating verification (regression) testing in the situation where the LLM is replaced with a different LLM or a version of the LLM is changed)
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to have included, among the start situations that initiate the verification testing of Auld, the situation in which the LLM is replaced with a different LLM or a version of the LLM is changed, as taught by Ma, in order to detect the performance regression that silent or forced LLM/version updates introduce into a deployed LLM solution, as Ma expressly demonstrates that such updates can cause significant, slice-level performance deterioration that would otherwise go undetected (Ma, section 3.2).
With respect to claim 10, the claim is directed to a non-transitory, computer-readable medium that corresponds to the method recited in claim 3, respectively (see the rejection of claim 3 above).
With respect to claim 17, the claim is directed to a system that corresponds to the method recited in claim 3, respectively (see the rejection of claim 3 above).
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Auld et al. (US Pub. No. 2024/0311094) in view of Eberlein et al. (US Pat. No. 11,379,217, hereinafter Eberlein – IDS 07/29/2024). With respect to claim 7, Auld is silent to disclose, however, in an analogous art, Eberlein teaches comprising, providing to a key user, a list of genAI functions which are impacted by either adjusting the artifacts of the LLM or updating the genAI function of the software application (Auld teaches that “the deployment and promotion service promotes the new solution using the LLM to production” (paragraph [0044]) and modifies/redeploys the LLM solution (paragraphs [0063], [0066]). Eberlein teaches a feature management cockpit providing a vendor cockpit and customer cockpits to manage the lifecycle of vendor-provided software features (column 1 line 64 – column 2 line 6, column 6 lines 47-63). Eberlein teaches that the vendor cockpit publishes “data representative of an update … including a component version … [and] features” that is received by “each of a plurality of customer cockpits” (column 10 lines 17-32, column 16 lines 28-41); that, upon such an update, “the customer can determine which instances and tenants are impacted by the change” through the customer cockpit (column 14 lines 51-59); and that the customer cockpit “provides features to display which component versions and related features are available” (column 10 lines 33-57). Mapping Eberlein’s software “features” to the claimed genAI functions, Eberlein teaches providing to a key user (the customer/administrator) a list of the functions impacted by an update.) It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to have provided, to a key user of the LLM-solution platform of Auld, a list of the genAI functions which are impacted by an adjustment of the LLM artifacts or an update of the genAI function, as taught by Eberlein, in order to enable the key user to identify, schedule, and test at their own controlled timing the functions affected by the change, thereby minimizing the risk and cascading effects (“blast radius”) of an incompatible or defective change, as Eberlein expressly teaches (Eberlein, column 5 lines 16-37, column 14 line 51-59).
With respect to claim 14, the claim is directed to a non-transitory, computer-readable medium that corresponds to the method recited in claim 7, respectively (see the rejection of claim 7 above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Bragaia et al. (US Pub. No. 2025/0103471) The techniques disclosed herein enhance software testing through automated test case generation and implementation. This is accomplished through utilizing a natural language interface and a generative probabilistic model, such as a large language model. A first user input can provide descriptions of a software feature to be tested and various use cases for the software feature. Accordingly, the probabilistic model generates a plurality of test cases for evaluating various aspects of the software feature in accordance with the first user input. The natural language descriptions can be accompanied by a contextual input to constrain the probabilistic model and ensure consistent outputs. Subsequently, a second user input selects a test case from the generated test cases to be implemented. In response, the probabilistic model generates a software implementation of the selected test case. The software implementation is then executed with log data extracted from the testing environment for processing results. (see abstract).
Sen (US Pub. No. 2025/0077397) An example operation may include one or more of generating a large language model via a user interface, executing the large language model on a repository of software test cases and requirements of the software test cases to train the large language model to understand connections between test case components and test case requirements, receiving a description of features of a software program to be tested, and in response to receiving the description of the features, generating a software test case based on execution of the large language model on the received descriptions, and displaying the software test case via the user interface. (see abstract). Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANIBAL RIVERACRUZ whose telephone number is (571)270-1200. The examiner can normally be reached Monday-Friday 9:30 AM-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hyung S Sough can be reached at 5712726799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANIBAL RIVERACRUZ/Primary Examiner, Art Unit 2192
Prosecution Insights