Notice of Pre-AIA or AIA Status
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This is the initial office action based on the application filed on October 03rd, 2024, which claims 1-20 are presented for examination.
Status of Claims
3. Claims 1-20 are pending, of which claims, of which claim 1, 10 and 16 are in independent form.
Priority
4. The instant application has a foreign priority INDIA 202411044834 06/10/2024.
Information Disclosure Statement
5. Information disclosure statement filed on 10/03/2024 and 12/16/2025, have been reviewed and considered by Examiner.
The Office's Note:
6. The Office has cited particular paragraphs / columns and line numbers in the reference(s) applied to the claims above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to specific limitations within the individual claim(s), other passages and figures may apply as well. It is respectfully requested from the Applicant in preparing responses, to fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the cited passages as taught by the prior art or relied upon by the Examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
7. Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claims 1-9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites “obtaining a request written at least partially in a natural language; determining an extent to which a software analyzer meets a requirement of the request, wherein the extent is a numeric value or an enumeration value; selecting a path, by (a) when the extent satisfies a threshold condition, selecting a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, and (b) when the extent does not satisfy the threshold condition, selecting a second path which specifies a second execution which executes at least one machine learning model; executing the selected path, including computationally performing software analysis work; and providing, via a user interface, a result of executing the selected path.” as drafted, are functions that, under its broadest reasonable interpretation, recite the abstract idea of a mental process. The limitations encompass a human mind carrying out the function through observation, evaluation judgment and /or opinion, or even with the aid of pen and paper. Thus, this limitation recites and falls within the “Mental Processes” grouping of abstract ideas under Prong 1.
Under Prong 2, this judicial exception is not integrated into a practical application. The additional elements ““memory”, and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “selecting a path, by (a) when the extent satisfies a threshold condition, selecting a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, and (b) when the extent does not satisfy the threshold condition, selecting a second path which specifies a second execution which executes at least one machine learning model.” do nothing more than add insignificant extra solution activity to the judicial exception of merely gathering, displaying, updating, transmitting and storing data/information. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g).
Under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of ““memory,” and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “ selecting a path, by (a) when the extent satisfies a threshold condition, selecting a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, and (b) when the extent does not satisfy the threshold condition, selecting a second path which specifies a second execution which executes at least one machine learning model”, the courts have identified merely gathering, displaying, updating, transmitting and storing data/information on a display is well-understood, routine and conventional activity. See MPEP 2106.05(d). The recitation of generic computer instruction and computer components to apply the judicial exception, and merely displaying data do not amount to significantly more, thus, cannot provide an inventive concept. Accordingly, the claims are not patent eligible under 35 USC 101.
Claim 10-15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 10 recites “ (a) obtaining a request written at least partially in a natural language, (b) determining an extent to which a software analyzer meets a functionality requirement of the request, (c) selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, or a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model executed for selecting the path, (d) triggering a performance of the path, the performance including computational software analysis work, and (e) providing, via the user interface, a result of the performance of the path.” as drafted, are functions that, under its broadest reasonable interpretation, recite the abstract idea of a mental process. The limitations encompass a human mind carrying out the function through observation, evaluation judgment and /or opinion, or even with the aid of pen and paper. Thus, this limitation recites and falls within the “Mental Processes” grouping of abstract ideas under Prong 1.
Under Prong 2, this judicial exception is not integrated into a practical application. The additional elements ““memory”, and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “(c) selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, or a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model executed for selecting the path.” do nothing more than add insignificant extra solution activity to the judicial exception of merely gathering, displaying, updating, transmitting and storing data/information. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g).
Under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of ““memory,” and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “ (c) selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, or a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model executed for selecting the path”, the courts have identified merely gathering, displaying, updating, transmitting and storing data/information on a display is well-understood, routine and conventional activity. See MPEP 2106.05(d). The recitation of generic computer instruction and computer components to apply the judicial exception, and merely displaying data do not amount to significantly more, thus, cannot provide an inventive concept. Accordingly, the claims are not patent eligible under 35 USC 101.
Claims 16-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 16 recites “ obtaining a request written at least partially in a natural language, the obtaining performed via a module plug interface of a development tool module and a module socket interface of a software development tool, the module plug interface adapted to the module socket interface, the development tool module external to the software development tool; determining, via an artificial intelligence model, an extent to which a software analyzer meets a functionality requirement of the request; selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, or a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model execution while selecting the path; triggering a performance of the path, the performance including computational software analysis work; and providing the software development tool with a result of the performance of the path.” as drafted, are functions that, under its broadest reasonable interpretation, recite the abstract idea of a mental process. The limitations encompass a human mind carrying out the function through observation, evaluation judgment and /or opinion, or even with the aid of pen and paper. Thus, this limitation recites and falls within the “Mental Processes” grouping of abstract ideas under Prong 1.
Under Prong 2, this judicial exception is not integrated into a practical application. The additional elements ““memory”, and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, or a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model execution while selecting the path” do nothing more than add insignificant extra solution activity to the judicial exception of merely gathering, displaying, updating, transmitting and storing data/information. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g).
Under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of ““memory,” and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “ selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model, or a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model execution while selecting the path”, the courts have identified merely gathering, displaying, updating, transmitting and storing data/information on a display is well-understood, routine and conventional activity. See MPEP 2106.05(d). The recitation of generic computer instruction and computer components to apply the judicial exception, and merely displaying data do not amount to significantly more, thus, cannot provide an inventive concept. Accordingly, the claims are not patent eligible under 35 USC 101.
In conclusion, claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Jain (US 12,106,205 – hereinafter Jain) and further in view of Athiwaratkum (US 20230418566 – hereinafter Athiwaratkum).
Claim 1 rejected, Jain teaches a software development method performed by a computing system, the method comprising automatically (Jain, abstract and summary):
obtaining a request written at least partially in a natural language (column 6, line 9 to 27, processing an input (e.g., a natural language prompt). Column 12, line 15-37, The prompt can include a natural language question or command (e.g., in English). For example, the prompt includes a request for an LLM to generate code (e.g., within a specified programming language) that executes a particular operation. Fig. 11, column 30, line 59 to column 31, line 8, At act 1102, the process 1100 can receive an input for generation of an output using a model. For example, the process 1100 receives, from a user device, an output generation request comprising an input (e.g., prompt) for generation of an output using a first model (e.g., an LLM) of a plurality of models. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) receives a prompt indicating a desired output, such as a text-based instruction for the generation of software-related code samples (e.g., associated with a particular function). The output generation request can include an indication of a selected model (e.g., LLM) for processing the prompt. As such, the data generation platform 102 can evaluate the effect of generating an output using the selected model based on the prompt (e.g., or other suitable inputs) on the basis of the content or nature of the request (e.g., based on a user identifier associated with the request).);
determining an extent to which a software analyzer meets a requirement of the request, wherein the extent is a numeric value or an enumeration value (column 4, line 10 to 30, The data generation platform can compare the performance metric value with an associated threshold or criterion. For example, the data generation platform determines that the estimated system resources required to process the prompt through the associated LLM is less than an allotment assigned to the user. As such, the data generation platform can proceed to provide the prompt to the LLM for generation of the requested output. In some implementations, the data generation platform further evaluates the output for accuracy, security, safety (e.g., with respect to associated policies, requirements, or criteria), compliance (e.g., compliance with regulations, rules, guidelines, etc.), and/or other requirements/recommendations. As an illustrative example, the data generation platform tests any generated code within a virtual machine or another suitable isolated environment to determine any security risks of the generated code. In response to validating the generated output, the data generation platform can transmit this information to an associated data store or deployment system (e.g., any relevant consumer of the generated data, such as a server that is accessible to the user). Fig. 11, column 31, line 9 to line 25, At act 1104, the process 1100 can determine a performance metric associated with processing the output generation request. For example, the process 1100 determines a performance metric associated with processing the output generation request. As an illustrative example, the data generation platform 102 can determine one or more performance metrics that characterize the behavior of the system (e.g., when providing inputs to a model for generation of an output). Such performance metrics can include CPU utilization, cost (e.g., associated with the operation of the system and/or the associated models), memory usage, storage space, and/or number of input or output tokens associated with LLMs. In some implementations, the data generation platform 102 (e.g., through the performance engine 118) determines multiple performance metrics (e.g., associated with the system state) for evaluation of the effects (e.g., of generating an output based on the prompt) on the system.);
selecting a path, by (a) when the extent satisfies a threshold condition, selecting a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model (column 4, line 41 to 63, the disclosed data generation platform enables evaluation of the user's prompt in a flexible, modular manner. For example, the data generation platform determines which prompt validation rules, criteria, or models with which to evaluate the user's prompt (e.g., based on the identity of the user, the nature of the prompt, and/or other suitable factors). Based on this determination, the data generation platform can evaluate the prompt with respect to relevant criteria, while avoiding the need to evaluate the prompt against unsuitable or unrelated criteria. In some implementations, the data generation platform evaluates the performance requirements associated with the prompt generate a recommendation for a suitable LLM for the received prompt (e.g., to improve the efficiency of system resource use). In some implementations, the data generation platform enables evaluation of model outputs in a flexible, modular manner (e.g., depending on the type of output). By doing so, the system can mitigate inaccuracies, security breaches, or other issues in data generated through LLMs in a user-dependent, application-dependent, and/or output-dependent manner. As such, the data generation platform enables targeted, configurable, modular, and flexible prompt and output evaluation. Fig. 11, column 31, line 26 to 50, At act 1106, the process 1100 can determine a system state associated with system resources. For example, the process 1100 determines a system state associated with system resources for processing requests using the first model of the plurality of models. As an illustrative example, the performance engine 118 dynamically determines a state of the system (e.g., with respect to the determined performance metrics). The system state can include an indication of values associated with performance metrics (e.g., usage values, such as CPU utilization metric values, memory usage values, hard disk space usage values, numbers of input tokens previously submitted to models within the system, and/or values of incurred cost). For example, the data generation platform 102, through communication engine 112 can query a diagnostic tool or program associated with the computing system and/or an associated database to determine values of the performance metrics. In some implementations, the system state includes maximum, minimum, or other limiting values associated with the performance metric values (e.g., a maximum cost/budget, or a maximum available memory value). By receiving information relating to the system state and associated restrictions, the data generation platform 102 can evaluate the received prompt to determine whether the selected model is suitable for generating an associated output. Fig. 11, column 34, line 56 to column 35, line 8, At act 1116, the process 1100 can provide the input (e.g., prompt) to the first model in response to determining that the first estimated performance metric value satisfies the threshold metric value. For example, in response to determining that the first estimated performance metric value satisfies the threshold metric value, the process 1100 provides the prompt to the first model to generate a first output by processing the input (e.g., prompt) included in the output generation request. As an illustrative example, the data generation platform 102 can transmit the prompt (e.g., through the communication engine 112 and/or via an associated API) to the first LLM for generation of an associated output. To illustrate, the data generation platform 102 can generate a vector representation of the prompt (e.g., through word2vec or another suitable algorithm) and generate a vector representation of the output via the first LLM. By doing so, the data generation platform 102 can process the user's output generation request with available system resources (e.g., monetary resources or computational resources).), and
(b) when the extent does not satisfy the threshold condition, selecting a second path which specifies a second execution which executes at least one machine learning model (column 18, line 49 to 67, The data generation platform 102 can determine that the estimated resource metric value (e.g., the estimated cost) does not satisfy the threshold metric value. For example, the data generation platform 102 determines that the estimated cost is greater than or equal to the threshold cost. Based on this determination, the performance evaluation 408 can determine to prevent provision of the associated prompt to one or more LLMs of the LLMs 410. Additionally or alternatively, the data generation platform 102 can determine another LLM (e.g., a second LLM) for which the associated performance metric value satisfies threshold metric value (e.g., where the estimated cost is less than the threshold cost) and re-route the prompt to this second LLM. As such, the data generation platform 102 can dynamically select LLMs for processing output generation requests on the basis of estimated system resources and associated limitations or requirements, thereby improving the efficiency, flexibility, and robustness of the associated development pipeline. Fig. 11, column 35 line 56 to column 36, line 5, At act 1126, the process 1100 can generate a second output by providing the prompt to the second model in response to determining that the second estimated performance metric value satisfies the threshold metric value. For example, the process 1100 provides the prompt to the second model to generate a second output by processing the input (e.g., prompt) included in the output generation request. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) can generate vector representations of the prompt and transmit these (e.g., via an API call) to a device associated with the second model for generation of the associated output. By doing so, the data generation platform 102 enables processing of the output generation request using a model (e.g., the second LLM) that satisfies system resource limitations or constraints, thereby improving the resilience and efficiency of the data generation platform 102.);
executing the selected path, including computationally performing software analysis work(fig. 11, column 35, line 4-8, By doing so, the data generation platform 102 can process the user's output generation request with available system resources (e.g., monetary resources or computational resources). fig. 11, column 36, line 25-49, In response to receiving the user instruction, the data generation platform 102 can provide the prompt to the second model. To illustrate, the data generation platform 102 can generate indications of one or more recommended LLMs with estimated performance metric values (e.g., estimated cost values) that are compatible with the associated threshold performance metric (e.g., a threshold cost metric). By doing so, the data generation platform 102 can present options for LLMs (e.g., that satisfy system performance constraints) for processing the user's prompt, conferring the user with increased control over output generation.); and
providing, via a user interface, a result of executing the selected path(fig. 11, column 35, line 9 to 21, At act 1118, the process 1100 can generate the output for display on a device associated with the user. For example, the process 1100 transmits the first output to a computing system enabling access to the first output by the user device. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) can transmit the output from the first LLM to a computing system (e.g., a server) from which the user can access the generated output (e.g., through an API call and/or via a user interface). By doing so, the data generation platform 102 enables generation of outputs (e.g., natural language outputs) using models specified by the user when system resources are available to process associated prompts. Fig. 11, column 36, line 50 to 58, At act 1128, the process 1100 can generate the output for display on a device associated with the user. For example, the process 1100 transmits the second output to the computing system enabling access to the second output by the user device. As an illustrative example, the data generation platform 102 (e.g., through communication engine 112) transmits the second output to a computing system that enables access to the output by the user (e.g., through an associated API or GUI).
The Office would like to use prior art Athiwaratkum to back up Jain to further teach limitation
obtaining a request written at least partially in a natural language (Athiwaratkum, US 20230418566, para [0122], As indicated at 1520, individual ones of the different items of the data set into a second programming language may be converted, in some embodiments. For example, conversion of prompts may include changing features such as the symbols used to indicate code comments (e.g., non-executable statements in the code), such as changing from “″″″” to “/* *”. Conversion of prompts may also include changing the natural language statements to replace source programming language terms to target programing language terms “Write a function in Python” to “Write a function in Java” to . . . or changing between terms such as “none” to “null”. In some scenarios, conversion rules may remove unnecessary or uncovertable source programming language specific statements. Para [0127], As indicated at 1550, a body of the converted function signature may be caused to be generated in the second programming language according to a prompt in the item used as input to a machine learning model trained to generate code in the second programming language, in some embodiments. For example, as discussed in detail below with regard to FIGS. 1, 2, 4, and 10, code generation techniques may take as input a given portion of code or natural language statements describing the code to generate, and generate suggested code. Thus a code conversion system (or other system, service, or application performing the recited techniques of FIG. 15) may invoke via an interface a code generation system or a locally implemented machine learning model to generate the function body code in the target programming language (e.g., by specifying the target programming language in order to cause the use of the machine learning model specific to that target programming language).)
It would have obvious to one having ordinary skill in the art before the effecting filing date of the claimed invention to combine the teachings of cited references. Thus, one of ordinary skill in the art before the effecting filing date of the claimed invention would have been motivated to incorporate Athiwaratkum into Jain to improve code performance and identify errors, which helps to overcome a developer's lack of familiarity with a programming language or an environment in which the programming language is deployed, so that high performing code still be written. Techniques that improve the speed and quality of code suggestion powered by machine learning improves the user experience as well as the quality of the applications, systems, or services produced using the feature as suggested by Athiwaratkum (See abstract and conclusion).
Claim 2 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein determining the extent to which the software analyzer meets the requirement of the request comprises submitting a prompt to a first machine learning model, the prompt comprising at least a portion of the request, the prompt also comprising at least one of: a description of the software analyzer, and an instruction to report the extent; or an instruction to identify at least one software analyzer which meets at least one requirement of the request, with an instruction to report the extent (Jain, Fig. 11, column 31, line 26 to 50, At act 1106, the process 1100 can determine a system state associated with system resources. For example, the process 1100 determines a system state associated with system resources for processing requests using the first model of the plurality of models. As an illustrative example, the performance engine 118 dynamically determines a state of the system (e.g., with respect to the determined performance metrics). The system state can include an indication of values associated with performance metrics (e.g., usage values, such as CPU utilization metric values, memory usage values, hard disk space usage values, numbers of input tokens previously submitted to models within the system, and/or values of incurred cost). For example, the data generation platform 102, through communication engine 112 can query a diagnostic tool or program associated with the computing system and/or an associated database to determine values of the performance metrics. In some implementations, the system state includes maximum, minimum, or other limiting values associated with the performance metric values (e.g., a maximum cost/budget, or a maximum available memory value). By receiving information relating to the system state and associated restrictions, the data generation platform 102 can evaluate the received prompt to determine whether the selected model is suitable for generating an associated output. Fig. 11, column 34, line 56 to column 35, line 8, At act 1116, the process 1100 can provide the input (e.g., prompt) to the first model in response to determining that the first estimated performance metric value satisfies the threshold metric value. For example, in response to determining that the first estimated performance metric value satisfies the threshold metric value, the process 1100 provides the prompt to the first model to generate a first output by processing the input (e.g., prompt) included in the output generation request. As an illustrative example, the data generation platform 102 can transmit the prompt (e.g., through the communication engine 112 and/or via an associated API) to the first LLM for generation of an associated output. To illustrate, the data generation platform 102 can generate a vector representation of the prompt (e.g., through word2vec or another suitable algorithm) and generate a vector representation of the output via the first LLM. By doing so, the data generation platform 102 can process the user's output generation request with available system resources (e.g., monetary resources or computational resources).).
Claim 3 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein the selecting selects the second path, and wherein determining the extent to which the software analyzer meets the requirement of the request comprises at least one of: ascertaining that the requirement includes summarizing a source code; ascertaining that the requirement includes decomposing a task into a plurality of smaller tasks; ascertaining that the requirement includes scheduling a plurality of tasks; or ascertaining that the requirement includes reviewing a change to a source code (Jain, column 30, line 50-58, the process 1100 enables selection of an LLM for generation of an output (e.g., software-related code samples) based on an input (e.g., a text-based prompt) to prevent overuse of system resources (e.g., to ensure that sufficient system resources are available to process the request). Column 25, line 11 to 34, the data generation platform 102 can generate a validation indicator based on determining that the output contains code and testing the code (and/or the compiled version of the code) in an isolated environment for potential adverse effects, viruses, or bugs. By doing so, the data generation platform 102 can ensure the safety and security of generated code, thereby protecting the software development pipeline from security breaches or unintended behavior.).
Claim 4 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein determining the extent to which the software analyzer meets the requirement of the request comprises finding that a first estimate of a first computational cost of the first path is lower than a second estimate of a second computational cost of the second path (Jain, fig. 10, column 29, line 49 to column 30, line 49, In some implementations, the GUI 1000 provides a breakdown of cost associated with various models. For example, the GUI 1000 enables a user to control a pointer (e.g., a mouse pointer) and/or a gesture (e.g., a finger) to hover over estimated cost metric values (e.g., the estimated cost metric value 1004). The data generation platform 102, based on detecting that the user is hovering over an indicator associated with the estimated cost metric value, can cause the GUI 1000 to display an indication of components associated with the estimated cost metric value 1004. For example, the GUI 1000 can display the component values making up the composite metric value for a given model (e.g., including associated weights), including CPU usage time, memory usage, and/or other estimated performance metric values. By doing so, the GUI 1000 confers information to the user relating to the estimated resource usage associated with processing the input through a given model, thereby enabling the user to select a model according to predicted system effects.).
Claim 5 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein determining the extent to which the software analyzer meets the requirement of the request comprises receiving an analysis plan from an analysis planning model, the analysis plan including a selection of either the first path or the second path (Jain, fig. 10, column 29, line 49 to column 30, line 49, In some implementations, the GUI 1000 provides a breakdown of cost associated with various models. For example, the GUI 1000 enables a user to control a pointer (e.g., a mouse pointer) and/or a gesture (e.g., a finger) to hover over estimated cost metric values (e.g., the estimated cost metric value 1004). The data generation platform 102, based on detecting that the user is hovering over an indicator associated with the estimated cost metric value, can cause the GUI 1000 to display an indication of components associated with the estimated cost metric value 1004. For example, the GUI 1000 can display the component values making up the composite metric value for a given model (e.g., including associated weights), including CPU usage time, memory usage, and/or other estimated performance metric values. By doing so, the GUI 1000 confers information to the user relating to the estimated resource usage associated with processing the input through a given model, thereby enabling the user to select a model according to predicted system effects.).
Claim 6 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein determining the extent to which the software analyzer meets the requirement of the request comprises receiving an analysis plan from an analysis planning model, wherein the analysis plan specifies a non-empty set of software analysis tasks, the analysis plan assigns a first non-empty portion of the set to the software analyzer, and the analysis plan assigns a second non-empty portion of the set to at least one machine learning model (Jain, fig. 10, column 29, line 49 to column 30, line 49, In some implementations, the GUI 1000 provides a breakdown of cost associated with various models. For example, the GUI 1000 enables a user to control a pointer (e.g., a mouse pointer) and/or a gesture (e.g., a finger) to hover over estimated cost metric values (e.g., the estimated cost metric value 1004). The data generation platform 102, based on detecting that the user is hovering over an indicator associated with the estimated cost metric value, can cause the GUI 1000 to display an indication of components associated with the estimated cost metric value 1004. For example, the GUI 1000 can display the component values making up the composite metric value for a given model (e.g., including associated weights), including CPU usage time, memory usage, and/or other estimated performance metric values. By doing so, the GUI 1000 confers information to the user relating to the estimated resource usage associated with processing the input through a given model, thereby enabling the user to select a model according to predicted system effects.).
Claim 7 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein determining the extent to which the software analyzer meets the requirement of the request comprises receiving at least part of an analysis plan from an analysis planning model, the analysis plan comprising: gathering a non-empty context, placing the context in a prompt, and submitting the prompt to at least one machine learning model, and wherein the context comprises at least one of: a symbol table; a call graph; an abstract syntax tree; control flow information at a callsite; or data flow information at a callsite(Jain, fig. 10, column 29, line 49 to column 30, line 49, In some implementations, the GUI 1000 provides a breakdown of cost associated with various models. For example, the GUI 1000 enables a user to control a pointer (e.g., a mouse pointer) and/or a gesture (e.g., a finger) to hover over estimated cost metric values (e.g., the estimated cost metric value 1004). The data generation platform 102, based on detecting that the user is hovering over an indicator associated with the estimated cost metric value, can cause the GUI 1000 to display an indication of components associated with the estimated cost metric value 1004. For example, the GUI 1000 can display the component values making up the composite metric value for a given model (e.g., including associated weights), including CPU usage time, memory usage, and/or other estimated performance metric values. By doing so, the GUI 1000 confers information to the user relating to the estimated resource usage associated with processing the input through a given model, thereby enabling the user to select a model according to predicted system effects. Column 17, line 4 to line 21, The data generation platform 102 (e.g., through the breach mitigation engine 116) can determine a subset of input controls 610 to implement for a given output generation request. For example, the data generation platform 102 selects a subset of the input controls 610 based on one or more attributes associated with the output generation request. To illustrate, the breach mitigation engine 116 can determine to implement input validation 612a, trace injection 612b and prompt injection 612f for a prompt that is related to software development, while adding secret redaction 612d and/or sensitive data detection 612e for a prompt that is related to user account analytics (e.g., for an application that is more likely to be associated with sensitive information). By modularly selecting controls depending on the application or use case of the output generation request, the data generation platform 102 can improve system efficiency by avoiding unnecessary controls, while maintaining prompt generation security and reliability.).
Claim 8 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein determining the extent to which the software analyzer meets the requirement of the request comprises receiving at least part of an analysis plan from an analysis planning model, the analysis plan comprising: gathering a non-empty context by execution of at least one software analyzer identified in the analysis plan or by execution of an analysis tool in at least one software analyzer category identified in the analysis plan, placing the context in a prompt, and submitting the prompt to at least one machine learning model(Jain, fig. 10, column 29, line 49 to column 30, line 49, In some implementations, the GUI 1000 provides a breakdown of cost associated with various models. For example, the GUI 1000 enables a user to control a pointer (e.g., a mouse pointer) and/or a gesture (e.g., a finger) to hover over estimated cost metric values (e.g., the estimated cost metric value 1004). The data generation platform 102, based on detecting that the user is hovering over an indicator associated with the estimated cost metric value, can cause the GUI 1000 to display an indication of components associated with the estimated cost metric value 1004. For example, the GUI 1000 can display the component values making up the composite metric value for a given model (e.g., including associated weights), including CPU usage time, memory usage, and/or other estimated performance metric values. By doing so, the GUI 1000 confers information to the user relating to the estimated resource usage associated with processing the input through a given model, thereby enabling the user to select a model according to predicted system effects.).
Claim 9 is rejected for the reasons set forth hereinabove for claim 1, Jain and Athiwaratkum teach the method of claim 1, wherein determining the extent to which the software analyzer meets the requirement of the request comprises receiving at least part of an analysis plan from an analysis planning model, the analysis plan comprising: executing at least one software analyzer to perform and complete the software analysis work without any further execution of any artificial intelligence model as part of the software analysis work(Jain, column 33, line 1 to 19, At act 1110, the process 1100 can determine an estimated performance metric value associated with processing the output generation request. For example, the process 1100 determines a first estimated performance metric value for the determined performance metric based on an indication of an estimated resource usage by the first model when processing the input included in the output generation request. As an illustrative example, the data generation platform 102 determines a prediction for resource usage for generating an output using the indicated model (e.g., an LLM associated with the determined performance metric). The data generation platform 102 (e.g., through the performance engine 118) can determine a number of input tokens within the input or prompt and predict a cost and/or a memory usage associated with processing the prompt using the selected model. By doing so, the data generation platform 102 can evaluate the effects of processing the input on system resources for evaluation of the suitability of the model for generating the requested output.).
Claim 10 rejected, Jain teaches a computing system, comprising:
at least one digital memory(column 10, line 42 to 57, fig. 3, computing devices 302a-302d);
a software development tool having a user interface ( column 10, line 42 to 57, environment 300 includes one or more client computing devices 302a-302d, examples of which can host graphical user interfaces associated with client devices. For example, one or more of the client computing devices 302a-302d includes user devices and/or devices associated with services requesting responses to queries from LLMs. );
at least one processor in operable communication with the at least one digital memory, the at least one processor configured to perform a software development method which comprises (Jain, abstract and summary):
(a) obtaining a request written at least partially in a natural language(column 6, line 9 to 27, processing an input (e.g., a natural language prompt). Column 12, line 15-37, The prompt can include a natural language question or command (e.g., in English). For example, the prompt includes a request for an LLM to generate code (e.g., within a specified programming language) that executes a particular operation. Fig. 11, column 30, line 59 to column 31, line 8, At act 1102, the process 1100 can receive an input for generation of an output using a model. For example, the process 1100 receives, from a user device, an output generation request comprising an input (e.g., prompt) for generation of an output using a first model (e.g., an LLM) of a plurality of models. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) receives a prompt indicating a desired output, such as a text-based instruction for the generation of software-related code samples (e.g., associated with a particular function). The output generation request can include an indication of a selected model (e.g., LLM) for processing the prompt. As such, the data generation platform 102 can evaluate the effect of generating an output using the selected model based on the prompt (e.g., or other suitable inputs) on the basis of the content or nature of the request (e.g., based on a user identifier associated with the request).),
(b) determining an extent to which a software analyzer meets a functionality requirement of the request(column 4, line 10 to 30, The data generation platform can compare the performance metric value with an associated threshold or criterion. For example, the data generation platform determines that the estimated system resources required to process the prompt through the associated LLM is less than an allotment assigned to the user. As such, the data generation platform can proceed to provide the prompt to the LLM for generation of the requested output. In some implementations, the data generation platform further evaluates the output for accuracy, security, safety (e.g., with respect to associated policies, requirements, or criteria), compliance (e.g., compliance with regulations, rules, guidelines, etc.), and/or other requirements/recommendations. As an illustrative example, the data generation platform tests any generated code within a virtual machine or another suitable isolated environment to determine any security risks of the generated code. In response to validating the generated output, the data generation platform can transmit this information to an associated data store or deployment system (e.g., any relevant consumer of the generated data, such as a server that is accessible to the user). Fig. 11, column 31, line 9 to line 25, At act 1104, the process 1100 can determine a performance metric associated with processing the output generation request. For example, the process 1100 determines a performance metric associated with processing the output generation request. As an illustrative example, the data generation platform 102 can determine one or more performance metrics that characterize the behavior of the system (e.g., when providing inputs to a model for generation of an output). Such performance metrics can include CPU utilization, cost (e.g., associated with the operation of the system and/or the associated models), memory usage, storage space, and/or number of input or output tokens associated with LLMs. In some implementations, the data generation platform 102 (e.g., through the performance engine 118) determines multiple performance metrics (e.g., associated with the system state) for evaluation of the effects (e.g., of generating an output based on the prompt) on the system.),
(c) selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model (column 4, line 41 to 63, the disclosed data generation platform enables evaluation of the user's prompt in a flexible, modular manner. For example, the data generation platform determines which prompt validation rules, criteria, or models with which to evaluate the user's prompt (e.g., based on the identity of the user, the nature of the prompt, and/or other suitable factors). Based on this determination, the data generation platform can evaluate the prompt with respect to relevant criteria, while avoiding the need to evaluate the prompt against unsuitable or unrelated criteria. In some implementations, the data generation platform evaluates the performance requirements associated with the prompt generate a recommendation for a suitable LLM for the received prompt (e.g., to improve the efficiency of system resource use). In some implementations, the data generation platform enables evaluation of model outputs in a flexible, modular manner (e.g., depending on the type of output). By doing so, the system can mitigate inaccuracies, security breaches, or other issues in data generated through LLMs in a user-dependent, application-dependent, and/or output-dependent manner. As such, the data generation platform enables targeted, configurable, modular, and flexible prompt and output evaluation. Fig. 11, column 31, line 26 to 50, At act 1106, the process 1100 can determine a system state associated with system resources. For example, the process 1100 determines a system state associated with system resources for processing requests using the first model of the plurality of models. As an illustrative example, the performance engine 118 dynamically determines a state of the system (e.g., with respect to the determined performance metrics). The system state can include an indication of values associated with performance metrics (e.g., usage values, such as CPU utilization metric values, memory usage values, hard disk space usage values, numbers of input tokens previously submitted to models within the system, and/or values of incurred cost). For example, the data generation platform 102, through communication engine 112 can query a diagnostic tool or program associated with the computing system and/or an associated database to determine values of the performance metrics. In some implementations, the system state includes maximum, minimum, or other limiting values associated with the performance metric values (e.g., a maximum cost/budget, or a maximum available memory value). By receiving information relating to the system state and associated restrictions, the data generation platform 102 can evaluate the received prompt to determine whether the selected model is suitable for generating an associated output. Fig. 11, column 34, line 56 to column 35, line 8, At act 1116, the process 1100 can provide the input (e.g., prompt) to the first model in response to determining that the first estimated performance metric value satisfies the threshold metric value. For example, in response to determining that the first estimated performance metric value satisfies the threshold metric value, the process 1100 provides the prompt to the first model to generate a first output by processing the input (e.g., prompt) included in the output generation request. As an illustrative example, the data generation platform 102 can transmit the prompt (e.g., through the communication engine 112 and/or via an associated API) to the first LLM for generation of an associated output. To illustrate, the data generation platform 102 can generate a vector representation of the prompt (e.g., through word2vec or another suitable algorithm) and generate a vector representation of the output via the first LLM. By doing so, the data generation platform 102 can process the user's output generation request with available system resources (e.g., monetary resources or computational resources).), or
a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model executed for selecting the path (column 18, line 49 to 67, The data generation platform 102 can determine that the estimated resource metric value (e.g., the estimated cost) does not satisfy the threshold metric value. For example, the data generation platform 102 determines that the estimated cost is greater than or equal to the threshold cost. Based on this determination, the performance evaluation 408 can determine to prevent provision of the associated prompt to one or more LLMs of the LLMs 410. Additionally or alternatively, the data generation platform 102 can determine another LLM (e.g., a second LLM) for which the associated performance metric value satisfies threshold metric value (e.g., where the estimated cost is less than the threshold cost) and re-route the prompt to this second LLM. As such, the data generation platform 102 can dynamically select LLMs for processing output generation requests on the basis of estimated system resources and associated limitations or requirements, thereby improving the efficiency, flexibility, and robustness of the associated development pipeline. Fig. 11, column 35 line 56 to column 36, line 5, At act 1126, the process 1100 can generate a second output by providing the prompt to the second model in response to determining that the second estimated performance metric value satisfies the threshold metric value. For example, the process 1100 provides the prompt to the second model to generate a second output by processing the input (e.g., prompt) included in the output generation request. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) can generate vector representations of the prompt and transmit these (e.g., via an API call) to a device associated with the second model for generation of the associated output. By doing so, the data generation platform 102 enables processing of the output generation request using a model (e.g., the second LLM) that satisfies system resource limitations or constraints, thereby improving the resilience and efficiency of the data generation platform 102.),
(d) triggering a performance of the path, the performance including computational software analysis work(fig. 11, column 35, line 4-8, By doing so, the data generation platform 102 can process the user's output generation request with available system resources (e.g., monetary resources or computational resources). fig. 11, column 36, line 25-49, In response to receiving the user instruction, the data generation platform 102 can provide the prompt to the second model. To illustrate, the data generation platform 102 can generate indications of one or more recommended LLMs with estimated performance metric values (e.g., estimated cost values) that are compatible with the associated threshold performance metric (e.g., a threshold cost metric). By doing so, the data generation platform 102 can present options for LLMs (e.g., that satisfy system performance constraints) for processing the user's prompt, conferring the user with increased control over output generation.), and
(e) providing, via the user interface, a result of the performance of the path (fig. 11, column 35, line 9 to 21, At act 1118, the process 1100 can generate the output for display on a device associated with the user. For example, the process 1100 transmits the first output to a computing system enabling access to the first output by the user device. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) can transmit the output from the first LLM to a computing system (e.g., a server) from which the user can access the generated output (e.g., through an API call and/or via a user interface). By doing so, the data generation platform 102 enables generation of outputs (e.g., natural language outputs) using models specified by the user when system resources are available to process associated prompts. Fig. 11, column 36, line 50 to 58, At act 1128, the process 1100 can generate the output for display on a device associated with the user. For example, the process 1100 transmits the second output to the computing system enabling access to the second output by the user device. As an illustrative example, the data generation platform 102 (e.g., through communication engine 112) transmits the second output to a computing system that enables access to the output by the user (e.g., through an associated API or GUI).
The Office would like to use prior art Athiwaratkum to back up Jain to further teach limitation
obtaining a request written at least partially in a natural language (Athiwaratkum, US 20230418566, para [0122], As indicated at 1520, individual ones of the different items of the data set into a second programming language may be converted, in some embodiments. For example, conversion of prompts may include changing features such as the symbols used to indicate code comments (e.g., non-executable statements in the code), such as changing from “″″″” to “/* *”. Conversion of prompts may also include changing the natural language statements to replace source programming language terms to target programing language terms “Write a function in Python” to “Write a function in Java” to . . . or changing between terms such as “none” to “null”. In some scenarios, conversion rules may remove unnecessary or uncovertable source programming language specific statements. Para [0127], As indicated at 1550, a body of the converted function signature may be caused to be generated in the second programming language according to a prompt in the item used as input to a machine learning model trained to generate code in the second programming language, in some embodiments. For example, as discussed in detail below with regard to FIGS. 1, 2, 4, and 10, code generation techniques may take as input a given portion of code or natural language statements describing the code to generate, and generate suggested code. Thus a code conversion system (or other system, service, or application performing the recited techniques of FIG. 15) may invoke via an interface a code generation system or a locally implemented machine learning model to generate the function body code in the target programming language (e.g., by specifying the target programming language in order to cause the use of the machine learning model specific to that target programming language).)
It would have obvious to one having ordinary skill in the art before the effecting filing date of the claimed invention to combine the teachings of cited references. Thus, one of ordinary skill in the art before the effecting filing date of the claimed invention would have been motivated to incorporate Athiwaratkum into Jain to improve code performance and identify errors, which helps to overcome a developer's lack of familiarity with a programming language or an environment in which the programming language is deployed, so that high performing code still be written. Techniques that improve the speed and quality of code suggestion powered by machine learning improves the user experience as well as the quality of the applications, systems, or services produced using the feature as suggested by Athiwaratkum (See abstract and conclusion).
Claim 11 is rejected for the reasons set forth hereinabove for claim 10, Jain and Athiwaratkum teach the computing system of claim 10, wherein the result of the performance of the path comprises at least one of: a code transformation, or a suggestion of the code transformation, and the method further comprises receiving a user input selecting the code transformation or the suggestion of the code transformation, and applying the code transformation to a source code in the software development tool (Jain, column 2, line 30 to 41, pre-existing systems can leverage LLMs to generate technical content, including software code, architectures, or code patches based on user prompts, such as in the case of a data analysis or software development pipeline. Column 12, line 1 to 14, The data generation platform 102 can transmit the generated output to the output validation model 420 for testing and validation of the output (e.g., to prevent security breaches). The output validation model 420 can transmit the validated output to a data consumption system 422, for exposure of the output to the user device 402a and/or the service 402b. ).
Claim 12 is rejected for the reasons set forth hereinabove for claim 10, Jain and Athiwaratkum teach the computing system of claim 10, comprising:
an analysis planning model, or an analysis planning model interface to an analysis planning model, the analysis planning model being an artificial intelligence model (Jain, column 9, line 8 to 25, The performance engine 118 can execute tasks relating to monitoring and controlling performance of the data generation platform 102 (e.g., or the associated development pipeline). For example, the performance engine 118 includes software components (e.g., performance monitoring modules), hardware components, or a combination thereof. To illustrate, the performance engine 118 can estimate performance metric values associated with processing a given prompt with a selected LLM (e.g., an estimated cost or memory usage). By doing so, the performance engine 118 can determine whether to allow access to a given LLM by a user, based on the user's requested output and the associated estimated system effects. The performance engine 118 can communicate with the communication engine 112, the access control engine 114, the performance engine 118, the generative model engine 120, and/or other components associated with the network 150 (e.g., the data node 104 and/or the third-party databases 108a-108n).);
an analysis model, or an analysis model interface to an analysis model, the analysis model being a machine learning model (Jain, column 9, line 26 to 45, The generative model engine 120 can execute tasks relating to machine learning inference (e.g., natural language generation based on a generative machine learning model, such as an LLM). The generative model engine 120 can include software components (e.g., one or more LLMs, and/or API calls to devices associated with such LLMs), hardware components, and/or a combination thereof. To illustrate, the generative model engine 120 can provide users' prompts to a requested, selected, or determined model (e.g., LLM) to generate a resulting output (e.g., to a user's query within the prompt). As such, the generative model engine 120 enables flexible, configurable generation of data (e.g., text, code, or other suitable information) based on user input, thereby improving the flexibility of software development or other such tasks. The generative model engine 120 can communicate with the communication engine 112, the access control engine 114, the performance engine 118, the generative model engine 120, and/or other components associated with the network 150 (e.g., the data node 104 and/or the third-party databases 108a-108n).); and
wherein the at least one processor is configured to communicate with the analysis planning model to receive an analysis plan which specifies a non-empty set of software analysis tasks (Jain, column 9, line 8 to 25, TThe performance engine 118 can execute tasks relating to monitoring and controlling performance of the data generation platform 102 (e.g., or the associated development pipeline). For example, the performance engine 118 includes software components (e.g., performance monitoring modules), hardware components, or a combination thereof. To illustrate, the performance engine 118 can estimate performance metric values associated with processing a given prompt with a selected LLM (e.g., an estimated cost or memory usage). By doing so, the performance engine 118 can determine whether to allow access to a given LLM by a user, based on the user's requested output and the associated estimated system effects. The performance engine 118 can communicate with the communication engine 112, the access control engine 114, the performance engine 118, the generative model engine 120, and/or other components associated with the network 150 (e.g., the data node 104 and/or the third-party databases 108a-108n).; and
wherein the at least one processor is configured to communicate with the analysis model in response to noting that the analysis plan assigns a non-empty portion of the set to at least one machine learning model(Jain, column 9, line 26 to 45, The generative model engine 120 can execute tasks relating to machine learning inference (e.g., natural language generation based on a generative machine learning model, such as an LLM). The generative model engine 120 can include software components (e.g., one or more LLMs, and/or API calls to devices associated with such LLMs), hardware components, and/or a combination thereof. To illustrate, the generative model engine 120 can provide users' prompts to a requested, selected, or determined model (e.g., LLM) to generate a resulting output (e.g., to a user's query within the prompt). As such, the generative model engine 120 enables flexible, configurable generation of data (e.g., text, code, or other suitable information) based on user input, thereby improving the flexibility of software development or other such tasks. The generative model engine 120 can communicate with the communication engine 112, the access control engine 114, the performance engine 118, the generative model engine 120, and/or other components associated with the network 150 (e.g., the data node 104 and/or the third-party databases 108a-108n).).
Claim 13 is rejected for the reasons set forth hereinabove for claim 12, Jain and Athiwaratkum teach the computing system of claim 12, comprising the analysis planning model and the analysis model interface, and wherein the analysis planning model is on a same machine or a same local area network as the at least one processor and the analysis model is not on the same machine and not on the same local area network as the at least one processor(Jain, column 6, line 59 to 65, In some implementations, the data generation platform 102 can be configured on a user device. Column 6, line 45 to 65, e.g., a laptop computer, smartphone, desktop computer, electronic tablet, or another suitable user device). Furthermore, the data generation platform 102 can reside on a server or node and/or can interface with third-party databases 108a-108n directly or indirectly.).
Claim 14 is rejected for the reasons set forth hereinabove for claim 10, Jain and Athiwaratkum teach the computing system of claim 10, wherein selecting the path comprises acquiring a first risk score which is associated with the first path, acquiring a second risk score which is associated with the second path, and comparing the first risk score to the second risk score (Jain, fig. 10, column 29, line 49 to column 30, line 49, In some implementations, the GUI 1000 provides a breakdown of cost associated with various models. For example, the GUI 1000 enables a user to control a pointer (e.g., a mouse pointer) and/or a gesture (e.g., a finger) to hover over estimated cost metric values (e.g., the estimated cost metric value 1004). The data generation platform 102, based on detecting that the user is hovering over an indicator associated with the estimated cost metric value, can cause the GUI 1000 to display an indication of components associated with the estimated cost metric value 1004. For example, the GUI 1000 can display the component values making up the composite metric value for a given model (e.g., including associated weights), including CPU usage time, memory usage, and/or other estimated performance metric values. By doing so, the GUI 1000 confers information to the user relating to the estimated resource usage associated with processing the input through a given model, thereby enabling the user to select a model according to predicted system effects.).
Claim 15 is rejected for the reasons set forth hereinabove for claim 10, Jain and Athiwaratkum teach the computing system of claim 10, wherein the performance of the path comprises a non-machine-learning software analyzer detecting a change, the change comprising at least one of: a change to a project-to-project reference; a change to a package reference; a change to a project property; an addition of a document to a project; a removal of a document from a project; a change to a tool-wide analysis setting; an addition of a development tool module to the software development tool; a removal of a development tool module from the software development tool; or a setting change in a development tool module of the software development tool(Jain, column 2, line 30 to 41, pre-existing systems can leverage LLMs to generate technical content, including software code, architectures, or code patches based on user prompts, such as in the case of a data analysis or software development pipeline.).
Claim 16 rejected, Jain teaches a computer-readable storage medium configured with data and instructions which upon execution by a processor perform a software development method in a computing system, the method comprising automatically (Jain, abstract and summary):
obtaining a request written at least partially in a natural language, the obtaining performed via a module plug interface of a development tool module and a module socket interface of a software development tool, the module plug interface adapted to the module socket interface, the development tool module external to the software development tool (column 6, line 9 to 27, processing an input (e.g., a natural language prompt). Column 12, line 15-37, The prompt can include a natural language question or command (e.g., in English). For example, the prompt includes a request for an LLM to generate code (e.g., within a specified programming language) that executes a particular operation. Fig. 11, column 30, line 59 to column 31, line 8, At act 1102, the process 1100 can receive an input for generation of an output using a model. For example, the process 1100 receives, from a user device, an output generation request comprising an input (e.g., prompt) for generation of an output using a first model (e.g., an LLM) of a plurality of models. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) receives a prompt indicating a desired output, such as a text-based instruction for the generation of software-related code samples (e.g., associated with a particular function). The output generation request can include an indication of a selected model (e.g., LLM) for processing the prompt. As such, the data generation platform 102 can evaluate the effect of generating an output using the selected model based on the prompt (e.g., or other suitable inputs) on the basis of the content or nature of the request (e.g., based on a user identifier associated with the request).);
determining, via an artificial intelligence model, an extent to which a software analyzer meets a functionality requirement of the request(column 4, line 10 to 30, The data generation platform can compare the performance metric value with an associated threshold or criterion. For example, the data generation platform determines that the estimated system resources required to process the prompt through the associated LLM is less than an allotment assigned to the user. As such, the data generation platform can proceed to provide the prompt to the LLM for generation of the requested output. In some implementations, the data generation platform further evaluates the output for accuracy, security, safety (e.g., with respect to associated policies, requirements, or criteria), compliance (e.g., compliance with regulations, rules, guidelines, etc.), and/or other requirements/recommendations. As an illustrative example, the data generation platform tests any generated code within a virtual machine or another suitable isolated environment to determine any security risks of the generated code. In response to validating the generated output, the data generation platform can transmit this information to an associated data store or deployment system (e.g., any relevant consumer of the generated data, such as a server that is accessible to the user). Fig. 11, column 31, line 9 to line 25, At act 1104, the process 1100 can determine a performance metric associated with processing the output generation request. For example, the process 1100 determines a performance metric associated with processing the output generation request. As an illustrative example, the data generation platform 102 can determine one or more performance metrics that characterize the behavior of the system (e.g., when providing inputs to a model for generation of an output). Such performance metrics can include CPU utilization, cost (e.g., associated with the operation of the system and/or the associated models), memory usage, storage space, and/or number of input or output tokens associated with LLMs. In some implementations, the data generation platform 102 (e.g., through the performance engine 118) determines multiple performance metrics (e.g., associated with the system state) for evaluation of the effects (e.g., of generating an output based on the prompt) on the system.);
selecting a path in response to at least the extent, wherein the path is one of: a first path which specifies a first execution which executes the software analyzer without specifying any execution of any machine learning model (column 4, line 41 to 63, the disclosed data generation platform enables evaluation of the user's prompt in a flexible, modular manner. For example, the data generation platform determines which prompt validation rules, criteria, or models with which to evaluate the user's prompt (e.g., based on the identity of the user, the nature of the prompt, and/or other suitable factors). Based on this determination, the data generation platform can evaluate the prompt with respect to relevant criteria, while avoiding the need to evaluate the prompt against unsuitable or unrelated criteria. In some implementations, the data generation platform evaluates the performance requirements associated with the prompt generate a recommendation for a suitable LLM for the received prompt (e.g., to improve the efficiency of system resource use). In some implementations, the data generation platform enables evaluation of model outputs in a flexible, modular manner (e.g., depending on the type of output). By doing so, the system can mitigate inaccuracies, security breaches, or other issues in data generated through LLMs in a user-dependent, application-dependent, and/or output-dependent manner. As such, the data generation platform enables targeted, configurable, modular, and flexible prompt and output evaluation. Fig. 11, column 31, line 26 to 50, At act 1106, the process 1100 can determine a system state associated with system resources. For example, the process 1100 determines a system state associated with system resources for processing requests using the first model of the plurality of models. As an illustrative example, the performance engine 118 dynamically determines a state of the system (e.g., with respect to the determined performance metrics). The system state can include an indication of values associated with performance metrics (e.g., usage values, such as CPU utilization metric values, memory usage values, hard disk space usage values, numbers of input tokens previously submitted to models within the system, and/or values of incurred cost). For example, the data generation platform 102, through communication engine 112 can query a diagnostic tool or program associated with the computing system and/or an associated database to determine values of the performance metrics. In some implementations, the system state includes maximum, minimum, or other limiting values associated with the performance metric values (e.g., a maximum cost/budget, or a maximum available memory value). By receiving information relating to the system state and associated restrictions, the data generation platform 102 can evaluate the received prompt to determine whether the selected model is suitable for generating an associated output. Fig. 11, column 34, line 56 to column 35, line 8, At act 1116, the process 1100 can provide the input (e.g., prompt) to the first model in response to determining that the first estimated performance metric value satisfies the threshold metric value. For example, in response to determining that the first estimated performance metric value satisfies the threshold metric value, the process 1100 provides the prompt to the first model to generate a first output by processing the input (e.g., prompt) included in the output generation request. As an illustrative example, the data generation platform 102 can transmit the prompt (e.g., through the communication engine 112 and/or via an associated API) to the first LLM for generation of an associated output. To illustrate, the data generation platform 102 can generate a vector representation of the prompt (e.g., through word2vec or another suitable algorithm) and generate a vector representation of the output via the first LLM. By doing so, the data generation platform 102 can process the user's output generation request with available system resources (e.g., monetary resources or computational resources).), or
a second path which specifies a second execution which executes at least one machine learning model in addition to any machine learning model execution while selecting the path (column 18, line 49 to 67, The data generation platform 102 can determine that the estimated resource metric value (e.g., the estimated cost) does not satisfy the threshold metric value. For example, the data generation platform 102 determines that the estimated cost is greater than or equal to the threshold cost. Based on this determination, the performance evaluation 408 can determine to prevent provision of the associated prompt to one or more LLMs of the LLMs 410. Additionally or alternatively, the data generation platform 102 can determine another LLM (e.g., a second LLM) for which the associated performance metric value satisfies threshold metric value (e.g., where the estimated cost is less than the threshold cost) and re-route the prompt to this second LLM. As such, the data generation platform 102 can dynamically select LLMs for processing output generation requests on the basis of estimated system resources and associated limitations or requirements, thereby improving the efficiency, flexibility, and robustness of the associated development pipeline. Fig. 11, column 35 line 56 to column 36, line 5, At act 1126, the process 1100 can generate a second output by providing the prompt to the second model in response to determining that the second estimated performance metric value satisfies the threshold metric value. For example, the process 1100 provides the prompt to the second model to generate a second output by processing the input (e.g., prompt) included in the output generation request. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) can generate vector representations of the prompt and transmit these (e.g., via an API call) to a device associated with the second model for generation of the associated output. By doing so, the data generation platform 102 enables processing of the output generation request using a model (e.g., the second LLM) that satisfies system resource limitations or constraints, thereby improving the resilience and efficiency of the data generation platform 102.);
triggering a performance of the path, the performance including computational software analysis work(fig. 11, column 35, line 4-8, By doing so, the data generation platform 102 can process the user's output generation request with available system resources (e.g., monetary resources or computational resources). fig. 11, column 36, line 25-49, In response to receiving the user instruction, the data generation platform 102 can provide the prompt to the second model. To illustrate, the data generation platform 102 can generate indications of one or more recommended LLMs with estimated performance metric values (e.g., estimated cost values) that are compatible with the associated threshold performance metric (e.g., a threshold cost metric). By doing so, the data generation platform 102 can present options for LLMs (e.g., that satisfy system performance constraints) for processing the user's prompt, conferring the user with increased control over output generation.); and
providing the software development tool with a result of the performance of the path(fig. 11, column 35, line 9 to 21, At act 1118, the process 1100 can generate the output for display on a device associated with the user. For example, the process 1100 transmits the first output to a computing system enabling access to the first output by the user device. As an illustrative example, the data generation platform 102 (e.g., through the communication engine 112) can transmit the output from the first LLM to a computing system (e.g., a server) from which the user can access the generated output (e.g., through an API call and/or via a user interface). By doing so, the data generation platform 102 enables generation of outputs (e.g., natural language outputs) using models specified by the user when system resources are available to process associated prompts. Fig. 11, column 36, line 50 to 58, At act 1128, the process 1100 can generate the output for display on a device associated with the user. For example, the process 1100 transmits the second output to the computing system enabling access to the second output by the user device. As an illustrative example, the data generation platform 102 (e.g., through communication engine 112) transmits the second output to a computing system that enables access to the output by the user (e.g., through an associated API or GUI).
The Office would like to use prior art Athiwaratkum to back up Jain to further teach limitation
obtaining a request written at least partially in a natural language (Athiwaratkum, US 20230418566, para [0122], As indicated at 1520, individual ones of the different items of the data set into a second programming language may be converted, in some embodiments. For example, conversion of prompts may include changing features such as the symbols used to indicate code comments (e.g., non-executable statements in the code), such as changing from “″″″” to “/* *”. Conversion of prompts may also include changing the natural language statements to replace source programming language terms to target programing language terms “Write a function in Python” to “Write a function in Java” to . . . or changing between terms such as “none” to “null”. In some scenarios, conversion rules may remove unnecessary or uncovertable source programming language specific statements. Para [0127], As indicated at 1550, a body of the converted function signature may be caused to be generated in the second programming language according to a prompt in the item used as input to a machine learning model trained to generate code in the second programming language, in some embodiments. For example, as discussed in detail below with regard to FIGS. 1, 2, 4, and 10, code generation techniques may take as input a given portion of code or natural language statements describing the code to generate, and generate suggested code. Thus a code conversion system (or other system, service, or application performing the recited techniques of FIG. 15) may invoke via an interface a code generation system or a locally implemented machine learning model to generate the function body code in the target programming language (e.g., by specifying the target programming language in order to cause the use of the machine learning model specific to that target programming language).)
It would have obvious to one having ordinary skill in the art before the effecting filing date of the claimed invention to combine the teachings of cited references. Thus, one of ordinary skill in the art before the effecting filing date of the claimed invention would have been motivated to incorporate Athiwaratkum into Jain to improve code performance and identify errors, which helps to overcome a developer's lack of familiarity with a programming language or an environment in which the programming language is deployed, so that high performing code still be written. Techniques that improve the speed and quality of code suggestion powered by machine learning improves the user experience as well as the quality of the applications, systems, or services produced using the feature as suggested by Athiwaratkum (See abstract and conclusion).
Claim 17 is rejected for the reasons set forth hereinabove for claim 16, Jain and Athiwaratkum teach the computer-readable storage medium of claim 16, wherein the method comprises:
executing the software analyzer, thereby generating a dependency graph (Athiwaratkun, para [0045-0046], Other context that may be extracted at 350 may be in-project context. In modern code development, classes and functions are usually defined in hierarchical structured files. Simple backward looking context does not include information outside of the current file, which may cause certain scenarios where machine learning model is not likely to generate the right code. As may code files use imported classes/functions/variables, adding this context may significantly improve code generation performance. Thus, in-project context may be added in some embodiments, where all imported classes, functions, and variables from the same project and used for obtaining a code completion suggestion. Para [0047], File and other context extraction 350 may perform regular expression based searches (e.g., for keywords such as “import”) and extractions to obtain the various types of context discussed above. In some embodiments, parsing-based extraction may be used (e.g., by generating a symbol tree or other parse graph of the code to obtain the other context information);
discerning that a first portion of a source code is dependent on a second portion of the source code according to the dependency graph(Athiwaratkun, para [0045-0046], Other context that may be extracted at 350 may be in-project context. In modern code development, classes and functions are usually defined in hierarchical structured files. Simple backward looking context does not include information outside of the current file, which may cause certain scenarios where machine learning model is not likely to generate the right code. As may code files use imported classes/functions/variables, adding this context may significantly improve code generation performance. Thus, in-project context may be added in some embodiments, where all imported classes, functions, and variables from the same project and used for obtaining a code completion suggestion. Para [0047], File and other context extraction 350 may perform regular expression based searches (e.g., for keywords such as “import”) and extractions to obtain the various types of context discussed above. In some embodiments, parsing-based extraction may be used (e.g., by generating a symbol tree or other parse graph of the code to obtain the other context information);
establishing that the second portion was changed after a submission of the first portion to at least one machine learning model(Athiwaratkun, para [0093], As indicated at 810, an event that triggers obtaining a code completion suggestion to include in a code file being edited using an integrated development environment may be detected, in some embodiments.); and
in response to the discerning and the establishing, resubmitting the first portion to at least one machine learning model with a prompt derived from the request (Athiwaratkun, para [0093-0094], As indicated at 820, generation of a code completion suggestion may be caused, in some embodiments. The code completion suggestion may be based on character(s) immediately preceding a cursor at a first time when the event that triggers the request for the code completion suggestion is detected, and where the code completion suggestion includes suggested character(s) to input to the code file immediately after the cursor at the first time, in some embodiments. In some embodiments, code suggestion may be locally implemented and performed (e.g., by a local sub-system). In some embodiments, the code suggestion may be generated remotely (e.g., as a feature of a code development service 210 in FIG. 2).).
Claim 18 is rejected for the reasons set forth hereinabove for claim 16, Jain and Athiwaratkum teach the computer-readable storage medium of claim 16, wherein the method further comprises:
discerning that a method in a source code was edited after a first submission of the method to a machine learning model, and after receiving a first result from the machine learning model in response to the first submission(Athiwaratkun, para [0093], As indicated at 810, an event that triggers obtaining a code completion suggestion to include in a code file being edited using an integrated development environment may be detected, in some embodiments.); and
in response to the discerning, submitting the method to the machine learning model in a second submission, while excluding from the second submission a portion of the source code which is changed but is independent of the method(Athiwaratkun, para [0093-0094], As indicated at 820, generation of a code completion suggestion may be caused, in some embodiments. The code completion suggestion may be based on character(s) immediately preceding a cursor at a first time when the event that triggers the request for the code completion suggestion is detected, and where the code completion suggestion includes suggested character(s) to input to the code file immediately after the cursor at the first time, in some embodiments. In some embodiments, code suggestion may be locally implemented and performed (e.g., by a local sub-system). In some embodiments, the code suggestion may be generated remotely (e.g., as a feature of a code development service 210 in FIG. 2). Para [0095-0096].).
Claim 19 is rejected for the reasons set forth hereinabove for claim 16, Jain and Athiwaratkum teach the computer-readable storage medium of claim 16, wherein determining the extent to which the software analyzer meets the requirement of the request comprises receiving at least part of an analysis plan from an analysis planning model, the analysis plan comprising: gathering a non-empty context, placing the context in a prompt, and submitting the prompt to at least one machine learning model, and wherein the context comprises control flow information(Jain, column 12, line 15 to 48, A user device (e.g., the user device 402a) and/or a module, component, or service of a development pipeline (e.g., a service 402b) can generate and transmit an output generation request to the data generation platform 102 (e.g., via the communication engine 112 of FIG. 1). An output generation request can include an indication of a requested output from a machine learning model. The output generation request can include an input, such as a prompt, an authentication token, and/or a user/device identifier of the requester. To illustrate, the output generation request can include a prompt (e.g., a query) requesting data, information, or data processing (e.g., from an LLM). The prompt can include a natural language question or command (e.g., in English). For example, the prompt includes a request for an LLM to generate code (e.g., within a specified programming language) that executes a particular operation. Additionally or alternatively, a prompt includes a data processing request, such as a request to extract or process information of a database (e.g., associated with one or more of the third-party databases 108a-108n). The output generation request can be transmitted to the data generation platform 102 using an API call to an API associated with the data generation platform 102 and/or through a graphical user interface (GUI).).
Claim 20 is rejected for the reasons set forth hereinabove for claim 16, Jain and Athiwaratkum teach the computer-readable storage medium of claim 16, wherein determining the extent to which the software analyzer meets the requirement of the request comprises receiving at least part of an analysis plan from an analysis planning model, the analysis plan comprising: gathering a non-empty context, placing the context in a prompt, and submitting the prompt to at least one machine learning model, and wherein the context comprises data flow information(Jain, column 11, line 50 to column 12, line 37d, FIG. 4 is a schematic illustrating a process 400 for validating model inputs and outputs, in accordance with some implementations of the present technology. For example, a user device 402a or a service 402b provides an output generation request (e.g., including an input, such as a prompt, and an authentication token) to the data generation platform 102 (e.g., to the access control engine 114 for access control 404 via the communication engine 112 of FIG. 1). The access control engine 114 can authenticate the user device 402a or service 402b by identifying stored tokens within an authentication database 412 that match the provided authentication token. The access control engine 114 can communicate the prompt to the breach mitigation engine 116 for input/output validation 406. The breach mitigation engine 116 can communicate with a sensitive token database 414 and/or a data-loss prevention engine 418, and/or an output validation model 420 for validation of prompts and/or LLM outputs. Following input validation, the performance engine 118 can evaluate the performance of LLMs to route the prompt to an appropriate LLM (e.g., large language model(s) 410). The data generation platform 102 can transmit the generated output to the output validation model 420 for testing and validation of the output (e.g., to prevent security breaches). The output validation model 420 can transmit the validated output to a data consumption system 422, for exposure of the output to the user device 402a and/or the service 402b. In some implementations, the data generation platform 102 can transmit metric values, records, or events associated with the data generation platform 102 to a metric evaluation database 416 (e.g., an event database) for monitoring, tracking, and evaluation of the data generation platform 102.).
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY KHUONG THANH NGUYEN whose telephone number is (571)270-7139. The examiner can normally be reached Monday - Friday 0800-1630.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lewis Bullock can be reached at 5712723759. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY KHUONG T NGUYEN/ Primary Examiner, Art Unit 2199
