DETAILED ACTION
This is the initial Office action based on the application submitted on March 22, 2024.
Claims 1-20 are pending.
In the interest of facilitating compact prosecution, the Examiner kindly asks the Applicant’s representative to authorize Internet communications with the Examiner by submitting Form PTO/SB/439 using Patent Center.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: LOGGING AND VISUALIZING EXECUTION FLOW IN A LARGE LANGUAGE MODEL (LLM) ENABLED USER QUERY APPLICATION.
Claim Objections
Claims 3, 8-10, 13, and 18-20 are objected to because of the following informalities:
Claims 3 and 13 recite “the stage.” It should read -- the stage of processing of the user query --.
Claims 8, 9, 18, and 19 recite “the summary.” It should read -- the summary of the structured log data --.
Claims 9 and 19 recite “processing user the user query.” It should read -- processing the user query --.
Claims 10 and 20 recite “the hierarchical view.” It should read -- the hierarchical view of the generated execution graph --.
Claims 10 and 20 recite “sub stages.” It should read -- sub-stages --.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claim Interpretation: Under the broadest reasonable interpretation (BRI), the limitations of Claim 1 are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP § 2111.
Claim 1 recites:
The step of “capturing, when processing a user query, structured log data from a plurality of software stacks in between a user application and a large language model, each of the plurality of software stacks following a predefined format to generate the structured log data.” The limitations “structured log data” and “a plurality of software stacks” are recognized as having the plain meanings of any structured log data and any software stacks, respectively. Furthermore, the claim does not impose any limits on how the structured log data is captured;
The step of “generating an execution graph as an ordered script from the captured structured log data.” This step only requires generating an execution graph from the captured structured log data. Furthermore, this step does not require the use of any specific process or component for making the generation; and
The step of “rendering a hierarchical view of the generated execution graph in a graphical user interface.” The claim does not limit the plain meaning of “rendering,” which, as understood by one of ordinary skill in the art, includes displaying. Furthermore, the claim does not impose any limits on how the hierarchical view of the generated execution graph is rendered or require any particular components that are used to perform the rendering.
In addition, these steps are recited as being implemented by a computer in the preamble of the claim. The computer is recited at a high level of generality, i.e., as a generic computer component performing generic computer functions.
Step 1: Claim 1 is directed to a computer-implemented method, which is a process (a series of steps or acts), and falls within one of the statutory categories of invention.
Step 2A, Prong One: Claim 1 recites the limitation:
(a) generating an execution graph as an ordered script from the captured structured log data.
The recited step, under the broadest reasonable interpretation (BRI), covers performance of the step in the human mind alone or with the aid of pen and paper. That is, other than reciting, in the preamble:
(1) [a] computer-implemented method.
Nothing in the claim precludes the step from practically being performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper. For example, the limitation (a) in the context of the claim encompasses a human observing structured log data in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper to generate an execution graph as an ordered script. See MPEP § 2106.04(a)(2)(III).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the human mind alone or with the aid of pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A, Prong Two: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional element, in the preamble:
(1) [a] computer-implemented method.
The additional element (1) is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components. The computer is used as a tool to perform the capturing, generating, and rendering steps of the claim. See MPEP § 2106.05(f).
Also, the claim recites the additional elements:
(2) capturing, when processing a user query, structured log data from a plurality of software stacks in between a user application and a large language model, each of the plurality of software stacks following a predefined format to generate the structured log data; and
(3) rendering a hierarchical view of the generated execution graph in a graphical user interface.
The additional elements (2) and (3) are mere data gathering/outputting recited at a high level of generality, and thus are insignificant extra-solution activities. See MPEP § 2106.05(g). Furthermore, all uses of the recited judicial exception require such data gathering/outputting, and, as such, the additional elements do not impose any meaningful limits on the claim. The additional elements amount to necessary data gathering/outputting. See MPEP § 2106.05.
Accordingly, even when viewed in combination, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as a combination do not amount to significantly more than the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the claim recites the additional element, in the preamble:
(1) [a] computer-implemented method.
The additional element (1) amounts to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept.
Also, the claim recites the additional elements:
(2) capturing, when processing a user query, structured log data from a plurality of software stacks in between a user application and a large language model, each of the plurality of software stacks following a predefined format to generate the structured log data; and
(3) rendering a hierarchical view of the generated execution graph in a graphical user interface.
The additional elements (2) and (3) simply append well-understood, routine, and conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception is not indicative of an inventive concept. MPEP § 2106.05(d)(II) expressly states that the courts have recognized the computer function of receiving or transmitting data over a network, e.g., using the Internet to gather data as a well‐understood, routine, and conventional computer function when it is claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activities. Thus, a person of ordinary skill in the art would readily comprehend that it is well-understood, routine, and conventional in the computing art to capture (or gather) structured log data and render a hierarchical view of an execution graph. Therefore, the limitations remain insignificant extra-solution activities even upon reconsideration and do not amount to significantly more.
Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the additional elements as a combination adds nothing that is not already present when looking at the additional elements taken individually. Even when considered in combination, the additional elements represent mere instructions to apply a judicial exception using generic computer components and insignificant extra-solution activities, and therefore do not provide an inventive concept. The claim is not patent eligible.
Claims 2-10 are dependent on Claim 1, but do not add any feature or subject matter that would solve the judicial exception deficiencies of Claim 1.
Claim 2 recites the limitations:
(a) the capturing of the structured log data comprising:
(b) capturing the structured log data from the plurality of software stacks within an orchestration layer in between the user application and the large language model.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 3 recites the limitations:
(a) the capturing of the structured log data comprising:
(b) capturing a stage of processing of the user query, an input into the stage, an output from the stage, a description of the stage, and metadata associated with the stage.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 4 recites the limitations:
(a) receiving the predefined format; and
(b) enforcing the predefined format across the plurality of software stacks.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 5 recites the limitations:
(a) the generating of the execution graph comprising:
(b) generating the execution graph using a portion of the captured structured log data having a predetermined persistency flag set.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 6 recites the limitation:
(a) pushing at least a portion of the captured structured log data to a long term storage.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 7 recites the limitation:
(a) pushing at least a portion of the captured structured log data organized by a transaction identification to a long term storage.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 8 recites the limitation:
(a) generating a summary of the structured log data, the summary comprising the execution graph.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 9 recites the limitation:
(a) generating a summary of the structured log data, the summary identifying the large language model and a latency of processing user the user query.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 10 recites the limitations:
(a) the rendering the hierarchical view comprising:
(b) rendering the hierarchical view configured to allow toggling between stages of processing the user query and corresponding sub stages.
Claims 4, 5, 8, and 9 recite further mental steps which can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper and thus, fail to make the claim any less abstract (see MPEP § 2106.04(a)(2)(III)).
Claims 2-4, 6, 7, and 10 recite further additional elements that do not integrate the judicial exception into a practical application of the judicial exception because they are mere data gathering/outputting recited at a high level of generality, and thus are insignificant extra-solution activities (see MPEP § 2106.05(g)), and thus, are not significantly more than the abstract idea.
Thus, Claims 2-10 do not add any steps or additional elements, when considered both individually and as a combination, that would convert Claim 1 into patent-eligible subject matter.
Therefore, Claims 1-10 are not drawn to patent-eligible subject matter as they are directed to an abstract idea without significantly more.
<<>> • × • <<>> • × • <<>> • × • <<>> • × • <<>> • × • <<>> • × • <<>> • × • <<>>
Claim Interpretation: Under the broadest reasonable interpretation (BRI), the limitations of Claim 11 are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP § 2111.
Claim 11 recites:
The step of “capturing, when processing a user query, structured log data from a plurality of software stacks in between a user application and a large language model, each of the plurality of software stacks following a predefined format to generate the structured log data.” The limitations “structured log data” and “a plurality of software stacks” are recognized as having the plain meanings of any structured log data and any software stacks, respectively. Furthermore, the claim does not impose any limits on how the structured log data is captured;
The step of “generating an execution graph as an ordered script from the captured structured log data.” This step only requires generating an execution graph from the captured structured log data. Furthermore, this step does not require the use of any specific process or component for making the generation; and
The step of “rendering a hierarchical view of the generated execution graph in a graphical user interface.” The claim does not limit the plain meaning of “rendering,” which, as understood by one of ordinary skill in the art, includes displaying. Furthermore, the claim does not impose any limits on how the hierarchical view of the generated execution graph is rendered or require any particular components that are used to perform the rendering.
In addition, these steps are recited as being performed by a non-transitory storage medium and a processor. The non-transitory storage medium and processor are recited at a high level of generality, i.e., as generic computer components performing generic computer functions.
Step 1: Claim 11 is directed to a system, which is a machine, and falls within one of the statutory categories of invention.
Step 2A, Prong One: Claim 11 recites the limitation:
(a) generating an execution graph as an ordered script from the captured structured log data.
The recited step, under the broadest reasonable interpretation (BRI), covers performance of the step in the human mind alone or with the aid of pen and paper. That is, other than reciting:
(1) a non-transitory storage medium storing computer program instructions; and
(2) a processor configured to execute the computer program instructions to cause operations […].
Nothing in the claim precludes the step from practically being performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper. For example, the limitation (a) in the context of the claim encompasses a human observing structured log data in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper to generate an execution graph as an ordered script. See MPEP § 2106.04(a)(2)(III).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the human mind alone or with the aid of pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A, Prong Two: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements:
(1) a non-transitory storage medium storing computer program instructions; and
(2) a processor configured to execute the computer program instructions to cause operations […].
The additional elements (1) and (2) are recited at a high-level of generality such that they amount to no more than mere instructions to apply the judicial exception using generic computer components. The non-transitory storage medium and processor are used as tools to perform the capturing, generating, and rendering steps of the claim. See MPEP § 2106.05(f).
Also, the claim recites the additional elements:
(3) capturing, when processing a user query, structured log data from a plurality of software stacks in between a user application and a large language model, each of the plurality of software stacks following a predefined format to generate the structured log data; and
(4) rendering a hierarchical view of the generated execution graph in a graphical user interface.
The additional elements (3) and (4) are mere data gathering/outputting recited at a high level of generality, and thus are insignificant extra-solution activities. See MPEP § 2106.05(g). Furthermore, all uses of the recited judicial exception require such data gathering/outputting, and, as such, the additional elements do not impose any meaningful limits on the claim. The additional elements amount to necessary data gathering/outputting. See MPEP § 2106.05.
Accordingly, even when viewed in combination, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as a combination do not amount to significantly more than the abstract idea. As discussed above with respect to integration of the abstract idea into a practical application, the claim recites the additional elements:
(1) a non-transitory storage medium storing computer program instructions; and
(2) a processor configured to execute the computer program instructions to cause operations […].
The additional elements (1) and (2) amount to no more than mere instructions to apply the judicial exception using generic computer components. Mere instructions to apply a judicial exception using generic computer components cannot provide an inventive concept.
Also, the claim recites the additional elements:
(3) capturing, when processing a user query, structured log data from a plurality of software stacks in between a user application and a large language model, each of the plurality of software stacks following a predefined format to generate the structured log data; and
(4) rendering a hierarchical view of the generated execution graph in a graphical user interface.
The additional elements (3) and (4) simply append well-understood, routine, and conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception is not indicative of an inventive concept. MPEP § 2106.05(d)(II) expressly states that the courts have recognized the computer function of receiving or transmitting data over a network, e.g., using the Internet to gather data as a well‐understood, routine, and conventional computer function when it is claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activities. Thus, a person of ordinary skill in the art would readily comprehend that it is well-understood, routine, and conventional in the computing art to capture (or gather) structured log data and render a hierarchical view of an execution graph. Therefore, the limitations remain insignificant extra-solution activities even upon reconsideration and do not amount to significantly more.
Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the additional elements as a combination adds nothing that is not already present when looking at the additional elements taken individually. Even when considered in combination, the additional elements represent mere instructions to apply a judicial exception using generic computer components and insignificant extra-solution activities, and therefore do not provide an inventive concept. The claim is not patent eligible.
Claims 12-20 are dependent on Claim 11, but do not add any feature or subject matter that would solve the judicial exception deficiencies of Claim 11.
Claim 12 recites the limitations:
(a) the capturing of the structured log data comprising:
(b) capturing the structured log data from the plurality of software stacks within an orchestration layer in between the user application and the large language model.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 13 recites the limitations:
(a) the capturing of the structured log data comprising:
(b) capturing a stage of processing of the user query, an input into the stage, an output from the stage, a description of the stage, and metadata associated with the stage.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 14 recites the limitations:
(a) receiving the predefined format; and
(b) enforcing the predefined format across the plurality of software stacks.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 15 recites the limitations:
(a) the generating of the execution graph comprising:
(b) generating the execution graph using a portion of the captured structured log data having a predetermined persistency flag set.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 16 recites the limitation:
(a) pushing at least a portion of the captured structured log data to a long term storage.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 17 recites the limitation:
(a) pushing at least a portion of the captured structured log data organized by a transaction identification to a long term storage.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 18 recites the limitation:
(a) generating a summary of the structured log data, the summary comprising the execution graph.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 19 recites the limitation:
(a) generating a summary of the structured log data, the summary identifying the large language model and a latency of processing user the user query.
<<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>> + <<>>
Claim 20 recites the limitations:
(a) the rendering the hierarchical view comprising:
(b) rendering the hierarchical view configured to allow toggling between stages of processing the user query and corresponding sub stages.
Claims 14, 15, 18, and 19 recite further mental steps which can be practically performed in the human mind alone using observation, evaluation, judgment, and opinion or with the aid of pen and paper and thus, fail to make the claim any less abstract (see MPEP § 2106.04(a)(2)(III)).
Claims 12-14, 16, 17, and 20 recite further additional elements that do not integrate the judicial exception into a practical application of the judicial exception because they are mere data gathering/outputting recited at a high level of generality, and thus are insignificant extra-solution activities (see MPEP § 2106.05(g)), and thus, are not significantly more than the abstract idea.
Thus, Claims 12-20 do not add any steps or additional elements, when considered both individually and as a combination, that would convert Claim 11 into patent-eligible subject matter.
Therefore, Claims 11-20 are not drawn to patent-eligible subject matter as they are directed to an abstract idea without significantly more.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-8, 11-14, and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over US 2018/0113783 (hereinafter “Luo”) in view of US 2005/0097110 (hereinafter “Nishanov”) and US 2025/0086439 (hereinafter “Paul”).
[Examiner’s Remarks: In order for a reference to be proper for use in an obviousness rejection under 35 U.S.C. 103, the reference must be analogous art to the claimed invention. In re Bigio, 381 F.3d 1320, 1325, 72 USPQ2d 1209, 1212 (Fed. Cir. 2004). A reference is analogous art to the claimed invention if: (1) the reference is from the same field of endeavor as the claimed invention (even if it addresses a different problem); or (2) the reference is reasonably pertinent to the problem faced by the inventor (even if it is not in the same field of endeavor as the claimed invention).
Note that the claimed invention is generally directed to logging and visualizing execution flow in large language model (LLM) enabled systems (specification, paragraph [0016]). As for the “same field of endeavor” test, Luo is generally directed to profiling the performance of an entire distributed software stack solely using unstructured log data generated by heterogeneous software components (Luo, paragraph [0022]). As for the “reasonably pertinent” test, Nishanov is generally directed to tracing program execution, for serializing data, such as an object data, into consistent, structured (e.g., XML) output (Nishanov, Abstract). And Paul is generally directed to integration of generative artificial intelligence (GAI) into enterprise operations (Paul, paragraph [0004]). Thus, Luo, Nishanov, and Paul are all analogous art to the claimed invention (even if they address different problems or are not in the same field of endeavor as the claimed invention). See MPEP § 2141.01(a)(I).]
As per Claim 1, Luo discloses:
A computer-implemented method (paragraph [0048], “FIG. 3 shows a process for obtaining and parsing log data.”) comprising:
capturing, when processing a user query, […] log data from a plurality of software stacks […] (paragraph [0034], “Each log processor 20 includes instructions to obtain and parse logs generated by the host 12 at which the log processor 20 is installed. Such logs may contain log messages that are generated by log statements written by human programmers and provided to an executable program operational at the host 12. This kind of log statement is often inserted by programmers to assist in debugging, and it is not necessary that such log statements conform to any standard or provide any specific information or hook. Additionally or alternatively, logs may contain log messages that are generated by an automated tool or program, such as a tracing tool [a plurality of software stacks]. This kind of log statement is often inserted automatically, for example under the direction of a human programmer, and may have a predictable format or hook, but not necessarily.”; paragraph [0081], “At block 90, processed log data is obtained. This may include receiving processed log data from a log processor via the network 14 [capturing {…} log data from a plurality of software stacks] (emphasis added).”; paragraph [0103], “As shown in FIG. 10, while a user can submit multiple queries, a query is always uniquely associated with a single user. Hence, USER to QUERY is a 1:n relationship [when processing a user query] (emphasis added).”);
generating an execution graph as an ordered script from the captured […] log data (paragraph [0077], “The log analyzer 24 may include instructions to construct an S3 graph, or similar visualization, and instantiate the S3 graph with object instances. Such a graph may have nodes defined by object identifiers. Events associated with the object identifiers may be determined and included in the visual representation. This may take the form of an event timeline.”; paragraph [0082], “At block 92, a graph, such as an S3 graph, is constructed by identifying how each object type is related to the other object types with respect to participating in the same event (emphasis added).”; paragraph [0092], “Information may be extracted from logs to identify objects, their interactions, and their hierarchical relationships [from the captured {…} log data]. An S3 graph, which may a directed acyclic graph (DAG), may be generated such that each node represents an object type and each directed edge captures a hierarchical relationship between a high-level object type (parent) and a low-level object type (child).”; paragraph [0116]1, “The GUI 26 may load the S3i graph as a JSON file [an execution graph as an ordered script] and displays each node and its events as a row in a two-panel layout as shown in FIGS. 12 and 13 (emphasis added).”); and
[1Examiner’s Remarks: Note that the Applicant’s specification expressly states that “[…] the server applications capture structured log data during the processing of a user request, generate an execution graph as an ordered script (e.g., in JSON format) from the captured structured log data, and render a hierarchical view of the generated execution graphical in the UIs 152” (paragraph [0018], emphasis added). Thus, under the broadest reasonable interpretation (BRI), the plain meaning of the limitation “an ordered script” includes a JSON file, which is consistent with the specification. Thus, the limitation “an ordered script,” given its plain meaning consistent with the specification, is mapped to Luo’s JSON file. See MPEP § 2173.01(I).]
rendering a hierarchical view of the generated execution graph in a graphical user interface (Figures 12 and 13; paragraph [0085], “At block 98, the result is outputted at a GUI. This may include displaying objects along a timeline and displaying a hierarchy of objects in play when servicing requests (emphasis added).”; paragraph [0116], “The GUI 26 may load the S3i graph as a JSON file and displays each node and its events as a row in a two-panel layout as shown in FIGS. 12 and 13 (emphasis added).”).
Luo discloses “log data,” but Luo does not explicitly disclose:
structured log data; and
each of the plurality of software stacks following a predefined format to generate the structured log data.
However, Nishanov discloses:
structured log data (paragraph [0046], “Via this architecture, checkpoints in an application program have their data (e.g., of objects) logged in a structured, serialized format (emphasis added).”); and
each of the plurality of software stacks following a predefined format to generate the structured log data (paragraph [0037], “In general, a managed application (assembly) 204 [each of the plurality of software stacks], comprising code 204a and metadata 204b, is configured for tracing by making calls (e.g., as coded by a tester) to a logging library of functions 206. For example, a tester, such as a person debugging a program, may add checkpoints at desired locations in the program code 204a, which when reached during execution will make the calls to the logging library 206 and provide associated data for logging, such as a reference to an object.”; paragraph [0046], “Via this architecture, checkpoints in an application program have their data (e.g., of objects) logged in a structured, serialized format (emphasis added).”).
As pointed out hereinabove, Luo and Nishanov are both analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Nishanov into the teaching of Luo to include “structured log data; and each of the plurality of software stacks following a predefined format to generate the structured log data.” The modification would be obvious because one of ordinary skill in the art would be motivated to perform queries, automated analysis, statistics gathering, and comparison of data against thresholds with structured log data (Nishanov, paragraph [0004]).
Luo discloses “a plurality of software stacks,” but the combination of Luo and Nishanov does not explicitly disclose:
[…] a plurality of software stacks in between a user application and a large language model.
However, Paul discloses:
[…] a plurality of software stacks in between a user application and a large language model (Figure 2; paragraph [0046], “In the example of FIG. 2, the orchestration component 220 includes workflow management, deployment and scaling, and API management. In some examples, the orchestration component 220 connects services with knowledge and datasets to orchestrate end-to-end flow of application interactions with LLMs (emphasis added).”).
As pointed out hereinabove, Paul is an analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Paul into the combined teachings of Luo and Nishanov to include “[…] a plurality of software stacks in between a user application and a large language model.” The modification would be obvious because one of ordinary skill in the art would be motivated to connect services with knowledge and datasets to orchestrate end-to-end flow of application interactions with LLMs (Paul, paragraph [0046]).
As per Claim 2, the rejection of Claim 1 is incorporated; and Luo further discloses:
the capturing of the structured log data comprising:
capturing the structured log data from the plurality of software stacks […] (paragraph [0034], “Each log processor 20 includes instructions to obtain and parse logs generated by the host 12 at which the log processor 20 is installed. Such logs may contain log messages that are generated by log statements written by human programmers and provided to an executable program operational at the host 12. This kind of log statement is often inserted by programmers to assist in debugging, and it is not necessary that such log statements conform to any standard or provide any specific information or hook. Additionally or alternatively, logs may contain log messages that are generated by an automated tool or program, such as a tracing tool. This kind of log statement is often inserted automatically, for example under the direction of a human programmer, and may have a predictable format or hook, but not necessarily.”; paragraph [0081], “At block 90, processed log data is obtained. This may include receiving processed log data from a log processor via the network 14.”).
Luo discloses “the plurality of software stacks,” but the combination of Luo and Nishanov does not explicitly disclose:
the plurality of software stacks within an orchestration layer in between the user application and the large language model.
However, Paul discloses:
the plurality of software stacks within an orchestration layer in between the user application and the large language model (Figure 2; paragraph [0046], “In the example of FIG. 2, the orchestration component 220 includes workflow management, deployment and scaling, and API management. In some examples, the orchestration component 220 connects services with knowledge and datasets to orchestrate end-to-end flow of application interactions with LLMs (emphasis added).”).
As pointed out hereinabove, Paul is an analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Paul into the combined teachings of Luo and Nishanov to include “the plurality of software stacks within an orchestration layer in between the user application and the large language model.” The modification would be obvious because one of ordinary skill in the art would be motivated to connect services with knowledge and datasets to orchestrate end-to-end flow of application interactions with LLMs (Paul, paragraph [0046]).
As per Claim 3, the rejection of Claim 1 is incorporated; and Luo further discloses:
the capturing of the structured log data comprising:
capturing a stage of processing of the user query […] (paragraph [0034], “Each log processor 20 includes instructions to obtain and parse logs generated by the host 12 at which the log processor 20 is installed. Such logs may contain log messages that are generated by log statements written by human programmers and provided to an executable program operational at the host 12. This kind of log statement is often inserted by programmers to assist in debugging, and it is not necessary that such log statements conform to any standard or provide any specific information or hook. Additionally or alternatively, logs may contain log messages that are generated by an automated tool or program, such as a tracing tool. This kind of log statement is often inserted automatically, for example under the direction of a human programmer, and may have a predictable format or hook, but not necessarily.”; paragraph [0081], “At block 90, processed log data is obtained. This may include receiving processed log data from a log processor via the network 14.”; paragraph [0103], “As shown in FIG. 10, while a user can submit multiple queries, a query is always uniquely associated with a single user. Hence, USER to QUERY is a 1:n relationship.”).
The combination of Luo and Nishanov does not explicitly disclose:
capturing […] an input into the stage, an output from the stage, a description of the stage, and metadata associated with the stage.
However, Paul discloses:
capturing […] an input into the stage, an output from the stage, a description of the stage, and metadata associated with the stage (Figure 2; paragraph [0036], “[…] the applications component 202 executes the logic and project-specific implementation for an application. In the example of FIG. 2, the applications component 202 includes non-limiting example applications of chatbots, voice assistants, and personalization engines.” and […] a personalization engine uses ML to analyze user data and behavior to provide personalized content, recommendations, and experiences. In some examples, and as described herein, the application includes a UI (e.g., a chatbot displayed in a GUI) or other user-accessible interface to enable users to provide user input and received output (e.g., textual content generated by a LLM that is responsive to the user input).”).
As pointed out hereinabove, Paul is an analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Paul into the combined teachings of Luo and Nishanov to include “capturing […] an input into the stage, an output from the stage, a description of the stage, and metadata associated with the stage.” The modification would be obvious because one of ordinary skill in the art would be motivated to connect services with knowledge and datasets to orchestrate end-to-end flow of application interactions with LLMs (Paul, paragraph [0046]).
As per Claim 4, the rejection of Claim 1 is incorporated; and the combination of Luo and Paul does not explicitly disclose:
receiving the predefined format; and
enforcing the predefined format across the plurality of software stacks.
However, Nishanov discloses:
receiving the predefined format (paragraph [0041], “In order for the serializer 210 to log an object into the appropriate structured, serialized format 216, the serializer 210 needs the type information for the various types of data that can be logged.”; paragraph [0046], “Via this architecture, checkpoints in an application program have their data (e.g., of objects) logged in a structured, serialized format.”); and
enforcing the predefined format across the plurality of software stacks (paragraph [0037], “In general, a managed application (assembly) 204, comprising code 204a and metadata 204b, is configured for tracing by making calls (e.g., as coded by a tester) to a logging library of functions 206. For example, a tester, such as a person debugging a program, may add checkpoints at desired locations in the program code 204a, which when reached during execution will make the calls to the logging library 206 and provide associated data for logging, such as a reference to an object.”; paragraph [0041], “In order for the serializer 210 to log an object into the appropriate structured, serialized format 216, the serializer 210 needs the type information for the various types of data that can be logged.”; paragraph [0046], “Via this architecture, checkpoints in an application program have their data (e.g., of objects) logged in a structured, serialized format.”).
As pointed out hereinabove, Nishanov is an analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Nishanov into the combined teachings of Luo and Paul to include “receiving the predefined format; and enforcing the predefined format across the plurality of software stacks.” The modification would be obvious because one of ordinary skill in the art would be motivated to perform queries, automated analysis, statistics gathering, and comparison of data against thresholds with structured log data (Nishanov, paragraph [0004]).
As per Claim 6, the rejection of Claim 1 is incorporated; and Luo further discloses:
pushing at least a portion of the captured structured log data to a long term storage (paragraph [0035], “A log processor 20 may be configured to locate active logs, parse log messages into a set of object identifiers and related data, and communicate such data to the server 22.”; paragraph [0040], “The results database 28 may store data received from the log processors 20 and data generated by the log analyzer 24.”; paragraph [0081], “At block 90, processed log data is obtained. This may include receiving processed log data from a log processor via the network 14.”).
As per Claim 7, the rejection of Claim 1 is incorporated; and Luo further discloses:
pushing at least a portion of the captured structured log data organized by a transaction identification to a long term storage (paragraph [0035], “A log processor 20 may be configured to locate active logs, parse log messages into a set of object identifiers and related data, and communicate such data to the server 22.”; paragraph [0040], “The results database 28 may store data received from the log processors 20 and data generated by the log analyzer 24.”; paragraph [0081], “At block 90, processed log data is obtained. This may include receiving processed log data from a log processor via the network 14.”).
As per Claim 8, the rejection of Claim 1 is incorporated; and Luo further discloses:
generating a summary of the structured log data, the summary comprising the execution graph (paragraph [0077], “The log analyzer 24 may include instructions to construct an S3 graph, or similar visualization, and instantiate the S3 graph with object instances. Such a graph may have nodes defined by object identifiers. Events associated with the object identifiers may be determined and included in the visual representation. This may take the form of an event timeline.”; paragraph [0082], “At block 92, a graph, such as an S3 graph, is constructed by identifying how each object type is related to the other object types with respect to participating in the same event.”; paragraph [0092], “Information may be extracted from logs to identify objects, their interactions, and their hierarchical relationships. An S3 graph, which may a directed acyclic graph (DAG), may be generated such that each node represents an object type and each directed edge captures a hierarchical relationship between a high-level object type (parent) and a low-level object type (child).”).
Claims 11-14 and 16-18 are system claims corresponding to the computer-implemented method claims hereinabove (Claims 1-4 and 6-8, respectively). Therefore, Claims 11-14 and 16-18 are rejected for the same reasons set forth in the rejections of Claims 1-4 and 6-8, respectively.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Nishanov and Paul as applied to Claims 1 and 11 above, and further in view of US 2020/0026787 (hereinafter “Alam”).
[Examiner’s Remarks: In order for a reference to be proper for use in an obviousness rejection under 35 U.S.C. 103, the reference must be analogous art to the claimed invention. In re Bigio, 381 F.3d 1320, 1325, 72 USPQ2d 1209, 1212 (Fed. Cir. 2004). A reference is analogous art to the claimed invention if: (1) the reference is from the same field of endeavor as the claimed invention (even if it addresses a different problem); or (2) the reference is reasonably pertinent to the problem faced by the inventor (even if it is not in the same field of endeavor as the claimed invention).
Note that the claimed invention is generally directed to logging and visualizing execution flow in large language model (LLM) enabled systems (specification, paragraph [0016]). As for the “reasonably pertinent” test, Alam is generally directed to implementing Internet of Things (IoT) applications (Alam, paragraph [0002]). Thus, Alam is an analogous art to the claimed invention (even if it addresses a different problem). See MPEP § 2141.01(a)(I).]
As per Claim 5, the rejection of Claim 1 is incorporated; and Luo further discloses:
the generating of the execution graph comprising:
generating the execution graph using a portion of the captured structured log data […] (paragraph [0077], “The log analyzer 24 may include instructions to construct an S3 graph, or similar visualization, and instantiate the S3 graph with object instances. Such a graph may have nodes defined by object identifiers. Events associated with the object identifiers may be determined and included in the visual representation. This may take the form of an event timeline.”; paragraph [0082], “At block 92, a graph, such as an S3 graph, is constructed by identifying how each object type is related to the other object types with respect to participating in the same event.”; paragraph [0092], “Information may be extracted from logs to identify objects, their interactions, and their hierarchical relationships. An S3 graph, which may a directed acyclic graph (DAG), may be generated such that each node represents an object type and each directed edge captures a hierarchical relationship between a high-level object type (parent) and a low-level object type (child).”).
The combination of Luo, Nishanov, and Paul does not explicitly disclose:
[…] a predetermined persistency flag set.
However, Alam discloses:
[…] a predetermined persistency flag set (paragraph [0041], “[…] the control logic 300 may include the steps of defining a set of events and a set of actions for the IT implementation based on a process flow and configuration metadata at step 301, creating a set of custom actions, for the IT implementation, based on the set of events, the set of actions, and the configuration metadata at step 302, and processing a set of messages received from the IoT device based on the process flow, the set of custom actions, and a set of pre-built actions at step 303.”; paragraph [0043], “It should be noted that the configuration metadata may include […] a persistency flag […].”).
As pointed out hereinabove, Alam is an analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Alam into the combined teachings of Luo, Nishanov, and Paul to include “[…] a predetermined persistency flag set.” The modification would be obvious because one of ordinary skill in the art would be motivated to allow a user to implement a persistency flag based on a process flow.
Claim 15 is a system claim corresponding to the computer-implemented method claim hereinabove (Claim 5). Therefore, Claim 15 is rejected for the same reason set forth in the rejection of Claim 5.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Nishanov and Paul as applied to Claims 1 and 11 above, and further in view of US 2024/0411824 (hereinafter “Sahu”).
[Examiner’s Remarks: In order for a reference to be proper for use in an obviousness rejection under 35 U.S.C. 103, the reference must be analogous art to the claimed invention. In re Bigio, 381 F.3d 1320, 1325, 72 USPQ2d 1209, 1212 (Fed. Cir. 2004). A reference is analogous art to the claimed invention if: (1) the reference is from the same field of endeavor as the claimed invention (even if it addresses a different problem); or (2) the reference is reasonably pertinent to the problem faced by the inventor (even if it is not in the same field of endeavor as the claimed invention).
Note that the claimed invention is generally directed to logging and visualizing execution flow in large language model (LLM) enabled systems (specification, paragraph [0016]). As for the “reasonably pertinent” test, Sahu is generally directed to a query gateway system that provides an efficient and flexible framework for providing context-retained autosuggest queries from an autosuggest query system to a generative language model system (Sahu, paragraph [0012]). Thus, Sahu is an analogous art to the claimed invention (even if it addresses a different problem). See MPEP § 2141.01(a)(I).]
As per Claim 9, the rejection of Claim 1 is incorporated; and Luo further discloses:
generating a summary of the structured log data (paragraph [0077], “The log analyzer 24 may include instructions to construct an S3 graph, or similar visualization, and instantiate the S3 graph with object instances. Such a graph may have nodes defined by object identifiers. Events associated with the object identifiers may be determined and included in the visual representation. This may take the form of an event timeline.”; paragraph [0082], “At block 92, a graph, such as an S3 graph, is constructed by identifying how each object type is related to the other object types with respect to participating in the same event.”; paragraph [0092], “Information may be extracted from logs to identify objects, their interactions, and their hierarchical relationships. An S3 graph, which may a directed acyclic graph (DAG), may be generated such that each node represents an object type and each directed edge captures a hierarchical relationship between a high-level object type (parent) and a low-level object type (child).”).
Luo discloses “the summary,” but the combination of Luo, Nishanov, and Paul does not explicitly disclose:
the summary identifying the large language model and a latency of processing user the user query.
However, Sahu discloses:
[…] identifying the large language model and a latency of processing user the user query (paragraph [0081], “In various implementations, the lightweight language model 532 is a smaller, simpler, and faster version of the sequence-to-sequence model 514. For example, while the sequence-to-sequence model 514 is an LLM, the lightweight language model 532 is an LSTM or another type of language machine-learning model. In many implementations, the lightweight language model 532 is a classifier-type model that operates within specified low latency parameters to ensure real-time processing (e.g., providing a reformulated autosuggest query within a few milliseconds).”; paragraph [0092], “As mentioned above, by utilizing the reformulated query cache 534 to identify reformulated autosuggest queries, the query gateway system 206 does not have to reprocess the same queries again, reducing wasted processing and improving efficiency. Furthermore, by utilizing an online lightweight language model and an offline classifier model, the query gateway system 206 capitalizes on resource efficiency to produce highly accurate results without sacrificing latency, making the user wait for the query gateway system 206 to process requests.”).
As pointed out hereinabove, Sahu is an analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Sahu into the combined teachings of Luo, Nishanov, and Paul to include “the summary identifying the large language model and a latency of processing user the user query.” The modification would be obvious because one of ordinary skill in the art would be motivated to utilize a classifier-type model that operates within specified low latency parameters to ensure real-time processing (e.g., providing a reformulated autosuggest query within a few milliseconds) (Sahu, paragraph [0081]).
Claim 19 is a system claim corresponding to the computer-implemented method claim hereinabove (Claim 9). Therefore, Claim 19 is rejected for the same reason set forth in the rejection of Claim 9.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Nishanov and Paul as applied to Claims 1 and 11 above, and further in view of US 2013/0204619 (hereinafter “Berman”).
[Examiner’s Remarks: In order for a reference to be proper for use in an obviousness rejection under 35 U.S.C. 103, the reference must be analogous art to the claimed invention. In re Bigio, 381 F.3d 1320, 1325, 72 USPQ2d 1209, 1212 (Fed. Cir. 2004). A reference is analogous art to the claimed invention if: (1) the reference is from the same field of endeavor as the claimed invention (even if it addresses a different problem); or (2) the reference is reasonably pertinent to the problem faced by the inventor (even if it is not in the same field of endeavor as the claimed invention).
Note that the claimed invention is generally directed to logging and visualizing execution flow in large language model (LLM) enabled systems (specification, paragraph [0016]). As for the “reasonably pertinent” test, Berman is generally directed to improving the applicability of spoken dialog systems to environments characterized by complex worker-system interaction (Berman, paragraph [0056]). Thus, Berman is an analogous art to the claimed invention (even if it addresses a different problem). See MPEP § 2141.01(a)(I).]
As per Claim 10, the rejection of Claim 1 is incorporated; and Luo further discloses:
rendering the hierarchical view (Figures 12 and 13; paragraph [0085], “At block 98, the result is outputted at a GUI. This may include displaying objects along a timeline and displaying a hierarchy of objects in play when servicing requests.”; paragraph [0116], “The GUI 26 may load the S3i graph as a JSON file and displays each node and its events as a row in a two-panel layout as shown in FIGS. 12 and 13.”).
Luo discloses “the hierarchical view,” but the combination of Luo, Nishanov, and Paul does not explicitly disclose:
[…] the hierarchical view configured to allow toggling between stages of processing the user query and corresponding sub stages.
However, Berman discloses:
[…] allow toggling between stages of processing the user query and corresponding sub stages (paragraph [0090], “Guidance may be provided by the system throughout a workflow 108, such as to prompt user to perform next step (such as asking which step the user would like to do); to display a current status for the install, with the new step highlighted; to toggle a display as visible/hidden; to change a display to show sub steps; to query whether the technician wants guidance or not; to provide business logic feedback using business rules and user input or user query; and to provide implicit or explicit confirmation for any and all procedures.”).
As pointed out hereinabove, Berman is an analogous art to the claimed invention. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Berman into the combined teachings of Luo, Nishanov, and Paul to include “[…] the hierarchical view configured to allow toggling between stages of processing the user query and corresponding sub stages.” The modification would be obvious because one of ordinary skill in the art would be motivated to prompt a user to perform next step (such as asking which step the user would like to do) (Berman, paragraph [0090]).
Claim 20 is a system claim corresponding to the computer-implemented method claim hereinabove (Claim 10). Therefore, Claim 20 is rejected for the same reason set forth in the rejection of Claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the Applicant’s disclosure. They are as follows:
US 2015/0347243 (hereinafter “Guerin”) discloses logging transactions within a distributed processing system.
US 2016/0098342 (hereinafter “Faizanullah”) discloses logging service requests in computer systems.
US 2017/0076012 (hereinafter “Sreenivasa”) discloses processing log files using a database system.
US 2019/0012254 (hereinafter “Gupta”) discloses a software analytics platform.
US 2020/0110654 (hereinafter “Oleszkiewicz”) discloses creating a log file.
US 2021/0392202 (hereinafter “Henning”) discloses artificial intelligence log processing and content distribution network (CDN) optimization.
Any inquiry concerning this communication or earlier communications from the Examiner should be directed to Qing Chen whose telephone number is 571-270-1071. The Examiner can normally be reached on Monday through Friday from 9:00 AM to 5:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, the Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at https://www.uspto.gov/ interviewpractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, Wei Mui, can be reached at 571-272-3708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for more information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO customer service representative, call 800-786-9199 (in USA or Canada) or 571-272-1000.
/Qing Chen/
Primary Examiner, Art Unit 2191