DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1, 3, 10, 12, 19, 22, 26 are amended. Claims 2, 4, 11, 13, and 21 are canceled. Claims 27-29 are new. Claims 1, 3, 7, 8, 10, 12, 16, 17, 19-20, 22-29 are pending and are examined herein.
Claims 3, 12, 26-29 are rejected under 35 U.S.C. 112(b).
Claims 1, 3, 7, 8, 10, 12, 16, 17, 19-20, 22-29 are rejected under 35 U.S.C. 101.
Claims 1, 3, 7, 8, 10, 12, 16, 17, 19-20, 22-29 are rejected under 35 U.S.C. 103.
Response to Amendment
The amendment filed April 28th, 2026 has been entered. Claims 1, 3, 10, 12, 19, 22, 26 are amended. Claims 2, 4, 11, 13, and 21 are canceled. Claims 27-29 are new. Claims 1, 3, 7, 8, 10, 12, 16, 17, 19-20, 22-29 are pending and are examined herein.
Response to Arguments
Applicant's arguments filed April 28th, 2026 regarding the 35 U.S.C. 101 rejection of claims 1 – 4, 7, 8, 10 – 13, 16, 17, 19 – 26 have been fully considered but they are not persuasive. Applicant argues, on pages 9-10, that the claims improve computer related technology and artificial intelligence technology by enabling a user to construct a logical graph from subgraphs, providing rewards and other factors through a GUI, using a reinforcement learning model to simulate paths through the graph, and refining the graph based on the simulations. Applicant relies on paragraphs [0005-0006],[0017-0023] of the specification. The cited portions of the specification have been considered. Also, Applicant’s reliance on the August 4, 2025 subject matter eligibility memorandum is acknowledged and examiner agrees that a claim itself does not need to explicitly recite the improvement. However, the claim must still recite the components or steps that provide the disclosed technical improvement.
The improvement discussed in paragraph [0005] relies on enabling rewards to be quickly inserted into the logical graph through the GUI. Independent claims 1, 10, and 19 do not require rewards. Although claims 3 and 12 recite receiving positive or negative rewards, those claims broadly use the rewards to influence which paths the agents select. The claims do not recite a particular improvement to the operation of the GUI or reinforcement learning model resulting from the use of the rewards.
The improvement discussed in paragraph [0006] relies on dividing a complex logical graph into smaller subsections that can be individually created, tested, combined, and simulated. The independent claims recite subgraphs that flow into one another and may provide alternative paths, but they do not require the subgraphs to be individually tested. Therefore, the claims do not recite the complete process identified in paragraph [0006] as providing the disclosed improvement.
Paragraphs [0017] – [0023] describe using a logical graph and reinforcement learning model to generate synthetic data and refining the graph over multiple simulations. Some portions of that process are now recited in the independent claims. However, the claims do not recite how the user created factors are applied to control path selection or how the simulations are analyzed to identify and add the additional nuanced steps. Instead, these limitations state the desired functions and results at a high level. Also, the resulting improvement concerns the accuracy or representativeness of the modeled information and synthetic data, rather than a claimed improvement to the operation of the computer, GUI, simulation system, or reinforcement learning model.
The dependent claims do not change this conclusion. Those claims further limit the information used or displayed during the simulations or broadly apply the synthetic data in model training and verification steps, but do not recite a specific technological mechanism that improves the operation of the computer or AI model. Accordingly, when considered as a whole, the claims use the GUI and reinforcement learning model as tools to create, simulate, and refine a logical representation of a situation and generate synthetic data. The additional elements do not integrate the abstract idea into a practical application and, for the reasons set forth in the rejection below, do not amount to significantly more than the abstract idea. Therefore, the rejection under 35 U.S.C. 101 is maintained.
Applicant's arguments filed April 28th, 2026 regarding the 35 U.S.C. 103 rejection of claims 1 – 4, 7, 8, 10 – 13, 16, 17, 19 – 26 have been fully considered but they are not persuasive.
Applicant argues that Floren, Dechene, and Gutierrez do not teach the reinforcement learning model choosing paths through the logical graph based on categorical or continuous factors created by the user within the GUI, or adding additional nuanced steps by analyzing the simulations.
As discussed in the rejection below, the rejection relies on the combined teachings of the references. Floren teaches a GUI through which a user creates and links graph elements and subgraphs. Dechene teaches a reinforcement learning routing agent whose actions are affected by rewards and user configured policy dimensions. Dechene’s selectable data type and scenario options correspond to categorical factors, while its slider based and numerical settings correspond to continuous factors. Gutierrez teaches repeated simulations that generate synthetic data and the iterative analysis, modification, and addition of simulation model behaviors and actions. Therefore, it would have been obvious to use Dechene’s reinforcement learning agent to choose paths through Floren’s user created graph based on Dechene’s user configured factors, and to use Floren’s graph editing functionality to add steps identified through Gutierrez’s simulation analysis and modification process.
Applicant does not present separate substantive arguments for the dependent claims but relies on their dependency from the independent claims. Because the arguments concerning the independent claims are not persuasive, the dependency argument is likewise not persuasive. The additional limitations of the dependent claims are addressed separately in the rejections below.
Claim Objections
Claims 1, 10, 19 are objected to because of the following informalities: “an artificial intelligence (AI) model the AI model being a reinforcement learning model” should include appropriate punctuation like “an artificial intelligence (AI) model, the AI model being a reinforcement learning model”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 12, 26-29 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 3 and 12 recite the limitation “simulating, using a graphical processing unit to graphically simulate, the number of times using the AI model” but does not identify what is being graphically simulated. “The number of times” only identifies a quantity recited in the underlying independent claim rather than an object capable of being simulated. Also, the repeated recitations of “simulating” and “to graphically simulate” make it unclear whether the limitation requires a separate simulation operation or merely specifies hardware for performing the simulation inherited from independent claims. Therefore, the scope of the claimed simulation operation is unclear. For examination purposes, the limitation would refer back to “the situation” as recited previously from independent claims to refer same situation.
Claim 26 recites “wherein the controller detect changes to states or the rewards when running a job and retraining the reinforcement learning model.” It is unclear whether the claim requires the controller to detect the changes while the controller is both running a job and already retraining the reinforcement learning model, or whether the controller detects the changes when running the job and, in response to the detected changes, retrains the reinforcement learning model. Based on paragraph [0036] of specification, the latter sequence seems more appropriate, but the claim language does not clearly recite that sequence. For examination purposes, it would be read as “wherein, in response to detecting changes to states or rewards when running a job, the controller retrains the reinforcement learning model.”
Claims 27 – 29 are dependent on claim 26. They do not resolve the issue of indefiniteness and are rejected with the same rationale.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3, 7, 8, 10, 12, 16, 17, 19-20, 22-29 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
MPEP § 2109(III) sets out steps for evaluating whether a claim is drawn to patent-eligible subject matter. The analysis of claims 1, 3, 7, 8, 10, 12, 16, 17, 19-20, 22-29 in accordance with these steps, follows.
Step 1 Analysis:
Step 1 is to determine whether the claim is directed to a statutory category (process, machine, manufacture, or composition of matter.
Claims 1, 3, 7, 8, 22 – 29 are directed to a method, meaning that it is directed to the statutory category of process. Claims 10, 12, 16, 17 are directed to a system, which is also the statutory category of machine. Claims 19 – 20 are directed to a computer program product, which can be an article of manufacture.
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
Regarding claim 1, the following claim elements are abstract ideas:
guiding… a user to generate a logical graph including a plurality of subgraphs that represents how real data is generated in a situation; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
,wherein subsequent subgraphs flow logically into each other to create the logical graph, (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
and wherein at least some subgraphs of the plurality of subgraphs are alternatives of each other within the situation; (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
and simulating … the situation a number of times by having the AI model choose paths through the logical graph based on factors created by the user within the GUI to generate synthetic data that is representative of the real data, such that the synthetic data can be used for training other AI models. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
and adding … additional nuanced steps within the logical graph (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
via analyzing the simulations of the AI model, (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
The following claim elements are additional elements which, taken alone or in combination with the other additional elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
using a graphical user interface (GUI), using an artificial intelligence (AI) model the AI model being a reinforcement learning model, (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
wherein the factors created by the user are categorial or continuous factors, … based on factors created by the user within the GUI (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 3, the rejection of claim 1 is incorporated herein. Further, claim 3 recites the following abstract ideas:
the positive or negative rewards define why agents of the AI model would choose different paths of the logical graph (Defining rule or reasons for selection is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components.)
Claim 3 further recites following additional elements:
receiving positive or negative rewards associated with steps of the logical graph (Receiving positive or negative rewards is a well-understood, routine conventional activity in the field of AI. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
wherein the logical graph includes a plurality of nodes connected via edges (Logical graph with plurality of nodes connected via edges is a well-understood, routine conventional activity in the field of logical or knowledge graph models. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
training the AI model by having the agents of the AI model choose the different paths based on the positive or negative rewards. (Training AI model to make choice based on the positive or negative rewards is a well-understood, routine conventional activity in the field of AI. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
simulating … the number of times using the AI model (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
using a graphical processing unit to graphically simulate (Using GPU to graphically simulate is a well-understood, routine conventional activity in the field of AI. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
Regarding claim 7, the rejection of claim 1 is incorporated herein. Further, claim 7 recites the following additional element:
training another AI model with the synthetic data such that no real-world data is used to train the another AI model. (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 8, the rejection of claim 1 is incorporated herein. Further, claim 8 recites the following additional element:
guiding the user to generate the logical graph includes recommending one or more steps for the user to add into the logical graph. (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 10, the following claim elements are additional elements:
a processor; and a memory in communication with the processor, the memory containing instructions that, when executed by the processor (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
The rest of claims 10 and 12, 16, 17 recite substantially similar subject matter to claims 1 and 3, 7,8 respectively and are rejected with the same rationale, mutatis mutandis.
Regarding claim 19, the following claim elements are additional elements:
the computer program product comprising a computer readable storage medium having program instructions embodied therewith (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
The rest of claim 19 recites substantially similar subject matter to claim 1 respectively and is rejected with the same rationale, mutatis mutandis.
Regarding claim 20, the rejection of claim 19 is incorporated herein. Further, claim 20 recites the following abstract ideas:
wherein the logical graph is a simplified version of the situation (Simplifying situation to create logical graph is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components, or by a human using a pen and paper.)
The rest of claim 20 recites substantially similar subject matter to claim 1 respectively and is rejected with the same rationale, mutatis mutandis.
Regarding claim 22, the rejection of claim 1 is incorporated herein. Further, claim 22 recites the following abstract ideas:
setting a statistical relationship between the factors (Setting a statistical relationship between the factors recites a mathematical relationship, which is mathematical concept.)
creating one or more simulation states; and (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
creating one or more simulation flow graphs … based on the one or more simulation states created, and wherein each node in the one or more simulation flow graphs represents a simulation step and each edge establishes a way-point between two states. (This is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 22 further recites following additional element:
created by the user within the GUI; (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
using a node-graph canvas (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 23, the rejection of claim 22 is incorporated herein. Further, claim 23 recites the following abstract idea:
to reinforce a particular outcome for a simulation using logical operators, (Selecting a desired outcome and assigning rewards is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
Claim 23 further recites following additional elements:
detecting, by a controller, the user connecting two state nodes; (This is mere data gathering, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
and prompting, by the controller, the user within the GUI to add rewards … (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
wherein the user is provided different quantities of the rewards to the way-point between the two states. (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 24, the rejection of claim 23 is incorporated herein. Further, claim 24 recites the following additional element:
providing a review dashboard to the user within the GUI, (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
wherein the review dashboard includes sectional insights to different aspects of a configuration of the simulation. (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 25, the rejection of claim 24 is incorporated herein. Further, claim 25 recites the following additional element:
wherein the sectional insights include at least factors, statistical relationships, time series, the one or more simulation states, and the rewards for the logical graph, and wherein the GUI further enables the user to combine the simulation with cumulative metrics of previously run simulations. (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 26, the rejection of claim 1 is incorporated herein. Further, claim 26 recites the following abstract idea:
retraining, by a controller, the reinforcement learning model based on edits to rewards or simulation flows by the user. (Iterative training of reinforcement learning with rewards merely recites mathematical relationship, which is mathematical concept.)
Claim 26 further recites following additional elements
wherein the controller detect changes to state or the rewards when running a job and retraining the reinforcement learning model. (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 27, the rejection of claim 1 is incorporated herein. Further, claim 26 recites the following abstract idea:
combining… a plurality of simulations, wherein each of the plurality of simulations correspond to one of the plurality of subgraphs of the logical graph… (Combining simulations is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
wherein the logical graph includes cumulative metrics from each of the plurality of simulations. (Cumulative metrics from each of the plurality of simulations could pertain aggregating or calculating results or statistical metrics, which recites mathematical concept.)
Claim 27 further recites following additional elements:
by the controller, (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 28, the rejection of claim 27 is incorporated herein. Further, claim 28 recites the following additional elements:
generating … a full set of synthetic data (This is mere data outputting, an insignificant extra solution activity, which is a well-understood, routine conventional activity. It does not integrate the judicial exception into a practical application. See MPEP § 2106.05(d). Therefore, this does not amount to significantly more than the judicial exception.)
… by the controller, (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
training the reinforcement learning model using the full set of synthetic data, (These are merely specifying a type of data to use. See MPEP § 2106.05(g). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
wherein the training includes both a training step and a verification stage, (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
wherein both the training step and the verification stage are conducted with the full set of synthetic data. (These are merely specifying a type of data to use. See MPEP § 2106.05(g). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Regarding claim 29, the rejection of claim 28 is incorporated herein. Further, claim 29 recites the following abstract idea:
threshold and preference data include thresholds that define a manner in which … the generating of the full set of synthetic data (Setting thresholds or rules for such computer task execution is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper.)
wherein the thresholds define when … the user provide the rewards or use a specific state or factor. (Comparing values against threshold to suggest specific action is practical to perform in the human mind under its broadest reasonable interpretation aside from the recitation of generic computer components or by a human using a pen and paper. It can also recite mathematical relationships, which is mathematical concept.)
Claim 29 further recites following additional element
the controller manages… the controller suggests… (These are mere instructions to apply abstract idea on a generic computer. See MPEP § 2106.05(f). Therefore, this does not amount to significantly more than the judicial exception and does not integrate the abstract idea into a practical applications.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 7, 10, 16, 19, 20, 22, 26 are rejected under 35 U.S.C. 103 as being unpatentable over Floren et al. (U.S. Pub. 2022/0075515) in view of Dechene et al. (U.S. Pub. 2022/0245462), further in view of Gutierrez et al. (U.S. Pub. 11847390).
Regarding Claim 1, Floren teaches
A computer-implemented method comprising: guiding, using a graphical user interface (GUI), a user to generate a logical graph including a plurality of subgraphs that represents how real data is generated in a situation, wherein subsequent subgraphs flow logically into each other to create the logical graph, ([0167] of Floren states “Referring to FIG. 8A, an example user interface 800 includes an interactive graph section 802 in which various systems, subsystems, and data objects can be represented by nodes or indicators, such as icons 804 and 806. For ease of description, the information shown in the GUIs of the present disclosure is generally referred to as objects, but as noted various systems and subsystems may similarly be represented. As described throughout the present disclosure, the systems, subsystems, and objects may represent various things, such as people, locations, facilities, and the like. Relationships among the various systems, subsystems, and objects are represented by edges, such as edge 808, which may optionally be directional (or bi-directional) to indicate, e.g., flows of information or items… For example, the user may add additional related objects/nodes to the graph section 802 in various ways, including by searching the system for objects related to already displayed objects/nodes. FIG. 8C illustrates a toolbar by which the user may select to add objects to the graph via button 818.” [0168] of Floren states “The example user interface 800 provides a view of a simulated technical system representing a real-world system. The view may include various technical systems, subsystems, objects, and the like. Although not shown in the user interface, the system can associate various data, including time-based data, and models with the systems, subsystems, and objects, such that simulations can be run.” [0183] of Floren states “Referring to FIGS. 8I-8J, example user interface portions 860-862 are shown which may comprise portions of, or updates to, user interface 800. The user interface portions 860-862 illustrate system functionality related to subgraphs. Subgraphs provide another way to abstract away parts of a larger, more complicated graph. User interface portion 860 illustrates that the user can select to create a subgraph from the ‘. . . ’ menu on the top navigation bar breadcrumbs. In other implementations other buttons or GUI functionality may be provided for the user to create a subgraph. In response, in user interface portion 861, which can comprise an overlaid GUI portion, or a separate GUI portion, the user can fill in details of the subgraph just like a regular graph, can name the subgraph, and can then link the subgraph back to the parent graph. User interface portion 862 (of FIG. 8J) illustrates that, back in the parent graph, the user can link the created subgraph to the parent graph (e.g., the user can add a ‘Region’ to link to the subgraph on click). The user can also add any number of ‘Text’ or ‘Note’ items to achieve the desired view. In various implementations, the user may also link the subgraph to related objects of the parent graph.” [0136] of Floren states “The model connector 404 may connect two or more models together via chaining, where the chaining occurs by linking predicted nodal relationships of parameter output nodes of one model with parameter input nodes of another model. In an embodiment, a link between a parameter output node of one model and parameter input node of another model may be established by the model connector 404 based on similar or matching parameter output nodes and parameter input nodes (e.g., the parameter output node of one model matches the parameter input node of another model), where the nodes may include model specific data comprised of subsystems, objects, and/or object properties (e.g., property types and/or property values).” Floren expressly teaches a GUI in which the user creates a node and edge graph, creates subgraphs, links the subgraphs to a parent graph, and uses the graph to represent a simulated real world system.)
Floren does not explicitly teach
and wherein at least some subgraphs of the plurality of subgraphs are alternatives of each other within the situation
and simulating, using an artificial intelligence (AI) model the AI model being a reinforcement learning model, the situation a number of times by having the AI model choose paths through the logical graph based on factors created by the user within the GUI to generate synthetic data that is representative of the real data, wherein the factors created by the user are categorical or continuous factors such that the synthetic data can be used for training other AI models;
adding, via analyzing the simulations of the AI model, additional nuanced steps within the logical graph.
However, Dechene teaches that
and wherein at least some subgraphs of the plurality of subgraphs are alternatives of each other within the situation ([0162] of Dechene states “In several embodiments, the reinforcement-learning model can include a hierarchical reinforcement learning model, as shown in FIGS. 5 and 9, and described above. In several embodiments, multiple alternative versions of the routing agent model can be trained on traffic generated from different traffic profiles.” [0155] of Dechene states “A model can be set as the primary, with additional models set as alternates. Alternate models can allow the system to quickly rollback in the event of model failure, while also maintaining different models to be quickly applied during operational scenarios” Floren teaches subgraphs in a graph based GUI, including creating subgraphs and linking subgraphs to a parent graph. Dechene teaches multiple alternative versions of an AI routing agent model trained under different traffic profiles and further teaches alternate models for different operational scenarios. Accordingly, it would have been obvious to represent at least some of Floren’s subgraphs as alternative subgraphs (e.g., alternative scenario branches) for the same simulated situation. )
and simulating, using an artificial intelligence (AI) model the AI model being a reinforcement learning model, the situation a number of times ([0034] of Dechene states “The method also can include training a routing agent model on the digital twin network simulation using a reinforcement-learning model on traffic that flows through nodes of the digital twin network simulation.” [0168] of Dechene states “In a number of embodiments, method 2000 further can include an activity 2040 of training the routing agent model on the digital twin network simulation using the reinforcement-learning model on traffic that flows through nodes of the digital twin network simulation. The routing agent model can be similar or identical to AI agent routing service 316 (FIG. 3), agent 430 (FIGS. 4 and 7), agent 530 (FIG. 5), agent model 1251 (FIG. 12), and AI agent 1264 (FIG. 12)… In some embodiments, the routing agent model can include a machine-learning model, such as a neural network, a random forest model, a gradient boosted model, and/or another suitable model. In a number of embodiments, the reinforcement-learning model can include a deep-Q meta-reinforcement learning model.” [0178] of Floren states “Via the simulation panel 842, the user can specify inputs, outputs, and models. Further, the user can run multiple simulations, as represented by columns 843 and 844.”. In combination, simulation can be done multiple times in reinforcement learning model operating in a node based digital twin simulation.)
by having the AI model choose paths through the logical graph based on factors created by the user within the GUI ([0062] of Dechene states “An AI agent, such as an agent 430, takes a series of actions (e.g., an action 421) within an episode (e.g., 410), known as steps (e.g., a step 420). Each action (e.g., 421) can be informed by observations of a state 432 of an environment 440 (e.g., a training or live environment of a computer network) and an expected reward (e.g., a reward 423).” [0151] of Dechene states “In several embodiments, the user can specify the training scenario through interactive buttons, sliders, and editable text fields in training scenario component 1730. The user can customize policy tradeoffs and optimize data flow through the network, effectively tuning the RL model and its hyperparameters in accordance with the user's subject matter expertise and intent. Network speed and reliability, priority data type, and expected seasonal traffic variation are examples of the type of dimensions the user can create and modify.” Dechene expressly teaches user created and modified factors within a GUI that control the reinforcement learning simulation and AI agent choosing routes of the action based on factors. It would have been obvious to apply Dechene’s node based routing agent to Floren’s logical graph.)
wherein the factors created by the user are categorical or continuous factors ([0151] of Dechene states “For example, as shown in training scenario component 1730, a user can select or de-select an option 1731 to prefer routes where the router CPU is low, select or de-select an option 1732 to include partial and/or total link failures, select or de-select an option 1733 to include partial and/or total node failures, use a slider 1734 to specify a setting between prioritizing voice and prioritizing video, use a slider 1735 to specify a setting between shortest path for delay sensitive traffic and stable path for jitter sensitive traffic, and/or use sliders 1736 to specify a level of seasonal demand, such as a slider 1737 for fall demand, a slider 1738 for winter demand, a slider 1739 for spring demand, and/or a slider 1740 for summer demand.” The binary or selectable options correspond to categorical factors and the adjustable slider values correspond to continuous factors.)
Gutierrez teaches that
to generate synthetic data that is representative of the real data, such that the synthetic data can be used for training other AI models. (Column 8 Lines 49 – 62 of Gutierrez states “A generative model, as used herein, is used to describe models that generate instances of output variables that may be used for machine learning. A generative model may generate synthetic data that may be input into various machine learning models. A generative model may be referred to as a representation of a data distribution that may be used to generate data points. In some situations, a good generative model may be treated as a source of synthetic data—e.g., data that is realistic but not actual, real-world data. Multiple approaches exist for generating synthetic data including, but not limited to, generative adversarial networks, variational auto encoders, probabilistic graphical models, and agent-based models.” Column 10 Lines 20 – 24 of Gutierrez states “FIG. 3 is an example of a flow chart describing a process for creating synthetic data from true-source data. The synthetic dataset may be used to train a machine learning model or may be used to augment existing data and the combination used to train the machine learning model.” )
adding, via analyzing the simulations of the AI model, additional nuanced steps within the logical graph. ([0167] of Floren states “For example, the user may add additional related objects/nodes to the graph section 802 in various ways, including by searching the system for objects related to already displayed objects/nodes. FIG. 8C illustrates a toolbar by which the user may select to add objects to the graph via button 818.” Column 23 lines 49 – 55 of Gutierrez states “Alternatively or additionally, in step 1111, the system may receive instructions to add a new agent probability distribution definition and/or a new behavior probability distribution definition. In step 1112, the new agent and/or new behavior probability distribution definition may be added to the simulation specification 1100 for the new generation of a specification state.” Column 37 lines 66 – Column 38 lines 20 of Gutierrez states “Alternatively or additionally, from reference E, the system may determine statistical and/or correlation parameters of the generated data in step 1502. After the determination of the statistical or correlation parameters in step 1502, the system may receive modifications (step 1503) of the statistical parameters or correlation parameters of the scrubbed data model and/or the generative model as described above. Alternatively or additionally, after the determination of the statistical and/or correlation parameters of the generated dataset, the parameters of the generated dataset may be compared, in step 1504, with the expected parameters of the scrubbed data model and/or those of the generative model. Based on the comparison of step 1504, modifications may be received in step 1503 of the statistical and/or correlation parameters, the scrubbed data model of 1409 and/or the generative model of 1410 may be modified in step 1413 of FIG. 14 . Another generative model may be trained based on the modified scrubbed data model 1409 and another synthetic dataset generated in step 1411 or, if modifying the generative model directly, the another synthetic dataset may be generated in step 1411 once the generative model has been modified.” Gutierrez teaches analyzing simulation generated data and modifying the model based on the analysis and adding additional steps corresponding to the analysis. The added behaviors or actions from Gutierrez would have been an additional graph steps in combination with Floren.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings from Floren with the combination of Gutierrez and Dechene. Floren teaches an interactive node/edge graph GUI with subgraphs for representing and modifying a simulated technical system. Gutierrez teaches simulation state generation and generation of synthetic data from simulation states. Dechene teaches use of reinforcement learning model in a simulated environment to make decisions and train a model, including dashboard functionality for simulation results. One with the ordinary skill in the art would be motivated to incorporate the teachings of Gutierrez and Dechene into Floren to improve Floren’s graph-based simulation interface with explicit simulation state semantics and configurable statistical relationships, and reinforcement learning driven path decisioning, user adjustable factors/reward behavior, and simulation review functionality through dashboard. These are directed to compatible computer implemented simulation workflows and would have predictably improved the ability to define, execute, and iteratively refine simulations for AI-driven applications.
Regarding Claim 7, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Floren, Gutierrez and Dechene teaches
training another AI model with the synthetic data such that no real-world data is used to train the another AI model. (Column 37, Lines 60 – 65 of Gutierrez states “The synthetic dataset may be used in various ways including, for instance, training another machine learning model, modeling a database, or comparing the synthetic dataset with other datasets to possibly determine whether the other datasets represent actual data or synthetic data.” Column 8 Lines 49 – 62 of Gutierrez states “A generative model, as used herein, is used to describe models that generate instances of output variables that may be used for machine learning. A generative model may generate synthetic data that may be input into various machine learning models. A generative model may be referred to as a representation of a data distribution that may be used to generate data points. In some situations, a good generative model may be treated as a source of synthetic data—e.g., data that is realistic but not actual, real-world data. Multiple approaches exist for generating synthetic data including, but not limited to, generative adversarial networks, variational auto encoders, probabilistic graphical models, and agent-based models.”)
Claims 10, 16 recite substantially similar subject matter as claims 1 and 7 respectively, and are rejected with the same rationale, mutatis mutandis.
Claim 19 recites substantially similar subject matter as claims 1 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 20, the rejection of claim 19 is incorporated herein. Furthermore, the combination of Floren, Gutierrez and Dechene teaches
wherein the logical graph is a simplified version of the situation ([0183] of Floren states “ The user interface portions 860-862 illustrate system functionality related to subgraphs. Subgraphs provide another way to abstract away parts of a larger, more complicated graph. “ Therefore, the graph and subgraphs of Floren would provide a simplified representation of a more complicated situation. )
The rest of claim 20 recites substantially similar subject matter as claims 1 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 22, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Floren, Gutierrez and Dechene teaches
setting a statistical relationship between the factors created by the user within the GUI; (Column 45 Lines 39 – 41 of Gutierrez states “The correlation parameter may comprise one of covariance, interclass correlation, intraclass correlation, or rank.” Column 46 Lines 27 – 33, 39 – 43 of Gutierrez states “generate, based on the data model, a user interface; receive user interactions with the user interface, the user interactions defining relationships between the fields of the data model; generate, based on the relationships, a generative model, wherein the generative model may be configured to generate generated datasets having records arranged in the fields;… determine, based on data in the one or more fields of the generated test dataset, a parameter, wherein the parameter may be one or more of a statistical parameter or a correlation parameter;” [0151] of Dechene states “In several embodiments, the user can specify the training scenario through interactive buttons, sliders, and editable text fields in training scenario component 1730. The user can customize policy tradeoffs and optimize data flow through the network, effectively tuning the RL model and its hyperparameters in accordance with the user's subject matter expertise and intent. Network speed and reliability, priority data type, and expected seasonal traffic variation are examples of the type of dimensions the user can create and modify. Several common training scenarios can be preloaded for users, with support for full customization.” Gutierrez provides the relationship, statistical, and correlation among user defined fields/factors. Dechene further provides the user-created GUI factors. )
creating one or more simulation states; (Column 20 Lines 63 – 68 of Gutierrez states “The simulation specification may be used, with instantiation data, to instantiate instances of agents who are defined in the simulation specification by sampling the simulation specification with a random number generator, resulting in a simulation state. That simulation state may be iteratively sampled, using the random number generator, to perform actions defined in behaviors associated with the instantiated agents. Each sampling of the simulation state may be as a simulation step.”)
and creating one or more simulation flow graphs using a node-graph canvas based on the one or more simulation states created, and wherein each node in the one or more simulation flow graphs represents a simulation step and each edge establishes a way-point between two states. ([0167] of Floren states “Referring to FIG. 8A, an example user interface 800 includes an interactive graph section 802 in which various systems, subsystems, and data objects can be represented by nodes or indicators, such as icons 804 and 806. For ease of description, the information shown in the GUIs of the present disclosure is generally referred to as objects, but as noted various systems and subsystems may similarly be represented. As described throughout the present disclosure, the systems, subsystems, and objects may represent various things, such as people, locations, facilities, and the like. Relationships among the various systems, subsystems, and objects are represented by edges, such as edge 808, which may optionally be directional (or bi-directional) to indicate, e.g., flows of information or items.” [0183] of Floren states “The user interface portions 860-862 illustrate system functionality related to subgraphs. Subgraphs provide another way to abstract away parts of a larger, more complicated graph. User interface portion 860 illustrates that the user can select to create a subgraph from the ‘. . . ’ menu on the top navigation bar breadcrumbs. In other implementations other buttons or GUI functionality may be provided for the user to create a subgraph. In response, in user interface portion 861, which can comprise an overlaid GUI portion, or a separate GUI portion, the user can fill in details of the subgraph just like a regular graph, can name the subgraph, and can then link the subgraph back to the parent graph.” Column 23 Lines 21 – 31 of Gutierrez states “In step 1105, time is set equal to zero (t=0) for the generation of the simulation state. In step 1106, the simulation specification 1100 is sampled to generate the simulation state. As no previous step of the simulation exists, the simulation state is generated based on the probability distribution definitions and other data of the simulation specification 1100. In step 1107, the simulation state of the instantiated agents is stored. If desired, synthetic data may be generated from the simulation state of the instantiated agents (simulation step t=0) and stored in step 1109.” Column 26 Lines 49 – 63 of Gutierrez states “The generating the synthetic dataset simulating may further comprise iteratively simulating additional simulation steps of the agent. The generating the synthetic dataset may be based on the additional simulation steps. The generated synthetic dataset may comprise synthetic data, of the agent instance, from two or more iterative simulation steps. The outputting may comprise streaming, per simulation step, the synthetic dataset. Additional instructions may be received to modify a quantity of the agent instances to be generated in the simulation state and the method may regenerate, based on the modified quantity of agent instances, the simulation state, and the regenerated simulation state may comprise a count of agent instances corresponding to the received modified quantity.” Floren teaches the node/edge graph representation and Gutierrez teaches simulation states and simulation step progression. It would have been obvious to use Floren’s nodes/edges to represent Gutierrez’s simulation steps and transitions between simulation states (i.e., way point between two states))
Regarding claim 26, the rejection of claim 1 is incorporated herein. The combination of Floren, Gutierrez, Dechene teaches
retraining, by a controller, the reinforcement learning model based on edits to rewards or simulation flows by the user. ([0177] of Dechene states “In a number of embodiments, method 2100 additionally can include an activity 2120 of training a neural network model using a reinforcement learning model with the policy settings as updated by the user to adjust rewards assigned in the reinforcement learning model. The neural network model can be similar or identical to neural network model 431 (FIG. 4) and/or neural network models 531 (FIG. 5).” [0036] of Dechene states “The method also can include receiving one or more inputs from the user. The inputs include one or more modifications of at least a portion of the one or more first interactive elements of the user interface to update the policy settings of the reinforcement learning model. The method additionally can include training a neural network model using a reinforcement learning model with the policy settings as updated by the user to adjust rewards assigned in the reinforcement learning model.” Dechene teaches the user edit and subsequent training portion. Training the existing reinforcement learning after the user updates its reward policy settings corresponds to the claimed retraining. While Dechene does not expressly teach detecting the reward change specifically when a job is run, that timing and detection would have been obvious for the learning model to be updated corresponding to the user update. )
Claims 3, 12 are rejected under 35 U.S.C. 103 as being unpatentable over Floren et al. (U.S. Pub. 2022/0075515) in view of Dechene et al. (U.S. Pub. 2022/0245462), Gutierrez et al. (U.S. Pub. 11847390), further in view of Mallya Kasaragod et al. (U.S. Pub. 11836577).
Regarding Claim 3, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Floren, Gutierrez and Dechene teaches
guiding the user to generate the logical graph includes receiving positive or negative rewards associated with steps of the logical graph, wherein: the positive or negative rewards define why agents of the AI model would choose different paths of the logical graph, wherein the logical graph includes a plurality of nodes connected via edges; (Paragraph [0036] of Dechene states “The user interface can include one or more first interactive elements… The inputs include one or more modifications of at least a portion of the one or more first interactive elements of the user interface to update the policy settings of the reinforcement learning model. The method additionally can include training a neural network model using a reinforcement learning model with the policy settings as updated by the user to adjust rewards assigned in the reinforcement learning model.” And paragraph [0062] of Dechene states “An AI agent, such as an agent 430, takes a series of actions (e.g., an action 421) within an episode (e.g., 410), known as steps (e.g., a step 420). Each action (e.g., 421) can be informed by observations of a state 432 of an environment 440 (e.g., a training or live environment of a computer network) and an expected reward (e.g., a reward 423).” [0063] of Dechene states “As a simple example, an action (e.g., 421) can be choosing to route through the public internet or instead through a MPLS (Multiprotocol Label Switching) network. One or more observations (e.g., 422) of environment 440 can include a link identifier, a current bandwidth, and an available bandwidth in the network, and such state information can be stored in state 432. The reward (e.g., 423) can assign reward scores for various actions, such as a score of 1 for using MPLS, in which there is guaranteed success, a score of 3 for using the public internet with enough bandwidth, a score of −2 for using the public internet with limited bandwidth, and a score of −5 for an error in the network.” [0167] of Floren states “As described throughout the present disclosure, the systems, subsystems, and objects may represent various things, such as people, locations, facilities, and the like. Relationships among the various systems, subsystems, and objects are represented by edges, such as edge 808, which may optionally be directional (or bi-directional) to indicate, e.g., flows of information or items. In the example user interface 800, a supply chain is represented, including node representing parts and goods suppliers, manufacturing plants, distributors, consumers (e.g., hospitals), and the like.”)
training the AI model by having the agents of the AI model choose the different paths based on the positive or negative rewards. (Paragraph [0034] of Dechene states “The method also can include training a routing agent model on the digital twin network simulation using a reinforcement-learning model on traffic that flows through nodes of the digital twin network simulation. The routing agent model includes a machine-learning model.” [0062] of Dechene states “An AI agent, such as an agent 430, takes a series of actions (e.g., an action 421) within an episode (e.g., 410), known as steps (e.g., a step 420). Each action (e.g., 421) can be informed by observations of a state 432 of an environment 440 (e.g., a training or live environment of a computer network) and an expected reward (e.g., a reward 423).” [0063] of Dechene states “As a simple example, an action (e.g., 421) can be choosing to route through the public internet or instead through a MPLS (Multiprotocol Label Switching) network. One or more observations (e.g., 422) of environment 440 can include a link identifier, a current bandwidth, and an available bandwidth in the network, and such state information can be stored in state 432. The reward (e.g., 423) can assign reward scores for various actions, such as a score of 1 for using MPLS, in which there is guaranteed success, a score of 3 for using the public internet with enough bandwidth, a score of −2 for using the public internet with limited bandwidth, and a score of −5 for an error in the network.” Dachene teaches actions informed by expected rewards and different positive, negative reward values for different routing actions. Applying the Dachene’s reward based reinforcement training through network nodes to the different paths represented in Floren’s graph would have been obvious combination.)
The combination of Floren, Gutierrez and Dechene does not teach
simulating, using a graphical processing unit to graphically simulate, the number of times using the AI model.
However, Mallya Kasaragod explicitly teaches
simulating, using a graphical processing unit to graphically simulate, the number of times using the AI model. (Column 7, Lines 64 – 67 of Mallya Kasaragod states “For instance, the set of simulation parameters may include the batch size for the simulation, which may be used to determine the GPU requirements for the simulation.” And Column 19, Lines 54 – 61 of Mallya Kasaragod states “In an embodiment, based on the simulation parameters and the system parameters, the simulation agent 304 executes one or more visualization applications 310 to allow the customer to interact and visualize the simulation as it is being performed. The one or more visualization applications 310 may generate a graphical representation of the simulation, which may include a graphical representation of the simulation environment.” Column 21 Lines 7 – 15 of Kasaragod states “In an embodiment, the system simulation agent 404 injects the reinforcement learning model 406 into the robotic device application and obtains, from a simulation components 408 datastore, the various simulation components that, if executed, are used to create the simulation environment and execute the simulation. As noted above, the simulation components 408 may include physics engines, rendering engines, and the like.”)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings from Mallya Kasaragod with the combination of Floren, Gutierrez and Dechene. Floren teaches an interactive node/edge graph GUI with subgraphs for representing and modifying a simulated technical system. Gutierrez teaches simulation state generation and generation of synthetic data from simulation states. Dechene teaches use of reinforcement learning model in a simulated environment to make decisions and train a model, including dashboard functionality for simulation results. Mallya Kasaragod teaches determining GPU requirements for a simulation and generating a graphical representation of the simulation environment. Using a GPU to execute and visualize simulations is a well-known approach to efficiently compute simulations and provide interactive graphical feedback. One with the ordinary skill in the art would be motivated to incorporate the teachings of Mallya Kasaragod into combination of Floren, Gutierrez and Dechene as it results in predictable improvement of processing speed, visualization quality, and allow AI model to simulate situations multiple times more efficiently. Therefore, combination of Floren, Gutierrez, Dechene, and Mallya Kasaragod would have been obvious for a POSITA.
Claim 12 recites substantially similar subject matter as claim 3 respectively, and is rejected with the same rationale, mutatis mutandis.
Claims 8, 17, 23, 24 are rejected under 35 U.S.C. 103 as being unpatentable over Floren et al. (U.S. Pub. 2022/0075515) in view of Dechene et al. (U.S. Pub. 2022/0245462), Gutierrez et al. (U.S. Pub. 11847390), further in view of Kumar et al. (U.S. Pub. 10528327).
Regarding claim 8, the rejection of claim 1 is incorporated herein. Furthermore, the combination of Floren, Gutierrez, Dechene does not explicitly teach
recommending one or more steps for the user to add into the logical graph.
However, Kumar teaches
recommending one or more steps for the user to add into the logical graph. (Column 7, Lines 58 – 68 of Kumar states “When a developer is editing a workflow, step selector 306 may enable the developer to select workflow steps for inclusion in the workflow, and to order the steps. The workflow steps may be accessed by step selector 306 in workflow library 118. For instance, step selector 306 may display a menu of workflow steps, a scrollable and/or searchable list of available workflow steps, or may provide the workflow steps in another manner, and may enable the developer to select any number of workflow steps from the list for inclusion in the workflow.” Column 13 Lines 40 – 43, 56 - 62 of Kumar states “ Such a natural language search engine may be capable of determining a user's intent based on the content of the query and provide highly-relevant workflow step suggestions… This step may entail, for example, step selector 306 presenting the identifiers to the developer in a menu that is displayed within a workflow designer GUI that displays a representation of the workflow currently being developed. The developer may be enabled to navigate or scroll up and down the menu to highlight and select a particular workflow step.” Kumar teaches recommending workflow steps and allowing the user to select and insert a recommended step into a graphical workflow. Applying these recommendations to Floren’s logical graph would have been obvious combination.)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings from Kumar with the combination of Floren, Gutierrez and Dechene. Floren teaches an interactive node/edge graph GUI with subgraphs for representing and modifying a simulated technical system. Gutierrez teaches simulation state generation and generation of synthetic data from simulation states. Dechene teaches use of reinforcement learning model in a simulated environment to make decisions and train a model, including dashboard functionality for simulation results. Kumar teaches recommending relevant workflow steps for user selection and configuring workflow branches using logical conditions. One with the ordinary skill in the art would be motivated to incorporate the teachings of Kumar into combination of Floren, Gutierrez and Dechene to help the user identify, insert, and configure appropriate simulation steps and conditional paths, including paths associated with rewards. It would have been a predictable combination of known workflow authoring and graphical simulation technique to yield the predictable result of assisting the user in constructing and configuring the logical graph while reducing manual searching and configuration errors.
Claim 17 recites substantially similar subject matter as claim 8 respectively, and is rejected with the same rationale, mutatis mutandis.
Regarding Claim 23, the rejection of claim 22 is incorporated herein. Furthermore, the combination of Floren, Gutierrez, Dechene, and Kumar teaches
detecting, by a controller, the user connecting two state nodes; ([0167] of Floren states “Referring to FIG. 8A, an example user interface 800 includes an interactive graph section 802 in which various systems, subsystems, and data objects can be represented by nodes or indicators, such as icons 804 and 806. For ease of description, the information shown in the GUIs of the present disclosure is generally referred to as objects, but as noted various systems and subsystems may similarly be represented. As described throughout the present disclosure, the systems, subsystems, and objects may represent various things, such as people, locations, facilities, and the like. Relationships among the various systems, subsystems, and objects are represented by edges, such as edge 808, which may optionally be directional (or bi-directional) to indicate, e.g., flows of information or items.” [0183] of Floren states “The user interface portions 860-862 illustrate system functionality related to subgraphs. Subgraphs provide another way to abstract away parts of a larger, more complicated graph. User interface portion 860 illustrates that the user can select to create a subgraph from the ‘. . . ’ menu on the top navigation bar breadcrumbs. In other implementations other buttons or GUI functionality may be provided for the user to create a subgraph. In response, in user interface portion 861, which can comprise an overlaid GUI portion, or a separate GUI portion, the user can fill in details of the subgraph just like a regular graph, can name the subgraph, and can then link the subgraph back to the parent graph.” Column 23 Lines 21 – 31 of Gutierrez states “In step 1105, time is set equal to zero (t=0) for the generation of the simulation state. In step 1106, the simulation specification 1100 is sampled to generate the simulation state. As no previous step of the simulation exists, the simulation state is generated based on the probability distribution definitions and other data of the simulation specification 1100. In step 1107, the simulation state of the instantiated agents is stored. If desired, synthetic data may be generated from the simulation state of the instantiated agents (simulation step t=0) and stored in step 1109.” Floren teaches a user editable node/edge graph interface and explicitly teaches linking graph/subgraph elements. Detecting a user connection between nodes is inherent in implementing this interface graph linking functionality.)
and prompting, by the controller, the user within the GUI to add rewards to reinforce a particular outcome for a simulation using logical operators, ([0174] of Dechene states “Referring to FIG. 21, method 2100 can include an activity 2110 of transmitting a user interface to be displayed to a user. The user interface can be provided by GUI service 311 of user interface system 310 (FIG. 3), and exemplary displayed of the user interface can be similar or identical to user interface displays 1500 (FIG. 15), 1600 (FIG. 16), 1700 (FIG. 17), 1800 (FIG. 18), and/or 1900 (FIG. 19). In some embodiments, the user interface can include one or more first interactive elements that display policy settings of a reinforcement learning model. For example, the policy settings can be similar or identical to policies 1220 (FIG. 12), and/or the first interactive elements can be similar or identical to one or more of the elements of training scenarios component 1730 (FIG. 17) and/or one or more of the elements of user interface display 1800 (FIG. 18). The reinforcement learning model can be similar or identical to RL model 400 (FIG. 4), HRL model 500 (FIG. 5), Meta-RL model 700 (FIG. 7), and/or RL model 1241 (FIG. 12). In a number of embodiments, the one or more first interactive elements can be configured to allow the user to update the policy settings of the reinforcement learning model. “ [0177] of Dechene states “In a number of embodiments, method 2100 additionally can include an activity 2120 of training a neural network model using a reinforcement learning model with the policy settings as updated by the user to adjust rewards assigned in the reinforcement learning model. The neural network model can be similar or identical to neural network model 431 (FIG. 4) and/or neural network models 531 (FIG. 5). In many embodiments, the neural network model can include a routing agent model configured to control a physical computer network through a software-defined-network (SDN) control system.” Column 10 Lines 16 – 24 of Kumar states “The condition of workflow step 702 enables the workflow to fork based on the determination of a condition (e.g., a variable value). The condition may include an object name, a relationship (e.g., a logical relationship, such as equal to, includes, not equal to, less than, greater than, etc.), and a value, which are all defined by the developer interacting with workflow step 702. Corresponding action steps may be performed depending on which way the workflow forks based on the condition.” Dechene teaches the RL reward adjustment context and user updated policy settings in a GUI. Kumar teaches explicit GUI defined logical relationships/operators used to control workflow outcomes. )
wherein the user is provided different quantities of the rewards to the way-point between the two states. ([0063] of Dechene states “The reward (e.g., 423) can assign reward scores for various actions, such as a score of 1 for using MPLS, in which there is guaranteed success, a score of 3 for using the public internet with enough bandwidth, a score of −2 for using the public internet with limited bandwidth, and a score of −5 for an error in the network.” [0167] of Floren states “Referring to FIG. 8A, an example user interface 800 includes an interactive graph section 802 in which various systems, subsystems, and data objects can be represented by nodes or indicators, such as icons 804 and 806. For ease of description, the information shown in the GUIs of the present disclosure is generally referred to as objects, but as noted various systems and subsystems may similarly be represented. As described throughout the present disclosure, the systems, subsystems, and objects may represent various things, such as people, locations, facilities, and the like. Relationships among the various systems, subsystems, and objects are represented by edges, such as edge 808, which may optionally be directional (or bi-directional) to indicate, e.g., flows of information or items.” [0183] of Floren states “The user interface portions 860-862 illustrate system functionality related to subgraphs. Subgraphs provide another way to abstract away parts of a larger, more complicated graph. User interface portion 860 illustrates that the user can select to create a subgraph from the ‘. . . ’ menu on the top navigation bar breadcrumbs. In other implementations other buttons or GUI functionality may be provided for the user to create a subgraph. In response, in user interface portion 861, which can comprise an overlaid GUI portion, or a separate GUI portion, the user can fill in details of the subgraph just like a regular graph, can name the subgraph, and can then link the subgraph back to the parent graph.” Column 23 Lines 21 – 31 of Gutierrez states “In step 1105, time is set equal to zero (t=0) for the generation of the simulation state. In step 1106, the simulation specification 1100 is sampled to generate the simulation state. As no previous step of the simulation exists, the simulation state is generated based on the probability distribution definitions and other data of the simulation specification 1100. In step 1107, the simulation state of the instantiated agents is stored. If desired, synthetic data may be generated from the simulation state of the instantiated agents (simulation step t=0) and stored in step 1109.” Dechene teaches different reward quantities in an RL context. Combine with Gutierrez and Floren teaches assigning different rewards to graph transitions (way points) between state nodes. )
Regarding Claim 24, the rejection of claim 23 is incorporated herein. Furthermore, the combination of Floren, Gutierrez, Dechene, and Kumar teaches
providing a review dashboard to the user within the GUI, wherein the review dashboard includes sectional insights to different aspects of a configuration of the simulation. ([0157] of Dechene states “In many embodiments, the user can select the current state monitoring option in menu 1910 to monitor the state and/or performance of an AI model once it is deployed on the live network. When the model is deployed, the user can have visibility into the live network through an interactive dashboard, such as dashboard 1940, which can assist in tracking performance against relevant benchmarks, as well as alerting the user to any performance issues or security threats. The dashboard can include metrics and/or visualizations describing the network's health. In some embodiments, a dashboard menu 1941 can allow the user to select various different dashboard display options, such as data, charts, and/or alerts.” [0041] of Floren states “In response, graphical user interfaces (“GUIs”) may be generated that can include, for example, graph-based GUIs, map-based GUIs, and panel-based GUIs, among others. The GUIs may include one or more panels to display data including technical data objects (also referred to herein as “objects”) (e.g., pumps, compressors, valves, machinery, welding stations, vats, containers, products or items, organizations, countries, counties, factories, customers, hospitals, etc.), technical object properties (e.g., flow rate, suction temperature, volume, capacity, order volume, sales amounts, sales quantity during a time period (e.g., a day, a week, a year, etc.), population density, patient volume, etc.), simulations, alerts, recommendations, and the like. The technical objects and technical object properties may represent the inputs and outputs of the simulated models. Various GUIs may further comprise at least one of information, trend, simulation, mapping, schematic, time, equipment, and toolbar panels. Various panels may display the objects, object properties, inputs, and outputs of the simulated models.”)
Regarding Claim 25, the rejection of claim 24 is incorporated herein. Furthermore, the combination of Floren, Gutierrez, Dechene, and Kumar teaches
wherein the sectional insights include at least factors, statistical relationships, time series, the one or more simulation states, and the rewards for the logical graph, and wherein the GUI further enables the user to combine the simulation with cumulative metrics of previously run simulations. ([0151] of Dechene states “In several embodiments, the user can specify the training scenario through interactive buttons, sliders, and editable text fields in training scenario component 1730. The user can customize policy tradeoffs and optimize data flow through the network, effectively tuning the RL model and its hyperparameters in accordance with the user's subject matter expertise and intent. Network speed and reliability, priority data type, and expected seasonal traffic variation are examples of the type of dimensions the user can create and modify. Several common training scenarios can be preloaded for users, with support for full customization.” [0157] of Dechene states “When the model is deployed, the user can have visibility into the live network through an interactive dashboard, such as dashboard 1940, which can assist in tracking performance against relevant benchmarks, as well as alerting the user to any performance issues or security threats. The dashboard can include metrics and/or visualizations describing the network's health. In some embodiments, a dashboard menu 1941 can allow the user to select various different dashboard display options, such as data, charts, and/or alerts.“ [0083] of Floren states “Additionally the interactive user interface may be configured to allow a user to view or edit, in at least one of the displayed panels, unusual (e.g., abnormal) or periodic events that have occurred and/or that may occur in the future during operation of the logical computations, sensors, and/or measuring devices 114 and/or real-world subsystems 112. Such events may also apply to the simulated virtual objects (e.g., virtual items or products, virtual measuring devices, and/or virtual subsystems).” [0178] of Floren states “As shown, the example user interface portion 840 can include a simulation panel 842 that can display simulation parameters and results. Via the simulation panel 842, the user can specify inputs, outputs, and models. Further, the user can run multiple simulations, as represented by columns 843 and 844. As described above, the user can specify which simulations are used to display values in the readouts 833 and 834. Additional details of GUI functionality related to simulations are provided herein, including in reference FIGS. 8N-8O and 9C-9F.”)
Claim 27 are rejected under 35 U.S.C. 103 as being unpatentable over Floren et al. (U.S. Pub. 2022/0075515) in view of Dechene et al. (U.S. Pub. 2022/0245462), Gutierrez et al. (U.S. Pub. 11847390), further in view of Canedo et al. (U.S. Pub. 2020/0090085).
Regarding Claim 27, the rejection of claim 26 is incorporated herein. Furthermore, the combination of Floren, Gutierrez, Dechene teaches
wherein the logical graph includes cumulative metrics from each of the plurality of simulations ([0011] of Floren states “In some embodiments, data may be presented in graphical representations, such as visual representations, such as charts and graphs, where appropriate, to allow the user to rapidly review the large amount of data and to take advantage of humans' particularly strong pattern recognition abilities related to visual stimuli. In some embodiments, the system may present aggregate quantities, such as totals, counts, and averages.” [0177] of Floren states “Still referring to FIG. 8D, via selection box 836, the user may select a particular simulation result to compare to (e.g., a simulation based on which values are displayed in the right column of readouts 833, 834).” [0178] of Floren states “As shown, the example user interface portion 840 can include a simulation panel 842 that can display simulation parameters and results. Via the simulation panel 842, the user can specify inputs, outputs, and models. Further, the user can run multiple simulations, as represented by columns 843 and 844. As described above, the user can specify which simulations are used to display values in the readouts 833 and 834.”)
The combination does not teach
combining, by the controller, a plurality of simulations, wherein each of the plurality of simulations correspond to one of the plurality of subgraphs of the logical graph
However, Canedo teaches
combining, by the controller, a plurality of simulations, wherein each of the plurality of simulations correspond to one of the plurality of subgraphs of the logical graph ([0007] of Canedo states “For example, assume that the DTG comprises a first sub-graph corresponding to a first physical object and a second-graph corresponding to a second physical object connected by an edge indicating that the first physical object is using the second physical object… In some embodiments, the time period is predicted using simulation models which simulate behavior of the first physical object and the second physical object. In these embodiments, the computing system may include a simulation platform configured to execute each respective simulation model using simulation engines executing in parallel across a plurality of processors.” [0028] of Canedo states “The Big Simulation Platform 510B included at the DS Layer 510 provides a structure which is similar to that employed by the Big Data Platform 510A, except that simulation tasks are automatically dispatched to simulation engines and the results are automatically aggregated.” The aggregation of the results produced by the respective simulation models corresponds to combining the plurality of simulations [0029] of Canedo states “In the example of FIG. 5, each DT comprise a graph database (GDB) which stores the sub-graph corresponding to a physical machine, structure, or other entity represented in the DTG… The GDB of each DT are further linked such that they collectively amount to the DTG of the entire system. As an alternative to having multiple GDBs, in some embodiments, a single GDB is used and the designation of each sub-graph (i.e., each DT) may be explicitly stored along with information describing the various nodes and edges comprising the DTG.” [0030] of Canedo states “In the example of FIG. 5, each DT also includes a simulation model (SM)… Additionally, although only one SM is shown in FIG. 5, it should be understood that a DT may have multiple SMs associated with it. The exact implementation of each SM will vary, depending the specific characteristics of the DT.” )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings from Canedo with the combination of Floren, Gutierrez and Dechene. Floren teaches an interactive node/edge graph GUI with subgraphs for representing and modifying a simulated technical system. Gutierrez teaches simulation state generation and generation of synthetic data from simulation states. Dechene teaches use of reinforcement learning model in a simulated environment to make decisions and train a model, including dashboard functionality for simulation results. Canedo teaches organizing a digital twin graph into respective subgraphs, associating respective simulation models with the corresponding digital twins and subgraphs, executing the respective simulations, and automatically aggregating the simulation results. One with the ordinary skill in the art would be motivated to incorporate the teachings of Canedo into combination of Floren, Gutierrez and Dechene so that different portions of the modeled system could be simulated separately and their results combined for evaluation of the complete system. Combining aggregate metrics from Floren with the corresponding subgraphs from Canedo would have allowed the user to identify the contribution of each subgraph associated simulation. It would have been a predictable combination of known graph based simulation and result aggregation techniques to obtain a logical graph containing cumulative metrics organized by the simulations.
Claims 28 - 29 are rejected under 35 U.S.C. 103 as being unpatentable over Floren et al. (U.S. Pub. 2022/0075515) in view of Dechene et al. (U.S. Pub. 2022/0245462), Gutierrez et al. (U.S. Pub. 11847390), Canedo et al. (U.S. Pub. 2020/0090085), further in view of Schulter et al. (U.S. Pub. 2020/0094824).
Regarding Claim 28, the rejection of claim 27 is incorporated herein. Furthermore, the combination of Floren, Gutierrez, Dechene, Canedo teaches
generating, by the controller, a full set of synthetic data (The claimed “a full set of synthetic data” is interpreted as the completed or resultant synthetic dataset produced after the simulation process, rather than as requiring that every individual record is required. The interpretation is consistent with Fig. 2 ref 218 and the specification, which describes the full set as the resultant synthetic data generated after the simulation is completed and determined to be satisfactory. [0084] of Dechene states “Generating synthetic traffic (e.g., using network traffic service 323 (FIG. 3) can allow the AI-defined networking solution to train on a multitude of scenarios. Synthetic traffic can be generated within the digital twin model for direct training usage in RL agent actions and rewards, or as noise that serves as competing traffic against the legitimate training traffic. Generated synthetic traffic can be actual like-for-like traffic from a model or fuzzy (i.e., realistically altered) to avoid overfitting the training model.” Column 23 lines 32 – 48 of Guiterrez states “Additionally or alternatively, a time step may be incremented to the next time step (e.g., t=t+1) in step 1108 and the simulation executed again, using the simulation specification information obtained in step 1100 and the simulation state of the instantiated agents from 1107. The process may repeat (next simulation steps) for a set number of iterations, until a given result is obtained (e.g., 30% home ownership), or the simulation reaches a steady state (no significant changes from a previous state—e.g., 99% of the collected states not changing between steps). In step 1110, the stored synthetic dataset may be sent to a user. The generated predictions may be sent (e.g., to the above user or a different user) in step 1110. Alternatively or additionally, the synthetic dataset may be used to train a machine-learning model in step 1114 and the trained machine-learning model used to generate predictions in step 1115 based on new true-source data.“)
However, the combination does not expressly teach
training the reinforcement learning model using the full set of synthetic data, wherein the training includes both a training step and a verification stage, wherein both the training step and the verification stage are conducted with the full set of synthetic data.
Schulter teaches that
training the reinforcement learning model using the full set of synthetic data, wherein the training includes both a training step and a verification stage, wherein both the training step and the verification stage are conducted with the full set of synthetic data. ([0005] of Schulter states “The method includes generating, by a hardware processor, fully-annotated simulated training data for a machine learning model responsive to receiving a set of computer-selected simulator-adjusting parameters. The method further includes training, by the hardware processor, the machine learning model using reinforcement learning on the fully-annotated simulated training data. The method also includes measuring, by the hardware processor, an accuracy of the trained machine learning model relative to learning a discriminative function for a given task. The discriminative function predicts a given label for a given image from the fully-annotated simulated training data.” [0061] of Schulter states “At block 410, train the machine learning model using reinforcement learning on the fully-annotated simulated training data.” [0062] of Schulter states “At block 415, measure an accuracy of the trained machine learning model relative to learning a discriminative function for a given task, the discriminative function predicting a given label for a given image from the fully-annotated simulated training data.” Schulter trains the ML model using reinforcement learning and the training step comprises training step and post training measurement which constitutes a verification stage as it evaluates the performance of the trained model using same generated fully annotated simulated training data. )
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to combine the teachings from Schulter with the combination of Floren, Gutierrez, Dechene, and Canedo. Floren teaches an interactive node/edge graph GUI with subgraphs for representing and modifying a simulated technical system. Gutierrez teaches simulation state generation and generation of synthetic data from simulation states. Dechene teaches use of reinforcement learning model in a simulated environment to make decisions and train a model, including dashboard functionality for simulation results. Canedo teaches organizing a digital twin graph into respective subgraphs, associating respective simulation models with the corresponding digital twins and subgraphs, executing the respective simulations, and automatically aggregating the simulation results. Schulter teaches generating simulated training data based on adjustable simulator parameters, training a ML model using reinforcement learning on the simulated data, measuring the trained model’s accuracy using data from the simulated training dataset, and adjusting the simulator parameters and repeating the training and validation step when the accuracy is below a threshold. One with the ordinary skill in the art would be motivated to incorporate the teachings of Schulter into combination of Floren, Gutierrez, Dechene, and Canedo to verify model performance and control further generation of synthetic training data based on the verification result. It would have been a predictable combination to apply known reinforcement learning training and validation technique to a known synthetic data simulation environment.
Regarding Claim 29, the rejection of claim 28 is incorporated herein. Furthermore, the combination of Floren, Gutierrez, Dechene, Canedo, and Schulter teaches
wherein threshold and preference data include thresholds that define a manner in which the controller manages the generating of the full set of synthetic data, wherein the thresholds defines when the controller suggests the user provide the rewards or use a specific state or factor. ([0151] of Dechene states “The user can customize policy tradeoffs and optimize data flow through the network, effectively tuning the RL model and its hyperparameters in accordance with the user's subject matter expertise and intent.” [0036] of Dechene states “ The one or more first interactive elements are configured to allow the user to update the policy settings of the reinforcement learning model… The method additionally can include training a neural network model using a reinforcement learning model with the policy settings as updated by the user to adjust rewards assigned in the reinforcement learning model.” Dechene’s user selected policy settings and tradeoffs correspond to the claimed preference data. [0060] of Schulter states “At block 405, generate fully-annotated simulated training data for a machine learning model responsive to receiving a set of computer-selected simulator-adjusting parameters. In an embodiment, the parameters are scene parameters that define a probability distribution of a set of scenes.” [0063] of Schulter states “At block 420, adjust the computer-selected simulator-adjusting parameters and repeating said training and measuring steps responsive to the accuracy being below a threshold accuracy. In an embodiment, the threshold accuracy can be derived from the reward (e.g., R-b, where R is the reward and b is a baseline as described further herein). In an embodiment, the adjusting block 420 can be skipped responsive to the accuracy being equal to or greater than a threshold accuracy. In an embodiment, a reward can be provided responsive to the accuracy being equal to or greater than the threshold accuracy. In an embodiment, the reward can quantify an error value, wherein the computer-selected simulator-adjusting parameters can be adjusted responsive to a magnitude of the error signal. In an embodiment, block 420 can involve updating a probability distribution of the computer-selected simulator-adjusting parameters.” Since the adjusted parameters determine the synthetic data subsequently generated by the simulator, the threshold determines whether and how the controller changes the generation of the synthetic dataset. Schulter further teaches that the threshold condition determines when a reward is called for. [0185] of Dechene states “In a number of embodiments, method 2100 additionally can include, after block 2135, an activity 2140 of transmitting alerts to be displayed to the user when one or more of the performance results are outside one or more predefined thresholds.” With respect to [0036] of Dechene above, it teaches informing the user when a threshold condition occurs and allowing the user to supply policy information that determines or adjusts the reward.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BYUNGKWON HAN whose telephone number is (571)272-5294. The examiner can normally be reached M-F: 9:00AM-6PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYUNGKWON HAN/ Examiner, Art Unit 2121
/Li B. Zhen/ Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights