DETAILED ACTION
A. This action is in response to the following communications: Amendment filed: 08/28/2026. This action is made Final.
B. Claims 1-20 remain pending.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being clearly anticipated by Lang, Ulrich et al. (US Pub. 2019/0258953), herein referred to as “Lang”.
As for claims 1, 2 and 12, Lang teaches. A system and corresponding method of claim 2 and non-transitory machine-readable media storing program instructions of claim 12 for responding to a user interface (UI) by generating and using hierarchical agents based on data stored in a runtime environment used to present the UI, the system comprising one or more processors and one or more non-transitory, machine-readable media storing program instructions that, when executed by the one or more processors, perform operations comprising (par. 14 overview of automating at least one agent for at least one environment including an agent action determination and action execution implementation; par. 52 hardware environment for executing software invention):
decomposing, into UI input elements, a dynamic web document that defines a UI within a first runtime environment, the dynamic web document being dynamically selected based on previous inputs (par. 56 the system utilizing machine learning will “decompose” a task into a set of steps for automation/ manual input by agents/user);
determining, within the first runtime environment, a dynamic set of hierarchical agents for the dynamic web document based on the UI input elements (par. 59 fully automates at least one agent for at least one environment);
generating interaction data for a target UI element of the UI input elements by delegating a task to populate the target UI element to a leaf agent of the dynamic set of hierarchical agents by providing, as an input, an element identifier of the target UI element to the dynamic set of hierarchical agents, the leaf agent using a machine learning model outside the first runtime environment to generate the interaction data based on the target UI element and a user-provided value obtained in a second runtime environment; (par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes) and
updating the dynamic web document by populating the target UI element with the interaction data (par. 59 An environment can for example pertain to people (e.g. instructions for human users), for example users guided by the present invention's actions to complete a task—involving inputs (e.g. audio/visual/sensor, textual etc.) and outputs (e.g. guidance, assistance, recommendations, alarms, notifications etc.). Actions can for example pertain to any combination of these and other environments, and any combination of automated and manual (involving humans) actions. The invention is a system, and therefore all action inputs and outputs have a technical manifestation (e.g. ingested data from video feed, outputted audio guidance), even if the environment it pertains to may not be technical (e.g. human user; par. 81-88 fig. 17 describes an example process for automatically configuring at least one action determination model for at least one agent that indicates at least one action option to be executed on at least one environment by at least one agent(s)).
As for claim 3, Lang teaches. The method of claim 2, wherein the agent is a lower-level agent, and wherein the UI element information comprises an identifier of the target UI element, and wherein delegating the task comprises providing the UI element information to a higher-level agent of the dynamic set of hierarchical agents to cause the higher-level agent to delegate the task to the lower-level agent based on a match between the identifier and a known element title mapped to a known input data type (par. 224, 226-227 FIG. 19 shows a flow chart of an example of the agent classifying entity; agent classifying entity is able to classify what task the user wants to monitor, be a UI for example or other instance such as image input; based upon learned model for classifying agents are assigned to action behavior models with weighted tree).
As for claim 4, Lang teaches. The method of claim 2, wherein the agent is a lower-level agent, and wherein the UI element information comprises an identifier of the target UI element, and wherein delegating the task comprises: providing the identifier to a higher-level agent of the dynamic set of hierarchical agents;
causing, via the higher-level agent, a neural network classifier to output a semantic element prediction matching a known semantic element and an associated confidence value based on the identifier; determining a result indicating that the associated confidence value satisfies a confidence value threshold; and delegating the task to the lower-level agent based on the result (par. 224, 226-227 FIG. 19 shows a flow chart of an example of the agent classifying entity; agent classifying entity is able to classify what task the user wants to monitor, be a UI for example or other instance such as image input; based upon learned model for classifying agents are assigned to action behavior models with weighted tree).
As for claim 5, Lang teaches. The method of claim 2, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the input data comprises: determining a set of options by providing the set of text corresponding with the selectable categories to a language model to generate the set of options; determining a result indicating that the user-provided value is within a boundary indicated by a first option of the set of options; and using the first option as the input data based on the result (fig. 16-17 and par. 91 the AI entity can for example comprise of several machine learning models (e.g. neural networks) to learn different relevant behaviors and information).
As for claim 6, Lang teaches. The method of claim 2, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the input data comprises: determining a set of options by providing the set of text corresponding with the selectable categories to a language model to generate the set of options; determining a result indicating that the user-provided value is equal to a first option of the set of options; and using the first option as the input data based on the result (par. 63,65 and 81-85 system is able to gather inputs from user and/or simulated inputs using models to generate output (options) for completing task based upon options selected manually/automatically).
As for claim 7, Lang teaches. The method of claim 2, wherein the UI element information comprises a first identifier of the target UI element, and wherein the agent is a first agent, further comprising: delegating a second task to populate a second UI element to a second agent of the dynamic set of hierarchical agents by providing, as an input, a second identifier associated with the second UI element to the dynamic set of hierarchical agents; retrieving, via the agent, an image based on the second identifier associated with the second UI element;
interacting with the second UI element to upload the image onto a server associated with the document; and causing a rendering of the document to display the image or a link to the image (par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete).
As for claim 8, Lang teaches. The method of claim 2, wherein the UI element information comprises an identifier of the target UI element, further comprising generating a rendering based on the document, wherein delegating the task to update the target UI element to the agent comprises: determining a portion of the rendering corresponding with the target UI element; sending the portion of the rendering to an image recognition model; determining an input method type based on the image recognition model and the identifier; and delegating the target UI element to a leaf node of the dynamic set of hierarchical agents based on the input method type (par. 57 use of image object recognition to aid automated agents using machine learning for completing tasks assigned to one or more agents; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete) .
As for claim 9, Lang teaches. The method of claim 2, wherein updating the target UI element comprises presenting, on a display device, a rendering of the document as the target UI element is updated (par.371-378 presenting user interactions and outputting onto user interfaces).
As for claim 10, Lang teaches. The method of claim 2, wherein the UI elements are a first set of UI elements, further comprising: decomposing a second document within a third runtime environment into second UI elements; generating a second hierarchy of agents based on the second UI elements; determining, within the second runtime environment, an agent hierarchy for a second dynamic set of hierarchical agents by categorizing each respective UI element of the UI elements; determining a set of input values using the second dynamic set of hierarchical agents; and updating the second document of the third runtime environment by populating a second set of UI elements indicated by the second document with the set of input values (par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete; par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes).
As for claim 11, Lang teaches. The method of claim 2, wherein the second runtime environment is ended before the first runtime environment is initialized, further comprising: storing the user-provided value into a memory; and retrieving the user-provided value from the memory, wherein generating the input data comprises providing the user-provided value stored in the memory to the dynamic set of hierarchical agents (par. 52 hardware environment for executing software invention; par. 61 using memory for execution of software steps analyzed in preceding claims).
As for claim 13, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the document is a first document, and wherein the set of hierarchical agents is a first set of hierarchical agents, and wherein the user information comprises a first user-provided value, and wherein the interaction data is a first input value, the operations further comprising: obtaining the first user-provided value and a second user-provided value in a second runtime environment, wherein generating the interaction data comprises generating the interaction data based on the first user-provided value; updating the document to access a new UI screen defined by a second document in a third runtime environment; generating, in the third runtime environment, a second set of hierarchical agents based on additional UI elements of the second document; and
providing the second user-provided value obtained in the second runtime environment to the third runtime environment to determine a second input value for a UI element of the third runtime environment (par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete; par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes).
As for claim 14, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, further comprising: detecting that an additional set of UI elements is visible in response to a previous user-provided entry; and
re-determining the set of hierarchical agents based on the additional set of UI elements (par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes).
As for claim 15, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the UI element information comprises a first identifier of a first UI element, further comprising: delegating a second task to populate a second UI element to a leaf agent of the set of hierarchical agents by: determining a result indicating that the second UI element is associated with a media file upload option; and selecting the leaf agent based on the result and a second element identifier of the second UI element; determining a target media file using the leaf agent by: retrieving metadata associated with a set of candidate media files; and selecting a first media file for use as the target media file of the set of candidate media files based on the metadata; and interacting with the second UI element to upload the target media file (par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete).
As for claim 16, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the interaction data comprises: determining a set of options by providing the set of text corresponding with the selectable categories to a language model to generate the set of options; determining a result indicating that a user-provided value of the user information is within a boundary indicated by a first option of the set of options; and using the first option as the interaction data based on the result (fig. 16-17 and par. 91 the AI entity can for example comprise of several machine learning models (e.g. neural networks) to learn different relevant behaviors and information)..
As for claim 17, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the interaction data comprises: determining a set of semantic vectors by providing the set of text corresponding with the selectable categories of the target UI element to a semantic space encoder to generate the set of semantic vectors;
generating an input semantic vector based on a user-provided value of the user information; determining a nearest neighbor semantic vector based on a distance between the input semantic vector and the nearest neighbor semantic vector, wherein the nearest neighbor semantic vector is derived from a text corresponding with a first option; and using the first option as the interaction data (par. 234, 251,261-276 the agent classifying entity checks whether the currently selected model is or is not eliminated. If yes, then the agent classification entity sets (in step 1990 of an embodiment) the most applicable classified agent behavior model as the currently selected, and, if there is a reason for executing further actions, returns to step 1930. The most applicable model can be determined based on numerous factors, incl. objectives, the environment, classified agent characteristics etc. It can for example be determined by selecting e.g. the nearest neighbor (on a one-, two- or n-dimensional scale of characteristics), picking the easiest alternative (e.g. least skilled if skill level is a characteristics) etc.).
As for claim 18, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, the operations further comprising: generating a rendering based on the document; and providing the rendering to an image recognition model to confirm an input data type of the target UI element (par. 57 use of image object recognition to aid automated agents using machine learning for completing tasks assigned to one or more agents; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete) ..
As for claim 19, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein updating the target UI element comprises presenting, on a display device, a series of updates to a rendering as a field of the target UI element is updated (par. 57 use of image object recognition to aid automated agents using machine learning for completing tasks assigned to one or more agents; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete) ..
As for claim 20, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, further comprising: obtaining a user identifier; and
retrieving at least one value of the user information from a shared database accessible from the first runtime environment and from a second runtime environment based on the user identifier, wherein generating the interaction data comprises providing the at least one value to the set of hierarchical agents (par. 80, 89, 101 and 227 utilizing multiple environments to retrieve multiple values of user information from database that is accessible remotely, wherein the system/software is used at an enterprise level with multiple users).
(Note :) It is noted that any citation to specific, pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006,1009, 158 USPQ 275, 277 (CCPA 1968)).
Response to Arguments
Applicant's arguments filed 08/28/2026 have been fully considered but they are not persuasive.
After careful review of the current claims (given the broadest reasonable interpretation) and the remarks provided by the Applicant along with the cited reference(s) the Examiner respectfully disagrees with the Applicant for at least the reasons provided below:
A1. Applicant argues that Lang does not teach decompose a dynamic web document.
R1. Examiner does not agree, without further clarification on “decomposing” the broadest reasonable interpretation for this limitation is to separate into parts, said parts are user interface elements of an arbitrary dynamic web document (e.g. website, web application or just application etc…) said document defines user interface within a first runtime environment without further defining what the environment is or if there is a second runtime to reference to or interact with so the runtime is arbitrary and lastly using some arbitrary previous inputs (system/user) to select document. Lang teaches assisted policy management and automation of supporting activities, assisted policy management and determining policies, agent automation, classifying agent characteristics, automation for vulnerability assessment and penetration testing, protection of neural networks, and risk mitigation of communication buses as mentioned in paragraph 2. Lang teaches a method of automating at least one agent for at least one environment including an agent action determination and action execution implementation; at least one action determination model for the at least one agent that indicates at least one action option to be executed by the at least one agent, the indication being determined by at least one objective of the at least one agent; determining, via the processor, whether the at least one action determination model indicates at least one next action option to be executed by the at least one agent. At paragraph 56 the system, utilizing machine learning, will “decompose” a task into a set of steps for automation/ manual input by agents/user. This allows a user to be guided on completing a task, such as on a user interface, examples are given such that the user takes pictures of items and the system can use computer vision to assist; the invention also mentions help with applications such that it would not be hard pressed to see that it would see application and offer help on user interface elements in order to complete a task relating to the application.
Applicant notes that Lang does not describe runtime environments; Examiner does not agree, the implementation of paragraph 56-59 is carried out by use of multiple virtual machines that can each host runtime environment python (par. 125-129; 200 (Machine Learning “ML” tool kit on the virtual machines; par. 324 python used on each ML tool kit per virtual machine to execute commands; thereby multiple runtime environments for carrying out agent tasks as mentioned in paragraphs 56-59 fully automates at least one agent for at least one environment that is either a real or simulated environment.
Examiner recommends clarification amendment to overcome Lang for runtime environment as it is arbitrary within the claim limitations currently and anticipated by Langs multiple examples of different environments.
A2. Applicant argues that Lang does not teach generating interaction data for a target UI element of the UI input elements by delegating a task to populate the target UI element to a leaf agent of the dynamic set of hierarchical agents by providing, as an input, an element identifier of the target UI element to the dynamic set of hierarchical agents, the leaf agent using a machine learning model outside the first runtime environment to generate the interaction data based on the target UI element and a user-provided value obtained in a second runtime environment.
Lang Does Not Discuss Documents Having UI Elements and
Lang Does Not Delegate a Task Related to a Target UI Element to an Agent of a Set of Hierarchical Agents.
R2. Examiner does not agree, Lang teaches the use of multiple agents wherein one depends on another thereby teaching a hierarchy of agents and the last agent by definition of hierarchy would be a leaf; Paragraph 57 details this hierarchy dependency.
For example, a step executed by the agent entity could involve capturing a photo of a circuit board (e.g. using a device camera operated by the user, e.g. smart glasses), then using text extraction machine learning approaches known to those skilled in the art (and for example implemented in Google Vision API, see cloud.google.com/vision) to determine location and content of symbols/text from the circuit board photo. This information then allows the (e.g. computer-implemented on a processor) agent entity to determine the next steps, e.g. which connectors are available, and where, may determine the next instructions to the user, e.g. where to plug in a connector; or e.g. determining CPU brand/model and thus selecting a particular automated testing tool for the particular processor architecture. The agent entity can therefore for example execute (e.g. on a processor) any combination of automated steps (e.g. based on particular inputs or already-discovered information, e.g. read by a processor from a memory or storage) and manual steps (e.g. guiding the user to carry out an action, such as taking a photo, entering something on a device, utilizing some tool etc., providing inputs into the agent entity etc.). The described examples of automated and/or manual actions sequenced by an agent entity could for example be used to guide a user intelligently through diagnosing, testing, assessing, repairing, assembling, disassembling, operating etc. a particular device. Note that in paragraph 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes.
Paragraph 59 describes workflow of figure 16 which discusses loading action model, indicating action to be executed, such as application environment (which known in the art to have UI elements) select an action, execute action on the environment, determine result and output. Further paragraph 59 teaches an environment can for example pertain to people (e.g. instructions for human users), for example users guided by the present invention's actions to complete a task—involving inputs (e.g. audio/visual/sensor, textual etc.) and outputs (e.g. guidance, assistance, recommendations, alarms, notifications etc.). Actions can for example pertain to any combination of these and other environments, and any combination of automated and manual (involving humans) actions. The invention is a system, and therefore all action inputs and outputs have a technical manifestation (e.g. ingested data from video feed, outputted audio guidance), even if the environment it pertains to may not be technical (e.g. human user). Paragraphs 81-88 and fig. 17 describes an example process for automatically configuring at least one action determination model for at least one agent that indicates at least one action option to be executed on at least one environment by at least one agent(s)).
Examiner recommends clarification amendment of a leaf agent in relation to other agents and their functions or clarifying the leaf agents function.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Inquires
Any inquiry concerning this communication should be directed to NICHOLAS AUGUSTINE at telephone number (571)270-1056.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
PNG
media_image1.png
213
559
media_image1.png
Greyscale
/NICHOLAS AUGUSTINE/Primary Examiner, Art Unit 2178 October 16, 2026