DETAILED ACTION
A. This action is in response to the following communications: Transmittal of New Application filed 04/15/2024.
B. Claims 1-20 remains pending.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being clearly anticipated by Lang, Ulrich et al. (US Pub. 2019/0258953), herein referred to as “Lang”.
As for claims 1, 2 and 12, Lang teaches. A system and corresponding method of claim 2 and non-transitory machine-readable media storing program instructions of claim 12 for responding to a user interface (UI) by generating and using hierarchical agents based on data stored in a runtime environment used to present the UI, the system comprising one or more processors and one or more non-transitory, machine-readable media storing program instructions that, when executed by the one or more processors, perform operations comprising (par. 14 overview of automating at least one agent for at least one environment including an agent action determination and action execution implementation; par. 52 hardware environment for executing software invention):
decomposing, into UI input elements, a dynamic web document that defines a UI within a first runtime environment, the dynamic web document being dynamically selected based on previous inputs (par. 56 the system utilizing machine learning will “decompose” a task into a set of steps for automation/ manual input by agents/user);
determining, within the first runtime environment, a dynamic set of hierarchical agents for the dynamic web document based on the UI input elements (par. 59 fully automates at least one agent for at least one environment);
generating interaction data for a target UI element of the UI input elements by delegating a task to populate the target UI element to a leaf agent of the dynamic set of hierarchical agents by providing, as an input, an element identifier of the target UI element to the dynamic set of hierarchical agents, the leaf agent using a machine learning model outside the first runtime environment to generate the interaction data based on the target UI element and a user-provided value obtained in a second runtime environment; (par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes) and
updating the dynamic web document by populating the target UI element with the interaction data (par. 59 An environment can for example pertain to people (e.g. instructions for human users), for example users guided by the present invention's actions to complete a task—involving inputs (e.g. audio/visual/sensor, textual etc.) and outputs (e.g. guidance, assistance, recommendations, alarms, notifications etc.). Actions can for example pertain to any combination of these and other environments, and any combination of automated and manual (involving humans) actions. The invention is a system, and therefore all action inputs and outputs have a technical manifestation (e.g. ingested data from video feed, outputted audio guidance), even if the environment it pertains to may not be technical (e.g. human user; par. 81-88 fig. 17 describes an example process for automatically configuring at least one action determination model for at least one agent that indicates at least one action option to be executed on at least one environment by at least one agent(s)).
As for claim 3, Lang teaches. The method of claim 2, wherein the agent is a lower-level agent, and wherein the UI element information comprises an identifier of the target UI element, and wherein delegating the task comprises providing the UI element information to a higher-level agent of the dynamic set of hierarchical agents to cause the higher-level agent to delegate the task to the lower-level agent based on a match between the identifier and a known element title mapped to a known input data type (par. 224, 226-227 FIG. 19 shows a flow chart of an example of the agent classifying entity; agent classifying entity is able to classify what task the user wants to monitor, be a UI for example or other instance such as image input; based upon learned model for classifying agents are assigned to action behavior models with weighted tree).
As for claim 4, Lang teaches. The method of claim 2, wherein the agent is a lower-level agent, and wherein the UI element information comprises an identifier of the target UI element, and wherein delegating the task comprises: providing the identifier to a higher-level agent of the dynamic set of hierarchical agents;
causing, via the higher-level agent, a neural network classifier to output a semantic element prediction matching a known semantic element and an associated confidence value based on the identifier; determining a result indicating that the associated confidence value satisfies a confidence value threshold; and delegating the task to the lower-level agent based on the result (par. 224, 226-227 FIG. 19 shows a flow chart of an example of the agent classifying entity; agent classifying entity is able to classify what task the user wants to monitor, be a UI for example or other instance such as image input; based upon learned model for classifying agents are assigned to action behavior models with weighted tree).
As for claim 5, Lang teaches. The method of claim 2, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the input data comprises: determining a set of options by providing the set of text corresponding with the selectable categories to a language model to generate the set of options; determining a result indicating that the user-provided value is within a boundary indicated by a first option of the set of options; and using the first option as the input data based on the result (fig. 16-17 and par. 91 the AI entity can for example comprise of several machine learning models (e.g. neural networks) to learn different relevant behaviors and information).
As for claim 6, Lang teaches. The method of claim 2, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the input data comprises: determining a set of options by providing the set of text corresponding with the selectable categories to a language model to generate the set of options; determining a result indicating that the user-provided value is equal to a first option of the set of options; and using the first option as the input data based on the result (par. 63,65 and 81-85 system is able to gather inputs from user and/or simulated inputs using models to generate output (options) for completing task based upon options selected manually/automatically).
As for claim 7, Lang teaches. The method of claim 2, wherein the UI element information comprises a first identifier of the target UI element, and wherein the agent is a first agent, further comprising: delegating a second task to populate a second UI element to a second agent of the dynamic set of hierarchical agents by providing, as an input, a second identifier associated with the second UI element to the dynamic set of hierarchical agents; retrieving, via the agent, an image based on the second identifier associated with the second UI element;
interacting with the second UI element to upload the image onto a server associated with the document; and causing a rendering of the document to display the image or a link to the image (par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete).
As for claim 8, Lang teaches. The method of claim 2, wherein the UI element information comprises an identifier of the target UI element, further comprising generating a rendering based on the document, wherein delegating the task to update the target UI element to the agent comprises: determining a portion of the rendering corresponding with the target UI element; sending the portion of the rendering to an image recognition model; determining an input method type based on the image recognition model and the identifier; and delegating the target UI element to a leaf node of the dynamic set of hierarchical agents based on the input method type (par. 57 use of image object recognition to aid automated agents using machine learning for completing tasks assigned to one or more agents; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete) .
As for claim 9, Lang teaches. The method of claim 2, wherein updating the target UI element comprises presenting, on a display device, a rendering of the document as the target UI element is updated (par.371-378 presenting user interactions and outputting onto user interfaces).
As for claim 10, Lang teaches. The method of claim 2, wherein the UI elements are a first set of UI elements, further comprising: decomposing a second document within a third runtime environment into second UI elements; generating a second hierarchy of agents based on the second UI elements; determining, within the second runtime environment, an agent hierarchy for a second dynamic set of hierarchical agents by categorizing each respective UI element of the UI elements; determining a set of input values using the second dynamic set of hierarchical agents; and updating the second document of the third runtime environment by populating a second set of UI elements indicated by the second document with the set of input values (par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete; par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes).
As for claim 11, Lang teaches. The method of claim 2, wherein the second runtime environment is ended before the first runtime environment is initialized, further comprising: storing the user-provided value into a memory; and retrieving the user-provided value from the memory, wherein generating the input data comprises providing the user-provided value stored in the memory to the dynamic set of hierarchical agents (par. 52 hardware environment for executing software invention; par. 61 using memory for execution of software steps analyzed in preceding claims).
As for claim 13, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the document is a first document, and wherein the set of hierarchical agents is a first set of hierarchical agents, and wherein the user information comprises a first user-provided value, and wherein the interaction data is a first input value, the operations further comprising: obtaining the first user-provided value and a second user-provided value in a second runtime environment, wherein generating the interaction data comprises generating the interaction data based on the first user-provided value; updating the document to access a new UI screen defined by a second document in a third runtime environment; generating, in the third runtime environment, a second set of hierarchical agents based on additional UI elements of the second document; and
providing the second user-provided value obtained in the second runtime environment to the third runtime environment to determine a second input value for a UI element of the third runtime environment (par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete; par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes).
As for claim 14, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, further comprising: detecting that an additional set of UI elements is visible in response to a previous user-provided entry; and
re-determining the set of hierarchical agents based on the additional set of UI elements (par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes).
As for claim 15, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the UI element information comprises a first identifier of a first UI element, further comprising: delegating a second task to populate a second UI element to a leaf agent of the set of hierarchical agents by: determining a result indicating that the second UI element is associated with a media file upload option; and selecting the leaf agent based on the result and a second element identifier of the second UI element; determining a target media file using the leaf agent by: retrieving metadata associated with a set of candidate media files; and selecting a first media file for use as the target media file of the set of candidate media files based on the metadata; and interacting with the second UI element to upload the target media file (par. 59 the agent's actions can include for example inputs, data processing, and outputs related to an IT environment, for example configuring/re-configuring, generating, reading/ingesting, writing, transmitting, configuring etc… In organizational environments, the agent's actions can include for example identification of the need for workflow/process changes, and instructions to change workflows/processes; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete).
As for claim 16, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the interaction data comprises: determining a set of options by providing the set of text corresponding with the selectable categories to a language model to generate the set of options; determining a result indicating that a user-provided value of the user information is within a boundary indicated by a first option of the set of options; and using the first option as the interaction data based on the result (fig. 16-17 and par. 91 the AI entity can for example comprise of several machine learning models (e.g. neural networks) to learn different relevant behaviors and information)..
As for claim 17, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein the target UI element comprises a set of text corresponding with selectable categories, and wherein generating the interaction data comprises: determining a set of semantic vectors by providing the set of text corresponding with the selectable categories of the target UI element to a semantic space encoder to generate the set of semantic vectors;
generating an input semantic vector based on a user-provided value of the user information; determining a nearest neighbor semantic vector based on a distance between the input semantic vector and the nearest neighbor semantic vector, wherein the nearest neighbor semantic vector is derived from a text corresponding with a first option; and using the first option as the interaction data (par. 234, 251,261-276 the agent classifying entity checks whether the currently selected model is or is not eliminated. If yes, then the agent classification entity sets (in step 1990 of an embodiment) the most applicable classified agent behavior model as the currently selected, and, if there is a reason for executing further actions, returns to step 1930. The most applicable model can be determined based on numerous factors, incl. objectives, the environment, classified agent characteristics etc. It can for example be determined by selecting e.g. the nearest neighbor (on a one-, two- or n-dimensional scale of characteristics), picking the easiest alternative (e.g. least skilled if skill level is a characteristics) etc.).
As for claim 18, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, the operations further comprising: generating a rendering based on the document; and providing the rendering to an image recognition model to confirm an input data type of the target UI element (par. 57 use of image object recognition to aid automated agents using machine learning for completing tasks assigned to one or more agents; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete) ..
As for claim 19, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, wherein updating the target UI element comprises presenting, on a display device, a series of updates to a rendering as a field of the target UI element is updated (par. 57 use of image object recognition to aid automated agents using machine learning for completing tasks assigned to one or more agents; par. 74-76; fig. 16 1670-1680 agent(s) completing tasks as noted above in claim 1 can move on to other tasks presented (e.g. second UI element) in a repeating function until all tasks are complete) ..
As for claim 20, Lang teaches. The one or more non-transitory, machine-readable media of claim 12, further comprising: obtaining a user identifier; and
retrieving at least one value of the user information from a shared database accessible from the first runtime environment and from a second runtime environment based on the user identifier, wherein generating the interaction data comprises providing the at least one value to the set of hierarchical agents (par. 80, 89, 101 and 227 utilizing multiple environments to retrieve multiple values of user information from database that is accessible remotely, wherein the system/software is used at an enterprise level with multiple users).
(Note :) It is noted that any citation to specific, pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006,1009, 158 USPQ 275, 277 (CCPA 1968)).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Virtual Assistant Architecture For Natural Language Understanding In A Customer Service System
Document ID
US 11606463 B1
Date Published
2023-03-14
Abstract
A virtual assistant system for communicating with customers uses human intelligence to correct any errors in the system AI, while collecting data for machine learning and future improvements for more automation. The system may use a modular design, with separate components for carrying out different system functions and sub-functions, and with frameworks for selecting the component best able to respond to a given customer conversation.
Transforming HTML Forms Into Mobile Native Forms
Document ID
US 20150363368 A1
Date Published
2015-12-17
Abstract
Techniques disclosed herein transform HTML forms into forms with graphical user interfaces (UIs) native to mobile devices. A user interface virtualization (UIV) agent divides an HTML form into rows based on row breaks. The UIV agent then identifies name-input pairs in the HTML form by applying a trained naïve Bayes classifier to determine name fields, and mapping the name fields to corresponding input fields. In addition, the UIV agent generates metadata which includes both information describing the rows in the form and the name-input information. Based on the metadata, a native form renderer running in the client device draws the form with native UI elements. In addition, the native form renderer forwards native UI events as HTML events.
Inquires
Any inquiry concerning this communication should be directed to NICHOLAS AUGUSTINE at telephone number (571)270-1056.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
PNG
media_image1.png
213
559
media_image1.png
Greyscale
/NICHOLAS AUGUSTINE/Primary Examiner, Art Unit 2178 May 22, 2026
Prosecution Insights