DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This action is in response to the arguments filed on 01/29/2026. Claims 1-4 and 6-21 are pending in the application and have been considered below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci).
As to claim 1, Holcomb teaches a system for providing artificial intelligence agents that automate software usage, comprising:
a memory that stores a plurality of processor executable instructions (paragraph [0004], memory); and
one or more hardware processors that read and execute the plurality of processor executable instructions from the memory (paragraph [0037], processors), wherein the plurality of processor-executable instructions are configurable to cause the system to perform operations comprising:
providing a training server (paragraph [0050], wherein Examiner interprets the general system training element 330 as a training server) a plurality of training datasets to train agents and thereby produce trained agents to perform a synthetic task on a user interface associated with a software application implemented on a client device (paragraphs [0005]-[0006] Providing the capability-based training includes acquiring first training data, generating a trained artificial intelligence (AI) agent by training the candidate AI agent to have at least one capability associated with using the first training data, generating synthetic validation data different from the synthetic training data, validating the trained AI agent by performing validation testing on the trained AI agent using first validation data that includes the synthetic validation data, and providing the trained AI agent as a validated AI agent after the validating the AI agent; wherein Examiner interprets” validating the trained AI agent by performing validation testing on the trained AI agent using first validation data that includes the synthetic validation data” as “trained agents to perform a synthetic task”; [0050] Thus, the AI agent training may continue with data generated by the general system training data generation element 334 until the AI agent reaches an acceptable accuracy in performing the training objective…; [0055] In some embodiments, a training data generation element 348 provides training data to an AI agent training element 346. The training data may be synthetic data or real data, or a combination of both….), [wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks];
However, Holcomb fails to explicitly teach:
wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks;
configuring a production server with the trained agents for use during the inference;
providing during the inference, multimodal interface automation prompts issued by clients to the production server to cause the production server to translate the prompts into multimodal interface automation agent calls, wherein each multimodal interface automation agent call specifies interface actions to cause the a generate one or more outputs of actuation commands that automate multimodal interface workflow in response to the multimodal prompts; and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device being actuated according to the automated multimodal interface workflow.
Lu, in combination with Holcomb, teaches:
wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks (Fig. 1 "load('calendar.google.com')",, element-wise tasks (Fig. 1 "click" and "input"), and action-wise tasks (Fig. 1 "please add 'Bring multiple copies of my resume' as the note") (Section 1 discloses recording "frames from demonstration-level video recordings" wherein the demos are used to the train the models; Fig. 2 and Table 2; Figs. 1 and 4 show example agent predicted trajectories of the usage of the software tools for completing particular tasks; Fig. 2 and Table 2; Fig. 1 shows an example trajectory of the usage of the software tools in which "screenshots" of the trajectory are interleaved with "textual website representation [s]" of the actions executed, such as "load", "click", "input", etc. and are further annotated with target symbols on clicks and areas in which information has been input; See also Fig. 5 and Appendix A.1; Fig. 1 shows an example trajectory of the usage of the software tools in which "screenshots" of the trajectory are interleaved with text descriptions of the tasks, such as "Create a task for a Career Fair on Google calendar" and "please add 'Bring multiple copies of my resume' as the note" along with annotations of target symbols on clicks and areas in which information has been input as a result of the task; See also Fig. 5 and Appendix A1; Section 5 discloses that the model is trained on browser screenshots including "images and text"; see Figs. 4 and 8, for example and Fig. 2 and Table 2).
It would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention to modify the system of Holcomb to add datasets to the system of Holcomb, as taught by Lu, above. The modification would have been obvious because one of ordinary skill would be motivated to efficiently prune HTML pages by ranking relevant elements, as suggested by Lu (Abstract).
However, Holcomb and Lu fail to explicitly teach:
configuring a production server with the trained agents for use during the inference;
providing, during inference, multimodal interface automation prompts issued by clients to the production server to cause the production server to translate the multimodal interface automation prompts into multimodal interface automation agent calls, wherein each multimodal interface automation agent call specifies interface actions to cause the a generate one or more outputs of actuation commands that automate the multimodal interface automation task workflow in response to the multimodal interface automation prompts; and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device to be actuated according to the automated multimodal interface workflow.
Cheng, in combination with Holcomb and Lu, teaches:
configuring a production server (paragraphs [0016], wherein, Examiner interprets
“building a customized generative artificial intelligence (AI) infrastructure at an enterprise server (interpreted as production sever by Examiner shown in FIGS. 1-10” to include the configuration) [0029], “enterprise server” interpreted as production sever by Examiner) with the trained agents for use during the inference (paragraphs [0031]- [0035] an inference engine 121 that conducts Al service at an inference stage, e.g., to execute a received task request. For example, the generative Al platform 110 may provide an Al agent conducting a conversation with a user for customer service, IT support, and/or the like, the inference engine 121 may then receive a user utterance via the Al agent and process the user utterance as an NLP task, e.g., question answering, document retrieval (to retrieve a support document to trouble shoot an IT issue, etc.), and/or the like. The data ingestion module 124 may integrate with various data sources including internal databases and/or any data cloud that is located remotely (e.g., data vendor servers 1045, 1070 and 1080 in FIG. 10), and provides domain adaptation during inference to serve the generative functions);
providing during inference, multimodal interface automation prompts issued by clients to the production server to cause the production server to translate the multimodal interface automation prompts into multimodal interface automation agent calls, [wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate the multimodal interface automation task workflow in response to the multimodal interface automation(paragraphs [0031 ]- [0035] an inference engine 121 that conducts Al service at an inference stage, e.g., to execute a received task request. For example, the generative Al platform 110 may provide an Al agent conducting a conversation with a user for customer service, IT support, and/or the like, the inference engine 121 may then receive a user utterance via the Al agent and process the user utterance as an NLP task, e.g., question answering, document retrieval (to retrieve a support document to trouble shoot an IT issue, etc.), and/or the like. The data ingestion module 124 may integrate with various data sources including internal databases and/or any data cloud that is located remotely (e.g., data vendor servers 1045, 1070 and 1080 in FIG. 10), and provides domain adaptation during inference to serve the generative functions).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb and Lu, to
add prompts translation to the combination system of Holcomb and Lu, as taught by
Cheng above. The modification would have been obvious because one of ordinary skill
would be motivated to generate contextualized and semantically meaningful output, as
suggested by Cheng ([0048]).
However, Holcomb, Lu and Cheng fail to explicitly teach:
wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate the multimodal interface automation task workflow in response to the multimodal interface automation prompts; and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device to be actuated according to the automated multimodal interface workflow.
Pierucci, in combination with Holcomb, Lu and Cheng, teaches
wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate the multimodal interface automation task workflow in response to the multimodal interface automation prompts (see, detailed description paragraphs [0065], a user (e.g., user organization) may use the custom model building process 10 to provide requests (Rs), e.g., first request R1, second request R2, and third request R3, etc. where each request may be requesting one type of custom model (e.g., request for a custom language model). Each request R1, R2, R3 may also include data sets related and useful to the process of building the requested custom model (e.g., data set related to customization for a language model may include input text that may have terms or words relevant to the user organization as provided below in the example). The user organization may use a user application 110 to generate these requests (e.g., Rs). In one example, a single user application 110 may be used to generate multiple different requests (e.g., R1, R2, and R3). In another example, a different user application may be used for different requests such that a first user application may generate a first request R1, a second distinct user application may generate a second request R2, and a third distinct user application may generate a third request R3. Use of different applications for different users and requests may be useful where the user); [0066] The user application 110 or user applications may direct these requests R1, R2, R3 to a front end of the system 100 which may be a custom machine learning model (MLM) server 112. The custom MLM server 112 may be used to receive multiple requests from one or more different organizations; [0067]; wherein, Examiner interprets “user application 110 to generate these requests (e.g., Rs)” as “outputs of actuation commands”); and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device to be actuated according to the automated multimodal interface workflow (paragraphs [0066] The user application 110 or user applications may direct (interpreted as “transmitting” by Examiner) these requests R1, R2, R3 to a front end of the system 100 which may be a custom machine learning model (MLM) server 112. The custom MLM server 112 may be used to receive multiple requests from one or more different organizations).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu and Cheng to
add outputs of actuation commands to the combination system of Holcomb, Lu and Cheng, as taught by Pierucci, above. The modification would have been obvious because one of ordinary skill would be motivated to effectively build the custom model based on the processing of the part of the request, as suggested by Pierucci, ([0034]).
As to claim 19, Holcomb teaches a computer-implemented method for providing artificial intelligence agents that automate software usage, the computer-implemented method comprising:
training, at least one server (paragraph [0050], wherein Examiner interprets the general system training element 330 as a training server), one or more artificial intelligence agents to perform a synthetic task on a user interface associated with a software application implemented on a client device, using a plurality of training datasets thereby to produce one or more trained artificial intelligence agents (paragraphs [0005]-[0006] Providing the capability-based training includes acquiring first training data, generating a trained artificial intelligence (AI) agent by training the candidate AI agent to have at least one capability associated with using the first training data, generating synthetic validation data different from the synthetic training data, validating the trained AI agent by performing validation testing on the trained AI agent using first validation data that includes the synthetic validation data, and providing the trained AI agent as a validated AI agent after the validating the AI agent; wherein Examiner interprets” validating the trained AI agent by performing validation testing on the trained AI agent using first validation data that includes the synthetic validation data” as “trained agents to perform a synthetic task; [0050] Thus, the AI agent training may continue with data generated by the general system training data generation element 334 until the AI agent reaches an acceptable accuracy in performing the training objective…; [0055] In some embodiments, a training data generation element 348 provides training data to an AI agent training element 346. The training data may be synthetic data or real data, or a combination of both….), [wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks];
However, Holcomb fails to explicitly teach:
wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks;
configuring the at least one server production server with the one or more trained artificial intelligence agents for use during the inference:
receiving, at the at least one server, during the inference, one or more texts of multimodal interface automation prompts from one or more clients;
translating the one or more texts of multimodal interface automation prompts into multimodal interface automation agent calls to actuate the user interface associated with the software application, wherein each multimodal interface automation agent call specifies one or more interface actions to cause the one or more trained artificial intelligence agents to generate one or more outputs of actuation commands that automate multimodal interface workflow in response to the multimodal interface automation prompts; and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device to be actuated according to the multimodal interface workflow.
Lu, in combination with Holcomb, teaches:
wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks (Fig. 1 "load('calendar.google.com')",, element-wise tasks (Fig. 1 "click" and "input"), and action-wise tasks (Fig. 1 "please add 'Bring multiple copies of my resume' as the note") (Section 1 discloses recording "frames from demonstration-level video recordings" wherein the demos are used to the train the models; Fig. 2 and Table 2; Figs. 1 and 4 show example agent predicted trajectories of the usage of the software tools for completing particular tasks; Fig. 2 and Table 2; Fig. 1 shows an example trajectory of the usage of the software tools in which "screenshots" of the trajectory are interleaved with "textual website representation [s]" of the actions executed, such as "load", "click", "input", etc. and are further annotated with target symbols on clicks and areas in which information has been input; See also Fig. 5 and Appendix A.1; Fig. 1 shows an example trajectory of the usage of the software tools in which "screenshots" of the trajectory are interleaved with text descriptions of the tasks, such as "Create a task for a Career Fair on Google calendar" and "please add 'Bring multiple copies of my resume' as the note" along with annotations of target symbols on clicks and areas in which information has been input as a result of the task; See also Fig. 5 and Appendix A1; Section 5 discloses that the model is trained on browser screenshots including "images and text"; see Figs. 4 and 8, for example and Fig. 2 and Table 2).
It would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention to modify the system of Holcomb to add datasets to the system of Holcomb, as taught by Lu, above. The modification would have been obvious because one of ordinary skill would be motivated to efficiently prune HTML pages by ranking relevant elements, as suggested by Lu (Abstract).
However, Holcomb and Lu fail to explicitly teach:
configuring the at least one server with the one or more trained artificial intelligence agents for use during the inference; providing during inference, multimodal interface automation prompts issued by clients to the production server to cause the production server to translate the prompts into multimodal interface automation agent calls, wherein each multimodal interface automation agent call specifies interface actions to cause the a generate one or more outputs of actuation commands that automate multimodal interface workflow in response to the multimodal prompts; and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device being actuated according to the automated multimodal interface workflow.
Cheng, in combination with Holcomb and Lu, teaches:
configuring the at least one server (paragraphs [0016], wherein, Examiner interprets
“building a customized generative artificial intelligence (AI) infrastructure at an enterprise server (interpreted as production sever by Examiner shown in FIGS. 1-10” to include the configuration) [0029], “enterprise server” interpreted as production sever by Examiner) with the trained agents for use during the inference configuring the at least one server with the one or more trained artificial intelligence agents for use during the inference (paragraphs [0031]- [0035] an inference engine 121 that conducts Al service at an inference stage, e.g., to execute a received task request. For example, the generative Al platform 110 may provide an Al agent conducting a conversation with a user for customer service, IT support, and/or the like, the inference engine 121 may then receive a user utterance via the Al agent and process the user utterance as an NLP task, e.g., question answering, document retrieval (to retrieve a support document to trouble shoot an IT issue, etc.), and/or the like. The data ingestion module 124 may integrate with various data sources including internal databases and/or any data cloud that is located remotely (e.g., data vendor servers 1045, 1070 and 1080 in FIG. 10), and provides domain adaptation during inference to serve the generative functions);
receiving, at the at least one server, during inference, one or more texts of multimodal interface automation prompts from one or more clients ;
translating the one or more texts of multimodal interface automation prompts into multimodal interface automation agent calls to actuate the user interface associated with the software application, wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate multimodal interface workflow in response to the multimodal prompts (paragraphs [0031 ]- [0035] an inference engine 121 that conducts Al service at an inference stage, e.g., to execute a received task request. For example, the generative Al platform 110 may provide an Al agent conducting a conversation with a user for customer service, IT support, and/or the like, the inference engine 121 may then receive a user utterance via the Al agent and process the user utterance as an NLP task, e.g., question answering, document retrieval (to retrieve a support document to trouble shoot an IT issue, etc.), and/or the like. The data ingestion module 124 may integrate with various data sources including internal databases and/or any data cloud that is located remotely (e.g., data vendor servers 1045, 1070 and 1080 in FIG. 10), and provides domain adaptation during inference to serve the generative functions).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb and Lu to
add prompts translation to the combination system of Holcomb and Lu, as taught by Cheng above. The modification would have been obvious because one of ordinary skill would be motivated to generate contextualized and semantically meaningful output, as suggested by Cheng ([0048]).
However, Holcomb, Lu and Cheng fail to explicitly teach:
wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate multimodal interface workflow in response to the multimodal prompts”); and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device being actuated according to the automated multimodal interface workflow
Pierucci, in combination with Holcomb, Lu and Cheng, teaches
wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate multimodal interface workflow in response to the multimodal prompts(see, detailed description paragraphs [0065], a user (e.g., user organization) may use the custom model building process 10 to provide requests (Rs), e.g., first request R1, second request R2, and third request R3, etc. where each request may be requesting one type of custom model (e.g., request for a custom language model). Each request R1, R2, R3 may also include data sets related and useful to the process of building the requested custom model (e.g., data set related to customization for a language model may include input text that may have terms or words relevant to the user organization as provided below in the example). The user organization may use a user application 110 to generate these requests (e.g., Rs). In one example, a single user application 110 may be used to generate multiple different requests (e.g., R1, R2, and R3). In another example, a different user application may be used for different requests such that a first user application may generate a first request R1, a second distinct user application may generate a second request R2, and a third distinct user application may generate a third request R3. Use of different applications for different users and requests may be useful where the user); [0066] The user application 110 or user applications may direct these requests R1, R2, R3 to a front end of the system 100 which may be a custom machine learning model (MLM) server 112. The custom MLM server 112 may be used to receive multiple requests from one or more different organizations; [0067]; wherein, Examiner interprets “user application 110 to generate these requests (e.g., Rs)” as “outputs of actuation commands”); and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device being actuated according to the automated multimodal interface workflow (paragraphs [0066] The user application 110 or user applications may direct (interpreted as “transmitting” by Examiner) these requests R1, R2, R3 to a front end of the system 100 which may be a custom machine learning model (MLM) server 112. The custom MLM server 112 may be used to receive multiple requests from one or more different organizations).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu and Cheng to
add outputs of actuation commands to the combination system of Holcomb, Lu and Cheng, as taught by Pierucci, above. The modification would have been obvious because one of ordinary skill would be motivated to effectively build the custom model based on the processing of the part of the request, as suggested by Pierucci, ([0034]).
As to claim 20, Holcomb teaches a non-transitory computer readable storage medium (paragraph [0034]) impressed with computer program instructions for providing artificial intelligence agents that automate software usage, wherein the computer program instructions, when executed on a processor (paragraph [0037), perform operations comprising:
training agents on a plurality of training datasets and thereby produce the trained agents to perform a synthetic task on a user interface associated with a software application implemented on a client device (paragraphs [0005]-[0006] Providing the capability-based training includes acquiring first training data, generating a trained artificial intelligence (AI) agent by training the candidate AI agent to have at least one capability associated with using the first training data, generating synthetic validation data different from the synthetic training data, validating the trained AI agent by performing validation testing on the trained AI agent using first validation data that includes the synthetic validation data, and providing the trained AI agent as a validated AI agent after the validating the AI agent; wherein Examiner interprets” validating the trained AI agent by performing validation testing on the trained AI agent using first validation data that includes the synthetic validation data” as “trained agents to perform a synthetic task; [0050] Thus, the AI agent training may continue with data generated by the general system training data generation element 334 until the AI agent reaches an acceptable accuracy in performing the training objective…; [0055] In some embodiments, a training data generation element 348 provides training data to an AI agent training element 346. The training data may be synthetic data or real data, or a combination of both….), [wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks].
However, Holcomb fails to explicitly teach:
wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks;
translating prompts issued by clients into multimodal interface automation
agent calls, wherein each multimodal interface automation agent call specifies
interface actions to cause the trained agents to generate one or more outputs of
actuation commands that automate multimodal interface workflow in response the multimodal prompts; and
transmitting one or more outputs of actuation commands to the client device thereby causing the user interface associated with the software application on the client device being actuated according to the automated multimodal interface workflow.
Lu, in combination with Holcomb, teaches:
wherein the plurality of datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks (Fig. 1 "load('calendar.google.com')",, element-wise tasks (Fig. 1 "click" and "input"), and action-wise tasks (Fig. 1 "please add 'Bring multiple copies of my resume' as the note") (Section 1 discloses recording "frames from demonstration-level video recordings" wherein the demos are used to the train the models; Fig. 2 and Table 2; Figs. 1 and 4 show example agent predicted trajectories of the usage of the software tools for completing particular tasks; Fig. 2 and Table 2; Fig. 1 shows an example trajectory of the usage of the software tools in which "screenshots" of the trajectory are interleaved with "textual website representation [s]" of the actions executed, such as "load", "click", "input", etc. and are further annotated with target symbols on clicks and areas in which information has been input; See also Fig. 5 and Appendix A.1; Fig. 1 shows an example trajectory of the usage of the software tools in which "screenshots" of the trajectory are interleaved with text descriptions of the tasks, such as "Create a task for a Career Fair on Google calendar" and "please add 'Bring multiple copies of my resume' as the note" along with annotations of target symbols on clicks and areas in which information has been input as a result of the task; See also Fig. 5 and Appendix A1; Section 5 discloses that the model is trained on browser screenshots including "images and text"; see Figs. 4 and 8, for example and Fig. 2 and Table 2).
It would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention to modify the system of Holcomb to add datasets to the system of Holcomb, as taught by Lu, above. The modification would have been obvious because one of ordinary skill would be motivated to efficiently prune HTML pages by ranking relevant elements, as suggested by Lu (Abstract).
However, Holcomb and Lu fail to explicitly teach:
translating prompts issued by clients into multimodal interface automation
agent calls, wherein each multimodal interface automation agent call specifies
interface actions to cause the trained agents to generate one or more outputs of
actuation commands that automate multimodal interface workflow in response the multimodal prompts; and
transmitting one or more outputs of actuation commands to the client device thereby causing the user interface associated with the software application on the client device being actuated according to the automated multimodal interface workflow.
Cheng, in combination with Holcomb and Lu, teaches:
translating prompts issued by clients into multimodal interface automation
agent calls, wherein each multimodal interface automation agent call specifies
interface actions to cause the trained agents to generate one or more outputs of
actuation commands that automate multimodal interface workflow in response the multimodal prompts (paragraphs [0031]- [0035] an inference engine 121 that conducts Al service at an inference stage, e.g., to execute a received task request. For example, the generative Al platform 110 may provide an Al agent conducting a conversation with a user for customer service, IT support, and/or the like, the inference engine 121 may then receive a user utterance via the Al agent and process the user utterance as an NLP task, e.g., question answering, document retrieval (to retrieve a support document to trouble shoot an IT issue, etc.), and/or the like. The data ingestion module 124 may integrate with various data sources including internal databases and/or any data cloud that is located remotely (e.g., data vendor servers 1045, 1070 and 1080 in FIG. 10), and provides domain adaptation during inference to serve the generative functions).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb and Lu, to
add prompts translation to the combination system of Holcomb and Lu, as taught by
Cheng above. The modification would have been obvious because one of ordinary skill
would be motivated to generate contextualized and semantically meaningful output, as
suggested by Cheng ([0048]).
However, Holcomb, Lu and Cheng fail to explicitly teach:
wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate multimodal interface workflow in response to the prompts; and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device to be actuated according to the multimodal interface workflow.
Pierucci, in combination with Holcomb, Lu and Cheng, teaches
wherein each multimodal interface automation agent call specifies interface actions to cause the trained agents to generate one or more outputs of actuation commands that automate a multimodal interface workflow in response to the prompts (see, detailed description paragraphs [0065], a user (e.g., user organization) may use the custom model building process 10 to provide requests (Rs), e.g., first request R1, second request R2, and third request R3, etc. where each request may be requesting one type of custom model (e.g., request for a custom language model). Each request R1, R2, R3 may also include data sets related and useful to the process of building the requested custom model (e.g., data set related to customization for a language model may include input text that may have terms or words relevant to the user organization as provided below in the example). The user organization may use a user application 110 to generate these requests (e.g., Rs). In one example, a single user application 110 may be used to generate multiple different requests (e.g., R1, R2, and R3). In another example, a different user application may be used for different requests such that a first user application may generate a first request R1, a second distinct user application may generate a second request R2, and a third distinct user application may generate a third request R3. Use of different applications for different users and requests may be useful where the user); [0066] The user application 110 or user applications may direct these requests R1, R2, R3 to a front end of the system 100 which may be a custom machine learning model (MLM) server 112. The custom MLM server 112 may be used to receive multiple requests from one or more different organizations; [0067]; wherein, Examiner interprets “user application 110 to generate these requests (e.g., Rs)” as “outputs of actuation commands”); and
transmitting the one or more outputs of actuation commands to the client
device thereby causing the user interface associated with the software application on the client device to be actuated according to the multimodal interface workflow (paragraphs [0066] The user application 110 or user applications may direct (interpreted as “transmitting” by Examiner) these requests R1, R2, R3 to a front end of the system 100 which may be a custom machine learning model (MLM) server 112. The custom MLM server 112 may be used to receive multiple requests from one or more different organizations).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu and Cheng to
add outputs of actuation commands to the combination system of Holcomb, Lu and Cheng, as taught by Pierucci, above. The modification would have been obvious because one of ordinary skill would be motivated to effectively build the custom model based on the processing of the part of the request, as suggested by Pierucci, ([0034]).
21. (New) The computer-implemented method of claim 19, further comprising: causing the at least one server to periodically retrain the trained one or more artificial intelligence agents.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and Poulis et al. (US 12,124,932 B1, hereinafter referred to as Poulis)
As to claim 2, which incorporates the rejection of claim 1, Holcomb, Lu, Cheng and Pierucci fail to explicitly teach: using one or more training datasets in the plurality of training datasets for pre-training the agents, for post-training the agents, for finisher training the agents, for combined fine-tuning of the agents, and for agentic-fine tuning of the agents for pre-training the agents, for post-training the agents, for finisher training the agents, for combined fine-tuning of the agents, and for agentic-fine tuning of the agents.
Poulis, in combination with Holcomb, Lu, Cheng and Pierucci, teaches using one or more training datasets in the plurality of training datasets for pre-training the agents, for post-training the agents, for finisher training the agents, for combined fine-tuning of the agents, and for agentic-fine tuning of the agents for pre-training the agents, for post-training the agents, for finisher training the agents, for combined fine-tuning of the agents, and for agentic-fine tuning of the agents (col. 3, lines 45-54, FIG. 2 illustrates more details of a training module 106B of the system in FIG. 1 that can post-train or fine tune the training of an LLM or LMM to align with different one or more domain specific principles; FIG. 3 illustrates a method for fine tuning the training of an LLM or LMM; and FIG. 4 illustrates a method and data flow for post-training an LLM or LMM to be aligned with domain specific principles..; col. 4, lines 8-25, an LLM or LMM may be aligned by the system to two or more domain specific principles either post-training or by generating instructions (See FIG. 2) that
25 are fine tuned in order to align the LLM/LMM to a set of principles specific for more than one domain...; col. 8, lines 65-67 col. 9, lines 1-6, the LLM/LMM 106A may be pre-trained on a large corpus of text or fine-tuned on news articles. For example, the .
LLM/LMM 106A may be pre-trained on a set of input/output instructions and definitions of journalistic principles for each of the principles described below. The journalistic
principles are used to pre-train the LLM in order to detect instances of these in the content but also to be able to generate content that respects these journalistic principles.)
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, and Pierucci
to add training datasets to the system of Holcomb, Lu, Cheng, and Pierucci, as taught by Poulis above. The modification would have been obvious because one of ordinary skill would be motivated to provide a LLM or LMM model that may be post-trained or fine-tuned during training with domain-specific knowledge, thus having a better understanding of the domain and being able to operate within the domain context more accurately and safely, as suggested by Poulis (col. 3, lines 35-39).
Claims 3 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and SINGH (US 2023/0360388 A1, hereinafter referred to as SINGH).
As to claim 3, which incorporates the rejection of claim 1, Holcomb, Lu, Cheng, and Pierucci fail to explicitly teach: causing the training servers to periodically retrain the trained agents.
SINGH, in combination with Holcomb, Lu, Cheng, and Pierucci, teaches causing the training servers to periodically retrain the trained agents (paragraphs [0055] This dynamic input may then be saved as training data for retraining AI/ML models 132, and may be stored in a database such as database 140, for example. The AI center may then schedule and execute training jobs to train the new versions of the AI/ML models using the training data. Both positive and negative examples may be stored and used for retraining of AI/ML models 132; [0061] AI/ML models may be trained and retrained, and the process may repeat itself. This facilitates a more robust and effective suite of automations; [0134]-[0135]).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, and Pierucci
to add trained and retrained to the system of Holcomb, Lu, Cheng, and Pierucci, as taught by SINGH above. The modification would have been obvious because one of ordinary skill would be motivated to facilitate robust and effective suite of automations, as suggested by SINGH ([0061]).
As to claim 21, which incorporates the rejection of claim 19, Holcomb, Lu, Cheng, and Pierucci fail to explicitly teach: causing the training servers to periodically retrain the trained agents.
SINGH, in combination with Holcomb, Lu, Cheng, and Pierucci, teaches causing the training servers to periodically retrain the trained agents (paragraphs [0055] This dynamic input may then be saved as training data for retraining AI/ML models 132, and may be stored in a database such as database 140, for example. The AI center may then schedule and execute training jobs to train the new versions of the AI/ML models using the training data. Both positive and negative examples may be stored and used for retraining of AI/ML models 132; [0061] AI/ML models may be trained and retrained, and the process may repeat itself. This facilitates a more robust and effective suite of automations; [0134]-[0135]).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, and Pierucci
to add trained and retrained to the system of Holcomb, Lu, Cheng, and Pierucci, as taught by SINGH above. The modification would have been obvious because one of ordinary skill would be motivated to facilitate robust and effective suite of automations, as suggested by SINGH ([0061]).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and SINGH (US 2023/0360388 A1, hereinafter referred to as SINGH), and Clodore et al. (US 2022/0046129 A1, hereinafter referred to as Clodore).
As to claim 4, which incorporates the rejection of claim 3, Holcomb, Lu, Cheng, and Pierucci fail to explicitly teach: periodically reconfiguring the production servers with the retrained agents responsive to reliability scores corresponding to multimodal task benchmark.
SINGH, in combination with Holcomb, Lu, Cheng, and Pierucci, teaches periodically reconfiguring the production servers with the retrained agents [responsive to reliability scores corresponding to multimodal task benchmark] (paragraph [0055] This dynamic input may then be saved as training data for retraining AI/ML models 132, and may be stored in a database such as database 140, for example. The AI center (i.e., include production servers) may then schedule and execute training jobs to train the new versions of the AI/ML models using the training data. Both positive and negative examples may be stored and used for retraining of AI/ML models 132; [0061] AI/ML models may be trained and retrained, and the process may repeat itself. This facilitates a more robust and effective suite of automations; [0134]-[0135]).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, and Pierucci to add trained and retrained to the system Holcomb, Lu, Cheng, and Pierucci, as taught by SINGH above. The modification would have been obvious because one of ordinary skill would be motivated to facilitate robust and effective suite of automations, as suggested by SINGH ([0061]).
However, Holcomb, Lu, Cheng, Pierucci and SINGH fail to explicitly teach:
responsive to reliability scores corresponding to multimodal task benchmark.
Clodore, in combination with Holcomb, Lu, Cheng, Tsai and SINGH, teaches responsive to reliability scores corresponding to multimodal task benchmark (paragraphs [0042], metrics corresponding to agent performance for these new conversations. Machine learning algorithm is retrained based on metrics corresponding
to agent performance for these new conversations; [0010]-[011], provides a client's performance against industry benchmarks for each of the performance pillars described above. For instance, the performance monitoring system may display industry benchmarks in 25 percentile increments, whereby client performance for each performance pillar may be quantified according to a particular industry benchmark quartile. The performance scores for each of the performance pillars may thus be adjusted or scaled according to the industry benchmarks to determine the client's performance against the industry benchmark (Examiner interprets “performance scores” as reliability scores and “industry benchmarks” as multimodal task benchmark).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, Pierucci and SINGH to add reliability scores to the system of Holcomb, Lu, Cheng, Pierucci and SINGH, as taught by Clodore above. The modification would have been obvious because one of ordinary skill would be motivated to generate more accurate or appropriate recommendations corresponding to the actions performed by the agents that resulting in the improved performance, as suggested by Clodore ([0042]).
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and Odinak et al. (US 2008/0118051A1, hereinafter referred to as Odinak).
As to claim 5, which incorporates the rejection of claim 1, Holcomb, Lu, and Pierucci fail to explicitly teach wherein the agent calls are multimodal interface automation agent calls.
Odinak, in combination with Holcomb, Lu, Cheng, and Pierucci, teaches wherein the agent calls are multimodal interface automation agent calls (paragraphs [0024]-[0025], providing a multi-modal communications infrastructure for automated call center operation. A multi-modal call is accepted from a caller through a telephony interface, which accommodates multi-modal calls including at least one of verbal speech and text messaging).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, and Pierucci to add multimodal interface to the system of Holcomb, Lu, Cheng, and Pierucci, as taught by Odinak above. The modification would have been obvious because one of ordinary skill would be motivated to provide a multi-modal communications infrastructure for automated call center operation, as suggested by Odinak ([0025]).
As to claim 6, which incorporates the rejection of claim 5, Holcomb, Lu, Cheng, and Pierucci fail to explicitly teach periodically configuring the clients with agent workflow logics that construct, based on the prompts, agent specifications that are configured to issue the multimodal interface automation agent calls to the trained agents.
Odinak, in combination with Holcomb, Lu, Cheng, and Pierucci, teaches periodically configuring the clients with agent workflow logics that construct, based on the prompts, agent specifications that are configured to issue the multimodal interface automation agent calls to the trained agents (paragraphs [0019] …agent can choose a predefined “script to prompt and collect or simply provide the customer with information in a step-by-step manner...; [0051]… the user engages in a customer Support scenario 22 with an agent, which is either a live person or an automated prompt. Such as with an automated Voice response system, to enable information collection and problem trouble-shooting…Importantly, however, from the perspective of the caller, the experience appears to be an interaction with an intelligent machine and the caller is aware that the agent is automated, not human...).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, and Pierucci to add multimodal interface to the system of Holcomb, Lu, Cheng, and Pierucci, as taught by Odinak above. The modification would have been obvious because one of ordinary skill would be motivated to provide a multi-modal communications infrastructure for automated call center operation, as suggested by Odinak ([0025]).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER).
As to claim 7, which incorporates the rejection of claim 1, Holcomb, Lu, Cheng, and Pierucci fail to explicitly teach wherein the plurality of training datasets includes: a first training dataset including documents containing text interleaved with images; a second training dataset including text embedded in images; a third training dataset including recorded videos of software usage; a fourth training dataset including portable document format (PDF) documents; a fifth training dataset including recorded videos of software tool usage trajectories, wherein; a sixth training dataset including images of open-domain web pages; a seventh training dataset including images of specific-domain web pages; and/or an eighth training dataset including images of agentic trajectories of the agent performing interface automation task workflows.
LODDENKEMPER, in combination with Holcomb, Lu, Cheng, and Pierucci, teaches wherein the plurality of training datasets includes: a first training dataset including documents containing text interleaved with images; a second training dataset including text embedded in images; a third training dataset including recorded videos of software usage; a fourth training dataset including portable document format (PDF) documents; a fifth training dataset including recorded videos of software tool usage trajectories, wherein; a sixth training dataset including images of open-domain web pages; a seventh training dataset including images of specific-domain web pages; and/or an eighth training dataset including images of agentic trajectories of the agent performing interface automation task workflows (paragraphs [0033]… automated software application… [0138]-[0140]…video data for training may include recorded video…; [0216]… biometric sensor data for training may include recorded videos…).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, and Pierucci to add recorded videos to the system of Holcomb, Lu, Cheng, and Pierucci, as taught by LODDENKEMPER above. The modification would have been obvious because one of ordinary skill would be motivated to evaluate the performance of the model upon training with the training dataset, as suggested by LODDENKEMPER ([0216]).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER) and Xie et al. (“Open Agents: AN OPEN PLATFORM FOR LANGUAGE AGENTS IN THE WILD,” hereinafter referred to as Xie).
As to claim 8, which incorporates the rejection of claim 7, Holcomb, Lu, Cheng, Pierucci and LODDENKEMPER fail to explicitly teach wherein images in the recorded videos of software tool usage trajectories are interleaved with text descriptions of tasks executed in the recorded videos through the software tool usage trajectories.
Xie, in combination with Holcomb, Lu, Cheng, Pierucci and LODDENKEMPER, teaches wherein images in the recorded videos of software tool usage trajectories are interleaved with text descriptions of tasks executed in the recorded videos through the software tool usage trajectories (page 15/34…. A.1.3 USER-CENTRIC INTERFACE…. we interleave images, code snippets, console outputs, and interactive visualizations with markdown text altogether…).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, Pierucci and LODDENKEMPER to add interleaved with text to the system of Poulis, Zheng, Cheng, Pierucci and LODDENKEMPER, as taught by Xie above. The modification would have been obvious because one of ordinary skill would be motivated to enhance user engagement and sustain analytical continuity, as suggested by Xie (A.1.3 USER-CENTRIC INTERFACE).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER) and Xie et al. (“Open Agents: AN OPEN PLATFORM FOR LANGUAGE AGENTS IN THE WILD,” hereinafter referred to as Xie) and Liang et al. (US 2018/031,4943 A1, hereinafter referred to as Liang).
As to claim 9, which incorporates the rejection of claim 8, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Xie fail to explicitly teach wherein the images in the recorded videos of software tool usage trajectories are further interleaved with text descriptions of actions executed on the images and image annotations resulting from execution of the actions.
Liang, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Xie, teaches wherein the images in the recorded videos of software tool usage trajectories are further interleaved with text descriptions of actions executed on the images and image annotations resulting from execution of the actions (paragraphs [0023]-[0024] select images for annotation, the images can be annotated (in any suitable manner), the annotated images can be used to fine-tune a classifier).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Xie to add images and image annotations to the system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Xie, as taught by Liang above. The modification would have been obvious because one of ordinary skill would be motivated to fine-tune the classifier by incorporating newly annotated samples in each iteration to enhance the classifier's performance gradually, as suggested by Liang ([0091]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER) and Xie et al. (“Open Agents: AN OPEN PLATFORM FOR LANGUAGE AGENTS IN THE WILD,” hereinafter referred to as Xie) and Liang et al. (US 2018/031,4943 A1, hereinafter referred to as Liang) and Dernancourt et al. (US 2019/0384807 A1, hereinafter referred to as Dernancourt (US 2019/0384807 A1, hereinafter referred to as Dernancourt).
As to claim 10, which incorporates the rejection of claim 9, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Xie and Liang fail to explicitly teach wherein the actions include clicking, scrolling, and typing.
Dernancourt, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Xie and Liang, teaches wherein the actions include clicking, scrolling, and typing (paragraphs [0093], annotator interactions include clicks, scrolls, dragging, typing, and touch gestures).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Xie and Liang to add clicking, scrolling, and typing to the combination system of Holcomb, Lu, Zheng, Cheng, Pierucci, LODDENKEMPER, Xie and Liang, as taught by Dernancourt above. The modification would have been obvious because one of ordinary skill would be motivated to utilize annotator interactions, as suggested by Dernancourt ([0093]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER)., and Barbosa et al. (US 2017/0091178 A1, hereinafter referred to as hereinafter referred to as Barbosa).
As to claim 11, which incorporates the rejection of claim 7, Holcomb, Lu, Cheng, Pierucci and LODDENKEMPER fail to explicitly teach wherein the images of open-domain web pages are automatically crawled.
Barbosa, in combination with Holcomb, Lu, Cheng, Pierucci and LODDENKEMPER, teaches wherein the images of open-domain web pages are automatically crawled (paragraphs [0042]-[0048]…the crawler automatically identifies the most important pages during the crawling process…)
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the system of Holcomb, Lu, Cheng, Pierucci and LODDENKEMPER to add web crawler to the system of Holcomb, Lu, Cheng, Pierucci and LODDENKEMPER, as taught by Barbosa above. The modification would have been obvious because one of ordinary skill would be motivated to perform a learning iteration by collecting the link neighborhood of the links, as suggested by Barbosa ([0049]).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER)., and Barbosa et al. (US 2017/0091178 A1, hereinafter referred to as hereinafter referred to as Barbosa) and Cross (US 2008/02288494 A1, hereinafter referred to as Cross).
As to claim 12, which incorporates the rejection of claim 11, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa fail to explicitly teach wherein the open-domain web pages are multimodal web pages.
Cross, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa, teaches wherein the open-domain web pages are multimodal web pages (paragraphs [0096] …multimodal browser (196) provides an execution environment for the web page (195); [0106]-[0107], multimodal browser (196) provides an execution environment for the web page (195)).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa to add web crawler to the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa, as taught by Cross above. The modification would have been obvious because one of ordinary skill would be motivated to use a multimodal browser that provides an execution environment for web page (195). The web page (195) is an information resource containing web content that can be accessed through a browser, as suggested by Cross ([0016]).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER)., and Barbosa et al. (US 2017/0091178 A1, hereinafter referred to as hereinafter referred to as Barbosa), and Agapi et al. (US 2007 /0233495 A1, hereinafter referred to as Agapi).
As to claim 13, which incorporates the rejection of claim 11, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa fail to explicitly teach wherein the open-domain web pages are part of software tools.
Agapi, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa, teaches wherein the open-domain web pages are part of software tools (paragraphs [0017], tools of a software design interface, such as a call flow development tool...; [0036] …. Automated software development tools…).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa to add software tools to the combination system of Poulis, Zheng, Cheng, Pierucci, LODDENKEMPER and Barbosa, as taught by Agapi above. The modification would have been obvious because one of ordinary skill would be motivated to use an automated software development tool for converting GUI elements into SUI elements in accordance with an embodiment of the inventive arrangements disclosed herein, as suggested by Agapi ([0036]).
Claims 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER)., and Barbosa et al. (US 2017/0091178 A1, hereinafter referred to as hereinafter referred to as Barbosa), and Pottorf et al. (US 2022/0130013 A1, hereinafter referred to as Pottorf).
As to claim 14, which incorporates the rejection of claim 11, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa fail to explicitly teach wherein the images of open-domain web pages are interleaved with text descriptions of synthetic tasks and image annotations resulting from execution of the synthetic tasks.
Pottorf, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa, teaches wherein the images of open-domain web pages are interleaved with text descriptions of synthetic tasks and image annotations resulting from execution of the synthetic tasks (paragraphs [0061], an image or sequence of images, is synthetically generated 402, such as by being rendered at a first resolution. In at least one embodiment, a second version of this training data (e.g., image or sequence) is synthetically generated 404… a wholly synthetic dataset; [0086] … image is to be synthetically generated…; [0347] …AI-assisted annotation 3110 may be used to aid in generating annotations corresponding to imaging data; [0361], annotating of imaging data 3108).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa to add synthetic tasks to the combination system Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa, as taught by Pottorf above. The modification would have been obvious because one of ordinary skill would be motivated to train a machine learning model using synthetically generated data, as suggested by Pottorf ([0061]).
As to claim 15, which incorporates the rejection of claim 14, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa fail to explicitly teach wherein the synthetic tasks include website-wise tasks, element-wise tasks, and action-wise tasks.
Pottorf, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa, teaches wherein the synthetic tasks include website-wise tasks, element-wise tasks, and action-wise tasks (paragraphs [0082], Internet web page (i.e., website-wise) search software (i.e., action-wise); e-mail virus scan software database software (i.e. action-wise), and streaming video content software (i.e. action-wise); [0050], temporal smoothing (i.e. element-wise) may require at least one sequence of images (i.e. element-wise) at one or more of these resolutions (i.e. element-wise)).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa to add synthetic tasks to the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER and Barbosa, as taught by Pottorf above. The modification would have been obvious because one of ordinary skill would be motivated to train a machine learning model using synthetically generated data, as suggested by Pottorf ([0061]).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER)., and Barbosa et al. (US 2017/0091178 A1, hereinafter referred to as hereinafter referred to as Barbosa), and Pottorf et al. (US 2022/0130013 A1, hereinafter referred to as Pottorf), and Zhang et al. (US 2023/0222285 A1, hereinafter referred to as Zhang).
As to claim 16, which incorporates the rejection of claim 15, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf fail to explicitly teach wherein the website-wise tasks include heading optical character recognition (OCR), captioning, and web question answering.
Zhang, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf, teaches wherein the website-wise tasks include heading optical character recognition (OCR), captioning, and web question answering (WebQA) (paragraphs [0061], optical character recognition; [0063], (OCR) engine); [0105], (e.g., header, paragraph, list, table, image, caption, and padding block); [0140], caption for the image block; [0042] and [0047], question answering).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf to add (OCR), captioning, and WebQA to the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf, as taught by Zhang above. The modification would have been obvious because one of ordinary skill would be motivated to generate an improved document-level representation for the document, as suggested by Zhang ([0035])
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER)., and Barbosa et al. (US 2017/0091178 A1, hereinafter referred to as hereinafter referred to as Barbosa), and Pottorf et al. (US 2022/0130013 A1, hereinafter referred to as Pottorf) and PRIESTAS et al. (US 2022/0246257 A1, hereinafter referred to as PRIESTAS).
As to claim 17, which incorporates the rejection of claim 15, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf fail to explicitly teach wherein the element-wise tasks include element optical character recognition (OCR), element grounding/localization, and key-value pair identification.
PRIESTAS, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf, teaches wherein the element-wise tasks include element optical character recognition (OCR), element grounding/localization, and key-value pair identification (paragraphs [0034]-[0035], optical character recognition (OCR); identification of specific key-value pairs; [0054], devices in one or more data centers (i.e. localization)).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf to add (OCR), localization, and key-value pair identification to the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf, as taught by PRIESTAS above. The modification would have been obvious because one of ordinary skill would be motivated to use a machine learning-based domain model that includes domain-specific terminology, definitions of industry terms, and/or possible fields of various data types that may be included in documents, as suggested by PRIESTAS ([0034]).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Holcomb et al. (US 2025/0258760 A1, hereinafter referred to as Holcomb), in view of Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue,” hereinafter referred to as Lu) and further in view of Cheng et al. (US 2024/0303443 A1, hereinafter referred to as Cheng), and Pierucci et al. (US 2024/0370765 A1, hereinafter referred to as Pierucci), and LODDENKEMPER et al. (US 20230386025 A1, hereinafter referred to as LODDENKEMPER)., and Barbosa et al. (US 2017/0091178 A1, hereinafter referred to as hereinafter referred to as Barbosa), and Pottorf et al. (US 2022/0130013 A1, hereinafter referred to as Pottorf) and Li et al. (US 2023/0031702 A1, hereinafter referred to as Li).
As to claim 18, which incorporates the rejection of claim 15, Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf fail to explicitly teach wherein the action-wise tasks include action grounding and action prediction.
Li, in combination with Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf, teaches wherein the action-wise tasks include action grounding and action prediction (paragraphs [0041], command grounding task 138; [0049], appability prediction task).
It would have been obvious to one of ordinary skill in the art before the effective filing of
the claimed invention to modify the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf, to add grounding and prediction to the combination system of Holcomb, Lu, Cheng, Pierucci, LODDENKEMPER, Barbosa and Pottorf, as taught by Li above. The modification would have been obvious because one of ordinary skill would be motivated to use a real-time developer tool that models a GUI, predicts modeling task outputs, and provides recommendations, in substantial real-time, as suggested by Li ([0091]).
Response to Applicant’s arguments
Applicant's arguments on file on 01/29/2026 with respect to claims 1-4 and 6-21 are moot in view of new ground(s) of rejection.
Argument (pages 8 of 10 and 9 of 10)
Applicant appears to assert that Poulis was filed on January 2, 2025 as a continuation-in-part application of U.S. Application 19/004,001, filed December 27, 2024, which in turn is a continuation of U.S. Application No. 18/646,104, filed April 25, 2024, which in turn is continuation-in-part application of U.S. Application No. 18/599,955, filed March 8, 2024. The cited portions of Poulis (e.g., col. 5, lines 19-50 of Poulis cited in the OA, pages 4-5, as allegedly relevant to the claimed "providing a training server a plurality of training datasets to train agents and thereby produce trained agents to perform synthetic task on a user interface associated with a software application implemented on a client device" of independent claims 1, 19, and 20) was only first added in Poulis at its filing date and therefore has a priority date of January 2, 2025. Poulis thus is not qualified prior art.
Zheng was cited as allegedly relevant to the claimed "wherein the plurality of
datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks" recited in claim 1 (OA, page 6). However, Zheng was published on March 26, 2024, which is later than the earliest priority date of the noted claims, and is also not qualified prior art.
Applicant respectfully submits that the remaining references, namely, Cheng and
Pierucci, do not disclose the limitations of "providing a training server a plurality of training datasets to train agents and thereby produce trained agents to perform a synthetic task on a user interface associated with a software application implemented on a client device, wherein the plurality of training datasets includes images of defined agentic trajectories for multimodal interface automation task workflows, interleaved with one or more of interface document object models (DOMs), unified resource locators (URLs), and text descriptions of synthetic tasks, wherein the synthetic tasks include one or more of website-wise tasks, element-wise tasks, and action-wise tasks" as recited in amended claim 1. Independent claims 19 and 20 also include similar limitations as claim 1.
Thus, Applicant respectfully submits that independent claims 1, 9, and 20 are submitted as patentable over the cited references. Claims 2-4, and 6-18 are also patentable by virtue of their dependencies of independent claim 1.
Accordingly, Applicant respectfully requests reconsideration and withdrawal of the
rejections under 35 U.S.C. § 103.
New Claim
Claim 21 is added and depends on claims 19. Therefore, claim 21 is submitted as
patentable over the cited references for at least the same reasons as claim 19 discussed above.
Examiner’s response:
Applicant’s arguments are moot in view of new ground(s) of rejection (Holcomb et al. (US 2025/0258760 A1)), (Lu et al. (“WEBLINX: Real-World Website Navigation with Multi-Turn Dialogue”) and Poulis et al. (US 12,124,932 B1).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABABACAR SECK whose telephone number is (571)270-7146. The examiner can normally be reached Monday-Friday 8:00 A.M.-6:00 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached on 5712705871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABABACAR SECK/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights