Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This Office Action is responsive to Applicants' Amendment filed on February 9, 2026, in which claims 1, 7-9, 14-15, and 20 are currently amended. Claims 1-5, 7-10, 12, and 14-20 are currently pending.
Response to Arguments
The rejections to claims 1 and 9 under 35 U.S.C. § 112(b) are hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections.
Applicant’s arguments with respect to rejection of claims 1-5, 7-10, 12, and 14-20 under 35 U.S.C. 103 based on amendment have been considered. The argument is moot in view of a new ground of rejection set forth below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-10, 12, 14-17, 19, and 20 are rejected under U.S.C. §103 as being unpatentable over the combination of Han (“BrowserPilot”, 2024) and Gur (“A REAL-WORLD WEBAGENT WITH PLANNING, LONG CONTEXT UNDERSTANDING, AND PROGRAM SYNTHESIS”, 2024).
Regarding claim 1, Han teaches A system for constructing prompts that cause an agent to automate multimodal interface workflows via a client device, comprising: ([p. 19] "BrowserPilot An intelligent web browsing agent controlled by natural language. […] Language is the most natural interface through which humans give and receive instructions. Instead of writing bespoke automation or scraping code which is brittle to changes, creating and adding agents should be as simple as writing plain English" web browser interpreted as multimodal interface)
a memory storing a plurality of processor-executable instructions: and one or more processors coupled to the memory and executing the plurality of processor-executable instructions to:([p. 22] "This package runs code output from the OpenAI API in Python using `exec`." BrowserPilot is a Python software package with executable instructions stored in source files and executed on a user machine)
generate, using agent specification logic on the client device, agent specification functions based on a natural language prompt, ([p. 19] "this project uses GPT-3 to translate natural language into code" [p. 20] " The prompts used can be found in [instruction compiler]" [p. 13] "class InstructionCompiler: [...] """Initialize the compiler. The compiler handles the sequencing of each set of instructions which are injected into the base prompt. [...] The primary entry point is step(). At each step, the compiler will take the current instruction and inject it into the base prompt, asking the language model to get the next action. Once it has the next action, it will inject the action into the base prompt, asking the language model to get the output for that action. [...] base_prompt (str): The base prompt to use. Defaults to BASE_PROMPT." InstructionCompiler is interpreted as the agent-specification logic. It receives the natural language instruction block, injects it into BASE_PROMPT, sends it to the model, and receives generated code)
wherein the agent specification functions are configured to automate a multimodal interface workflow, ([p. 19] "An intelligent web browsing agent controlled by natural language.")
and wherein the agent specification functions include multiple levels of prompts including click-level prompts, ([p. 21] "`InstructionCompiler` is used to parse user input into semantically cogent blocks of actions" [p. 23] "compiled: […] env.click(visible_text_box)" [p. 23] "Find all anchor elements that link to Wikipedia.- Click on the first one." See also FIG. 7 706.)
and task-level prompts, ([p. 23] "Find all anchor elements that link to Wikipedia.- Click on the first one." full task prompt interpreted as task level prompt)
and wherein the agent specification functions conceal complexities of a webpage document object model (DOM) by abstracting interface elements into higher-level references;([p. 20] "Find the first visible textarea. Click on the first visible textarea. [...] to preprocess HTML and use GPT-3 to intelligently pick elements out" [p. 10] "Clean the HTML from self.driver, ask GPT-Index to find the element, and return Selenium code to access it. Return a GPTWebElement" BrowserPilot lets the user write natural language descriptions like "first visible textarea" while internally it preprocesses HTML, asks the LLM to locate an element, generates Xpath values, uses Selenium WebElements, searches iframes, and wraps elements at GPTWebElement, which conceals the DOM complexity behind natural-language prompts and env.* functions)
transmit, from the client device, the agent specification functions to a server; ([p. 19] "Download Chromedriver" [p. 3] "# Instantiate Service with the path to the chromedriver and the options. service = Service(chromedriver_path) self.driver = webdriver.Chrome(service=service" [p. 12] "BASE_PROMPT = """You have an instance `env` with methods:- `env.driver`, the Selenium webdriver" Chromedriver interpreted as a webserver. The client transmits the agent specification functions to the chromedriver server to interact with the browser based on the natural language prompt)
receive an intermediate representation from the server in response to the transmitted agent specification functions, wherein the intermediate representation encodes the agent specification functions in a modified form;([p. 8] "First, it finds all elements on the page that match the given `by` and `value`. Then, it finds all iframes on the page and switches to each one. It then finds all elements on the page that match the given `by` and `value`. It then switches back to the original frame and repeats the process for each iframe. Finally, it returns the list of all elements found on the page and in all iframes. Returns a list of GPTWebElement objects [...] elements = self.driver.find_elements(by, value) [...] elements = [GPTWebElement(element) for element in elements]" Han explicitly receives an intermediate representation "elements" from the server (self.driver) in response to the transmitted agent specification functions. BrowserPilot then wraps the elements as GPTWebElement objects using the same variable)
detect, by runtime interpretation logic on the client device, one or more agent functions in the intermediate representation; generate one or more multimodal agent calls based on the one or more agent functions;([p. 12] "`env.send_keys(element, text)` sends `text` to element. […] `env.click(element)` clicks the WebElement" BrowserPilot executes generated code with env bound to the local GPTSeleniumAgent. The returned GPTWebElement "element" handles are then used as operands for env.click(element), env.send_keys(element, text), env.screenshot(element, filename), etc.)
transmit, from the client device, the one or more multimodal agent calls to an agent on the server; ([p. 1] "from selenium import webdriver from selenium.webdriver.chrome.service import Service" [p. 3] "# Instantiate Service with the path to the chromedriver and the options. service = Service(chromedriver_path) self.driver = webdriver.Chrome(service=service, options=_chrome_options)" [p. 12] "BASE_PROMPT = """You have an instance `env` with methods:- `env.driver`, the Selenium [...] env.click(element)" BrowserPilot explicitly creates a ChromeDriver server and a Selenium Webdriver session. The generated env. Calls are transmitted as WebDriver (agent) commands through that session to the ChromeDriver server which controls the browser, including, importantly the elements which were obtained responsive to scraping the page with a first transmission of an agent specification function)
receive runtime actuation functions from the agent in response to the one or more multimodal agent calls; ([p. 1] "class GPTWebElement(webdriver.remote.webelement.WebElement): """Wrapper over Selenium's WebElement with an additional iframe ivar for recordkeeping."""" [p. 8] "it returns the list of all elements found on the page and in all iframes. Returns a list of GPTWebElement objects." [p. 11] "return GPTWebElement(element, iframe=iframe_of_element)" ChromeDriver (server) returns WebElement handles in response to WebDriver (agent) calls. BrowserPilot wraps them as GPTWebElement preserving the Selenium element's parent/session and element id.)
and trigger, by the client device, one or more machine-actuated actions based on the runtime actuation functions as synthetic actions that automate the multimodal interface workflow, ([p. 7] "def is_element_visible_in_viewport(self, element: GPTWebElement) -> bool: is_visible = self.driver.execute_script(" [p. 9] "ActionChains(self.driver).pause(wait_time).move_to_element(element).pause(wait_time).click(element).perform()" BrowserPilot uses the returned GPTWebElement handle to trigger synthetic behavior actions. Clicks, for example, are performed using Selenium ActionChains.)
wherein automating the multimodal interface workflow includes modifying elements of a multimodal interface, ([p. 9] "url_before_click = self.driver.current_url ActionChains(self.driver).pause(wait_time).move_to_element(element).pause(wait_time).click(element).perform() url_after_click = self.driver.current_url […] # If the URL changed, then add the page to memory. if self.memory_folder and (url_before_click != url_after_click):" Han explicitly anticipates the automation modifying elements of the multimodal interface and performs explicitly checks for it)
and wherein the multimodal interface is a user interface.([p. 19] "instructions = """Go to Google.com Find all textareas. Find the first visible textarea. Click on the first visible textarea.").
However, Han does not explicitly teach step-level prompts, .
Gur, in the same field of endeavor, teaches step-level prompts, ([p. 18] "HTML-T5 iteratively predicts a decomposed sub-instruction and task-relevant snippet (orange) in a closed-loop manner, conditioning on the HTML documents, instruction (yellow), and history of past predictions" See FIG. 6 where sub-instruction is interpreted as step-level).
Han as well as Gur are directed towards natural language web browser automation. Therefore, Han as well as Gur are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Han with the teachings of Gur by using HTML-T5 to partition task level prompts into sub-instructions (step-level prompts) and use those and the BrowserPilot input. Gur provides as additional motivation for combination ([Abstract] "HTML T5 is the best model to solve various HTML understanding tasks; achieving 18.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation"). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Han and Gur teaches The system of claim 1, wherein the agent specification functions are constructable using various degrees of expressiveness(Gur [p. 9 §5] "we have exhibited WebAgent can follow given complex and sometimes ambiguous instructions on real estate, social media and map websites" See also Examples on p. 20 §F which demonstrate varying degrees of expressiveness.).
Regarding claim 3, the combination of Han and Gur teaches The system of claim 2, wherein the agent specification functions are constructed using natural language commands(Gur [Abstract] "We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions" See also FIG. 6).
Regarding claim 4, the combination of Han and Gur teaches The system of claim 3, wherein the agent specification functions are constructed using prescriptive commands(Gur [p. 3 §3] "Users initiate natural language interactions with a clear intent, such as apartment searching. Upon receiving the initial request, HTML-T5 formulates a “go to <URL>” sub-instruction, triggering Flan-U-PaLM to generate a corresponding Python program that navigates to the specified website" Go to <URL> interpreted as prescriptive command).
Regarding claim 5, the combination of Han and Gur teaches The system of claim 4, wherein the agent specification functions are constructed using a combination of the natural language commands and the prescriptive commands(Gur [p. 3 §3] "Users initiate natural language interactions with a clear intent, such as apartment searching. Upon receiving the initial request, HTML-T5 formulates a “go to <URL>” sub-instruction, triggering Flan-U-PaLM to generate a corresponding Python program that navigates to the specified website").
Regarding claim 7, the combination of Han and Gur teaches The system of claim 2, wherein a first agent specification function of the agent specification functions includes interface operations and interface elements(Gur [p. 22] "driver.find_element(By.CSS_SELECTOR,"#Cycling").click()" CSS_SELECTOR "#Cycling" is interpreted as interface element, click() interpreted as interface operation.).
Regarding claim 8, the combination of Han and Gur teaches The system of claim 7, wherein a second agent specification function of the agent specification functions does not include the interface operations and the interface elements(Gur [p. 22] "driver.get("https://www.(map website).com/")").
Regarding claim 9, the combination of Han and Gur teaches The system of claim 8, wherein the first agent specification function and the second agent specification function apply the interface operations to the interface elements(Gur [p. 22] "driver.find_element(By.CSS_SELECTOR,"#Cycling").click()" CSS_SELECTOR "#Cycling" is interpreted as interface element, click() interpreted as interface operation.).
Regarding claim 10, the combination of Han and Gur teaches The system of claim 1, wherein the one or more machine-actuated actions are a sequence of actuation commands(Gur [p. 5] "Scripted Planning and Prompted Programming We employ a rule-based parser to decompose instructions into sequences of sub-instructions").
Regarding claim 12, the combination of Han and Gur teaches The system of claim 1, wherein the one or more processors further execute the plurality of processor-executed instructions to: receive the runtime activation functions from the agent, and process the runtime actuation functions to trigger the one or more machine-actuated actions as synthetic actions that automate the multimodal interface workflow(Gur [p. 6 §3.3] "This conditional program synthesis requires LLMs to not only generate code to follow natural language instructions but also understand the semantics and functionality of HTML elements. We provide several Python snippet examples generated by Flan-U-PaLM as follows (sub-instructions are treated as comments in the script)" See FIG. 5 and See lines 1-12 of generated Selenium Python code (multimodal agent calls like click()) which is received as an HTTP POST to the element-click endpoint on the server which triggers the one or more machine actuated actions based on the runtime actuation functions.).
Regarding claim 14, the combination of Han and Gur teaches The system of claim 1, wherein the agent specification functions include at least one of an answerQuestionAboutScreen function , goToURL function, an act function, a typeIntoElement function, a click function, a type function, a wait function, a goToSong function, a compose function, an answerTrueFalseQuestion functionAboutScreen, a composeAndType function, a getCurrentDate function, a pickdate function, a fillform function, a isVisible function, a keydown function, a print function, a scroll function, or a spotlight function(Han [p. 21] "`InstructionCompiler` is used to parse user input into semantically cogent blocks of actions" [p. 23] "compiled: […] env.click(visible_text_box)" [p. 23] "Find all anchor elements that link to Wikipedia.- Click on the first one." See also FIG. 7 706.).
Regarding claim 15, the combination of Han and Gur teaches The system of claim 7, wherein the interface operations include at least one of clicks, hovers, scrolls, picks, text entries, or form fills(Han [p. 21] "`InstructionCompiler` is used to parse user input into semantically cogent blocks of actions" [p. 23] "compiled: […] env.click(visible_text_box)" [p. 23] "Find all anchor elements that link to Wikipedia.- Click on the first one." See also FIG. 7 706.).
Regarding claim 16, the combination of Han and Gur teaches The system of claim 1, wherein the one or more processors further execute the plurality of processor-executable instructions to construct the agent specification functions using another agent(Gur [p. 2] "We combine two LLMs to form WebAgent: newly introduced HTML-T5, a domain-expert pre-trained language model, for task planning and conditional HTML summarization and Flan-UPaLM" Gur combines a number of agents to construct the agent specifications.).
Regarding claim 17, the combination of Han and Gur teaches The system of claim 16, wherein the another agent is a large language model (LLM)(Gur [p. 2] "We combine two LLMs to form WebAgent: newly introduced HTML-T5, a domain-expert pre-trained language model, for task planning and conditional HTML summarization and Flan-UPaLM").
Regarding claim 19, the combination of Han and Gur teaches The system of claim 16, wherein the one or more processors further execute the plurality of processor-executable instructions to construct the agent specification functions using atomic actions(Gur [p. 6] "To address this open-domain challenge, we introduce the act via programming paradigm in web automation by utilizing the conditional code generation capabilities of LLMs (Chen et al., 2021a; Liang et al., 2023). Provided with few-shot generic examples (such as selecting checkboxes, entering text into inputs, clicking on options, and scrolling etc.)" Click, scroll, type, etc. are interpreted as atomic actions).
Regarding claim 20, the combination of Han and Gur teaches The system of claim 19, wherein the atomic actions include click, type, scroll, and act(Gur See Table 12 on p. 27).
Claim 18 is rejected under U.S.C. §103 as being unpatentable over the combination of Han and Gur and in further view of Chen ("MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence", 2023).
Regarding claim 18, the combination of Han and Gur teaches The system of claim 16.
However, the combination of Han and Gur doesn't explicitly teach wherein the one or more processors further execute the plurality of processor-executable instructions to receive a preliminary agent specification from the another agent, and to receive edits from a user to the preliminary agent specification to generate a final agent specification.
Chen, in the same field of endeavor, teaches the one or more processors further execute the plurality of processor-executable instructions to receive a preliminary agent specification from the another agent, and to receive edits from a user to the preliminary agent specification to generate a final agent specification ([p. 1 Abstract] "We provide MIWA, a mixed-initiative web automation system that enables users to create web automation scripts by demonstrating what content they want from the targeted websites. Compared to existing web automation tools, MIWA helps users better understand a generated script and build trust in it by (1) providing a step-by-step explanation of the script’s behavior with visual correspondence to the target website, (2) supporting greater autonomy and control over web automation via step-through debugging and fine-grained demonstration refinement, and (3) automatically detecting potential corner cases that are handled improperly by the generated script" [p. 3 §3.3] "This helps PBD systems better reduce ambiguity in user demonstrations. Such multimodal specifications have also been adopted by other PBD systems").
The combination of Han and Gur as well as Chen are directed towards using machine learning models for web browser automation. Therefore, the combination of Han and Gur as well as Chen are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Han and Gur with the teachings of Chen by allowing the user to edit generated automation commands. Chen provides as additional motivation for combination ([p. 3 §2.3] "Existing PBD systems adopt various interaction mechanisms to improve the usability of PBD. For instance, Krosnick and Oney recently proposed a tool to allow users to specify which parts of a natural language query can be generalized, in addition to providing demonstration traces [30]. This helps PBD systems better reduce ambiguity in user demonstrations").
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Zhang (US12174906B1) is directed towards a natural language web browser automation system.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
Prosecution Insights