DETAILED ACTION
This action is responsive to the application filed on July 26, 2024, which claims priority from Provisional Application 63/529,130 filed on July 26, 2023.
Claims 1-20 are pending and presented to examination.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Drawings
The drawings filed on August 14, 2024 are acceptable for examination purposes.
Claim Objections
Claims 1-20 are objected to because of the following informalities: Claim 5 recites the limitation “wherein generating the first prompt comprises generating the first prompt comprising the test statement, the first screenshot, and the first textual representation of the first set of target content, and a description of the first action executed within the first instance of the target webpage.” For claims 1-20, remove the bullets from the claim language and rewrite the claims with proper indentation and semicolons. Appropriate correction is required. Please amend the claim language indicated in bold.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 12 and 18-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 18 recites the limitation "executing the second sequence of actions within the target webpage according to the third set of code" in lines 14-15. There is insufficient antecedent basis for this limitation in the claim.
Claim 19 recites the limitation "capturing a second screenshot of a region of a second instance of the target webpage depicting a second set of target content rendered within the viewport responsive to execution of the first sequence of actions;" in lines 32-33. There is insufficient antecedent basis for this limitation in the claim.
Claim 12 recites "wherein generating the second response comprises generating the textual response representing occurrence of the target outcome at the target webpage and describing the first sequence of actions completed to achieve the target outcome.". There is insufficient antecedent basis for this limitation in the claim.
Dependent claim 20 does not overcome the deficiency of the base claim and, therefore, are rejected for the same reasons as the base claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention recites a judicial exception, is directed to that judicial exception, an abstract idea, as it has not been integrated into practical application and the claims further do not recite significantly more than the judicial exception. Examiner has evaluated the claims under the framework provided in the 2019 Patent Eligibility Guidance published in the Federal Register 01/07/2019 and has provided such analysis below.
Step 1: Claims 1-20 are directed to methods and fall within the statutory category of processes. Therefore, “Are the claims to a process, machine, manufacture or composition of matter?” Yes.
In order to evaluate the Step 2A inquiry “Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?” we must determine, at Step 2A Prong 1, whether the claim recites a law of nature, a natural phenomenon or an abstract idea and further whether the claim recites additional elements that integrate the judicial exception into a practical application.
Step 2A Prong 1:
Claims 1, 9 and 19 as drafted, recite a process that, under its broadest reasonable interpretation, covers steps that could reasonably be performed in the mind, including with the aid of pen and paper, but for the recitation of generic computer components. That is, the limitations: Claim 1
a) “based on the language model and the first prompt, generating a textual response to the test statement representing occurrence of the target outcome at the target webpage;” – Mental Process, See MPEP 2106.04(a)(2), III. Claim 9
a) “based on the language model and the first prompt, generating a first response: describing a first sequence of actions executable within the target webpage and predicted to yield the target outcome; and comprising a first set of code corresponding to the first sequence of actions;” – Mental Process, See MPEP 2106.04(a)(2), III.
b) “based on the language model and the second prompt, generating a second response to the test statement representing occurrence of the target outcome;” – Mental Process, See MPEP 2106.04(a)(2), III. Claim 19
a) “based on the language model and the first prompt, generating a first response: describing a first action executable within the target webpage and predicted to yield the target outcome; and comprising a first set of code corresponding to the first action;” – Mental Process, See MPEP 2106.04(a)(2), III.
b) “in response to failure to execute the first action, generating a second prompt comprising the test statement, the first screenshot, the first textual representation, a description of the first action and the first set of code, and a first instruction to not repeat the first action;” – Mental Process, See MPEP 2106.04(a)(2), III.
c) “based on the language model and the second prompt, generating a second response: describing a second action executable within the target webpage and predicted to yield the target outcome; and comprising a second set of code corresponding to the second action;” – Mental Process, See MPEP 2106.04(a)(2), III. d) “based on the language model and the third prompt, generating a third response to the test statement representing occurrence of the target outcome and describing the second action;” – Mental Process, See MPEP 2106.04(a)(2), III. That is, nothing in the claim elements precludes the step from practically being performed in the mind or with a pen and paper. These limitations can be performed in the human mind though observation, evaluation, judgment, opinion with the aid of pen and paper. Thus, these limitations fall within the “Mental Processes” grouping of abstract ideas.
Therefore, Yes, claims 1, 9 and 19 recite judicial exceptions.
The claims have been identified to recite judicial exceptions, Step 2A Prong 2 will evaluate whether the claims are directed to the judicial exception.
Step 2A Prong 2:
This judicial exception is not integrated into a practical application. Claim 1
Additional element 1 — “accessing a test statement defining a target outcome associated with contents of a target webpage affiliated with an organization;” -insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 2 — “capturing a first screenshot of a first region of the target webpage depicting a first set of target content and rendered within a viewport;”. - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 3 — “accessing a set of webpage code defined for the target webpage and corresponding to the first set of target content depicted in the first screenshot;”. - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 4 — “generating a first prompt comprising the test statement, an address corresponding to the target webpage, the first screenshot, and the first textual representation of the first set of target content;”. - insignificant extra-solution activity (Data Formatting/assembly), See MPEP 2106.05(g).
Additional element 5 — “accessing a language model configured to generate responses to test statements based on visual and textual content extracted from corresponding prompts;”. – Mere instructions to Apply an Exception, See MPEP 2106.05(f).
Additional element 6 — “serving the textual response to a user associated with the organization”. - insignificant extra-solution activity (Data Output/Transmission), See MPEP 2106.05(g).
Claim 9
Additional element 1 — “accessing a test statement defining a target outcome associated with contents of a target webpage;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 2 — “capturing a first screenshot of a first region of a first instance of the target webpage depicting a first set of target content rendered within a viewport;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 3 — “accessing a set of webpage code defined for the target webpage and corresponding to contents of the target webpage;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 4 — “generating a first prompt comprising the test statement, an address corresponding to the target webpage, the first screenshot, and the first textual representation of the first set of target content;” - insignificant extra-solution activity (Data Formatting/Assembly), See MPEP 2106.05(g).
Additional element 5 — “accessing a language model configured to generate responses to test statements based on visual and textual content extracted from corresponding prompts;” - Mere instructions to Apply an Exception, See MPEP 2106.05(f).
Additional element 6 — “executing the first sequence of actions within the target webpage according to the first set of code;” - Mere instructions to Apply an Exception, See MPEP 2106.05(f).
Additional element 7 — “capturing a second screenshot of a region of a second instance of the target webpage depicting a second set of target content rendered within the viewport responsive to execution of the first sequence of actions;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 8 — “generating a second prompt comprising the test statement, the second screenshot, and the second textual representation;” - insignificant extra-solution activity (Data Formatting/Assembly), See MPEP 2106.05(g).
Additional element 9 — “serving the second response to a user associated with the target webpage.” - insignificant extra-solution activity (Data Output/Transmission), See MPEP 2106.05(g).
Claim 19
Additional element 1 — “accessing a test statement defining a target outcome associated with contents of a target webpage;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 2 — “capturing a first screenshot of a region of a first target webpage, in a set of webpages, depicting a first set of target content rendered within a viewport;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 3 — “accessing a first set of webpage code defined for the first target webpage and corresponding to contents of the first target webpage;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 4 — “generating a first prompt comprising the test statement, the first screenshot, and the first textual representation of the first set of target content;” - insignificant extra-solution activity (Data Formatting/Assembly), See MPEP 2106.05(g).
Additional element 5 — “accessing a language model configured to generate responses to test statements based on visual and textual content extracted from corresponding prompts;” - Mere instructions to Apply an Exception, See MPEP 2106.05(f).
Additional element 6 — “triggering execution of the first action within the target webpage according to the first set of code;” - Mere instructions to Apply an Exception, See MPEP 2106.05(f).
Additional element 7 — “triggering execution of the first action within the target triggering execution of the second action within the target webpage according to the second set of code;” - Mere instructions to Apply an Exception, See MPEP 2106.05(f).
Additional element 8 — “in response to execution of the second action: capturing a second screenshot of a region of a second instance of the target webpage depicting a second set of target content rendered within the viewport responsive to execution of the first sequence of actions;” - insignificant extra-solution activity (Data Gathering), See MPEP 2106.05(g).
Additional element 9 — “generating a third prompt comprising the test statement, the second screenshot, the second textual representation, and a description of the second action;” - insignificant extra-solution activity (Data Formatting/Assembly), See MPEP 2106.05(g).
Additional element 10 — “serving the third response to a user associated with the target webpage.” - insignificant extra-solution activity (Data Output/Transmission), See MPEP 2106.05(g).
Accordingly, the additional elements recited in the claims do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea, thus failing to integrate the abstract idea into a practical application.
Therefore, “Do the claims recite additional elements that integrate the judicial exception into a practical application? No, these additional elements do not integrate the abstract idea into a practical application and they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
After having evaluating the inquires set forth in Steps 2A Prong 1 and 2, it has been concluded that claims 1, 9 and 19 not only recites a judicial exception but that the claim is directed to the judicial exception as the judicial exception has not been integrated into practical application.
Step 2B:
Additional elements for claims 1, 9 and 19: Additional element 1(claim 1) — “transforming the set of webpage code into a first sequence of contextual tags corresponding to the first set of target content depicted in the first screenshot to generate a first textual representation of the target webpage;” - Well-Understood, Routine, Conventional Activity, See MPEP 2106.05(d).
Additional element 2 (claim 9) — “generating a first textual representation of the target webpage by transforming the set of webpage code into a first sequence of contextual tags corresponding to the first set of target content;” - Well-Understood, Routine, Conventional Activity, See MPEP 2106.05(d).
Additional element 3 (claim 9) — “generating a second textual representation of the target webpage by transforming the set of webpage code into a second sequence of contextual tags corresponding to the second set of target content;” - Well-Understood, Routine, Conventional Activity, See MPEP 2106.05(d).
Additional element 4 (claim 19) — “generating a first textual representation of the first target webpage by transforming the set of webpage code into a first sequence of contextual tags corresponding to the first set of target content depicted in the first screenshot;” - Well-Understood, Routine, Conventional Activity, See MPEP 2106.05(d).
Additional element 5 (claim 19) — “generating a second textual representation of the target webpage by transforming the set of webpage code into a second sequence of contextual tags corresponding to the second set of target content;” - Well-Understood, Routine, Conventional Activity, See MPEP 2106.05(d). Accordingly, the additional elements recited in the claims cannot provide an inventive concept. In addition, after further evaluation the claim as a whole doesn’t improve any function of a computer or to any other technology or technical field. Thus, the claims are not patent eligible.
Therefore, “Do the claims recite additional elements that amount to significantly more than the judicial exception? No, these additional elements, alone or in combination, do not amount to significantly more than the judicial exception.
Having concluded analysis within the provided framework, Claims 1, 9 and 19 do not recite patent eligible subject matter under 35 U.S.C. § 101. Claim 2
The additional elements of claim 2 recite annotation of the first screenshot with visual markings each corresponding to a contextual tag of the first textual representation, and supply of the annotated screenshot as part of the first prompt. Image annotation tied to data identifiers is well-understood, routine, and conventional in graphical and document-processing systems, and the further inclusion of the annotated screenshot in the prompt is insignificant pre-solution data formatting/assembly. These additional elements do not integrate the abstract idea of claim 1 into a practical application, nor do they amount to significantly more than the abstract idea itself, and claim 2 therefore remains directed to the same mental process of evaluating whether a target webpage exhibits a target outcome.
Claim 3
The additional elements of claim 3 narrow the contextual-tag transformation to a specific tag/identifier data structure (each tag comprising a text string describing an interactive feature and an identifier linked to the text string) and tie each visual marking to its corresponding identifier. Both narrowings recite well-understood, routine, and conventional operations in HTML element extraction and graphical annotation, and they do not integrate the abstract idea into a practical application. Claim 3 remains directed to the same mental process as claims 1 and 2.
Claim 4
The additional elements of claim 4 narrow the recited target outcome to presence or absence of a target element on the target webpage, and narrow the recited textual response to a binary “true” or “false” output. Both narrowings recite further mental processes — a quintessential observational judgment (does the element appear or not) and a binary truth-value assignment. Rather than integrating the abstract idea into a practical application, claim 4 adds additional mental-process recitations on top of claim 1’s abstract idea, and remains directed to the same mental process.
Claim 5
The additional elements of claim 5 recite a closed-loop iteration of the method of claim 1 — pre-action screenshot capture, pre-action webpage code access, contextual-tag transformation, prompt construction, language-model-based action prediction, action execution, and post-action observation cycle. Each successive screenshot capture and webpage code access is insignificant pre-solution data gathering for the next iteration; each prompt construction is insignificant pre-solution data formatting/assembly; the contextual-tag transformation is well-understood, routine, and conventional; the language-model-based action prediction is a mental decision implemented via generic AI invocation; and the action execution is mere instructions to apply the mental decision via generic computing. Iterating the abstract idea does not change its character. Claim 5 remains directed to the same mental process as claim 1.
Claim 6
The additional elements of claim 6 characterize the accessed webpage code by a first data size and require the textual representation to be of a second, smaller data size. Pre-solution acquisition of input data with size characterization remains insignificant pre-solution data gathering, and HTML-to-snippet size reduction is well-understood, routine, and conventional in document-processing systems. Neither additional element integrates the abstract idea into a practical application. Claim 6 remains directed to the same mental process as claim 1.
Claim 7
The additional elements of claim 7 recite receipt of the test statement from, and delivery of the textual response to, a generic “test portal executing on a computing device accessed by the user.” Both recitations invoke generic computing hardware for input/output and amount to mere instructions to apply the abstract idea using a generic computing environment. Reciting a generic user-facing interface does not integrate the abstract idea into a practical application. Claim 7 remains directed to the same mental process as claim 1.
Claim 8
The additional elements of claim 8 recite acquisition of an additional input data structure (a website map representing a corpus of action sequences across a set of webpages) and inclusion of that additional input within the prompt. Acquisition of an additional input is insignificant pre-solution data gathering, and inclusion of that additional input in the prompt is insignificant pre-solution data formatting/assembly. Supplying more input data to the abstract idea does not integrate it into a practical application. Claim 8 remains directed to the same mental process as claim 1.
Claim 10
The additional elements of claim 10 recite annotation of both the first and second screenshots with visual markings each corresponding to a contextual tag of the respective textual representation, and supply of the annotated screenshots as part of the respective prompts. Image annotation tied to data identifiers is well-understood, routine, and conventional, and the further inclusion of the annotated screenshots in the prompts is insignificant pre-solution data formatting/assembly. These additional elements do not integrate the abstract idea of claim 9 into a practical application. Claim 10 remains directed to the same mental process as claim 9.
Claim 11
Claim 11 recites the same narrowing of the contextual-tag transformation and marking correspondence as claim 3, applied within the framework of claim 9. The analysis is the same as set forth for claim 3: the narrowed transformation and marking correspondence are well-understood, routine, and conventional operations and do not integrate the abstract idea into a practical application. Claim 11 remains directed to the same mental process as claim 9.
Claim 12
The additional element of claim 12 narrows the recited second response to also describe the first sequence of actions completed to achieve the target outcome. The retrospective description of completed actions is itself a mental process — a human recollection of what actions were performed — combined with the underlying mental evaluation of target outcome. The dependent thus adds a further mental-process recitation rather than an additional element that integrates the abstract idea. Claim 12 remains directed to the same mental process as claim 9.
Claim 13
Claim 13 recites the same website-map limitations as claim 8, applied within the framework of claim 9. The analysis is the same as set forth for claim 8: acquisition of an additional input data structure and its inclusion in the prompt are insignificant pre-solution data gathering and pre-solution data formatting/assembly respectively, and do not integrate the abstract idea. Claim 13 remains directed to the same mental process as claim 9.
Claim 14
The additional elements of claim 14 narrow the recited “first sequence of actions” to a singular “first action” across the response, execution, and post-action observation. The narrowing does not change the § 101 character of the underlying recitations: the mental action decision remains a mental process, the execution remains mere instructions to apply the abstract idea via generic computing, and the post-action observation remains insignificant pre-solution data gathering for the next iteration. Claim 14 remains directed to the same mental process as claim 9.
Claim 15
The additional elements of claim 15 recite a further iteration of the agent loop after execution of the first action, comprising a third screenshot capture, a third contextual-tag transformation, a third prompt construction (which itself includes a “first instruction to not repeat the first action”), a third language-model-based action prediction, and execution of a second action, with the second screenshot recharacterized as responsive to execution of both actions. The “instruction to not repeat” is itself a mental corrective directive (a human reasoning step of the form “don’t repeat that prior failed action”). The remaining additions are insignificant pre-solution data gathering and formatting/assembly, well-understood routine conventional HTML parsing, further mental action decisions, and mere instructions to apply via generic computing. Adding a mental corrective directive to the iterative abstract idea does not integrate it into a practical application. Claim 15 remains directed to the same mental process as claim 14.
Claim 16
The additional elements of claim 16 narrow the recited target outcome to submission of information within a text field rendered on the target webpage, and narrow the recited first and second actions to writing text in the text field and clicking an adjacent submit button. The narrowed mental criterion (form-submission target outcome) remains a mental process, and the narrowed actions remain mere instructions to apply the abstract idea via routine browser interactions. Claim 16 remains directed to the same mental process as claim 15.
Claim 17
The additional elements of claim 17 recite a fourth iteration of the agent loop after execution of the second action, with the fourth prompt including descriptions of both prior executed actions and both the first and second instructions to not repeat those prior actions. The cumulative corrective directives remain mental processes (cumulative human reasoning of the form “don’t repeat any prior failed action”). The remaining additions — fourth screenshot capture, contextual-tag transformation, prompt construction, mental action prediction, execution, cumulative observation — are insignificant pre-solution data gathering and formatting/assembly, well-understood routine conventional HTML parsing, further mental decisions, and mere instructions to apply via generic computing. Claim 17 remains directed to the same mental process as claim 15.
Claim 18
The additional elements of claim 18 recite a failure-triggered prompt generation in which, upon a mental observation of failure to execute the first sequence of actions, a new prompt is generated comprising a description of the failed action sequence and a “first instruction to not repeat the first sequence of actions,” from which a replacement action sequence is generated by the language model and executed. The failure observation and the instruction-to-not-repeat are both mental processes (a human observation followed by a mental corrective directive). The replacement action prediction is a further mental decision implemented via generic AI invocation; the prompt construction is insignificant pre-solution data formatting/assembly; the execution is mere instructions to apply via generic computing; and the post-recovery screenshot capture is insignificant pre-solution data gathering. Adding a failure-triggered corrective directive to the abstract idea does not integrate it into a practical application. Claim 18 remains directed to the same mental process as claim 9.
Claim 20
Claim 20 recites the same visual-marking limitations as claim 10, applied within the framework of claim 19. The analysis is the same as set forth for claim 10: image annotation tied to data identifiers and inclusion of the annotated screenshots in the prompts are well-understood routine conventional operations and insignificant pre-solution data formatting/assembly respectively, and do not integrate the abstract idea. Claim 20 remains directed to the same mental process as claim 19.
Therefore, Claims 1-20 do not recite patent eligible subject matter under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 5-7, 9, 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Izzeddin Gur et al. (“A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis”, hereinafter Gur) in view of Hiroki Furuta et al. (“Multimodal Web Navigation with Instruction-Finetuned Foundation Models”, hereinafter Furuta).
With respect to claim 1, Gur teaches a method comprising:
accessing a test statement defining a target outcome associated with contents of a target webpage affiliated with an organization (Gur discloses accessing a natural-language instruction (the test statement) that defines a target outcome on a target webpage hosted by a real-world organization. Gur, section 1 Introduction (“WebAgent, an LLM-driven agent that can complete the tasks on real websites following natural language instructions”); Section 4.1 Real-world Web Navigation (“WebAgent receives natural language instructions (e.g. Can you search for a studio bedroom, 1+ bathroom apartments in oroville, ca for corporate housing on real estate website?, or Could you present the most new thread of Python community filtered by Tutorial tag on social media website?)”), where the real estate website and the social media website each constitute an organization with which the target webpage is affiliated. The user instruction directed to the organization’s target webpage reads on the recited “test statement defining a target outcome associated with contents of a target webpage affiliated with an organization.”).
accessing a set of webpage code defined for the target webpage and corresponding to the first set of target content depicted in the first screenshot (Gur discloses accessing the raw HTML defined for the target webpage, where the HTML by definition corresponds to the content rendered on the target webpage (and therefore corresponds to the set of target content that would be depicted in any screenshot of that target webpage). Gur, section 1 Introduction (describing real-world HTML inputs to the agent that encode the content of the target webpage); Section 3 WebAgent, Figure 2 (depicting “HTML” as an input to HTML-T5); Appendix C, Figure 5 (showing the input HTML document of the target webpage, the elements of which constitute the content that would be visible in a screenshot of that target webpage). The HTML accessed reads on the recited “set of webpage code defined for the target webpage and corresponding to the first set of target content depicted in the first screenshot.”).
transforming the set of webpage code into a first sequence of contextual tags corresponding to the first set of target content depicted in the first screenshot to generate a first textual representation of the target webpage (Gur discloses transforming the raw HTML of the target webpage into a structured snippet representation by HTML-T5, where the snippet is a sequence of contextual tags each carrying a data-ref identifier of an interactive element of the target webpage (and therefore corresponding to a portion of the content that would be depicted in a screenshot of that target webpage), and where the resulting snippet constitutes a textual representation of the target webpage. Gur, section 3 WebAgent (“summarizes long HTML pages into task-relevant snippets”); Section 3.1 HTML-T5 (HTML-to-snippet transformation by HTML-T5); Appendix C, Figure 5 (“Snippet References data-ref=129, data-ref=156,” the snippet being a sequence of data-ref-tagged elements corresponding to interactive features of the target webpage). The resulting snippet reads on the recited “first sequence of contextual tags corresponding to the first set of target content depicted in the first screenshot” and the recited “first textual representation of the target webpage.”).
generating a first prompt comprising the test statement, an address corresponding to the target webpage, [[the first screenshot]] and the first textual representation of the first set of target content (Gur discloses generating a prompt to the language model comprising the user instruction (test statement), an address identifying the target webpage, and the HTML-T5-produced textual representation. Gur, section 3 WebAgent, Figure 2 (depicting prompt construction comprising “Navigation Instruction,” “History,” and “HTML” inputs along with the resulting “Sub-Instruction” and “HTML Snippets” fed to Flan-U-PaLM); Appendix C, Figure 5 (showing the prompt construction with the natural-language instruction, the URL/page-address context of the target webpage embedded in the HTML and snippet, and the snippet textual representation). The recited “test statement” reads on WebAgent’s instruction; the recited “address corresponding to the target webpage” reads on the URL/page-address context retained within Gur’s HTML and snippet inputs; and the recited “first textual representation” reads on Gur’s HTML-T5-produced snippet (as mapped above)).
accessing a language model configured to generate responses to test statements based on visual and textual content extracted from corresponding prompts (Gur discloses accessing language models that generate responses based on textual content extracted from the prompt. Gur, section 3 WebAgent (“WebAgent is composed of interactions between HTML-T5, a domain-expert language model … and Flan-U-PaLM, an instruction-finetuned LLM for grounded program synthesis”); Section 3.2 Grounded Program Synthesis (“Flan-U-PaLM with 540B parameters decodes an executable Python program”). The HTML-T5 + Flan-U-PaLM combination reads on the recited “language model.”).
based on the language model and the first prompt, generating a textual response to the test statement representing occurrence of the target outcome at the target webpage (Gur discloses generating, from the language model based on the prompt, a textual response that ultimately represents whether the target outcome has occurred at the target webpage. Gur, section 4.1 Real-world Web Navigation (describing the per-episode response of the agent and reporting the “success rate” and “score” indicating the proportion of required attributes covered during the episode); Section 4.1 (“The score represents the percentage of required attributes covered during the episode … When the agents could achieve 100 score, that episode would mark as success”). Under the broadest reasonable interpretation, Gur’s response indicating the score/success outcome of the episode reads on the recited “textual response … representing occurrence of the target outcome at the target webpage.”) and
serving the textual response to a user associated with the organization (Gur discloses serving the agent’s response back to the user who issued the instruction to the organization’s website. Gur, section 1 Introduction (describing WebAgent as an autonomous agent that completes user-issued tasks on real websites and returns its outcome to the user); Section 4.1 Real-world Web Navigation (evaluating WebAgent under human supervision in which the agent’s outputs are surfaced to the supervising user, where the user is a user of the organization’s real-world website). The agent’s response returned through this user-facing interface to the user of the organization’s website reads on the recited “serving the textual response to a user associated with the organization.”). Gur is silent to disclose, however in an analogous art, Furuta teaches:
capturing a first screenshot of a first region of the target webpage depicting a first set of target content and rendered within a viewport (Furuta discloses capturing a screenshot of a region of an instance of the target webpage rendered within a viewport. Furuta, section 3 Preliminaries (“the state sₜ ∈ ᵊ is a web page consisting of the raw HTML as a text sequence and a screenshot as an image”); Section 4.1 Image Encoder for Visual Tokens (“To map image observations (screenshots) into visual tokens for T5 encoder … We crop the screenshots of MiniWoB++ to remove the yellow instruction part … and the image size becomes 160 × 160”); Figure 1 (depicting Step-by-step screenshots captured at successive instances of the target webpage). The recited “first screenshot” of the “first region of the target webpage” reads on Furuta’s screenshot capture.).
the first screenshot (Furuta discloses inclusion of the screenshot as part of the multimodal prompt fed to the language model. Furuta, section 4.1 Multimodal Transformer Models (“the encoder considers a set of ViT-encoded visual tokens from a history of screenshots (H = 2 steps) and a set of text tokens from action history, user instruction, and raw HTML”); Figure 2 (depicting prompt construction comprising image observations alongside instruction and HTML tokens fed jointly into the T5 encoder). The screenshot included in Furuta’s multimodal prompt reads on the recited “first screenshot” included in the first prompt.).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the Gur system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. Gur operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, section 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis.
With respect to claim 5, Gur teaches further comprising:
accessing a second set of webpage code defined for the first instance of the target webpage and corresponding to the second set of target content depicted in the second screenshot (Gur discloses accessing the raw HTML of each successive instance of the target webpage as part of its closed-loop iteration. Gur, section 3 WebAgent, Figure 2 (depicting “HTML” as a per-iteration input to HTML-T5); Appendix C, Figure 5 (showing HTML access at each step). The HTML accessed at the pre-action time step reads on the recited “second set of webpage code” corresponding to the “second screenshot” of the pre-action (first) instance of the target webpage).
transforming the second set of webpage code into a second sequence of contextual tags corresponding to the second set of target content to generate a second textual representation of the first instance of the target webpage (Gur discloses transforming the raw HTML at each iteration via HTML-T5 into a structured snippet representation that is a sequence of contextual tags, each carrying a data-ref identifier of an interactive element of the target webpage. Gur, section 3 WebAgent (“summarizes long HTML pages into task-relevant snippets based on sub-instructions”); § 3.1 HTML-T5 (HTML‐to‐snippet transformation by HTML-T5); Appendix C, Figure 5 (“Snippet References data-ref=129, data-ref=156”, the snippet being a sequence of data-ref-tagged elements). The transformed snippet at the pre-action time step reads on the recited “second sequence of contextual tags” and “second textual representation of the first instance of the target webpage.”).
generating a second prompt comprising the test statement, [[the second screenshot,]] and the second textual representation of the second set of target content (Gur discloses constructing, at each iteration of the closed-loop agent, a prompt comprising the user instruction (test statement) and the HTML-T5-produced snippet (textual representation). Gur, section 3 WebAgent, Figure 2 (depicting per-iteration prompt construction with “Navigation Instruction,” “History,” and “HTML” inputs along with the “Sub-Instruction” and “HTML Snippets” outputs of HTML-T5 fed into Flan-U-PaLM); Appendix C, Figure 5 (showing per-iteration prompt construction). The recited “test statement” and “second textual representation” within the second prompt read on Gur’s per-iteration prompt-construction format applied at the pre-action time step).
based on the language model and the second prompt, generating a second response: describing a first action executable within the second instance of the target webpage and predicted to yield the target outcome; and comprising a first set of code corresponding to the first action (Gur discloses generating, from the language model based on the prompt, a response comprising both (i) a natural-language sub-instruction (description of the action to be executed) and (ii) the corresponding executable Python code. Gur, section 3.2 Grounded Program Synthesis (“Flan-U-PaLM with 540B parameters decodes an executable Python program … using Selenium WebDriver, a library for browser automation”; “we treat sub-instructions as comments in the script”); the Python-snippet example in section 3.2 expressly showing a sub-instruction comment (“# Type in walnut creek, ca into search”) paired with the corresponding executable Python code (“driver.find_element(By.CSS_SELECTOR, ‘[data-ref=”175”]’).send_keys(“walnut creek, ca”)”). Gur’s sub-instruction reads on the recited description of “a first action executable within … the target webpage and predicted to yield the target outcome,” and Gur’s corresponding Python code reads on the recited “first set of code corresponding to the first action.”) and
executing the first action within the first instance of the target webpage according to the first set of code (Gur discloses executing the generated Python code against the current instance of the target webpage via the Selenium WebDriver. Gur, section 3.2 Grounded Program Synthesis (Selenium-based browser automation); Section 4.1 Real-world Web Navigation (acting on real websites via the generated program). Execution at the pre-action instance reads on the recited “executing the first action within the first instance of the target webpage according to the first set of code.”).
wherein accessing the set of webpage code defined for the target webpage comprises accessing the set of webpage code defined for the second instance of the target webpage (Gur discloses per-iteration access to the raw HTML of each successive webpage state, including the post-action instance. Gur, section 3 WebAgent, Figure 2 (per-iteration HTML input); Appendix C, Figure 5 (“Previous Snippet IDs” structure, showing the iterative production of new snippets from successive HTML states). Access to the HTML at the post-action time step reads on the recited “set of webpage code defined for the second instance of the target webpage.”) wherein transforming the set of webpage code into the first sequence of contextual tags to generate the first textual representation of the target webpage comprises transforming the set of webpage code into the first sequence of contextual tags to generate the first textual representation of the second instance of the target webpage (Gur discloses per-iteration application of the HTML-T5 contextual-tag transformation, including at the post-action instance. Gur,section 3 WebAgent (closed-loop transformation at each step); Section 3.1 HTML-T5; Appendix C, Figure 5 (per-step snippet production). Application of the transformation at the post-action time step reads on the recited “first sequence of contextual tags” and “first textual representation of the second instance of the target webpage.”).
wherein generating the first prompt comprises generating the first prompt comprising the test statement, [[the first screenshot,]] and first textual representation of the first set of target content, and a description of the first action executed within the first instance of the target webpage (Gur discloses constructing the per-iteration prompt with a persistent History input that retains the prior executed sub-instructions across successive iterations, alongside the current per-iteration textual representation. Gur, § section WebAgent, Figure 2 (depicting “History” as a persistent input across iterations); Appendix C, Figure 5 (“Previous Planning Steps: Go to realestatewebsite.com,” expressly showing the prior executed sub-instructions retained in the per-iteration input). Gur’s persistent action-history input within the post-action prompt reads on the recited “description of the first action executed within the first instance of the target webpage.”).
Gur is silent to disclose, however in an analogous art, Furuta teaches:
capturing a second screenshot of a second region of a first instance of the target webpage depicting a second set of target content and rendered within the viewport (Furuta discloses capturing a screenshot of a region of an instance of the target webpage rendered within the viewport at each time step of the agent’s sequential interaction with the webpage. Furuta, section 3 Preliminaries (“the state sₜ ∈ ᵊ is a web page consisting of the raw HTML as a text sequence and a screenshot as an image”); Section 4.1 Image Encoder for Visual Tokens (screenshot pre-processing); Figure 1 (successive Step screenshots of different instances of the target webpage). The pre-action screenshot reads on the recited “second screenshot” of the “first instance” of the target webpage).
the second screenshot (Furuta discloses inclusion of the per-step screenshot in the multimodal prompt to the language model. Furuta, section 4.1 Multimodal Transformer Models (“the encoder considers a set of ViT-encoded visual tokens from a history of screenshots (H = 2 steps) and a set of text tokens from action history, user instruction, and raw HTML”); Figure 2 (depicting prompt construction with image observations and instruction tokens fed jointly into the T5 encoder). The pre-action screenshot included in Furuta’s multimodal prompt reads on the recited “second screenshot” portion of the second prompt.).
wherein capturing the first screenshot of the first region of the target webpage comprises capturing the first screenshot of the first region of a second instance of the target webpage depicting the first set of target content rendered within the viewport responsive to execution of the first action (Furuta discloses capture of a post-action screenshot of the target webpage at the time step immediately following execution of the agent’s action. Furuta, section 3 Preliminaries (state transition sₜ₊₁ = T(sₜ, aₜ), where the next-step state — including its screenshot — is obtained in response to execution of action aₜ); Figure 1 (depicting post-action Step screenshots responsive to each predicted action); Appendix H, Figure 8 (post-action target-webpage screenshots captured at each successive step of the episode). The post-action screenshot reads on the recited “first screenshot … of a second instance of the target webpage … responsive to execution of the first action.”).
the first screenshot (Furuta discloses inclusion of the per-step screenshot in the multimodal prompt at each successive time step. Furuta, section 4.1 Multimodal Transformer Models (history of screenshots H = 2 included in the encoder input at each time step). The post-action screenshot included in Furuta’s multimodal prompt reads on the recited “first screenshot” portion of the first prompt at the post-action iteration.).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the WebAgent system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. WebAgent operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, § 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis.
With respect to claim 6, Gur teaches wherein accessing the set of webpage code defined for the target webpage comprises accessing the set of webpage code defined for the target webpage and defining a first data size (Gur discloses accessing the raw HTML of the target webpage, which inherently defines a first data size measurable in tokens, and expressly characterizes the raw-HTML input of real-world webpages by its data size. Gur, section 1 Introduction (“the lack of pre-defined action space, much longer HTML observations than simulators … Most LLMs have shorter context lengths compared to the average tokens of HTML in real websites”); Figure 3 (“Statistics of HTML tokens among real websites,” expressly quantifying the data size of the HTML of real-world target webpages, ranging from approximately 7,000 to 14,000 tokens for the depicted websites). Gur’s explicit characterization of the raw-HTML input by its token-measured data size reads on the recited “accessing the set of webpage code … and defining a first data size.”) wherein transforming the set of webpage code into the first sequence of contextual tags to generate the first textual representation comprises transforming the set of webpage code into the first sequence of contextual tags to generate the first textual representation of a second data size less than the first data size (Gur discloses that the HTML-T5 transformation produces a textual representation (snippet) of smaller data size than the raw HTML from which it is derived, as the express purpose of the HTML-T5 architecture. Gur, section 1 Introduction (“it is prohibitively costly to treat such long documents as inputs directly … we introduce WebAgent … (ii) summarizes long HTML pages into task-relevant snippets based on sub-instructions”), expressly disclosing that the HTML-T5-produced snippet has a smaller data size than the raw HTML input; Section 3 WebAgent, Figure 2 (depicting HTML-T5’s role as transforming long HTML into compact snippets fed to Flan-U-PaLM); Appendix C, Figure 5 (showing the HTML document input being reduced to a concise snippet identified by a small number of data-ref values). The HTML-T5-produced contextual-tag snippet, of smaller data size than the raw HTML, reads on the recited “first textual representation of a second data size less than the first data size.”). With respect to claim 7, Gur is silent to disclose, however in an analogous art, Furuta teaches wherein accessing the test statement comprises receiving the test statement from an instance of a test portal executing on a computing device accessed by the user (Furuta discloses a web-agent system in which the user provides a natural-language instruction to the agent through a user-facing interface executing on a computing device. Furuta, section 1 Introduction (“there has been a growing interest in developing autonomous web agents to automate these actions and release humans from repetitive interactions with computer interfaces … our model takes in a command for a web-based task via a natural language instruction”); Section 5.3 Ability of Multi-Step Reasoning (describing the WebShop benchmark interface through which the user provides an instruction such as “I need a long clip-in hair extension which is natural looking, and price lower than 20.00 dollars”); Appendix H, Table 13 (depicting the WebShop interface receiving the user’s instruction at the start of the episode). Under the broadest reasonable interpretation, the recited “test portal executing on a computing device accessed by the user” reads on the user-facing interface (e.g., the WebShop or MiniWoB++ interface) through which the user submits the instruction to the agent) and
wherein serving the textual response to the user comprises serving the textual response to the instance of the test portal accessed by the user (Furuta discloses returning the language model’s textual outputs through the same user-facing interface that received the instruction. Furuta, section 4.1 Multimodal Transformer Models (“decoder predicts actions in text formats”); Appendix H, Table 13 (depicting successive Action/Observation/Termination textual outputs returned through the WebShop interface to the user, culminating in a textual outcome signal such as “Termination: Your score (min 0.0, max 1.0): 1.0”). Under the broadest reasonable interpretation, returning the agent’s textual output through the user-facing interface reads on the recited “serving the textual response to the instance of the test portal.”). It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the Gur system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. Gur operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, section 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis. With respect to claim 9, Gur teaches a method comprising:
accessing a test statement defining a target outcome associated with contents of a target webpage (Gur discloses accessing a natural-language instruction (the test statement) that defines a target outcome on a target webpage. Gur, section 1 Introduction (“WebAgent, an LLM-driven agent that can complete the tasks on real websites following natural language instructions”); Section 4.1 Real-world Web Navigation (“WebAgent receives natural language instructions (e.g. Can you search for a studio bedroom, 1+ bathroom apartments in oroville, ca for corporate housing on real estate website?, or Could you present the most new thread of Python community filtered by Tutorial tag on social media website?)”); Appendix C, Figure 5 (depicting the natural-language instruction “Can you find me a 1 bedroom apartment in San Diego that has a fitness center?” as input to the WebAgent system). The instruction defines the target outcome to be achieved on the target webpage and reads on the recited “test statement defining a target outcome associated with contents of a target webpage.”).
accessing a set of webpage code defined for the target webpage and corresponding to contents of the target webpage (Gur discloses accessing the raw HTML defined for the target webpage. Gur, section 1 Introduction (describing inputs to the agent including the HTML of the real-world target website); Section 3 Gur, Figure 2 (depicting “HTML” as an input to HTML-T5); Appendix C, Figure 5 (showing the input HTML document of the target webpage). The HTML accessed reads on the recited “set of webpage code defined for the target webpage.”).
generating a first textual representation of the target webpage by transforming the set of webpage code into a first sequence of contextual tags corresponding to the first set of target content (Gur discloses transforming the raw HTML of the target webpage into a structured snippet representation by HTML-T5, where the snippet is a sequence of contextual tags each carrying a data-ref identifier corresponding to an interactive element of the target webpage. Gur, section 3 WebAgent (“WebAgent (i) plans sub-instructions per step by decomposing natural language instructions, (ii) summarizes long HTML pages into task-relevant snippets based on sub-instructions …”); section 3.1 HTML-T5 (describing HTML-T5’s transformation of long HTML inputs into structured task-relevant snippets); Appendix C, Figure 5 (“Snippet References data-ref=129, data-ref=156”), showing the produced snippet as a sequence of data-ref-tagged elements corresponding to interactive features of the target webpage. The resulting snippet reads on the recited “first sequence of contextual tags” and “first textual representation.”).
generating a first prompt comprising the test statement, an address corresponding to the target webpage, [[the first screenshot]], and the first textual representation of the first set of target content (Gur discloses generating, as input to the language model, a prompt comprising the user instruction (the test statement), an address identifying the target webpage, and the HTML-T5-produced textual representation. Gur, section 3 WebAgent, Figure 2 (depicting prompt construction comprising “Navigation Instruction,” “History,” “HTML” and “Sub-Instruction,” “HTML Snippets” fed to the language model); Appendix C, Figure 5 (showing the prompt construction with the natural-language instruction, the URL/page address of the target webpage embedded in the HTML document and snippet, and the snippet textual representation as inputs to the system). The recited “test statement” reads on Gur’s instruction; the recited “address corresponding to the target webpage” reads on the URL/page-address context retained within Gur’s HTML and snippet; and the recited “first textual representation of the first set of target content” reads on Gur’s HTML-T5-produced snippet (as mapped in limitation above)).
accessing a language model configured to generate responses to test statements based on visual and textual content extracted from corresponding prompts (Gur discloses accessing language models that generate responses based on textual content from the prompt. Gur, section 3 WebAgent (“WebAgent is composed of interactions between HTML-T5, a domain-expert language model to predict the sub-instruction for the next-step program and to conditionally summarize long HTML documents, and Flan-U-PaLM, an instruction-finetuned LLM for grounded program synthesis”); Section 3.2 Grounded Program Synthesis (“Flan-U-PaLM with 540B parameters decodes an executable Python program”). The HTML-T5 + Flan-U-PaLM combination reads on the recited “language model” configured to generate responses to instructions based on textual content extracted from the prompts).
based on the language model and the first prompt, generating a first response: describing a first sequence of actions executable within the target webpage and predicted to yield the target outcome; and comprising a first set of code corresponding to the first sequence of actions (Gur discloses generating, from the language model based on the prompt, a first response comprising both (i) a natural-language description of the action(s) to be executed (the sub-instruction) and (ii) an executable code corresponding to that action. Gur, section 3.2 Grounded Program Synthesis (“Given a few canonical examples for program generation, next sub-instruction, and extracted HTML snippet from HTML-T5, Flan-U-PaLM with 540B parameters decodes an executable Python program … using Selenium WebDriver, a library for browser automation”); Section 3.2 (“we treat sub-instructions as comments in the script”); the Python-snippet example in section 3.2 expressly showing a sub-instruction comment (“# Type in walnut creek, ca into search”) paired with the corresponding executable Python code (“driver.find_element(By.CSS_SELECTOR, ‘[data-ref=”175”]’).send_keys(“walnut creek, ca”)”). The sub-instruction reads on the recited description of the first sequence of actions “executable within the target webpage and predicted to yield the target outcome,” and the corresponding Python code reads on the recited “first set of code corresponding to the first sequence of actions.”).
executing the first sequence of actions within the target webpage according to the first set of code (Gur discloses executing the generated Python program against the target webpage by means of the Selenium WebDriver. Gur, section 3.2 Grounded Program Synthesis (using “Selenium WebDriver, a library for browser automation,” and decoding “an executable Python program” to be carried out on the website); Section 4.1 Real-world Web Navigation (describing WebAgent acting on real estate and social media websites via the generated program). The execution of the Python code through the browser reads on the recited “executing the first sequence of actions within the target webpage according to the first set of code.”)
generating a second textual representation of the target webpage by transforming the set of webpage code into a second sequence of contextual tags corresponding to the second set of target content (Gur discloses applying the HTML-T5 transformation at each successive iteration of the agent’s closed-loop interaction with the target webpage, producing a new contextual-tag snippet at each step. Gur, section 3 WebAgent (closed-loop plan/summarize/program flow applied iteratively); Appendix C, Figure 5 (showing the iterative “Previous Planning Steps” and “Previous Snippet IDs” structure, where a new snippet is produced at each iteration based on the current HTML state). Application of the HTML-T5 transformation at the post-action time step reads on the recited “second sequence of contextual tags” and “second textual representation of the target webpage.”).
generating a second prompt comprising the test statement, the [[second screenshot]], and the second textual representation (Gur discloses constructing, at each subsequent iteration, a second prompt comprising the user instruction (test statement) and the newly produced HTML-T5 snippet (second textual representation). Gur, section 3 WebAgent, Figure 2 (depicting the iterative prompt-construction loop in which each subsequent step uses the same instruction and the HTML/snippet of the current webpage state); Appendix C, Figure 5 (showing the per-step prompt construction with the navigation instruction and the per-step snippet). The recited “second prompt” and its constituent “test statement” and “second textual representation” read on Gur’s iterative prompt construction.). based on the language model and the second prompt, generating a second response to the test statement representing occurrence of the target outcome (Gur discloses generating, at each iteration of the closed-loop agent, a response from the language model based on the prompt that ultimately culminates in a representation of whether the target outcome has occurred. Gur, section 4.1 Real-world Web Navigation (describing the iterative loop until task completion and reporting the “success rate” and “score” indicating the proportion of required attributes covered by the agent’s actions); Section 4.1 (“The score represents the percentage of required attributes covered during the episode … When the agents could achieve 100 score, that episode would mark as success”). Under the broadest reasonable interpretation, the agent’s textual response at the terminal iteration represents the occurrence of the target outcome at the target webpage.) and serving the second response to a user associated with the target webpage (Gur discloses serving the agent’s response back to the user who issued the instruction. Gur, section 1 Introduction (describing WebAgent as an autonomous agent that completes tasks on real websites following user instructions and returns its outcome to the user); Section 4.1 Real-world Web Navigation (evaluating WebAgent under human supervision in which the agent’s outputs are surfaced to the supervising user). The agent’s response returned through this user-facing interface reads on the recited “serving the second response to a user associated with the target webpage.”). Gur is silent to disclose, however in an analogous art, Furuta teaches:
capturing a first screenshot of a first region of a first instance of the target webpage depicting a first set of target content rendered within a viewport (Furuta discloses capturing a screenshot of a region of an instance of the target webpage rendered within a viewport. Furuta, section 3 Preliminaries (“the state sₜ ∈ ᵊ is a web page consisting of the raw HTML as a text sequence and a screenshot as an image”); Section 4.1 Image Encoder for Visual Tokens (“To map image observations (screenshots) into visual tokens for T5 encoder … We crop the screenshots of MiniWoB++ to remove the yellow instruction part … and the image size becomes 160 × 160”); Figure 1 (depicting Step-by-step screenshots captured at successive instances of the target webpage). The recited “first screenshot” of the “first instance of the target webpage” reads on Furuta’s screenshot captured at the initial time step.).
the first screenshot (Furuta discloses including the screenshot as part of the multimodal prompt fed to the language model. Furuta, section 4.1 Multimodal Transformer Models (“the encoder considers a set of ViT-encoded visual tokens from a history of screenshots (H = 2 steps) and a set of text tokens from action history, user instruction, and raw HTML”); Figure 2 (depicting the prompt construction comprising image observations alongside instruction and HTML tokens fed jointly into the T5 encoder transformer). The screenshot included in Furuta’s multimodal prompt reads on the recited “first screenshot” included in the first prompt.).
capturing a second screenshot of a region of a second instance of the target webpage depicting a second set of target content rendered within the viewport responsive to execution of the first sequence of actions (Furuta discloses capturing a screenshot of the post-action instance of the target webpage as part of the agent’s sequential observation. Furuta, section 3 Preliminaries (next state sₜ₊₁ = T(sₜ, aₜ) obtained from state-transition after execution of action aₜ); Section 4.1 Multimodal Transformer Models (history of screenshots H = 2 expressly including both pre- and post-action observations); Figure 1 (depicting successive Step-2, Step-3, Step-4 screenshots responsive to each predicted action); Appendix H, Figure 8 (showing successive post-action screenshots of MiniWoB++ episodes). The post-action screenshot reads on the recited “second screenshot … responsive to execution of the first sequence of actions.”).
second screenshot (Furuta discloses including the post-action screenshot in the multimodal prompt at the subsequent time step. Furuta, section 4.1 Multimodal Transformer Models (history of screenshots H = 2 included in the encoder input at each subsequent time step). The post-action screenshot included in Furuta’s multimodal prompt reads on the recited “second screenshot” included in the second prompt.).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the WebAgent system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. WebAgent operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, § 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis. With respect to claim 12, Gur teaches wherein generating the second response comprises generating the textual response representing occurrence of the target outcome at the target webpage and describing the first sequence of actions completed to achieve the target outcome (Gur discloses that the agent’s response at each iteration includes a description of the actions completed in prior steps, by carrying forward the history of previous sub-instructions and executed actions. Gur, section 3 WebAgent, Figure 2 (depicting “History” as an input retained across iterations); Appendix C, Figure 5 (showing the “Previous Planning Steps” field expressly listing the prior executed sub-instructions, e.g., “Go to realestatewebsite.com,” as part of the iterative state carried forward into subsequent responses). The agent’s sub-instruction trace and its retained history together read on the recited “describing the first sequence of actions completed to achieve the target outcome.” The “representing occurrence of the target outcome” portion of the limitation is mapped under Gur as set forth in claim 9.).
With respect to claim 14, Gur teaches wherein generating the first response describing the first sequence of actions and comprising the first set of code corresponding to the first sequence of actions comprises generating the first response describing a first action and comprising the first set of code corresponding to the first action (Gur discloses generating a first response that describes a single action and comprises the corresponding executable code. Gur, section 3.2 Grounded Program Synthesis (Python-snippet example showing one sub-instruction “# Type in walnut creek, ca into search” paired with one corresponding Python statement “driver.find_element … .send_keys(”walnut creek, ca”)”); Appendix C, Figure 5 (“Sub-Instruction: Type in San Diego into search,” followed by the single corresponding Python statement). Gur’s per-step output of a single sub-instruction paired with its single corresponding Python statement reads on the recited narrowing to “a first action” and “the first set of code corresponding to the first action.”).
wherein executing the first sequence of actions within the target webpage according to the first set of code comprises executing the first action within the target webpage according to the first set of code (Gur discloses executing the single corresponding Python statement against the target webpage via Selenium. Gur, section 3.2 Grounded Program Synthesis (single-step Selenium execution per sub-instruction). This reads on the recited “executing the first action within the target webpage according to the first set of code.”), Gur is silent to disclose, however in an analogous art, Furuta teaches:
wherein capturing the second screenshot of the region of the second instance of the target webpage responsive to execution of the first sequence of actions comprises capturing the second screenshot of the region of the second instance of the target webpage responsive to execution of the first action (Furuta discloses capturing a post-action screenshot following execution of a single action. Furuta, section 3 Preliminaries (state transition sₜ₊₁ = T(sₜ, aₜ) where aₜ is a single action); Figure 1 (each Step depicts the screenshot of the target webpage captured following execution of a single action of the agent). The post-single-action screenshot reads on the recited narrowing to “responsive to execution of the first action.”).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the WebAgent system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. WebAgent operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, § 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis.
Claims 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Izzeddin Gur et al. (“A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis”, hereinafter Gur) in view of Hiroki Furuta et al. (“Multimodal Web Navigation with Instruction-Finetuned Foundation Models”, hereinafter Furuta) and further in view of Noah Shinn et al. (“Reflexion: Language Agents with Verbal Reinforcement Learning”, hereinafter Shinn). With respect to claim 15, Gur teaches generating a third textual representation of the target webpage by transforming the set of webpage code into a third sequence of contextual tags corresponding to the third set of target content (Gur discloses applying the HTML-T5 contextual-tag transformation at each iteration of the closed-loop agent. Gur, section 3 WebAgent; § 3.1 HTML-T5; Appendix C, Figure 5 (per-step snippet production). Application of the transformation at the third iteration reads on the recited “third sequence of contextual tags” and “third textual representation.”).
generating a third prompt comprising the test statement, [[the third screenshot,]] the third textual representation, a description of the first action executed within the target webpage [[, and a first instruction to not repeat the first action;]] (Gur discloses constructing the per-iteration prompt comprising the user instruction (test statement), the per-iteration HTML-T5 snippet (third textual representation), and the history of previously executed sub-instructions/actions (the description of the first action executed). Gur, § 3 WebAgent, Figure 2 (depicting “History” as a persistent input across iterations); Appendix C, Figure 5 (“Previous Planning Steps: Go to realestatewebsite.com,” showing prior executed sub-instructions retained in the per-iteration input). The recited “test statement,” “third textual representation,” and “description of the first action executed” within the third prompt read on Gur’s iterative prompt-construction format.).
based on the language model and the third prompt, generating a third response: describing a second action executable within the target webpage and predicted to yield the target outcome; and comprising a second set of code corresponding to the second action (Gur discloses generating, at each iteration of the agent, a response comprising a sub-instruction (description of the next action) paired with the corresponding executable Python code, as mapped in limitation (9-g). Gur, section 3.2 Grounded Program Synthesis; Appendix C, Figure 5. Application of Gur’s per-iteration response generation at the third iteration reads on the recited “third response” comprising the description of a “second action” and the “second set of code corresponding to the second action.”) and
executing the second action within the target webpage according to the second set of code (Gur discloses executing the corresponding Python code against the target webpage via Selenium at each iteration of the agent. Gur, section 3.2 Grounded Program Synthesis. Application of Gur’s execution at the third iteration reads on the recited “executing the second action within the target webpage according to the second set of code.”). Gur is silent to disclose, however in an analogous art, Furuta teaches: capturing a third screenshot of a region of a third instance of the target webpage depicting a third set of target content rendered within the viewport responsive to execution of the first action (Furuta discloses capturing a screenshot of each successive post-action instance of the target webpage. Furuta, section 3 Preliminaries (state transition sₜ₊₁ = T(sₜ, aₜ)); § 4.1 Multimodal Transformer Models (history of screenshots across successive time steps); Figure 1 (depicting successive Step-1, Step-2, Step-3, Step-4 screenshots of the target webpage). The screenshot captured at the third successive time step reads on the recited “third screenshot … responsive to execution of the first action.”).
the third screenshot (Furuta discloses including each per-step screenshot in the multimodal prompt to the language model at the corresponding time step. Furuta, section 4.1 Multimodal Transformer Models (per-step inclusion of screenshot in the encoder input). The third-step screenshot included in Furuta’s multimodal prompt reads on the recited “third screenshot” included in the third prompt.)
wherein capturing the second screenshot of the region of the second instance of the target webpage responsive to execution of the first action comprises capturing the second screenshot of the region of the second instance of the target webpage responsive to execution of the first action and the second action (Furuta discloses iterative capture of post-action screenshots across multiple sequential actions of the agent. Furuta, section 3 Preliminaries (successive state transitions following each action); Figure 1 (each Step screenshot captured following the cumulative actions executed up to that step). The cumulative post-action screenshot reads on the recited “responsive to execution of the first action and the second action.”).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the WebAgent system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. WebAgent operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, § 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis.
Gur in view of Furuta is silent to disclose, however in an analogous art, Shinn teaches a first instruction to not repeat the first action (Shinn discloses generating, after a prior action of the agent, a verbal self-reflection text that identifies the prior action as undesirable and that instructs the agent to not repeat that prior action in subsequent steps, and including the resulting self-reflection text in the prompt to the language model at the subsequent step. Shinn, § section Reflexion: reinforcement via verbal reflection (“the agent can verbally state that it should have taken a different action, a′ᵢ, which would have resulted in a′ᵢ₊₁ and a′ᵢ₊₂, and store this experience in its memory. In subsequent trials, the agent can leverage its past experiences to adapt its decision-making approach at time t by choosing action a′ᵢ”); Section 3 Self-reflection (self-reflection model M_sr “generates nuanced and specific feedback … then stored in the agent’s memory (mem)”); Algorithm 1 (“Generate self-reflection sr_t using M_sr; Append sr_t to mem”, where mem is part of the policy πθ conditioning subsequent action generation); Appendix B, Figure 5 (AlfWorld example, in which the Trial #1 failure produces a Reflection text “I should have looked for the desklamp first, then looked for the mug … In the next trial, I will go to desk 1, find the lamp, then look for the mug and examine it with the desklamp,” which is then included as additional context in the Trial #2 prompt and which expressly directs the agent not to repeat the prior incorrect action). Shinn’s self-reflection text appended to memory and supplied as part of the subsequent prompt reads on the recited “first instruction to not repeat the first action.”). It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to further modify the Gur + Furuta combination to incorporate Shinn’s verbal self-reflection mechanism for the per-iteration prompts. Gur and Furuta both operate iterative agent loops in which the agent can fail to achieve the target outcome (Furuta, section 3 Preliminaries: “as a failure if the agent takes a invalid action or reaches a wrong terminal state”), and Gur itself notes that its evaluations involve substantial iteration before task completion (Gur, section 4.1). Shinn expressly addresses this failure-recovery problem in iterative language-agent tasks by appending verbal self-reflection text to memory across trials, where the self-reflection identifies prior failed actions and steers the agent away from them on subsequent trials (Shinn, section 3 Reflexion; Algorithm 1). Shinn is evaluated on AlfWorld and Webshop (Shinn, section 4.1; section 4.2), the same web-navigation and decision-making benchmarks targeted by Furuta and built upon by Gur. The combination is no more than the use of a known technique (Shinn’s verbal self-reflection appended to memory) to improve a similar method (Gur + Furuta’s iterative web-navigation agent) in the same way (adding self-reflection-based feedback to per-iteration prompts), yielding the predictable result of improved agent task success across multiple iterations by avoiding repetition of prior incorrect actions. With respect to claim 16, Gur teaches wherein accessing the test statement defining the target outcome comprises accessing the test statement defining the target outcome associated with submission of information within a text field rendered on the target webpage (Gur discloses receiving natural-language instructions whose target outcome is the submission of information through a text-field-and-button-style search form. Gur, section 4.1 Real-world Web Navigation (“Can you search for a studio bedroom, 1+ bathroom apartments in oroville, ca for corporate housing on real estate website?”, where the target outcome involves submitting search criteria via a search text field); Appendix C, Figure 5 (“Can you find me a 1 bedroom apartment in San Diego that has a fitness center?”, similarly involving text-field submission). The instructions whose target outcome is text-field submission read on the recited narrowing.).
wherein executing the first action within the target webpage comprises writing text within the text field rendered on the target webpage (Gur discloses executing a write-text-in-text-field action via a Selenium send_keys call against the search text field. Gur, section 3.2 Grounded Program Synthesis (Python-snippet example showing “driver.find_element … .send_keys(”walnut creek, ca”)” being executed to type into the search text field). This reads on the recited “writing text within the text field rendered on the target webpage.”) and
wherein executing the second action within the target webpage comprises clicking a submit button rendered adjacent the text field within the target webpage (Gur discloses executing a submit-the-search action via a Selenium submit/click call. Gur, section 3.2 Grounded Program Synthesis (Python-snippet example: “# Submit the search … driver.find_element … .submit()” being executed against the search button rendered on the target webpage). This reads on the recited “clicking a submit button rendered adjacent the text field within the target webpage.”). With respect to claim 17, Gur teaches generating a fourth textual representation of the target webpage by transforming the set of webpage code into a fourth sequence of contextual tags corresponding to the fourth set of target content (Gur discloses application of the HTML-T5 contextual-tag transformation at each iteration. Gur, section 3 WebAgent; Section 3.1 HTML-T5; Appendix C, Figure 5. Application of the transformation at the fourth iteration reads on the recited “fourth sequence of contextual tags” and “fourth textual representation.”).
generating a fourth prompt comprising the test statement, [[the fourth screenshot,]] the fourth textual representation, a description of the first action executed within the target webpage, a description of the second action executed within the target webpage [[, the first instruction to not repeat the first action, and a second instruction to not repeat the second action;]] (Gur discloses construction of the per-iteration prompt with the persistent history of all previously executed sub-instructions/actions across the prior iterations. Gur, section 3 WebAgent, Figure 2 (“History” input accumulating across iterations); Appendix C, Figure 5 (“Previous Planning Steps” field as a running list of prior executed sub-instructions). The recited “description of the first action executed” and “description of the second action executed” within the fourth prompt read on Gur’s accumulating history.).
based on the language model and the fourth prompt, generating a fourth response: describing a third action executable within the target webpage and predicted to yield the target outcome; and comprising a third set of code corresponding to the third action (Gur discloses per-iteration generation of a response comprising the next sub-instruction (description of action) and corresponding executable Python code (set of code). Gur, section 3.2 Grounded Program Synthesis; Appendix C, Figure 5. Application at the fourth iteration reads on the recited “third action” and “third set of code corresponding to the third action.”) and executing the third action within the target webpage according to the third set of code (Gur discloses Selenium-based execution of the per-iteration Python code. Gur, section 3.2 Grounded Program Synthesis. Application at the fourth iteration reads on the recited “executing the third action within the target webpage according to the third set of code.”). Gur is silent to disclose, however in an analogous art, Furuta teaches
capturing a fourth screenshot of a region of a fourth instance of the target webpage depicting a fourth set of target content rendered within the viewport responsive to execution of the second action (Furuta discloses capture of successive post-action screenshots across multiple time steps of the agent. Furuta, section 3 Preliminaries; section 4.1 Multimodal Transformer Models; Figure 1. The fourth-step screenshot reads on the recited “fourth screenshot … responsive to execution of the second action.”).
the fourth screenshot (Furuta discloses inclusion of each per-step screenshot in the multimodal prompt. Furuta, section 4.1 Multimodal Transformer Models. The fourth-step screenshot included in the fourth prompt reads on the recited “fourth screenshot.”).
wherein capturing the second screenshot of the region of the second instance of the target webpage responsive to execution of the first action comprises capturing the second screenshot of the region of the second instance of the target webpage responsive to execution of the first action, the second action, and the third action (Furuta discloses cumulative iterative state-transition across multiple sequential agent actions. Furuta, section 3 Preliminaries (successive state transitions); Figure 1 (each Step screenshot reflecting the cumulative actions executed up to that point). The cumulative post-action screenshot reads on the recited “responsive to execution of the first action, the second action, and the third action.”).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the WebAgent system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. WebAgent operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, § 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis.
Gur in view of Furuta is silent to disclose, however in an analogous art, Shinn teaches the first instruction to not repeat the first action, and a second instruction to not repeat the second action (Shinn discloses generating a verbal self-reflection at each failed prior action and accumulating multiple such self-reflections across successive trials by appending each to the agent’s memory, where the entire accumulated memory is supplied to the language model in subsequent prompts. Shinn, section 3 Reflexion: reinforcement via verbal reflection (“this self-reflective feedback acts as a ‘semantic’ gradient signal by providing the agent with a concrete direction to improve upon, helping it learn from prior mistakes”); Algorithm 1 (“Append sr_t to mem … Increment t”, with mem accumulating self-reflections across successive trials); Section 3 Memory (“we bound mem by a maximum number of stored experiences, Ω (usually set to 1-3)”, demonstrating that multiple self-reflections are concurrently retained as input context). The accumulation of plural self-reflections in memory and their inclusion in the subsequent prompt reads on the recited combination of “the first instruction to not repeat the first action, and a second instruction to not repeat the second action” within the fourth prompt.).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to further modify the Gur + Furuta combination to incorporate Shinn’s verbal self-reflection mechanism for the per-iteration prompts. Gur and Furuta both operate iterative agent loops in which the agent can fail to achieve the target outcome (Furuta, section 3 Preliminaries: “as a failure if the agent takes a invalid action or reaches a wrong terminal state”), and Gur itself notes that its evaluations involve substantial iteration before task completion (Gur, section 4.1). Shinn expressly addresses this failure-recovery problem in iterative language-agent tasks by appending verbal self-reflection text to memory across trials, where the self-reflection identifies prior failed actions and steers the agent away from them on subsequent trials (Shinn, section 3 Reflexion; Algorithm 1). Shinn is evaluated on AlfWorld and Webshop (Shinn, section 4.1; section 4.2), the same web-navigation and decision-making benchmarks targeted by Furuta and built upon by Gur. The combination is no more than the use of a known technique (Shinn’s verbal self-reflection appended to memory) to improve a similar method (Gur + Furuta’s iterative web-navigation agent) in the same way (adding self-reflection-based feedback to per-iteration prompts), yielding the predictable result of improved agent task success across multiple iterations by avoiding repetition of prior incorrect actions. With respect to claim 18, [[in response to failure to execute the first sequence of actions according to the first set of code, generating a third prompt comprising:]] the test statement; the address corresponding to the target webpage; [[the first screenshot;]] the first textual representation; a description of the first sequence of actions and the first set of code; [[and a first instruction to not repeat the first sequence of actions;]] (Gur discloses per-iteration prompt construction comprising the user instruction, the address of the target webpage, the prior HTML-T5 textual representation, and the history of prior sub-instructions and corresponding code. Gur, section 3 WebAgent, Figure 2 (“History” input accumulating across iterations); Appendix C, Figure 5 (Previous Planning Steps and Previous Snippet IDs retained from the prior iteration into the current prompt). The recited “test statement,” “address corresponding to the target webpage,” “first textual representation,” and “description of the first sequence of actions and the first set of code” within the recovery prompt read on Gur’s iterative prompt-construction format.).
based on the language model and the third prompt, generating a second set of code corresponding to a second sequence of actions executable within the target webpage and predicted to yield the target outcome (Gur discloses per-iteration generation of a replacement Python program corresponding to the next sub-instruction. Gur, section 3.2 Grounded Program Synthesis. Application at the recovery iteration reads on the recited “second set of code corresponding to a second sequence of actions.”) and
executing the second sequence of actions within the target webpage according to the third set of code (Gur discloses Selenium-based execution of the per-iteration Python code. Gur, section 3.2 Grounded Program Synthesis. Application at the recovery iteration reads on the recited “executing the second sequence of actions within the target webpage.”). Gur is silent to disclose, however in an analogous art, Furuta teaches wherein capturing the second screenshot of the second region of the target webpage comprises capturing the second screenshot of the second region of the target webpage in response to execution of the second sequence of actions according to the second set of code (Gur in view of Furuta discloses capture of the post-action screenshot following execution of the replacement action sequence. Furuta, section 3 Preliminaries (state transition following each action); Figure 1 (post-action screenshot capture). Capture of the screenshot at the post-recovery time step reads on the recited “responsive to execution of the second sequence of actions.”).
the first screenshot (Furuta discloses inclusion of the prior screenshot in the multimodal prompt at the recovery iteration. Furuta, section 4.1 Multimodal Transformer Models (history of screenshots H = 2 included in the per-iteration encoder input). The screenshot included in the recovery prompt reads on the recited “first screenshot.”). It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to modify the WebAgent system of Gur to incorporate the multimodal observation (raw HTML plus screenshot) of Furuta’s WebGUM. WebAgent operates on HTML alone as input to its language models (Gur, section 3 WebAgent; Figure 2), but Furuta expressly demonstrates that adding screenshot input improves agent grounding and task success on multi-step web tasks (Furuta, § 5.1 Image Modality for Grounded Understanding: “WebGUM leverages visual inputs for multi-step tasks with dynamic page transitions … or the tasks that require global context perception of the page”). Gur is built on the same line of research as Furuta and expressly cites Furuta’s WebGUM as prior multimodal HTML-based agent work (Gur, References [19]). The combination is no more than the use of a known technique (Furuta’s multimodal HTML-plus-screenshot observation) to improve a similar method (Gur’s plan/summarize/program web-navigation agent) in the same way (richer state representation comprising both HTML structure and visual rendering), yielding the predictable result of an agent better grounded in webpage content for both planning and program synthesis.
Gur in view of Furuta is silent to disclose, however in an analogous art, Shinn teaches in response to failure to execute the first sequence of actions, generating a third prompt and a first instruction to not repeat the first sequence of actions (Shinn discloses an explicit failure-triggered self-reflection mechanism in which a binary failure signal is converted into a verbal self-reflection that identifies the failed action sequence and instructs the agent not to repeat it on subsequent attempts, and in which the resulting self-reflection text is supplied as part of the next-trial prompt. Shinn, section 1 Introduction (“Reflexion converts binary or scalar feedback from the environment into verbal feedback in the form of a textual summary, which is then added as additional context for the LLM agent in the next episode”); Section 3 Reflexion: reinforcement via verbal reflection (“The Reflexion process … the Evaluator then produces a score r_0 … to amplify r_0 to a feedback form that can be used for improvement by an LLM, the Self-Reflection model analyzes the set of {τ_0, r_0} to produce a summary sr_0 which is stored in the memory mem”); Algorithm 1 (“while M_e not pass or t < max trials do … Generate self-reflection sr_t using M_sr; Append sr_t to mem”, with the while-not-pass loop being expressly failure-triggered); Appendix B, Figure 5 (AlfWorld example expressly showing the Trial #1 failure producing a Reflection text that instructs the agent not to repeat the prior failed plan in Trial #2). The failure-triggered generation of a self-reflection text and its inclusion in the next-trial prompt reads on the recited “in response to failure to execute the first sequence of actions … generating a third prompt comprising … a first instruction to not repeat the first sequence of actions.”).
It would have been obvious to one of ordinary skill in the art at the time the invention was made, and before the effective filing date of the claimed invention, to further modify the Gur + Furuta combination to incorporate Shinn’s verbal self-reflection mechanism for the per-iteration prompts. Gur and Furuta both operate iterative agent loops in which the agent can fail to achieve the target outcome (Furuta, section 3 Preliminaries: “as a failure if the agent takes a invalid action or reaches a wrong terminal state”), and Gur itself notes that its evaluations involve substantial iteration before task completion (Gur, section 4.1). Shinn expressly addresses this failure-recovery problem in iterative language-agent tasks by appending verbal self-reflection text to memory across trials, where the self-reflection identifies prior failed actions and steers the agent away from them on subsequent trials (Shinn, section 3 Reflexion; Algorithm 1). Shinn is evaluated on AlfWorld and Webshop (Shinn, section 4.1; section 4.2), the same web-navigation and decision-making benchmarks targeted by Furuta and built upon by Gur. The combination is no more than the use of a known technique (Shinn’s verbal self-reflection appended to memory) to improve a similar method (Gur + Furuta’s iterative web-navigation agent) in the same way (adding self-reflection-based feedback to per-iteration prompts), yielding the predictable result of improved agent task success across multiple iterations by avoiding repetition of prior incorrect actions.
With respect to claim 19, this is the independent method claim equivalent to the combination of claims 9, 14, 18, and 12, and recites no new substantive subject matter beyond that combination. Specifically: the base independent-method structure of claim 19 corresponds to claim 9 (accessing the test statement, capturing the first screenshot, accessing the webpage code, transforming the webpage code into a first sequence of contextual tags, generating the first prompt, accessing the language model, generating the first response, executing the action, capturing the second screenshot, generating the second textual representation, generating a subsequent prompt, generating a response representing occurrence, and serving the response); the single-action narrowing (“first action,” “second action”) corresponds to claim 14; the failure-triggered generation of a second prompt comprising a description of the first action and the first set of code and a first instruction to not repeat the first action corresponds to claim 18; and the recitation of the terminal response as both representing occurrence of the target outcome and describing the prior executed action corresponds to claim 12. The mappings of each limitation of claim 19 to Gur, Furuta, and Shinn, and the motivations to combine those references, are accordingly set forth in the rejections of claims 9, 14, 18, and 12 above.
Allowable Subject Matter
Claims 2-4, 8, 10-11, 13 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The correctness of these claims under 35 USC 101 is also needed as indicated above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Brown (US Pub. No. 2022/0374337) addresses technical challenges related to software testing and make substantial technical improvements to improving the computational efficiency and operational reliability of test automation platforms, as well as to the operational reliability of software applications that are tested using the software application platforms. Various embodiments of the present invention provide methods, apparatuses, systems, computing devices, computing entities, and/or the like for performing efficient and techniques for visual software test management using captured test case data entities, annotation-based test case data entities, and dynamic test case data entity cloning. (see abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANIBAL RIVERACRUZ whose telephone number is (571)270-1200. The examiner can normally be reached Monday-Friday 9:30 AM-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hyung S Sough can be reached at 5712726799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANIBAL RIVERACRUZ/Primary Examiner, Art Unit 2192
Prosecution Insights