DETAILED ACTION
This action is in response to the original filing on 07/31/2024. Claims 1-20 are pending and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 2, 10, 12, and 20 are objected to because of the following informalities:
Claims 2 and 12 recite ‘the context’; however, they should recite - - a context - -.
Claims 10 and 20 recite ‘click event’; however, they should recite - - a click event - -.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, claim 1 recites “the at least one instruction based on the scope of the agent”. It is unclear how this limitation is intended to relate to the previously recited “at least one instruction within the information”. For the purposes of examination, this limitation is interpreted as: a second instruction based on the scope of the agent
Claim 1 further recites “wherein the generative response engine is trained based on a safety discriminator to supplement inputs into the generative response engine and an input discriminator to improve mouse input accuracy”. It is unclear whether “to supplement inputs” is intended to modify the generative response engine, the training, or the discriminator. It is further unclear which previous limitation “and an input discriminator” is intended to modify. It is further unclear which previous limitation “to improve mouse input accuracy” is intended to modify. For the purposes of examination, this limitation is interpreted as: wherein the generative response engine is trained based on a safety discriminator and an input discriminator, wherein the generative response engine is trained to supplement inputs into the generative response engine and to improve mouse input accuracy
Regarding claim 11, claim 11 contains substantially similar limitations to those found in claim 1. Consequently, claim 11 is rejected for the same reasons.
Regarding claims 2-10 and 12-20, claims 2-10 and 12-20 are also rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for depending on an indefinite parent claim.
Regarding claims 3 and 13, the claims recite “the scope”. It is unclear how this limitation is intended to relate to the previously recited scope identified by the user and scope of the agent. For the purposes of examination, this limitation is interpreted as: a second scope
Regarding claims 8 and 18, the claims recite “the input”. It is unclear if this limitation is intended to relate to the previously recited first input or at least one type of input. For the purposes of examination, this limitation is interpreted as: a second input
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1 and 11
Step 1: Claims 1 and 11 recite a method and a device; therefore, they are directed to the statutory categories of a method and a machine.
Step 2A Prong 1: The claims recite, inter alia:
obtaining, from the generative response engine, information pertaining to a task being performed in conjunction with the generative response engine; validating at least one instruction within the information to ensure that the at least one instruction corresponds to a scope of the agent; Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of determining information relating to a task and validating the information, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The additional elements of “A method of interacting with a generative response engine based on a scope identified by a user, comprising”, and “A computing device for interacting with a generative response engine based on a scope identified by a user, comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to” amount to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h). The claimed computer components are recited at a high level of generality and are merely invoked as tool to perform the abstract idea. The additional elements of “providing, from an agent to a generative response engine, a first data of an application being monitored by the agent, wherein the first data comprises an initial screenshot prior to a first input and a subsequent screenshot after the first input, and wherein the subsequent screenshot is separated into a plurality of fragments“ amount to insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)). The additional element of “permitting the agent to input the at least one instruction based on the scope of the agent, wherein the generative response engine is trained based on a safety discriminator to supplement inputs into the generative response engine and an input discriminator to improve mouse input accuracy” is merely a post-solution step, a nominal addition to the claim that does not meaningfully limit the claim, and is therefore insignificant extra-solution activity (see MPEP 2106.05(g)). Even when viewed in combination, these additional element do not integrate the abstract idea into a practical application and the claims are thus directed to the abstract idea.
Step 2B: The claims do not contain significantly more than the judicial exception. “A method of interacting with a generative response engine based on a scope identified by a user, comprising”, and “A computing device for interacting with a generative response engine based on a scope identified by a user, comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to” amount to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The additional elements of “providing, from an agent to a generative response engine, a first data of an application being monitored by the agent, wherein the first data comprises an initial screenshot prior to a first input and a subsequent screenshot after the first input, and wherein the subsequent screenshot is separated into a plurality of fragments” amounts to insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”). The additional element of “permitting the agent to input the at least one instruction based on the scope of the agent, wherein the generative response engine is trained based on a safety discriminator to supplement inputs into the generative response engine and an input discriminator to improve mouse input accuracy” is merely a post-solution step, a nominal addition to the claim that does not meaningfully limit the claim, and is therefore insignificant extra-solution activity (see MPEP 2106.05(g)). Nothing in the claims provides significantly more than that abstract idea. As such, the claims are ineligible.
Claims 2-10 and 12-20
Step 1: Claims 2-10 and 12-20 recite methods and devices; therefore, they are directed to the statutory category of a method and a device.
Step 2: claims 2-10 and 12-20 merely narrow the previously recited abstract idea limitations. For the reasons described above with respect to claims 1 and 11, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claims above and do not provide anything more than the mental processes that are practically capable of being performed in the human mind with the assistance of pen and paper.
Claims 2 and 12 further recite the additional element of “wherein the generative response engine is configured to identify features in images at the first resolution”. Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of identifying features in images, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. The additional elements of “wherein the generative response engine stores the context comprising the initial screenshot at a first resolution, previous screenshots at the first resolution, and the subsequent screenshot” amount to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)).
Claims 3 and 13 further recite the additional element of “in response to determining that an asynchronous event in the application has resolved, capturing the screen including the scope as the subsequent screenshot, wherein the generative response engine is configured to separate the subsequent screenshot into the plurality of fragments”. Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of capturing an image in response to a determination that an event has resolved and separating the image into fragments, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. The additional element of “prior to the first input, capturing a screen including the scope as the initial screenshot” amounts to insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)).
Claims 4 and 14 further recite the additional element of “wherein the generative response engine is configured to receive a hint”. These elements amount to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)). The additional elements of “from a different machine learning model to improve a generated input for the agent based on a type of input” amount to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)).
Claims 5 and 15 further recite the additional element of “wherein the type of input corresponds to at least one of a primary click, a secondary click for generating contextual options, or a click that is modified based on a key press event”. These elements amount to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)).
Claims 6 and 16 further recite the additional element of “wherein the generative response engine is configured to generate coordinates for an input based on hints provided from the input discriminator that identifies mouse event misses”. Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating coordinates based on hints, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper.
Claims 7 and 17 further recite the additional element of “wherein the generative response engine is configured to generate data based on the inner monologue and the at least one type of input”. Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of generating data based on an inner monologue and input, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. The additional element of “receiving an instruction from the generative response engine to request supervisor guidance to complete a portion of the task, the instruction including an inner monologue of the generative response engine; and receiving at least one type of input from the supervisor resolving the portion of the task” amounts to insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)).
Claims 8 and 18 further recite the additional elements of “wherein the input comprises a text description of the at least one type of input provided by the supervisor”. These elements amount to insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”).
Claims 9 and 19 further recite the additional elements of “wherein the generative response engine is trained based on a dataset including the portion of the task, the inner monologue, the input, and the text description”. These elements amount to insignificant extra-solution activity in the form of mere data gathering and output (see MPEP § 2106.05(g)), and is a well-understood, routine, conventional activity (see MPEP § 2106.05(d); “Receiving or transmitting data over a network”). These elements amount to merely a post-solution step, a nominal addition to the claim that does not meaningfully limit the claim, and is therefore insignificant extra-solution activity (see MPEP 2106.05(g)).
Claims 10 and 20 further recite the additional element of “wherein the at least one instruction comprises a human input device command, wherein the human input device command comprises at least one of a mouse move and click event and key press events”. These elements amount to no more than generally linking the use of a judicial exception to a particular technological environment or field of use (see MPEP § 2106.05(h)).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Bose et al. (US 20240338248 A1, published 10/10/2024), hereinafter Bose.
Regarding claim 11, Bose teaches the claim comprising:
A computing device for interacting with a generative response engine based on a scope identified by a user, comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to (Bose Figs. 1-12; [0076], Receiving an automation request S100 can function to determine the automation request 110 representing the procedure that the user wants to automate; The automation request 110 can be received from the user; [0122], Alternative embodiments implement the above methods and/or processing modules in non-transitory computer-readable media, storing computer-readable instructions that, when executed by a processing system, cause the processing system to perform the method(s) discussed herein. The set of instructions 35 can be executed by computer-executable components integrated with the computer-readable medium and/or processing system; claim 12: A computing system, comprising: a storage device; and a processing system coupled to the storage device, the storage device storing software instructions for controlling the processor that, when executed, configure the processor to):
provide, from an agent to a generative response engine, a first data of an application being monitored by the agent, wherein the first data comprises an initial screenshot prior to a first input and a subsequent screenshot after the first input, and wherein the subsequent screenshot is separated into a plurality of fragments (Bose Figs. 1-12; [0023], generating a set of instructions 35 (e.g., a code snippet or set thereof) for each task (e.g., using a second machine learning “instruction” model), based on an application representation 130 (e.g., a document object model, a screenshot or video depicting the application, a set of interaction element segments or locations extracted from screenshots or video frames, etc.); [0036], The system can function to facilitate generation of an RPA bot 30 based on an automation request 110. As shown in FIG. 2, in variants, the system can include a computing environment 10 running a set of applications 20, a robotic process automation (RPA) bot 30, a set of instructions 35, a set of inputs (e.g., an automation request 110, a task 120, an application representation 130, including an optional set of element representations, etc.), a set of models, and/or other components; [0075], All or portions of the method can be performed before runtime (e.g., runtime of the RPA bot 30), during, or after runtime. In a first example, the RPA bot 30 can be created and then deployed to control the applications 20 within the computing environment 10. In a second example, the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application. The application representation 130 can be predetermined or be determined in real time (e.g., during runtime). In a specific example, the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0078], The screen recording can be determined from the automation request 110, recorded during runtime (e.g., before the set of instructions 35 for the next task is executed, after execution of a set of instructions 35 for a prior task, etc.). In a first example, the application representation 130 is the screen recording. In a second example, generating the application representation 130 includes segmenting frames (e.g., key frames) from the screen recording using a set of semantic segmentation models and/or detecting target objects using a set of object detectors, and wherein the application representation 130 includes the set of segments, detected objects, and/or the associated attributes (e.g., size, location, number of frames, etc.). In a fifth variant, the application representation 130 is determined based on an existing application representation 130 (e.g., in variants where S200 is performed multiple times). In an example of this variant, the application representation 130 is captured from a screen recording during a first instance of S200 and segmented into a set of interaction element segments for each of a set of distinct frames or keyframes of the screen recording during a second instance of S200; [0081], S200 can be performed one or multiple times in a row, optionally using the prior iteration's output as an input for each iteration. In a first variant, the application representation 130 is captured via screen recording during a first iteration and the screen recording is parsed to generate a hierarchy of segmented elements (e.g., element representations) during a second iteration (e.g., example shown in FIG. 10); [0108], S600 can use the validation model 250 to determine if the task 120 was accomplished based on pre-execution computing environment state and a post-execution computing environment state (e.g., whether the state has changed, whether a classification of the state change is a target class associated with the task, etc.). The computing environment states can be screenshots, application representations 130, and/or any other suitable information about the computing environment 10),
obtain, from the generative response engine, information pertaining to a task being performed in conjunction with the generative response engine (Bose Figs. 1-12; [0055], The task model 210 can function to break down an automation request 110 into a set of tasks 120 for an automation request or workflow (e.g., example shown in FIG. 11B); [0057], The task model 210 can determine the tasks from: visual information (e.g., video, screenshots, etc.), audio (e.g., a user describing the workflow, button tones, etc.), text, and/or any other suitable input (e.g., from the automation request or from another source). In a first variant, the task model 210 is a computer vision-based model which can determine tasks 120 being performed based on information from a set of frames. In this variant, the task model 210 can determine which frames within the set of frames include information about a task 120 being performed. The task model 210 can additionally determine the segment of a frame relevant to a performed task 120 (e.g., a UI element). The task model 210 can use a 3D CNN, TCN, RNN, attention mechanisms, one-stage detector, two-stage detector, GCNs, transformers, GPT, an LLM (llama, bard, etc.), a VLM (e.g., donut), an MLM, and/or another type of machine learning-based method; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0083], Generating a set of tasks based on the automation request S300 can function to determine an intermediary set of tasks 120 that collectively describe the workflow; S300 can be performed responsive to a failure and/or error (e.g., of determining a set of tasks 120, determining a set of instructions 35, executing the set of instructions 35, etc.); [0091], S500 is performed when the RPA bot 30 encounters an error; S500 is performed when the output of a previous iteration of S500 is not validated (e.g., fails in S600); [0098], S500 can generate the instructions using: generative models);
validate at least one instruction within the information to ensure that the at least one instruction corresponds to a scope of the agent (Bose Figs. 1-12; [0064], The optional validation model 250 functions to evaluate the set of instructions 35 against the set of tasks 120 (e.g., example shown in FIG. 6). In a first variant, the validation model 250 validates that the set of instructions 35 accomplishes the respective task 120; [0105], Optionally validating the set of instructions S600 can function to evaluate the set of instructions 35 determined in S500. In variants, the method can validate the set of instructions 35 for each task 120, all sets of instructions 35 for all tasks 120 in the set of tasks (e.g., the RPA bot 30 as a whole), each individual instruction, and/or any other suitable set of instructions 35. S600 can occur before or after S300, S200, S400, S600, S700, and/or at any other suitable time. In a first variant, S600 is performed whenever the set of instructions 35 is determined and/or updated. In a second variant, S600 is performed whenever the application representation 130 is determined and/or updated. In a third variant, S600 is performed when the application 20 and/or computing environment is updated. In a fourth variant, S600 is performed at a predetermined frequency. In a fifth variant, S600 is performed during execution of the RPA bot's instructions 35 (e.g., at every iteration, after every X iterations, responsive to an error event, etc.); [0108], S600 can use the validation model 250 to determine if the task 120 was accomplished based on pre-execution computing environment state and a post-execution computing environment state (e.g., whether the state has changed, whether a classification of the state change is a target class associated with the task, etc.). The computing environment states can be screenshots, application representations 130, and/or any other suitable information about the computing environment 10; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; [0116], remediating the RPA bot can include: executing a remediation option (e.g., remediation instruction, remediation code, remediation modules, etc.) from a set of remediation options; re-executing the instruction set for the failed task (e.g., the last task before the error was thrown); repeating the remediation using another remediation option from the set when the instruction set execution fails (e.g., determined using S600); and adding the remediation option (e.g., the code) to the RPA bot before the instruction set for the task when the instruction set execution succeeds);
and permit the agent to input the at least one instruction based on the scope of the agent, wherein the generative response engine is trained based on a safety discriminator to supplement inputs into the generative response engine and an input discriminator to improve mouse input accuracy (Bose Figs. 1-12; [0047], A task action can describe what to do with the element (e.g., click, drag, input value, delete information, hover, etc.). Examples of action parameters can include: duration, valence (e.g., up, down, left, right, etc.), distance (e.g., in pixels, in frames, in windows, etc.; etc.), location (e.g., coordinates), text values, and/or other parameters; [0057], The task model 210 can use a 3D CNN, TCN, RNN, attention mechanisms, one-stage detector, two-stage detector, GCNs, transformers, GPT, an LLM (llama, bard, etc.), a VLM (e.g., donut), an MLM, and/or another type of machine learning-based method; [0064], The optional validation model 250 functions to evaluate the set of instructions 35 against the set of tasks 120 (e.g., example shown in FIG. 6). In a first variant, the validation model 250 validates that the set of instructions 35 accomplishes the respective task 120; the validation model 250 detects an error message generated by the application 20 and/or computing environment 10; [0067], Examples of remediation options include “scroll up,” “scroll down,” “scroll right,” “scroll left,” “close modal/popup,” “click on button X,” “go back to prior page/frame,” “view history,” “open help bar,” and/or any other suitable remediation option. In an example, remediation options can include amending a set of pixel coordinates within a set of instructions (e.g., when the set of instructions fails due to a change in the UI); [0070], Models can be trained before the method is performed (e.g., before S100, etc.) and/or can be updated while the method is being performed (e.g., responsive to a failure of a deployed RPA bot 30). The models can be trained using information about failure (e.g., an error message), the set of tasks 120 during failure, the set of instructions 35 during failure, and/or any other suitable information. However, the models can be trained at any other suitable time. The models can optionally be validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; or be updated based on any other suitable data. Training data can be manually generated and/or automatically determined. In an example, the models use sets of tasks 120 corresponding to successfully-executed sets of instructions 35 to train the task model; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time)
Regarding claim 1, claim 1 contains substantially similar limitations to those found in claim 11. Consequently, claim 1 is rejected for the same reasons.
Regarding claim 2, Bose teaches all the limitations of claim 1, further comprising:
wherein the generative response engine stores the context comprising the initial screenshot at a first resolution, previous screenshots at the first resolution, and the subsequent screenshot, and wherein the generative response engine is configured to identify features in images at the first resolution (Bose Figs. 1-12; [0045], a screen recording or a video captured by a camera filming a screen; example shown in FIG. 8A; [0047], a task 120 includes text describing an instruction (e.g., “create a blank user profile”). In a second variant, a task 120 includes a task action and a task element (e.g., example shown in FIG. 3). The task element can be a reference (e.g., a descriptor, an index, a title, image segment, etc.) to the interaction element. Examples of task elements include an element representation, a semantic descriptor of the element (e.g., “start button”) and/or element representation, a segment of the automation request 110 material (e.g., a segment of a frame of a video, an image segment of the application interface, etc.), an encoding (e.g., of the element appearance), a semantic segment (e.g., visual segment associated with a semantic label), a bounding box (e.g., associated with a semantic label and coordinate location, determined by an object detector, etc.), and/or any other suitable types of task elements. The task element can be identified and/or determined based on the application representation 130, task 120, current set of instructions 35, and/or any other suitable system component; [0075], The application representation 130 can be predetermined or be determined in real time (e.g., during runtime). In a specific example, the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0078], The screen recording can be determined from the automation request 110, recorded during runtime (e.g., before the set of instructions 35 for the next task is executed, after execution of a set of instructions 35 for a prior task, etc.). In a first example, the application representation 130 is the screen recording. In a second example, generating the application representation 130 includes segmenting frames (e.g., key frames) from the screen recording using a set of semantic segmentation models and/or detecting target objects using a set of object detectors, and wherein the application representation 130 includes the set of segments, detected objects, and/or the associated attributes (e.g., size, location, number of frames, etc.). In a fifth variant, the application representation 130 is determined based on an existing application representation 130 (e.g., in variants where S200 is performed multiple times). In an example of this variant, the application representation 130 is captured from a screen recording during a first instance of S200 and segmented into a set of interaction element segments for each of a set of distinct frames or keyframes of the screen recording during a second instance of S200; [0081], S200 can be performed one or multiple times in a row, optionally using the prior iteration's output as an input for each iteration. In a first variant, the application representation 130 is captured via screen recording during a first iteration and the screen recording is parsed to generate a hierarchy of segmented elements (e.g., element representations) during a second iteration (e.g., example shown in FIG. 10); [0108], S600 can use the validation model 250 to determine if the task 120 was accomplished based on pre-execution computing environment state and a post-execution computing environment state (e.g., whether the state has changed, whether a classification of the state change is a target class associated with the task, etc.). The computing environment states can be screenshots, application representations 130, and/or any other suitable information about the computing environment 10; see also [0023], [0036])
Regarding claim 12, claim 12 contains substantially similar limitations to those found in claim 2. Consequently, claim 12 is rejected for the same reasons.
Regarding claim 3, Bose teaches all the limitations of claim 2, further comprising:
further comprising: prior to the first input, capturing a screen including the scope as the initial screenshot; and in response to determining that an asynchronous event in the application has resolved, capturing the screen including the scope as the subsequent screenshot, wherein the generative response engine is configured to separate the subsequent screenshot into the plurality of fragments (Bose Figs. 1-12; [0045], a screen recording or a video captured by a camera filming a screen; example shown in FIG. 8A; [0047], a task 120 includes text describing an instruction (e.g., “create a blank user profile”). In a second variant, a task 120 includes a task action and a task element (e.g., example shown in FIG. 3). The task element can be a reference (e.g., a descriptor, an index, a title, image segment, etc.) to the interaction element. Examples of task elements include an element representation, a semantic descriptor of the element (e.g., “start button”) and/or element representation, a segment of the automation request 110 material (e.g., a segment of a frame of a video, an image segment of the application interface, etc.), an encoding (e.g., of the element appearance), a semantic segment (e.g., visual segment associated with a semantic label), a bounding box (e.g., associated with a semantic label and coordinate location, determined by an object detector, etc.), and/or any other suitable types of task elements. The task element can be identified and/or determined based on the application representation 130, task 120, current set of instructions 35, and/or any other suitable system component; [0075], The application representation 130 can be predetermined or be determined in real time (e.g., during runtime). In a specific example, the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0078], The screen recording can be determined from the automation request 110, recorded during runtime (e.g., before the set of instructions 35 for the next task is executed, after execution of a set of instructions 35 for a prior task, etc.). In a first example, the application representation 130 is the screen recording. In a second example, generating the application representation 130 includes segmenting frames (e.g., key frames) from the screen recording using a set of semantic segmentation models and/or detecting target objects using a set of object detectors, and wherein the application representation 130 includes the set of segments, detected objects, and/or the associated attributes (e.g., size, location, number of frames, etc.). In a fifth variant, the application representation 130 is determined based on an existing application representation 130 (e.g., in variants where S200 is performed multiple times). In an example of this variant, the application representation 130 is captured from a screen recording during a first instance of S200 and segmented into a set of interaction element segments for each of a set of distinct frames or keyframes of the screen recording during a second instance of S200; [0081], S200 can be performed one or multiple times in a row, optionally using the prior iteration's output as an input for each iteration. In a first variant, the application representation 130 is captured via screen recording during a first iteration and the screen recording is parsed to generate a hierarchy of segmented elements (e.g., element representations) during a second iteration (e.g., example shown in FIG. 10); [0108], S600 can use the validation model 250 to determine if the task 120 was accomplished based on pre-execution computing environment state and a post-execution computing environment state (e.g., whether the state has changed, whether a classification of the state change is a target class associated with the task, etc.). The computing environment states can be screenshots, application representations 130, and/or any other suitable information about the computing environment 10; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; [0116], remediating the RPA bot can include: executing a remediation option (e.g., remediation instruction, remediation code, remediation modules, etc.) from a set of remediation options; re-executing the instruction set for the failed task (e.g., the last task before the error was thrown); repeating the remediation using another remediation option from the set when the instruction set execution fails (e.g., determined using S600); and adding the remediation option (e.g., the code) to the RPA bot before the instruction set for the task when the instruction set execution succeeds; see also [0023], [0036])
Regarding claim 13, claim 13 contains substantially similar limitations to those found in claim 3. Consequently, claim 13 is rejected for the same reasons.
Regarding claim 4, Bose teaches all the limitations of claim 3, further comprising:
wherein the generative response engine is configured to receive a hint from a different machine learning model to improve a generated input for the agent based on a type of input (Bose Figs. 1-12; [0067], the remediation model 260 can be a machine learning model; Examples of remediation options include “scroll up,” “scroll down,” “scroll right,” “scroll left,” “close modal/popup,” “click on button X,” “go back to prior page/frame,” “view history,” “open help bar,” and/or any other suitable remediation option. In an example, remediation options can include amending a set of pixel coordinates within a set of instructions (e.g., when the set of instructions fails due to a change in the UI); [0070], Models can be trained before the method is performed (e.g., before S100, etc.) and/or can be updated while the method is being performed (e.g., responsive to a failure of a deployed RPA bot 30). The models can be trained using information about failure (e.g., an error message), the set of tasks 120 during failure, the set of instructions 35 during failure, and/or any other suitable information. However, the models can be trained at any other suitable time. The models can optionally be validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; or be updated based on any other suitable data. Training data can be manually generated and/or automatically determined. In an example, the models use sets of tasks 120 corresponding to successfully-executed sets of instructions 35 to train the task model; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0091], S500 can be run using the remediation model and optionally an updated application representation to generate additional remediation code to insert into the RPA bot (e.g., set of instruction sets); S500 is performed when the output of a previous iteration of S500 is not validated (e.g., fails in S600); [0098], S500 can generate the instructions using: generative models; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; see also [0023], [0036])
Regarding claim 14, claim 14 contains substantially similar limitations to those found in claim 4. Consequently, claim 14 is rejected for the same reasons.
Regarding claim 5, Bose teaches all the limitations of claim 4, further comprising:
wherein the type of input corresponds to at least one of a primary click, a secondary click for generating contextual options, or a click that is modified based on a key press event (Bose Figs. 1-12; [0047], A task action can describe what to do with the element (e.g., click, drag, input value, delete information, hover, etc.). Examples of action parameters can include: duration, valence (e.g., up, down, left, right, etc.), distance (e.g., in pixels, in frames, in windows, etc.; etc.), location (e.g., coordinates), text values, and/or other parameters; [0067], Examples of remediation options include “scroll up,” “scroll down,” “scroll right,” “scroll left,” “close modal/popup,” “click on button X,” “go back to prior page/frame,” “view history,” “open help bar,” and/or any other suitable remediation option. In an example, remediation options can include amending a set of pixel coordinates within a set of instructions (e.g., when the set of instructions fails due to a change in the UI); [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; see also [0023], [0036], [0070], [0091], [0098])
Regarding claim 15, claim 15 contains substantially similar limitations to those found in claim 5. Consequently, claim 15 is rejected for the same reasons.
Regarding claim 6, Bose teaches all the limitations of claim 4, further comprising:
wherein the generative response engine is configured to generate coordinates for an input based on hints provided from the input discriminator that identifies mouse event misses (Bose Figs. 1-12; [0042], the set of instructions 35 includes coordinates (e.g., mouse/tap location commands); [0047], A task action can describe what to do with the element (e.g., click, drag, input value, delete information, hover, etc.). Examples of action parameters can include: duration, valence (e.g., up, down, left, right, etc.), distance (e.g., in pixels, in frames, in windows, etc.; etc.), location (e.g., coordinates), text values, and/or other parameters; [0067], the remediation model 260 can be a machine learning model; Examples of remediation options include “scroll up,” “scroll down,” “scroll right,” “scroll left,” “close modal/popup,” “click on button X,” “go back to prior page/frame,” “view history,” “open help bar,” and/or any other suitable remediation option. In an example, remediation options can include amending a set of pixel coordinates within a set of instructions (e.g., when the set of instructions fails due to a change in the UI); [0070], Models can be trained before the method is performed (e.g., before S100, etc.) and/or can be updated while the method is being performed (e.g., responsive to a failure of a deployed RPA bot 30). The models can be trained using information about failure (e.g., an error message), the set of tasks 120 during failure, the set of instructions 35 during failure, and/or any other suitable information. However, the models can be trained at any other suitable time. The models can optionally be validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; or be updated based on any other suitable data. Training data can be manually generated and/or automatically determined. In an example, the models use sets of tasks 120 corresponding to successfully-executed sets of instructions 35 to train the task model; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0091], S500 can be run using the remediation model and optionally an updated application representation to generate additional remediation code to insert into the RPA bot (e.g., set of instruction sets); S500 is performed when the output of a previous iteration of S500 is not validated (e.g., fails in S600); [0098], S500 can generate the instructions using: generative models; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; see also [0023], [0036])
Regarding claim 16, claim 16 contains substantially similar limitations to those found in claim 6. Consequently, claim 16 is rejected for the same reasons.
Regarding claim 7, Bose teaches all the limitations of claim 1, further comprising:
receiving an instruction from the generative response engine to request supervisor guidance to complete a portion of the task, the instruction including an inner monologue of the generative response engine; and receiving at least one type of input from the supervisor resolving the portion of the task, wherein the generative response engine is configured to generate data based on the inner monologue and the at least one type of input (Bose Figs. 1-12; [0055], The task model 210 can function to break down an automation request 110 into a set of tasks 120 for an automation request or workflow (e.g., example shown in FIG. 11B); [0087], the tasks 120 are displayed to the user, who edits the tasks 120; [0070], Models can be trained before the method is performed (e.g., before S100, etc.) and/or can be updated while the method is being performed (e.g., responsive to a failure of a deployed RPA bot 30). The models can be trained using information about failure (e.g., an error message), the set of tasks 120 during failure, the set of instructions 35 during failure, and/or any other suitable information. However, the models can be trained at any other suitable time. The models can optionally be validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; or be updated based on any other suitable data. Training data can be manually generated and/or automatically determined. In an example, the models use sets of tasks 120 corresponding to successfully-executed sets of instructions 35 to train the task model; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0091], S500 can be run using the remediation model and optionally an updated application representation to generate additional remediation code to insert into the RPA bot (e.g., set of instruction sets); S500 is performed when the output of a previous iteration of S500 is not validated (e.g., fails in S600); [0098], S500 can generate the instructions using: generative models; [0112], when the set of instructions 35 are determined to be invalid (e.g., incorrect, don't compile, don't accomplish the desired task 120, generate an error, etc.), then: the instruction set or task can be evaluated via an affordance function, any step between S100 and S500 can be re-run, S600 (instruction set validation) can be performed, S500 (instruction set remediation) can be performed, the set of instructions 35 can be manually edited by a user, the set of tasks 120 can be automatically edited and/or manually edited by a user, and/or any other instruction amendment step can be performed; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; see also [0023], [0036], [0067])
Regarding claim 17, claim 17 contains substantially similar limitations to those found in claim 7. Consequently, claim 17 is rejected for the same reasons.
Regarding claim 8, Bose teaches all the limitations of claim 7, further comprising:
wherein the input comprises a text description of the at least one type of input provided by the supervisor (Bose Figs. 1-12; [0055], The task model 210 can function to break down an automation request 110 into a set of tasks 120 for an automation request or workflow (e.g., example shown in FIG. 11B); [0087], the tasks 120 are displayed to the user, who edits the tasks 120; [0070], Models can be trained before the method is performed (e.g., before S100, etc.) and/or can be updated while the method is being performed (e.g., responsive to a failure of a deployed RPA bot 30). The models can be trained using information about failure (e.g., an error message), the set of tasks 120 during failure, the set of instructions 35 during failure, and/or any other suitable information. However, the models can be trained at any other suitable time. The models can optionally be validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; or be updated based on any other suitable data. Training data can be manually generated and/or automatically determined. In an example, the models use sets of tasks 120 corresponding to successfully-executed sets of instructions 35 to train the task model; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0091], S500 can be run using the remediation model and optionally an updated application representation to generate additional remediation code to insert into the RPA bot (e.g., set of instruction sets); S500 is performed when the output of a previous iteration of S500 is not validated (e.g., fails in S600); [0098], S500 can generate the instructions using: generative models; [0112], when the set of instructions 35 are determined to be invalid (e.g., incorrect, don't compile, don't accomplish the desired task 120, generate an error, etc.), then: the instruction set or task can be evaluated via an affordance function, any step between S100 and S500 can be re-run, S600 (instruction set validation) can be performed, S500 (instruction set remediation) can be performed, the set of instructions 35 can be manually edited by a user, the set of tasks 120 can be automatically edited and/or manually edited by a user, and/or any other instruction amendment step can be performed; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; see also [0023], [0036], [0067],)
Regarding claim 18, claim 18 contains substantially similar limitations to those found in claim 8. Consequently, claim 18 is rejected for the same reasons.
Regarding claim 9, Bose teaches all the limitations of claim 8, further comprising:
wherein the generative response engine is trained based on a dataset including the portion of the task, the inner monologue, the input, and the text description (Bose Figs. 1-12; [0055], The task model 210 can function to break down an automation request 110 into a set of tasks 120 for an automation request or workflow (e.g., example shown in FIG. 11B); [0070], Models can be trained before the method is performed (e.g., before S100, etc.) and/or can be updated while the method is being performed (e.g., responsive to a failure of a deployed RPA bot 30). The models can be trained using information about failure (e.g., an error message), the set of tasks 120 during failure, the set of instructions 35 during failure, and/or any other suitable information. However, the models can be trained at any other suitable time. The models can optionally be validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; or be updated based on any other suitable data. Training data can be manually generated and/or automatically determined. In an example, the models use sets of tasks 120 corresponding to successfully-executed sets of instructions 35 to train the task model; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0087], the tasks 120 are displayed to the user, who edits the tasks 120; [0091], S500 can be run using the remediation model and optionally an updated application representation to generate additional remediation code to insert into the RPA bot (e.g., set of instruction sets); S500 is performed when the output of a previous iteration of S500 is not validated (e.g., fails in S600); [0098], S500 can generate the instructions using: generative models; [0112], when the set of instructions 35 are determined to be invalid (e.g., incorrect, don't compile, don't accomplish the desired task 120, generate an error, etc.), then: the instruction set or task can be evaluated via an affordance function, any step between S100 and S500 can be re-run, S600 (instruction set validation) can be performed, S500 (instruction set remediation) can be performed, the set of instructions 35 can be manually edited by a user, the set of tasks 120 can be automatically edited and/or manually edited by a user, and/or any other instruction amendment step can be performed; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; see also [0023], [0036], [0067],)
Regarding claim 19, claim 19 contains substantially similar limitations to those found in claim 9. Consequently, claim 19 is rejected for the same reasons.
Regarding claim 10, Bose teaches all the limitations of claim 1, further comprising:
wherein the at least one instruction comprises a human input device command, wherein the human input device command comprises at least one of a mouse move and click event and key press events (Bose Figs. 1-12; [0042], the set of instructions 35 includes coordinates (e.g., mouse/tap location commands); [0047], A task action can describe what to do with the element (e.g., click, drag, input value, delete information, hover, etc.). Examples of action parameters can include: duration, valence (e.g., up, down, left, right, etc.), distance (e.g., in pixels, in frames, in windows, etc.; etc.), location (e.g., coordinates), text values, and/or other parameters; [0067], the remediation model 260 can be a machine learning model; Examples of remediation options include “scroll up,” “scroll down,” “scroll right,” “scroll left,” “close modal/popup,” “click on button X,” “go back to prior page/frame,” “view history,” “open help bar,” and/or any other suitable remediation option. In an example, remediation options can include amending a set of pixel coordinates within a set of instructions (e.g., when the set of instructions fails due to a change in the UI); [0070], Models can be trained before the method is performed (e.g., before S100, etc.) and/or can be updated while the method is being performed (e.g., responsive to a failure of a deployed RPA bot 30). The models can be trained using information about failure (e.g., an error message), the set of tasks 120 during failure, the set of instructions 35 during failure, and/or any other suitable information. However, the models can be trained at any other suitable time. The models can optionally be validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; or be updated based on any other suitable data. Training data can be manually generated and/or automatically determined. In an example, the models use sets of tasks 120 corresponding to successfully-executed sets of instructions 35 to train the task model; [0075], the RPA can execute a portion of its instructions 35 (e.g., a portion of the set of instructions 35 generated during a prior instance of the method) and can iteratively perform S500 when a failure condition is met (e.g., a task is failed, an instruction fails, etc.) until a success condition is met (e.g., a task is accomplished, an instruction succeeds, a target application state is achieved, etc.). In a third example, the system can generate an RPA bot 30 on-the-fly during runtime using the set of tasks 120 and an application representation 130 of the application; the system can, for each task 120, determine an application representation 130 (e.g., taking a screenshot and segmenting out interaction elements, etc.) and generate a set of instructions 35 for the upcoming task 120 in the set of tasks 120 using the application representation 130; [0091], S500 can be run using the remediation model and optionally an updated application representation to generate additional remediation code to insert into the RPA bot (e.g., set of instruction sets); S500 is performed when the output of a previous iteration of S500 is not validated (e.g., fails in S600); [0098], S500 can generate the instructions using: generative models; [0114], The RPA bot can be remediated: when a runtime error occurs, when an instruction set is invalid, and/or when any other suitable condition is met. The RPA bot can be remediated: during runtime (e.g., in real-time, during S700, etc.), after runtime (e.g., after S700), before runtime, and/or at any other time; see also [0023], [0036])
Regarding claim 20, claim 20 contains substantially similar limitations to those found in claim 10. Consequently, claim 20 is rejected for the same reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Furuta (US 20250272350 A1) see Figs. 1-12 and [0025-0029], [0036-0044], [0052].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN T REPSHER III whose telephone number is (571)272-7487. The examiner can normally be reached Monday - Friday, 8AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN T REPSHER III/ Primary Examiner, Art Unit 2143
Prosecution Insights