Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under, including the fee set forth in 37 CFR1.17(e), was filed in this application after final rejection. Since this application is eligiblefor continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e)has been timely paid, the finality of the previous Office action has been withdrawnpursuant to 37 CFR 1.114. Applicant's submission filed on 12/23/2025 has been entered.
Status of the Claims
Claims 1, 3-7, 9-11, 13-17, and 19-22 are pending.
Response to Applicant’s Arguments
In response to “Particularly, Contractor and/or Joshi are devoid of "initiating the semantic automation builder in response to a selection of a menu item that triggers the semantic automation builder;""receiving one or more user inputs corresponding to a target application, a written automation task, or one or more steps to identify one or more actions of the robotic process automation;""mapping each action of the one or more actions to an activity to generate one or more mapped activities including generating am automation sequence using written natural language;" and/or "transforming the one or more actions into the one or more semantic automations that automatically navigate between a first website, database, file system, or application to one or more different websites, databases, file systems, and applications based on the one or more mapped activities and the automation sequence," as recited in claim 1”.
In view of such combination of limitations set forth in claims 1 and 11, rejection under Contractor and Joshi has been withdrawn. Upon further search and consideration, please see a new citation of references set forth in details below.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
(a) NOVELTY; PRIOR ART.—A person shall be entitled to a patent unless—
(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention; or
(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
PNG
media_image1.png
18
19
media_image1.png
Greyscale
(b) EXCEPTIONS.—
(1) DISCLOSURES MADE 1 YEAR OR LESS BEFORE THE EFFECTIVE FILING DATE OF THE CLAIMED INVENTION.—A disclosure made 1 year or less before the effective filing date of a claimed invention shall not be prior art to the claimed invention under subsection (a)(1) if—
(A) the disclosure was made by the inventor or joint inventor or by another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor; or
(B) the subject matter disclosed had, before such disclosure, been publicly disclosed by the inventor or a joint inventor or another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor.
(2) DISCLOSURES APPEARING IN APPLICATIONS AND PATENTS.—A disclosure shall not be prior art to a claimed invention under subsection (a)(2) if—
(A) the subject matter disclosed was obtained directly or indirectly from the inventor or a joint inventor;
(B) the subject matter disclosed had, before such subject matter was effectively filed under subsection (a)(2), been publicly disclosed by the inventor or a joint inventor or another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor; or
(C) the subject matter disclosed and the claimed invention, not later than the effective filing date of the claimed invention, were owned by the same person or subject to an obligation of assignment to the same person.
Claims 1, 3-6, 9-11, 13-16, and 19-21 are rejected under 35 USC 102(a)(2) as being anticipated by Srivastava et al. (US 11966724 B1).
Regarding Claims 1 and 11, Srivastava discloses a system (Col 3, Rows 45-46, computer desktop with a graphical user interface or GUI) comprising:
one or more memories storing software of a semantic automation builder that generates one or more semantic automations of a robotic process automation (Col 30, Rows 17-21, computer program product encoded on non-transitory machine readable storage media; per Col 3, Rows 45-61, an assisted automation computer program for building assisted automations from recordings (Col 4, Rows 38-40, a recording can be a specification of how automation is expected to act in a specific scenario) of the world (Col 3, Rows 45-46, the word is a computer desktop with a GUI) and actions); and
at least one processor executing the software to cause the system (Col 3, Rows 45-61, computer desktop executing assisted automation computer program that can prompt user with on-screen instructions to perform desktop actions) to:
initiating the semantic automation builder in response to a selection of a menu item that triggers the semantic automation builder (Col 16, Rows 40-54 and Fig. 17, in drive-assist mode, Warpdrive (per Col 15, Rows 60-65, Warpdrive is an operational system that utilizes intent framework to build a desktop automation system) guides human with an on-screen descriptive prompt such that when the human performs the action, Warpdrive auto-detects that action and auto-advance to subsequent step; see e.g., Col 17, Rows 8-13, on screen pointer 1703 is a visible pointer drawing attention to a menu item for the user to interact with);
receive one or more user inputs corresponding to a target application, a written automation task, or one or more steps to identify one or more actions of the robotic process automation (Col 16, Rows 55-62, instruction panel 1702 direct the human to add descriptive annotations such as rich text descriptions; Col 19, Rows 21-24 and Rows 55-60, using Warpdrive for Robotic Process Automation, the Warpdrive includes an intent programming language designed for specifying UI interactions called Warpian target language that is easy to read and write by a human; e.g., high level intent statements like “call invoke on button Close”; per Col 21, Row 55 – Col 22, Row 5, a user could manually write high level programs that interact with their desktop computer in a declarative way where a Compiler performs semantic inference of observed mouse / keyboard events to emit Warpian program, the Warpian program captures high level intent of each action performed by the desktop user, and Warpdrive automatically infer desktop actions from human actions);
map each action of the one or more actions to an activity to generate one or more mapped activities including generating an automation sequence using written natural language (Col 20, Rows 1-14, compile desktop UI actions to high level intent statements in Warpian; in view of Col 5, Rows 20-32, warping interpreter builds an action that adjusts for differences between expected runtime world state and current runtime world state; e.g., example of high level command “navigate to salesforce.com in the open web browser”);
transform the one or more actions into the one or more semantic automations that automatically navigate between a first website, database, file system, or application to one or more different websites, databases, file systems, and applications based on the one or more mapped activities and the automation sequence (Col 5, Rows 24-36, for high level command “navigate to salesforce.com in the open web browser”, the runtime warping interpreter locates the web browser (i.e., first application) on the desktop, identifies the address bar within the web browser, send typing commands to update the address bar to reflect “salesforce.com” (i.e., different website), and invoke the go to url operation; see also Col 4, Rows 49-50, high level command “move a file from explorer to chrome”; i.e., automate the moving of a file from a first application to a different application); and
provide the one or more semantic automations in a user interface of the semantic automation builder to enable editing of the one or more actions (Col 4, Rows 50-52, automatically execute and/or present high level commands to user; Col 16, Rows 63-66, instruction panel 1702 in Fig. 17 introduces controls to edit the prompt to add descriptions; i.e., prompting the user to edit high level commands).
Regarding Claims 3 and 13, Srivastava discloses wherein the semantic automation builder enables adding one or more additional steps to the written automation task (Col 16, Rows 63-66, instruction panel 1702 in Fig. 17 introduces controls to edit the prompt to add descriptions; i.e., prompting the user to edit high level commands by adding descriptions).
Regarding Claims 4 and 14, Srivastava discloses wherein the semantic automation builder comprises a generator trained to determine the one or more actions from the written automation task (Col 10, Rows 28-30, self-learning to synthesize an iteratively improving automation by recording its own executions on new worlds).
Regarding Claims 5 and 15, Srivastava discloses wherein the semantic automation builder comprises a generator trained to process speech as the written automation task (Col 16, Rows 40-62, when directing human to perform the task of adding audio narration (Col 16, Row 61), and the human performs the action, the Warpdrive interpreter can autodetect that action and execute the actions on the human’s behalf resulting in a completely automated execution).
Regarding Claims 9 and 19, Srivastava discloses wherein the robotic process automation is exported to a designer for execution (Col 16, Rows 10-12 and 19-21, using Warpdrive to implement Robotic Process Automation to build desktop automations in the timespan of hours or days with no programming required by algorithmically turning recordings into code).
Regarding Claims 10 and 20, Srivastava discloses wherein the semantic automation builder trains one or more artificial intelligence and machine learning models using stored actions, edits of those actions, or workflows in a semantic workflow database (Col 4, Rows 15-31, Col 21, Rows 43-53, self-learning model observing human deviation from prompted script and using the recording to update assisted automation, making it more robust / intelligent) and utilizes the one or more artificial intelligence and machine learning models to determine the actions to activities mappings (Col 21, Rows 49-51, overtime, there should remain no unseen scenarios that the automation cannot handle).
Regarding Claim 21, Srivastava discloses wherein the transforming the one or more actions into the one or more semantic automations includes utilizing an automation sequence for the one or more actions and mapping steps of the automation sequence to the automatic navigation between the first website, database, file system, or application to the one or more different websites, databases, file systems, and applications based on the one or more mapped activities (Col 5, Rows 24-36, for high level command “navigate to salesforce.com in the open web browser”, the runtime warping interpreter locates the web browser (i.e., first application) on the desktop, identifies the address bar within the web browser, send typing commands to update the address bar to reflect “salesforce.com” (i.e., different website), and invoke the go to url operation; see also Col 4, Rows 49-50, high level command “move a file from explorer to chrome”; i.e., automate the moving of a file from a first application to a different application).
Claim Rejections - 35 USC § 103
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 103 that form the basis for the rejections under this section made in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6 and 16 are rejected under 35 USC 103(a) as being unpatentable over Srivastava et al. (US 11966724 B1) in view of Singh et al. (US 2021/0110301 A1).
Regarding Claims 6 and 16, Srivastava does not disclose wherein computer vision software captures user interface elements on an interface of the target application to correspond to the one or more actions.
Singh teaches computer vision software captures user interface elements on an interface of the target application to correspond to the one or more actions (¶21, if a monitoring ML model determines that a user is frequently going into QuickBooks and correcting a certain field after a robot processes invoices, the ML model may retrieve a different computer vision model and modify text recognition activity to use this model instead).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to implement computer vision software to capture user interface elements on an interface of the target application to correspond to the one or more actions to monitor behavior of an executing robot and automatically determine that certain activities should be included in the workflow, replaced with other activities, or be modified (Singh, ¶21).
Claims 7 and 17 are rejected under 35 USC 103(a) as being unpatentable over Srivastava et al. (US 11966724 B1) in view of Joshi et al. (US 11856038 B2).
Regarding Claims 7 and 17, Srivastava discloses wherein the semantic automation builder comprises a generator that maps the one or more actions (Col 5, Rows 20-32, warping interpreter interpreting “navigate to salesforce.com in the open web browser” to locate web browser, identifying address bar within the web browser, update address bar “salesforce.com”, and invoke the “go to url” operation).
Srivastava does not disclose using an autoregressive language model to map the one or more actions.
Joshi teaches a script-generating, conditional conversational agent “SGC-CA” (Col 5, Rows 55-58) comprising agent activity module using robotic process automation (RPA) algorithm to execute scripts to simulate live-agent actions in a synchronous conferencing system (Col 6, Rows 20-36, agent activity module (e.g., a robotic process automation (RPA) algorithm configured to execute computer instructions of script), the SGC-CA configured to understand meaning of a user query (Col 6, Rows 12-16) by using an autoregressive language model to map one or more actions corresponding to user input (Col 6, Rows 37-56, using pre-trained transformer model like GPT (Generative Pre-trained transformer including GPT1, GPT2, and the like) to perform above-described operations).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to use an autoregressive language model to map the one or more actions in order to understand the meaning of user inputs (Joshi, Col 6, Rows 15-16; compare Srivastava, Col 5, Rows 20-26, using GPT based language model to interpret “navigate to salesforce.com in the open web browser”).
Claim 22 is rejected under 35 USC 103(a) as being unpatentable over Srivastava et al. (US 11966724 B1) in view of Patil (US 2020/0211031 A1).
Regarding Claim 22, Srivastava discloses wherein the automation sequence comprises mapping each action of the one or more actions to an activity to generate one or more mapped activities including generating an automation sequence using written natural language (Col 19, Rows 55-60, Warpdrive includes an intent programming language designed for specifying UI interactions called Warpian that is easy to read and write by a human; in view of Col 16, Rows 40-43 and Rows 55-59, Warpdrive can present user with on-screen prompts 1702 where instruction panel 1702 includes information to direct the human to perform the task based on machine generated, human readable prompt) and comprising click and type activities in a single interface (Col 3, Rows 45-50, assisted automation computer program prompt the user with on-screen instructions to perform clicking or typing on particular elements on the screen).
Srivastava does not disclose the click and type activities in a single interface for filling a form.
Patil teaches conversion of language to text for form filing combined with robotic process automation to auto-fill forms (¶80).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to implement click and type activities in a single interface for filling a form for scenarios like filling out police reports or credit reports, making or checking airline bookings or hotel bookings (Patil, ¶80; compare Srivastava, Col 15, Rows 65-67, automating tasks like operations, human resources, finance (i.e., filling out credit reports), legal (i.e., filling out police reports), procurement and supply chain, customer success and support (making airline bookings or hotel bookings), marketing and sales, and accounting).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to examiner Richard Z. Zhu whose telephone number is 571-270-1587 or examiner’s supervisor Hai Phan whose telephone number is 571-272-6338. Examiner Richard Zhu can normally be reached on M-Th, 0730:1700.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICHARD Z ZHU/Primary Examiner, Art Unit 2654 02/04/2026