DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 6-14, 16-19, 21-24 are rejected under 35 U.S.C. 102(a)(1) ss being anticipated by Marin et al (US Pub 2024/0142917).
Regarding claim 1, Marin discloses a non-transitory computer-readable medium storing a computer program (sec 0088, 0092) for providing automatic annotations for robotic process automation (RPA) workflows (sec 0127; builder 903 generates a workflows for the RPA 905 using ML from natural language described task), the computer program configured to cause at least one processor to:
provide code for an RPA (sec 0127; builder 903 generates a workflows for the RPA 905 using a code such as an ML from natural language described task) or a process definition document (PPD) to a cognitive artificial intelligence (AI) layer [figs. 3, 9; sec 0078, 0127, 0128-0136; Listener 330 communicates to and from with database 355; database 355 is part of layer 336 and layer 366 comprises an AI layer 360; listener 330 provides a natural language description of a processor also known as a process definition document (PDD) or recorder of user actions, a task capture, a process mining, etc to the layer 336 having the AI layer 360], the cognitive Al layer configured to analyze and automatically generate annotations describing activities for the RPA workflow (AI generating and analyzing or editing annotations for an RPA workflow for the robot; sec 0035, 0036, 0038) or an RPA workflow pertaining to a process described in the PDD (sec 0035, 0037, 0038);
process the code for the RPA workflow or the PDD (figs. 3, 9; restructuring, reconfiguring, etc; sec 0078, 0127, 0128-0136), by the cognitive AI layer; and
generate annotations for the RPA workflow or RPA workflow code comprising the annotations at output (sec 0127; builder 903 generates a workflows for the RPA 905 using ML from natural language described task), by the cognitive AI layer.
Regarding claim 2, Marin discloses the non-transitory computer-readable medium of claim 1, wherein the computer program is configured to cause the at least one processor to:
generate the PDD describing a process to be automated using a generative AI model of the cognitive AI layer or another generative AI model [figs. 3, 9; sec 0078, 0127, 0128-0136; Listener 330 communicates to and from with database 355; database 355 is part of layer 336 and layer 366 comprises an AI layer 360; listener 330 provides a natural language description of a processor also known as a process definition document (PDD) or recorder of user actions, a task capture, a process mining, etc to the layer 336 having the AI layer 360
Regarding claim 3, Marin discloses the non-transitory computer-readable medium of claim 1, wherein the annotations comprise a description of changes between a current version of the RPA workflow and one or more previous versions of the RPA workflow (sec 0129; the RPA 905 can be a complex flow with multiple steps being added to one or more written automation tasks).
Regarding claim 4, Marin discloses the non-transitory computer-readable medium of claim 1, wherein the computer program is further configured to cause the at least one processor to:
display the annotated RPA workflow in a user interface of an RPA designer application (figs. 1, 5; i.e. screen; sec 0083-0087, 0091, 0098, 0105, 0108, 0109).
Regarding claim 6, Marin discloses the non-transitory computer-readable medium of claim 1, wherein the computer program is further configured to cause the at least one processor to:
provide one or more screenshots including one or more visual representations of at least a portion of the RPA workflow (sec 0043, 0049, 0101, 0102) to a computer vision (CV) model (sec 0043, 0049, 0101, 0102) and/or an optical character recognition (OCR) model (sec 0043, 0049, 0101, 0102) of the cognitive AI layer to identify text and/or images therein (sec 0043, 0049, 0050, 0101, 0102; figs. 1, 4).
Regarding claim 7, Marin discloses the non-transitory computer-readable medium of claim 1, wherein the annotations provide a description of an overall process of the RPA workflow, describe what each activity in the RPA workflow is doing, or both (sec 0043, 0049, 0050, 0101, 0102; figs. 1, 4).
Regarding claim 8, Marin discloses the non-transitory computer-readable medium of claim 1, wherein the computer program is further configured to cause the at least one processor to:
monitor development of the RPA workflow in an RPA designer application (sec 0043-0051);
determine that one or more activities were added to and/or modified in the RPA workflow (sec 0043-0051); and
provide the code for the RPA workflow to the cognitive AI layer responsive to the determination that the one or more activities were added to and/or modified in the RPA workflow (sec 0043-0051).
Regarding claim 9, Marin discloses the non-transitory computer-readable medium of claim 8, wherein the determination that the one or more activities were added to and/or modified in the RPA workflow comprises:
determining that an activity of the RPA workflow was completed and a user moved on to a next activity (sec 0043-0051, 0057-0061),
periodically checking for changes to the RPA workflow (sec 0043-0051, 0057-0061),
determining that the user saved the RPA workflow (sec 0043-0051, 0057-0061),
determining that the user clicked a button for annotating the RPA workflow in an RPA designer application (sec 0043-0051, 0057-0061).
Regarding claim 10, Marin discloses the non-transitory computer-readable medium of claim 8, wherein the computer program is further configured to cause the at least one processor to:
repeat the steps of claim 8 after the cognitive AI layer provides the annotations for the RPA workflow or the RPA workflow code comprising the annotations as output (sec 0043-0051, 0057-0061).
Regarding claim 11, Marin discloses the non-transitory computer-readable medium of claim 1, wherein the cognitive AI layer comprises:
a generative AI model configured to generate annotated RPA workflows, provide sematic associations between text on the screen, logically group classes of RPA workflow activities, infer subsequent activities to add based on context of the RPA workflow, convert RPA workflows from a format of one RPA vendor to a format of another RPA vendor, or any combination thereof (sec 0043-0051, 0057-0061).
Regarding claim 12, Marin discloses one or more computing systems (sec 0088, 0092), comprising:
memory storing computer program instructions for providing automatic annotations for robotic process automation (RPA) workflows (sec 0088, 0092); and
at least one processor configured to execute the computer program instructions (sec 0088, 0092), wherein the computer program instructions are configured to cause the at least one processor to:
provide code for an RPA workflow or a process definition document (PDD) to a cognitive artificial intelligence (AI) layer that comprises a generative AI model configured to generate annotated RPA workflows [figs. 3, 9; sec 0078, 0127, 0128-0136; Listener 330 communicates to and from with database 355; database 355 is part of layer 336 and layer 366 comprises an AI layer 360; listener 330 provides a natural language description of a processor also known as a process definition document (PDD) or recorder of user actions, a task capture, a process mining, etc to the layer 336 having the AI layer 360];
provide sematic associations between text on the screen, logically group classes of RPA workflow activities (figs. 3, 9; providing, restructuring, reconfiguring, etc; sec 0078, 0127, 0128-0136),
infer subsequent activities to add based on context of the RPA workflow (figs. 3, 9; inferring, deducing, restructuring, reconfiguring, generating, providing, processing, etc; sec 0078, 0127, 0128-0136),
convert RPA workflows from a format of one RPA vendor to a format of another RPA vendor, or any combination thereof (figs. 3, 9; inferring, deducing, restructuring, reconfiguring, generating, providing, processing, etc; sec 0078, 0127, 0128-0136),
process the code for the RPA workflow or the PDD, by the cognitive AI layer (figs. 3, 9; inferring, deducing, restructuring, reconfiguring, generating, providing, processing, etc; sec 0078, 0127, 0128-0136), and
provide annotations for the RPA workflow or RPA workflow code comprising the annotations as output, by the cognitive AI layer (figs. 3, 9; inferring, deducing, restructuring, reconfiguring, generating, providing, processing, etc; sec 0035-0038, 0078, 0127, 0128-0136),
wherein the annotations provide a description of an overall process of the RPA workflow, describe what each activity in the RPA workflow is doing, or both (figs. 3, 9; inferring, deducing, restructuring, reconfiguring, generating, providing, processing, etc; sec 0078, 0127, 0128-0136), and
the cognitive Al layer configured to analyze and automatically generate annotations describing activities for the RPA workflow (AI generating and analyzing or editing annotations for an RPA workflow for the robot; sec 0035, 0036, 0038) or an RPA workflow pertaining to a process described in the PDD (sec 0035, 0037, 0038)..
Regarding claim 13, Marin discloses the one or more computing systems of claim 12, wherein the annotations comprise a description of changes between a current version of the RPA workflow and one or more previous versions of the RPA workflow (sec 0129; the RPA 905 can be a complex flow with multiple steps being added to one or more written automation tasks).
Regarding claim 14, Marin discloses the one or more computing systems of claim 12, wherein the computer program instructions are further configured to cause the at least one processor to:
display the annotated RPA workflow in a user interface of an RPA designer application (figs. 1, 5; sec 0083-0087, 0091, 0098, 0105, 0098, 0099, 0108, 0109).
Regarding claim 16, Marin discloses the one or more computing systems of claim 12, wherein the computer program instructions are further configured to cause the at least one processor to:
provide one or more screenshots including one or more visual representations of at least a portion of the RPA workflow (sec 0043, 0049) to a computer vision (CV) model (sec 0043, 0049) and/or an optical character recognition (OCR) model (sec 0043, 0049) of the cognitive AI layer to identify text and/or images therein (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4),
wherein the CV model and/or the OCR model ARE configured to provide output to a generative AI model of the cognitive AI layer (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
Regarding claim 17, Marin discloses the one or more computing systems of claim 12, wherein the computer program instructions are further configured to cause the at least one processor to:
monitor development of the RPA workflow in an RPA designer application (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
determine that one or more activities were added to and/or modified in the RPA workflow (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4); and
provide the code for the RPA workflow to the cognitive AI layer responsive to the determination that the one or more activities were added to and/or modified in the RPA workflow (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
Regarding claim 18, Marin discloses the one or more computing systems of claim 17, wherein the determination that the one or more activities were added to and/or modified in the RPA workflow comprises:
determining that an activity of the RPA workflow was completed and a user moved on to a next activity (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4),
periodically checking for changes to the RPA workflow (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4),
determining that the user saved the RPA workflow, or determining that the user clicked a button for annotating the RPA workflow in an RPA designer application (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
Regarding claim 19, Marin discloses a computer-implemented method for providing automatic annotations for robotic process automation (RPA) workflows, comprising:
providing code for an RPA workflow or a process definition document (PDD) to a cognitive artificial intelligence (AI) layer, by an RPA designer application executing on a computing system, the cognitive AI layer configured to process the code for the RPA workflow and/or the PDD (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4);
receiving annotations for the RPA workflow or RPA workflow code comprising the annotations from the cognitive AI layer, by the RPA designer application (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4); and
displaying the annotated RPA workflow, by the RPA designer application, wherein the annotations provide a description of an overall process of the RPA workflow (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4),
describe what each activity in the RPA workflow is doing, a description of changes between a current version of the RPA workflow and one or more previous versions of the RPA workflow, or any combination thereof (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4), wherein
the cognitive Al layer configured to analyze and automatically generate annotations describing activities for the RPA workflow (AI generating and analyzing or editing annotations for an RPA workflow for the robot; sec 0035, 0036, 0038) or an RPA workflow pertaining to a process described in the PDD (sec 0035, 0037, 0038).
Regarding claim 21, Marin discloses the computer-implemented method of claim 19, further comprising:
providing one or more screenshots including one or more visual representations of at least a portion of the RPA workflow, by the RPA designer application, to a computer vision (CV) model and/or an optical character recognition (OCR) model of the cognitive AI layer to identify text and/or images therein, wherein the CV model and/or the OCR model and configured to provide output to a generative AI model of the cognitive AI layer (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
Regarding claim 22, Marin discloses the computer-implemented method of claim 19, further comprising:
monitoring development of the RPA workflow, by the RPA designer application (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4);
determining that one or more activities were added to and/or modified in the RPA workflow, by the RPA designer application; and providing the code for the RPA workflow to the cognitive AI layer, by the RPA designer application, responsive to the determination that the one or more activities were added to and/or modified in the RPA workflow (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4),
wherein the determination that the one or more activities were added to and/or modified in the RPA workflow comprises determining that an activity of the RPA workflow was completed and a user moved on to a next activity, periodically checking for changes to the RPA workflow, determining that the user saved the RPA workflow, or determining that the user clicked a button for annotating the RPA workflow in an RPA designer application (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
Regarding claim 23, Marin discloses the computer-implemented method of claim 22, further comprising: repeating the steps of claim 22 after the cognitive AI layer provides the annotations for the RPA workflow or the RPA workflow code comprising the annotations as output, by the RPA designer application (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
Regarding claim 24, Marin discloses the computer-implemented method of claim 19, wherein the cognitive Al layer comprises:
a generative AI model configured to generate annotated RPA workflows (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4);
provide sematic associations between text on the screen, logically group classes of RPA workflow activities (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4);
infer subsequent activities to add based on context of the RPA workflow (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4);
convert RPA workflows from a format of one RPA vendor to a format of another RPA vendor, or any combination thereof (sec 0043, 0049, 0050, 0098, 0099, 0101, 0102; figs. 1, 4).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 5, 15, 20 are rejected under 35 U.S.C. 103 as being obvious over Marin et al (US Pub 2024/0142917) in view of Subramanian (US 20240028586)
The applied reference has a common Assignee, UiPath, Inc with the instant application. Based upon the earlier effectively filed date of the reference, it constitutes prior art under 35 U.S.C. 102(a)(2).
This rejection under 35 U.S.C. 103 might be overcome by: (1) a showing under 37 CFR 1.130(a) that the subject matter disclosed in the reference was obtained directly or indirectly from the inventor or a joint inventor of this application and is thus not prior art in accordance with 35 U.S.C.102(b)(2)(A); (2) a showing under 37 CFR 1.130(b) of a prior public disclosure under 35 U.S.C. 102(b)(2)(B); or (3) a statement pursuant to 35 U.S.C. 102(b)(2)(C) establishing that, not later than the effective filing date of the claimed invention, the subject matter disclosed and the claimed invention were either owned by the same person or subject to an obligation of assignment to the same person or subject to a joint research agreement. See generally MPEP § 717.02.
Regarding claim 5, Brons discloses the non-transitory computer-readable medium of claim 1, but did not particularly recite, a large language model (LLM) “wherein the cognitive AI layer comprises the large language model (LLM) that has been fine-tuned during training to understand natures of and interrelationships between RPA workflow activities.”
However, Subramanian (sec 0052, 0059) teaches of the limitation, “wherein the cognitive AI layer comprises a large language model (LLM) that has been fine-tuned during training to understand natures of and interrelationships between RPA workflow activities.”
Therefore, it would have been to one having ordinary skill in the art at the time then invention was filed to modify Marin as taught by Subramanian for the purpose of augmenting the Marin device with a cognitive layer that comprises a large language model (LLM), to provide a more advanced and sophisticated user experience, as well as provide access to state-of-the-art natural language processing (NLP) and other ML capabilities that these companies offer (see Subramanian; sec 0059).
Regarding claim 15, Brons discloses the one or more computing systems of claim 12, but did not particularly recite, a large language model (LLM) “wherein the cognitive AI layer comprises a large language model (LLM) that has been fine-tuned during training to understand natures of and interrelationships between RPA workflow activities.”
However, Subramanian (sec 0052, 0059) teaches of the limitation, “wherein the cognitive AI layer comprises a large language model (LLM) that has been fine-tuned during training to understand natures of and interrelationships between RPA workflow activities.”
Therefore, it would have been to one having ordinary skill in the art at the time then invention was filed to modify Marin as taught by Subramanian for the purpose of augmenting the Marin device with a cognitive layer that comprises a large language model (LLM), to provide a more advanced and sophisticated user experience, as well as provide access to state-of-the-art natural language processing (NLP) and other ML capabilities that these companies offer (see Subramanian; sec 0059).
Regarding claim 20, Brons discloses the computer-implemented method of claim 19, but did not particularly recite, a large language model (LLM), wherein the cognitive AI layer comprises a large language model (LLM) that has been fine-tuned during training to understand natures of and interrelationships between RPA workflow activities.”
However, Subramanian (sec 0052, 0059) teaches of the limitation, “wherein the cognitive AI layer comprises a large language model (LLM) that has been fine-tuned during training to understand natures of and interrelationships between RPA workflow activities.”
Therefore, it would have been to one having ordinary skill in the art at the time then invention was filed to modify Marin as taught by Subramanian for the purpose of augmenting the Marin device with a cognitive layer that comprises a large language model (LLM), to provide a more advanced and sophisticated user experience, as well as provide access to state-of-the-art natural language processing (NLP) and other ML capabilities that these companies offer (see Subramanian; sec 0059).
Response to Arguments
Applicant's arguments filed 02/13/2026 have been fully considered but they are not persuasive.
Applicant amended the claims and generally argued making conclusory remarks that the prior art Marin does not disclose the amended limitations in the claims. Applicant fails to particularly point out what part of the limitation in the claim Marin did not disclose. The examiner respectfully disagrees with applicant’s conclusory remarks. It is respectfully submitted that the prior art, Marin at least at sec 0036-0038 uses AI to automatically generate annotations that describe activities for a Robotic Process Automation (RPA). Merlin for example performs a Robotic Process Automation (RPA) automatically e.g. by using AI (Artificial Intelligence). In performing the RPA Merlin disclose that, “One or more AI/ML models 132 may be employed to uncover recurring task patterns in the data. Repetitive tasks that are ripe for automation may then be identified. This information may initially be provided by listeners and analyzed on servers of core hyper-automation system 120, such as server 130, in some embodiments. The findings from task mining (e.g., extensive application markup language (XAML) process data) may be exported to process documents or to a designer application”. The again Merlin discloses that, “Task mining in some embodiments may include taking screenshots with user actions (e.g., mouse click locations, keyboard inputs, application windows and graphical elements the user was interacting with, timestamps for the interactions, etc.), collecting statistical data (e.g., execution time, number of actions, text entries, etc.), editing and annotating screenshots, specifying types of actions to be recorded, etc”. For describing the RPA process, Merlin uses markup language which is a system of tags that annotates a document to describe the structure and presentation thereof. Markup language uses tags to define elements like headings, paragraphs, or bold text, instructing a computer on how to display the content. Common examples include HTML (Hypertext Markup Language), XML (eXtensible Markup Language), and Markdown.
As such Merlin disclose Marin discloses a non-transitory computer-readable medium storing a computer program (sec 0088, 0092) for providing automatic annotations for robotic process automation (RPA) workflows (sec 0127; builder 903 generates a workflows for the RPA 905 using ML from natural language described task), the computer program configured to cause at least one processor to:
provide code for an RPA (sec 0127; builder 903 generates a workflows for the RPA 905 using a code such as an ML from natural language described task) or a process definition document (PPD) to a cognitive artificial intelligence (AI) layer [figs. 3, 9; sec 0078, 0127, 0128-0136; Listener 330 communicates to and from with database 355; database 355 is part of layer 336 and layer 366 comprises an AI layer 360; listener 330 provides a natural language description of a processor also known as a process definition document (PDD) or ecorder of user actions, a task capture, a process mining, etc to the layer 336 having the AI layer 360], the cognitive Al layer configured to analyze and automatically generate annotations describing activities for the RPA workflow (AI generating and analyzing or editing annotations for an RPA workflow for the robot; sec 0035, 0036, 0038) or an RPA workflow pertaining to a process described in the PDD (sec 0035, 0037, 0038);
process the code for the RPA workflow or the PDD (figs. 3, 9; restructuring, reconfiguring, etc; sec 0078, 0127, 0128-0136), by the cognitive AI layer; and
generate annotations for the RPA workflow or RPA workflow code comprising the annotations at output (sec 0127; builder 903 generates a workflows for the RPA 905 using ML from natural language described task), by the cognitive AI layer.
Conclusion
The prior art, Monakova (US pub 2022/0083330) made of record and not relied upon is considered pertinent to applicant's disclosure.
Communication
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RONNIE MANCHO whose telephone number is (571)272-6984. The examiner can normally be reached Mon-Thurs.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached at 571 270 5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RONNIE M MANCHO/