Detailed Action
Notice of Pre-AIA or AIA status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Restriction/Election
Applicant’s election with traverse of Species I in the reply filed on May 22, 2026 is acknowledged. For the reasons set forth below, the requirement is still deemed proper and is therefore made FINAL.
The Applicant contends that “Species I and Species II overlap in scope and are therefore not patentably distinct under MPEP § 806.05(j),” because “[b]oth species are directed to the same overarching invention of locating repeatable user-driven processes from recordings of user interactions with application programs by analyzing UI controls extracted from captured UI screen images.” (Response 2). The Examiner respectfully disagrees.
This argument misunderstands the special meaning of “overlap in scope” as it pertains to restriction practice. The requirement that distinct inventions “not overlap in scope” does not preclude distinct claims from sharing limitations. The rule against overlapping scope simply means that their scope is “mutually exclusive.” MPEP § 806.05(j). “Claims to different species are mutually exclusive if one claim recites limitations disclosed for a first species but not a second, while a second claim recites limitations disclosed only for the second species and not the first.” MPEP § 806.04(f).
Every claim in every species of the restriction adheres to this requirement.
Specifically, every claim identified in Species I has limitations describing “a subset of UI controls that are ranked with respect to each UI control’s likelihood of being selected by a user to start or stop a user-initiated process,” because that limitation appears in parent claim 1. Several claims in Species I further recite limitations for ranking the UI screen images as well (claims 5, 6, and 13–16). Meanwhile, none of the claims identified in Species II recite anything about ranking UI controls or UI screen images, because every claim in Species II depends from an independent claim that does not recite any limitations pertaining to ranking.
Likewise, every claim identified in Species II has limitations directed to optical character recognition (“OCR”), whereas none of the claims identified in Species I recite anything about OCR.
Consequently, the two species meet the distinctness test for non-overlapping scope, i.e., mutual exclusion. The Notice of Restriction provided this finding on page 2, paragraph 3, but the Applicant’s traversal never disputes it. As such, the traversal cannot prevail on the distinctness prong of the election of species requirement, because it never addresses with the basis for the requirement. Nevertheless, the Applicant’s second distinctness argument on page 3 of the Response will be addressed for completeness.
Specifically, the Applicant contends that the election requirement is improper because “the specification expressly discloses that ranking (Species I) and OCR text analysis (Species II) are not alternative approaches but rather complementary and sequential sub-steps within the same unified pre-processing pipeline.” (Response 3). “Because the OCR text analysis of Species II feeds directly into the UI-control ranking of Species I as part of the same technical process, the two species are not independent or mutually exclusive.” (Response 3).
Respectfully, this argument misunderstands both the facts and the law. Factually, this argument is simply untrue: the disclosure provides several examples of ranking that do not even involve text—let alone OCR—such as frequency of occurrence, user interaction frequency, and weighted control values based on screen signatures. (Spec. ¶¶ 38 and 53–57). In fact, OCR is not even required for the embodiments that rank UI components based on text, because the specification further discloses that object and text detection service 304 can interact directly with the operating system via operating system interface 306 to obtain the text data, rather than attempting interpret text from the literal pictures in the screenshot. (Spec. ¶ 47).
Legally, whether or not the disclosure provides for OCR to be used in the ranking is irrelevant. While it is true that species themselves may be described in terms of what the applicant disclosed, the legal basis for restriction turns on what is claimed. See 35 U.S.C. § 121 (restriction is based on whether multiple distinct inventions “are claimed in one application”) and MPEP § 806.04(f) (“Where two or more species are claimed, a requirement for restriction to a single species may be proper if the species are mutually exclusive.”). The Applicant may have disclosed an embodiment that has both ranking and OCR, but that embodiment is not claimed.
Therefore, the Examiner is not persuaded by the traversal of the distinctness requirement.
With respect to the burden requirement, the Applicant focuses on the content of the disclosure, and speculates (without evidence) that a prior art search for automated identification of repeatable user-driven processes from UI screen recordings would necessarily encompass techniques for both extracting UI control metadata (including via OCR) and ranking those controls based on process-start/stop likelihood. (Response 3). The Applicant also alleges that the Examiner’s reference to separate classification and divergent subject matter was “conclusory,” and lacks “further explanation” needed for a showing of serious burden to justify restriction. (Response 3). The Examiner respectfully disagrees.
As mentioned above, the disclosure provides several examples of ranking that do not even involve text, let alone OCR, such as frequency of occurrence, user interaction frequency, and weighted control values based on screen signatures. (Spec. ¶¶ 38 and 53–57). Such non-OCR methods are not only disclosed, they are also claimed in Species I. On the other hand, Species II sacrifices any claim scope concerning ranking criteria in favor of a deeper discussion of OCR.
These two species require different areas of search because the inventions analyze different things: species I analyzes user activity in software and thus falls within G06F 11/3438 subclass of the Cooperative Patent Classification scheme, while species II analyzes images for text, and thus fall within G06V 30/10 subclass of CPC. A showing of separate classifications and separate fields of search are, themselves, sufficient evidence for a prima facie case of search and/or examination burden. MPEP § 808.02.
Consequently, since the inventions are distinct and a search and examination burden exists, the Examiner is not persuaded to withdraw the restriction.
The requirement is still deemed proper and is therefore made FINAL.
Claim Rejections – 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1–16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 lacks antecedent basis for “the subset of UI controls that are known to indicate a start or stop of user-initiated processes.” Prior to this step, the subset is merely introduced as a subset of controls “ranked with respect to each UI control’s likelihood of being selected by a user to start or stop a user-initiated process.” UI controls that the computer merely believes are “likely” to start or stop a user-initiated process are not the same as a UI control that is actually “known” to start or stop a user-initiated process.
Claims 2–16 depend from claim 1, and are therefore rejected under 35 U.S.C. § 112(b) for their incorporation of the indefinite subject matter by reference. Claim 3 is also further rejected for directly referring back to the indefinite subject matter (“…the subset of UI controls that are known to indicate a start or stop of repeatable user-driven processes.”).
Claim Rejections – 35 U.S.C. § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1–21, 28, and 29 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by U.S. Patent Application Publication No. 2017/0001308 A1 (“Bataller”).
Claim 1
Bataller discloses:
A computer-implemented method for identifying user-initiated processes, the method comprising:
Reference is made to FIGs. 1 and 2, which illustrate computer systems 100 and 200 that automate manual processes, and perform said automation, respectively. See Bataller ¶¶ 17 and 38. As will be discussed below, systems 100 and 200 (but primarily 100) perform the claimed invention as part of their normal operation, and therefore, their disclosure of the system(s) also discloses the method. MPEP § 2111.02.
Bataller also discloses the method itself as “process 300,” see Bataller ¶ 54 and FIG. 3, but since Bataller provides a more detailed discussion of the systems 100, 200 that perform the method, this rejection will focus on Bataller’s disclosure of those systems.
acquiring recordings of at least one user interacting with at least one application program operating on at least one user computing device, the recordings including at least user-initiated events, and a user interface (UI) screen image captured for each of the user-initiated events;
“The image capturer 110 may obtain images of a display of a computer while a user is interacting with the computer in manually performing a process. For example, the image capturer may obtain a first image of a touchscreen showing a user interface with ‘Yes’ and ‘No’ buttons and a second image of a touchscreen showing the user interface with the ‘Yes’ button being highlighted in response to being touched by the user.” Bataller ¶ 18.
acquiring UI controls and metadata from the captured UI screen images;
“The image capturer 110 may provide the obtained images to the activity identifier 120,” which discerns “activities” performed by the user depicted in the images and provides those activities to an “activity information generator 130,” Bataller ¶¶ 21–25, thus allowing an “activity information generator 130 [to] generate activity information associated with the activity.” Bataller ¶ 26.
This “activity information” includes both the claimed UI controls and the metadata. Regarding the UI controls, “the activity information for a screen touch may describe the coordinates for a touch on a touch screen, a snapshot . . . and a screenshot of the display after the touch screen was touched.” Bataller ¶ 26. Regarding the metadata, the activity information may further comprise data about the inputs corresponding to each activity that were sent from a mouse, a keyboard, a touchscreen, or another input device. Bataller ¶¶ 28–30.
identifying, from the acquired UI controls, a set of selected UI controls that a user selected during the recordings;
System 100 also utilizes a “process splitter,” which takes as input “a list of sequential activities,” such as “click on application icon, application window opens, and click on menu button.” Bataller ¶ 20.
obtaining, from the set of selected UI controls, a subset of UI controls that are ranked with respect to each UI control’s likelihood of being selected by a user to start or stop a user-initiated process;
From this list, the process splitter will discover “when a recurring sequence of activities is identified,” and based on the recurrence, “identify a first activity in the recurring sequence as an activity associated with a start of a process and identify a last activity in the recurring sequence as an activity associated with an end of a process.” Bataller ¶ 20.
Analyzing recurring sequences is only one example of how the process splitter decides which activities are more likely to start/end a process than others (and thus, out rank them). As another example, “the system 100 may use machine-learning to determine when a process begins and when a process ends,” or, there may be “specific activities” that are pre-understood to bookend processes, “e.g., a specific software window popping into a foreground or a specific logo being detected.” Bataller ¶ 20. As yet another example, the user may manually indicate which activities start and end the process. See Bataller ¶ 33.
In each example, the process splitter “ranks” the activities (which include identifications of UI elements) in the sense that some activities are superior to others with respect to whether or not they terminate the segments.
and selecting a set of UI screen images from the captured UI screen images, based on the subset of UI controls that are known to indicate a start or stop of user-initiated processes, that are candidate UI screen images for corresponding to a start or a stop of a user-initiated process.
All of the foregoing data, including the screenshots, is then provided to a “process definition generator 140,” which uses the data to “generate a process definition for use in causing a robot to automatically perform a process by interacting with another computer.” Bataller ¶ 31. The process definition includes screenshots for each of the UI interactions in the process, see Bataller ¶ 32, and from these, an “activity trigger engine 220” selects which of the screenshots from the process definition it will use to automatically trigger the process during runtime. “The activity trigger engine 220 may trigger an activity when conditions for triggering the activity are satisfied. For example, the conditions may be when a portion of an image of a display matches a snapshot associated with the activity or that a previous activity is completed.” Bataller ¶ 44.
Claim 2
Bataller discloses a computer-implemented method as recited in claim 1,
wherein the identifying, the obtaining, and the selecting are performed on at least one server computer, and wherein the at least one server computer operable to couple to at least one network to acquire the recordings from the at least one user computer device, the at least one server computer being remote from the at least one user computer device.
“Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (‘LAN’) and a wide area network (‘WAN’), e.g., the Internet.” Bataller ¶ 74.
Claim 3
Bataller discloses the computer-implemented method as recited in claim 1,
wherein each of the candidate UI screen images include at least one of the UI controls within the subset of UI controls that are known to indicate a start or stop of repeatable user-driven processes.
“The activity trigger engine 220 may trigger an activity when conditions for triggering the activity are satisfied. For example, the conditions may be when a portion of an image of a display matches a snapshot associated with the activity or that a previous activity is completed.” Bataller ¶ 44.
Claim 4
Bataller discloses the computer-implemented method as recited in claim 1,
wherein the selecting each of the set of the candidate UI screen images is based on a frequency-based selection scheme, with those of the candidate UI screen images that occur more frequently in the captured UI screen images being more likely selected.
The screenshots chosen for inclusion in a process definition are selected based on the activities selected for that process definition, and the activities selected for a process definition are selected “when a recurring sequence of activities is identified.” Bataller ¶ 20.
Claim 5
Bataller discloses the computer-implemented method as recited in claim 1, wherein the selecting the set of the candidate UI screen images comprises:
ranking the captured UI screen images based on the subset of UI controls that are ranked with respect to each UI control’s likelihood of being selected by a user to start or stop a user-initiated processes; and selecting the set of candidate UI screen images from the ranked UI screen images.
The activity trigger engine 220 triggers processes “when a portion of an image of a display matches a snapshot associated with the activity,” and the activity trigger engine 220 decides which snapshot(s) to match against based on the order of their corresponding activities in the process definition: “For example, the activity trigger engine 220 may first analyze the images for portions that match Snapshot X for a first activity” before triggering the automated process, rather than matching the current display’s image against Snapshot Y, because the activity for Snapshot Y cannot occur until the activity for Snapshot X occurs. Bataller ¶ 44. Snapshot X is thus ranked over Snapshot Y in terms of whether it’s the first snapshot in the order of snapshots in the process definition.
Claim 6
Bataller discloses the computer-implemented method as recited in claim 5,
wherein the ranking of the UI screen images uses at least a heuristic ranking.
The ranking described in the rejection of claim 5 above was based on the order of the activities that correspond to the snapshots in the process definition, and in some embodiments, that order may be determined using heuristics such as “a specific image being shown on a display, e.g., a specific software window popping into a foreground or a specific logo being detected.” Bataller ¶ 20.
Claim 7
Bataller discloses a computer-implemented method as recited in claim 1, wherein the method comprises:
identifying a particular repeatable user-driven process based on the set of candidate UI screen images;
“[I]mage capturer 110 may be continuously obtaining images,” and the system 100 may use a process splitter “to identify recurring sequences of activities.” Bataller ¶ 20.
and converting the particular repeatable user-driven process into a software robot.
The screenshots (and other data) are provided to the “process definition generator 140,” which uses the data to “generate a process definition for use in causing a robot to automatically perform a process by interacting with another computer.” Bataller ¶ 31.
Claim 8
Bataller discloses a computer-implemented method as recited in claim 7,
wherein the particular repeatable user-driven process is a business process associated with a user.
“In general, an aspect of the subject matter described in this specification may involve a process for automating a process that is manually performed by a person. To automate a manual process, a system may use computer vision techniques to analyze images of a display of a computer while a person is manually interacting with the computer while performing the process. From the analysis, the system may determine the activities associated with a process, e.g., keystrokes, mouse clicks, or touchscreen touches. Based on the determined activities, the system may then cause a robot to interact with a computer to automatically repeat the process.” Bataller ¶ 4. “The process definitions may be inspected by a developer or a business analyst, which may be a human being or a computer, e.g. in order to choose an optimized process definition for a particular process among a plurality of process definitions that are the result of generating multiple process definitions for the same process different machines or users.” Bataller ¶ 61.
Claim 9
Bataller discloses a computer-implemented method as recited in claim 1, wherein the obtaining of the set of UI controls that are known to indicate a start or stop of repeatable user-driven processes comprises:
obtaining a set of start UI controls that are known to indicate a start of repeatable user-driven processes; and obtaining a set of stop UI controls that are known to indicate a stop of repeatable user-driven processes.
The process splitter will discover “when a recurring sequence of activities is identified,” and based on the recurrence, “identify a first activity in the recurring sequence as an activity associated with a start of a process and identify a last activity in the recurring sequence as an activity associated with an end of a process.” Bataller ¶ 20.
Analyzing recurring sequences is only one example of how the process splitter decides which activities are more likely to start/end a process than others (and thus, out rank them). As another example, “the system 100 may use machine-learning to determine when a process begins and when a process ends,” or, there may be “specific activities” that are pre-understood to bookend processes, “e.g., a specific software window popping into a foreground or a specific logo being detected.” Bataller ¶ 20. As yet another example, the user may manually indicate which activities start and end the process. See Bataller ¶ 33.
Claim 10
Bataller discloses a computer-implemented method as recited in claim 9, wherein the selecting the set of the candidate UI screen images comprises:
selecting a set of candidate start UI screen images from the captured UI screen images based on the set of start UI controls; and selecting a set of candidate stop UI screen images from the captured UI screen images based on the set of stop UI controls.
All the foregoing data obtained by the modules in system 100, including the screenshots, is provided to a “process definition generator 140,” which uses the data to “generate a process definition for use in causing a robot to automatically perform a process by interacting with another computer.” Bataller ¶ 31. The process definition includes screenshots for each of the UI interactions in the process, see Bataller ¶ 32, and from these, an “activity trigger engine 220” selects which of the screenshots from the process definition it will use to automatically trigger the process during runtime. “The activity trigger engine 220 may trigger an activity when conditions for triggering the activity are satisfied. For example, the conditions may be when a portion of an image of a display matches a snapshot associated with the activity or that a previous activity is completed.” Bataller ¶ 44.
Claim 11
Bataller discloses the computer-implemented method as recited in claim 10,
wherein the selecting each of the set of the candidate start and stop UI screen images is based on a frequency-based selection, with those of the candidate start and stop UI screen images that occur more frequently in the captured UI screen images being more likely selected.
The screenshots chosen for inclusion in a process definition are selected based on the activities selected for that process definition, and the activities selected for a process definition are selected “when a recurring sequence of activities is identified.” Bataller ¶ 20.
Claim 12
12. A computer-implemented method as recited in claim 10,
wherein the selecting each of the set of candidate start and stop UI screen images is based on a frequency-based selection, with those of the candidate start and stop UI screen images in which a user selects a start or stop UI control, respectively, more frequently being more likely selected.
The screenshots chosen for inclusion in a process definition are selected based on the activities selected for that process definition, and the activities selected for a process definition are selected “when a recurring sequence of activities is identified.” Bataller ¶ 20.
Claim 13
Bataller discloses the computer-implemented method as recited in claim 10, wherein the selecting the set of the candidate UI screen images comprises:
ranking the candidate start and stop UI screen images, respectively, based on the subset of UI controls that are known to indicate a start or stop of repeatable user-driven processes; and selecting the set of candidate start and stop UI screen images from the ranked, start and stop candidate UI screen images, respectively.
The activity trigger engine 220 triggers processes “when a portion of an image of a display matches a snapshot associated with the activity,” and the activity trigger engine 220 decides which snapshot(s) to match against based on the order of their corresponding activities in the process definition: “For example, the activity trigger engine 220 may first analyze the images for portions that match Snapshot X for a first activity” before triggering the automated process, rather than matching the current display’s image against Snapshot Y, because the activity for Snapshot Y cannot occur until the activity for Snapshot X occurs. Bataller ¶ 44. Snapshot X is thus ranked over Snapshot Y in terms of whether it’s the first snapshot in the order of snapshots in the process definition.
Claim 14
Bataller discloses the computer-implemented method as recited in claim 13,
wherein the ranking of the UI screen images uses at least a heuristic ranking.
The ranking described in the rejection of claim 13 above was based on the order of the activities that correspond to the snapshots in the process definition, and in some embodiments, that order may be determined using heuristics such as “a specific image being shown on a display, e.g., a specific software window popping into a foreground or a specific logo being detected.” Bataller ¶ 20.
Claim 15
Bataller discloses the computer-implemented method as recited in claim 13, wherein the method comprises:
identifying a particular repeatable user-driven process based on the set of candidate start and stop UI screen images;
“[I]mage capturer 110 may be continuously obtaining images,” and the system 100 may use a process splitter “to identify recurring sequences of activities.” Bataller ¶ 20.
and converting the particular repeatable user-driven process into a software robot.
The screenshots (and other data) are provided to the “process definition generator 140,” which uses the data to “generate a process definition for use in causing a robot to automatically perform a process by interacting with another computer.” Bataller ¶ 31.
Claim 16
Bataller discloses the computer-implemented method as recited in claim 15,
wherein the particular repeatable user-driven process is a business process associated with a user.
“In general, an aspect of the subject matter described in this specification may involve a process for automating a process that is manually performed by a person. To automate a manual process, a system may use computer vision techniques to analyze images of a display of a computer while a person is manually interacting with the computer while performing the process. From the analysis, the system may determine the activities associated with a process, e.g., keystrokes, mouse clicks, or touchscreen touches. Based on the determined activities, the system may then cause a robot to interact with a computer to automatically repeat the process.” Bataller ¶ 4. “The process definitions may be inspected by a developer or a business analyst, which may be a human being or a computer, e.g. in order to choose an optimized process definition for a particular process among a plurality of process definitions that are the result of generating multiple process definitions for the same process different machines or users.” Bataller ¶ 61.
Claim 17
Bataller discloses
A computer-implemented method for locating repeatable user-driven processes for conversion into software robots, the method comprising:
Reference is made to FIGs. 1 and 2, which illustrate computer systems 100 and 200 that automate manual processes, and perform said automation, respectively. See Bataller ¶¶ 17 and 38. As will be discussed below, systems 100 and 200 (but primarily 100) perform the claimed invention as part of their normal operation, and therefore, their disclosure of the system(s) also discloses the method. MPEP § 2111.02.
Bataller also discloses the method itself as “process 300,” see Bataller ¶ 54 and FIG. 3, but since Bataller provides a more detailed discussion of the systems 100, 200 that perform the method, this rejection will focus on Bataller’s disclosure of those systems.
acquiring recordings of a plurality of users interacting with at least application programs operating on user computing devices, the recordings including at least user-triggered events with corresponding user interface (UI) screen images captured for the user-triggered events;
“The image capturer 110 may obtain images of a display of a computer while a user is interacting with the computer in manually performing a process. For example, the image capturer may obtain a first image of a touchscreen showing a user interface with ‘Yes’ and ‘No’ buttons and a second image of a touchscreen showing the user interface with the ‘Yes’ button being highlighted in response to being touched by the user.” Bataller ¶ 18.
acquiring UI controls and metadata from the UI screen images within the recordings;
“The image capturer 110 may provide the obtained images to the activity identifier 120,” which discerns “activities” performed by the user depicted in the images and provides those activities to an “activity information generator 130,” Bataller ¶¶ 21–25, thus allowing an “activity information generator 130 [to] generate activity information associated with the activity.” Bataller ¶ 26.
This “activity information” includes both the claimed UI controls and the metadata. Regarding the UI controls, “the activity information for a screen touch may describe the coordinates for a touch on a touch screen, a snapshot . . . and a screenshot of the display after the touch screen was touched.” Bataller ¶ 26. Regarding the metadata, the activity information may further comprise data about the inputs corresponding to each activity that were sent from a mouse, a keyboard, a touchscreen, or another input device. Bataller ¶¶ 28–30.
identifying, from the acquired UI controls, a set of selected UI controls that a user selected during the recordings;
System 100 also utilizes a “process splitter,” which takes as input “a list of sequential activities,” such as “click on application icon, application window opens, and click on menu button.” Bataller ¶ 20.
obtaining a set of UI controls that are known to indicate a start or stop of repeatable user-driven processes;
From this list, the process splitter will discover “when a recurring sequence of activities is identified,” and based on the recurrence, “identify a first activity in the recurring sequence as an activity associated with a start of a process and identify a last activity in the recurring sequence as an activity associated with an end of a process.” Bataller ¶ 20.
Analyzing recurring sequences is only one example of how the process splitter decides which activities are more likely to start/end a process than others (and thus, out rank them). As another example, “the system 100 may use machine-learning to determine when a process begins and when a process ends,” or, there may be “specific activities” that are pre-understood to bookend processes, “e.g., a specific software window popping into a foreground or a specific logo being detected.” Bataller ¶ 20. As yet another example, the user may manually indicate which activities start and end the process. See Bataller ¶ 33.
and selecting a set of candidate UI screen images from the UI screen images within the recordings based on those of the set of selected UI controls that a user selected during the recordings which are within the set of UI controls that are known to indicate a start or stop of repeatable user-driven processes.
All of the foregoing data, including the screenshots, is then provided to a “process definition generator 140,” which uses the data to “generate a process definition for use in causing a robot to automatically perform a process by interacting with another computer.” Bataller ¶ 31. The process definition includes screenshots for each of the UI interactions in the process, see Bataller ¶ 32, and from these, an “activity trigger engine 220” selects which of the screenshots from the process definition it will use to automatically trigger the process during runtime. “The activity trigger engine 220 may trigger an activity when conditions for triggering the activity are satisfied. For example, the conditions may be when a portion of an image of a display matches a snapshot associated with the activity or that a previous activity is completed.” Bataller ¶ 44.
Claim 18
Bataller discloses the computer-implemented method as recited in claim 17,
wherein the selecting of the set of candidate UI screen images is based at least in part on the frequency in which the UI screen images having the set of UI controls that are known to indicate a start or stop of repeatable user-driven processes occur within the recordings.
The screenshots chosen for inclusion in a process definition are selected based on the activities selected for that process definition, and the activities selected for a process definition are selected “when a recurring sequence of activities is identified.” Bataller ¶ 20.
Claim 19
Bataller discloses the computer-implemented method as recited in claim 17,
wherein the obtaining a set of UI controls that are known to indicate a start or stop of repeatable user-driven processes comprises determining at least a part of the set of UI controls that are known to indicate a start or stop of repeatable user-driven processes by use of machine learning.
Analyzing recurring sequences is only one example of how the process splitter decides which activities are more likely to start/end a process than others (and thus, out rank them). As another example, “the system 100 may use machine-learning to determine when a process begins and when a process ends.” Bataller ¶ 20.
Claim 20
Bataller discloses the computer-implemented method as recited in claim 17,
wherein the selecting of the set of candidate UI screen images includes at least ranking the candidate UI screen images.
The process splitter discussed in the rejection of claim 17 “ranks” the activities (which include identifications of UI elements) in the sense that some activities are superior to others with respect to whether or not they terminate the segments.
Claim 21
Bataller discloses the computer-implemented method as recited in claim 20,
wherein the ranking uses at least a heuristic ranking.
The ranking described in the rejection of claim 20 above was based on the order of the activities that correspond to the snapshots in the process definition, and in some embodiments, that order may be determined using heuristics such as “a specific image being shown on a display, e.g., a specific software window popping into a foreground or a specific logo being detected.” Bataller ¶ 20.
Claims 28 and 29
Claim 29, read together with all of the elements it incorporates by reference to claim 28, is directed to a non-transitory computer readable medium with program code that causes a computer to perform exactly the same method as recited in claim 17.
Accordingly, the Examiner hereby incorporates by reference all of the findings set forth in the rejection of claim 17 into this rejection, and additionally finds that Bataller discloses the computer readable medium embodiment of the method. See Bataller ¶¶ 64–69. In view of these findings, claim 29 is anticipated by Bataller. Claim 28 is also anticipated over the same findings, as the transitional phrase “comprising” in claim 28 simply covers the additional unrecited elements that Bataller discloses for claim 29. See MPEP § 2111.03.
Claim Rejections – 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were effectively filed absent any evidence to the contrary. Applicant is advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned at the time a later invention was effectively filed in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claim 27 is rejected under 35 U.S.C. § 103 as being unpatentable over Bataller as applied to claim 17 above, and further in view of U.S. Patent Application Publication No. 2020/0206920 A1 (“Ma”).
Claim 27
Bataller teaches the computer-implemented method as recited in claim 17,
wherein the acquiring of the recordings of a plurality of users interacting with at least application programs operating on their respective user computing devices acquires a dataset
“[S]ystem 100 to detect process anomalies or discrepancies between users, and make sure the final documentation created illustrates the optimal way of executing a process.” Bataller ¶ 34 (emphasis added).
formed over a period of time
“The image capturer 110 may obtain images at various times. For example, the image capturer 110 may obtain an image at predetermined intervals, e.g., every one, five, twenty five, one hundred milliseconds, or some other interval.” Bataller ¶ 19.
Bataller does not appear to explicitly disclose that the period of time is at least two days in duration.
Ma, however, teaches a method 300 and computer implementation thereof that, much like Bataller and the claimed invention, mines/analyzes users’ command logs for automatable tasks, see Ma ¶¶ 2 and 6, but further teaches:
the acquiring of the recordings of a plurality of users interacting with at least application programs operating on their respective user computing devices acquires a dataset formed over a period of time, the period of time being at least two days in duration.
“Preferred implementations of the presently disclosed inventive concepts leverage directed, acyclic graphs (DAGs) as a conceptual/organizational model for robotic process automation applications,” using “traces” that were “generated by any number of users . . . over a predetermined timespan, e.g. a day, a week, a month, etc.” Ma ¶ 240; see also ¶ 252 (providing the example modeling traces from “20 employees working typical hours over a one month period”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to improve Bataller’s method and system of discovering repeatable tasks in the same way that Ma’s method and system were improved, i.e., by providing more data from users over a longer period of time. One would have been motivated to improve Bataller in the same way as Ma because in situations where general boundaries are not capable of exact definition, it is preferably to favor longer repetitive sequence patterns over shorter ones.” Ma ¶ 143.
Additional Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
US 20240211836 A1 teaches a method for extracting features, i.e text features from screen images for performing mining task using ML based network to facilitate the automation of tasks, involves resizing screen image to predetermined size and normalizing pixel values in screen image to generate preprocessed screen image
US 20210086354 A1 teaches a computing device for discovering, mining or identifying process sequences or patterns, has processor that is configured to automatically generate automated workflow based on generated hierarchy for process automation by robot
US 20240220581 A1 teaches a non-transitory computer-readable medium for storing program for automatic data transfer between source and target using task mining, has set of instructions for checking for matches between values in source and target in user interface using recorded data pertaining to user interactions
US 20190187987 A1 teaches a computer-implemented method for facilitating automation of sequences of actions, involves providing information of first sequence of actions to user in response to similarity measurement between second and first sequence of actions
US 20240134685 A1 teaches a method for detecting variants of automatable tasks for robotic process automation using software robots to automate repetitive and/or labor-intensive tasks by computing system e.g. cellphone, involves determining multiple variants of automatable task, and outputting variants of automatable task
US 20100114851 A1 teaches computer implemented user interface controls searching method for providing user input to software applications, involves providing retrieved interface controls with data entry fields for accepting user input data of application.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Justin R. Blaufeld whose telephone number is (571)272-4372. The examiner can normally be reached M-F 9:00am - 4:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James K Trujillo can be reached at (571) 272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Justin R. Blaufeld
Primary Examiner
Art Unit 2151
/Justin R. Blaufeld/Primary Examiner, Art Unit 2151
Prosecution Insights