Detailed Action
Notice of Pre-AIA or AIA status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
I. Response to Applicant’s “Formal” Request for Supervisory Review under MPEP § 707.02
On page 9 of the Response, the Applicant says that it “formally requests review of the rejections, rejections [sic], and merits of this case and assistance in expediting prosecution of the subject application by the Supervisory Patent Examiner, James K. Trujillo.”
In response, the Supervisory Patent Examiner (“SPE”) was made aware of the Applicant’s request.
The Examiner also reviewed MPEP § 707.02, and observed that it does not say anything about a separate Applicant-initiated path for making a “formal request” of SPE review (i.e., distinct from the Applicant’s existing ability to call the SPE at the number listed on every Office Action). It also does not say anything about the SPE reviewing the primary examiner’s “rejections, rejections, and merits” of third action cases pursuant to the so-called “formal request.”
Rather, MPEP § 707.02 simply outlines the USPTO’s internal procedure for ensuring that SPEs “personally check on the pendency of every application which is up for the third or subsequent Office action with a view to finally concluding its prosecution.” (Emphasis added). The words “rejection,” “merits,” “request,” or synonyms thereof do not appear anywhere in that section of the MPEP, and indeed, merely checking on pendency is quite different from a mandated substantive review.
The Examiner assumes that the Applicant’s misunderstanding of MPEP § 707.02 was merely an oversight. However, now that the Applicant’s representative has written notice of the contents of MPEP § 707.02, he is reminded of his duty under 37 C.F.R § 11.303(a) not to make known false statements to the tribunal going forward. To that end, the Examiner trusts that the Applicant’s representative will refrain from inaccurately representing the contents of that chapter, either here or in other cases.
II. Clarification of Applicant’s Interview Summary
The Applicant’s interview summary (Response 10) says that “Applicant's representative and the Examiner discussed . . . that Yeh et al. lacks [] an interface that provides users to select which attributes to use for a given graphical element detection technique.” The Examiner wishes to clarify that the Applicant’s representative discussed why he believes Yeh lacks such an interface, while the Examiner discussed why he disagrees. (See Examiner Interview Summary for Oct. 22 Interview) (mailed Oct. 28, 2025) (recording, contemporaneously with the interview, that “[n]o agreement was reached”).
III. Rejection under 35 U.S.C. § 102
Claims 1–23 stand rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller, Sikuli: Using GUI Screenshots for Search and Automation, Proceedings of the 22nd annual ACM symposium on User interface software and technology (Oct. 4, 2009) (“Yeh”). The Applicant’s arguments have been considered, but are not persuasive.
In an effort to comply with the novelty requirement of 35 U.S.C. § 102 of the Patent Act, the Applicant submitted an amendment that it asserts to be narrower than what the Yeh reference discloses. According to the Applicant, “Yeh et al. lacks what one skilled in the art would consider to be a visual graphical element detection technique selection and configuration interface.” (Response 11) (emphasis added). The Examiner respectfully disagrees for several reasons.
For one, the Applicant offers absolutely no evidence about “what one skilled in the art would consider to be a visual graphical element detection technique selection and configuration interface,” apart from the opinion of its representative. “Arguments presented by applicant cannot take the place of evidence in the record.” MPEP § 2145 (subsection I.). Such evidence is especially necessary here, because the Applicant’s assertion of the amended language’s customary meaning belies the plain meaning of the words in the amendment: the word “visual” simply describes something that is perceivable by sight. There is nothing about the word “visual” that excludes the manual typing of functions into Yeh’s code editor, because Yeh’s code editor and the code that its user types into the code editor are indeed displayed on a screen, and visible to the human eye. See Yeh Figure 7.
Indeed, the Applicant’s assertion about the claim scope belies not only the plain meaning of “visual,” but also the Applicant’s own written description of the invention, which attempts to hedge the scope of the invention by explaining that in some embodiments, “the developer may enter the attributes manually,” in contrast to others, where he selects them from a list. (Spec. ¶ 22); (see also Spec. ¶ 82) (contemplating that the claimed graphical element detection techniques may be configured via “manual editing of XML”).
Furthermore, writing code is not even the exclusive mode of input for Yeh’s code editor. As the figures show, “user can click on the camera button (a) in the toolbar to enter the screen capture mode” as part of entering the find() function, and the code editor “can also preview how a pattern matches the current desktop (Figure 8) under different parameters.” Yeh 189 ¶¶ 1–2.
Additionally, even assuming for the sake of argument that the narrow claim interpretation asserted by the Applicant’s representative is correct, that narrow interpretation still reads on Yeh’s disclosure. This is because the Applicant is overlooking the word “facilitates,” which substantially broadens the scope of the claimed interface:
providing, by a designer application, a visual graphical element detection technique selection and configuration interface that facilitates selection of graphical element detection techniques
(Claim 1) (emphasis added).
By the plain terms of the claim, the so-called “visual graphical element detection technique selection and configuration interface” does not need to exclusively select the graphical element detection techniques; it merely facilitates selection of those techniques. In other words, even if the word “visual” somehow requires a purely code-free interface, such an interface doesn’t actually need to select the graphical element detection technique, it merely needs to assist the selection, or have some hand in its completion, no matter how small.
To that end, even if we ignore that typing code onto a visual display is indeed visual, the toolbar portion of the software is an interface that at least facilitates selection and configuration of a graphical element detection technique, and the toolbar is clearly “visual” within even the Applicant’s definition of that term, given its buttons for selecting a screenshot source (Yeh Fig. 7) and sliders for selecting similarity thresholds (Yeh Fig. 8). That is, if a user manually types the code, but then specifies parts of the code with the toolbar interface, he has at least used the toolbar interface to “facilitate” selection and configuration of the graphical element detection technique used by the now-complete code.
Accordingly, the Applicant’s arguments concerning “providing, by a designer application, a visual graphical element detection technique selection and configuration interface that facilitates selection of graphical element detection techniques of a plurality of different graphical element detection techniques” is not persuasive.
The Applicant’s arguments concerning whether users of Yeh’s software “select activities from a visual graphical element detection selection and configuration interface” (Response 13–15) are not persuasive either. Here, the Applicant raises several arguments, none of which are persuasive.
First, before addressing the Applicant’s explicit arguments, the Examiner will address an argument that the Applicant seemingly raises through implication, rather than an explicitly written argument. Specifically, the Examiner observes that the Applicant’s amendment changes the name of the previously-claimed “selection interface” to a “visual graphical element detection technique selection and configuration interface,” and that the Applicant repeatedly underlines the phrase “and configuration” throughout the arguments, perhaps to imply that the interface now has two “elements”—one for selection, and one for configuration.
If the Applicant is attempting to highlight this as a distinction, it is not persuasive for two reasons. First, merely tacking noun adjuncts (“selection and configuration”) onto the noun (“interface”) does not have the same effect as actually reciting the desired structure and/or function of the interface in the claim. To be sure, a “visual graphical element detection technique selection and configuration interface” needs to provide a visual mechanism for selecting and configuring a graphical element detection technique, but it does not need to receive separate inputs to achieve both the selection and the configuration, because the Applicant simply has not recited that in the claim yet. In fact, the presence of claim 2 (which automatically configures the graphical element detection techniques) requires the scope of “selection and configuration interface” to include interfaces where a single input for merely selecting a graphical element detection technique results in both the selection and configuration of said technique (e.g., because the interface pre-configures the technique selected by the user). See Littelfuse, Inc. v. Mersen USA EP Corp., 29 F. 4th 1376, 1380 (Fed. Cir. 2022) (citing Baxalta Inc. v. Genentech, Inc., 972 F.3d 1341, 1346 (Fed. Cir. 2020)) (“if a dependent claim reads on a particular embodiment of the claimed invention, the corresponding independent claim must cover that embodiment as well”).
Second, Yeh’s script editor is, in fact, an interface that provides for both the selection and configuration of visual graphical element detection techniques. Users can select the graphical element detection technique by typing the name of the technique (e.g., the find() function), and configure it by specifying, with the visual interface, the source of the query image that the find() function will use to find on-screen elements. See Yeh 189 ¶ 1.
Moving on to the Applicant’s explicit arguments, we begin with the argument that “users do not select activities from a visual graphical element detection technique selection and configuration interface in Yeh,” because “users code functions into the code editor thereof, per the above.” (Response 14). In other words, this argument repeats all of the arguments from earlier that typing into an on-screen program is somehow non-visual. The Examiner’s response to those arguments is therefore the same.
The Applicant’s next argument, “that there is no activity of an RPA workflow, as claimed under any interpretation that would be permissible under Phillips and Toro,” (Response 14) is also unpersuasive. This argument is unpersuasive because, later in the Response, the Applicant reveals that its idea of a permissible claim interpretation is one that demands identity of terminology. (See Response 15) (arguing that Yeh uses different words from the words of the claim).
Moreover, the terms “activity of an RPA workflow” have been interpreted in exactly the same manner asserted by the Applicant, well within the bounds of Phillips and Toro.1 That is, the Applicant points to paragraph 44 of its specification to describe the scope of this term without any further explanation, even though the rejection explicitly points to paragraph 44 and explains how the scope corresponds to Yeh’s disclosure. (See Non-Final Office Action 3–4, ¶¶ 9–10). By simply repeating the quote from the specification without even addressing the Examiner’s reading of that same passage, the Applicant is ignoring the findings presented in the rejection. Arguments that aren’t grounded in the facts of the rejection cannot possibly persuade the Examiner of an error.
Moreover, per the Applicant’s own definition, an RPA workflow is “a custom set of steps,” where each step is an “activity,” and each activity “may include an action, such as clicking a button, reading a file, writing to a log panel, etc.” (Spec. ¶ 44). The custom set of action commands in each of Yeh’s Sikuli scripts are activities, because they specify an action to perform in a user interface responsive to the user interface conforming to a rule for that action. And since Yeh’s Sikuli scripts comprise one or more such activities, they indeed fall within the scope of the claimed RPA workflows, exactly as they are described in the Applicant’s specification. There is no daylight between the Applicant’s interpretation of this claim limitation and Sikuli’s disclosure of it. This argument is not persuasive.
The Applicant’s next argues that “Yeh et al. still does not cure the deficiencies of Hanke et al. that were identified by the Board in the dissent.” (Response 14). Respectfully, this entire argument appears to be some kind of typographical error: Hanke was not applied in the last rejection, nor any other Office Action during the prosecution of this application (let alone the rejection leading to the Board decision), nor did the dissent ever mention Hanke, nor are dissents binding on the examiner, nor is there even any obviousness rejection on the record for which the “cure” of a deficiency is asserted. The Applicant’s representative might have been thinking about a reversal written by the same judge in sister application 17/693,791, but even there, Hanke’s deficiency was not with respect to the scope of an RPA workflow activity. See Ex parte Dines, PTAB Appeal No. 2024-001056 at *4 (July 31, 2025) (acknowledging that Hanke relates to automated testing of RPA scripts).
The Applicant next argues that both “the getActiveWindow() and find() functions of Yeh et al. are not similar to what is claimed.” (Response 14). The Applicant argues these two functions separately, so the Examiner will address them separately.
With respect to find(), the Applicant’s arguments completely ignore the findings set forth in the rejection, by mischaracterizing the findings with respect to the find() function as “using the find() function multiple times.” (Response 14). That is not what Yeh or the rejection say. Both Yeh and the rejection show that there are multiple versions of the find() function, each using different graphical detection techniques, and that these multiple versions may be used together to perform a single action—hence disclosing multiple different graphical element detection techniques in a single activity. See Non-Final Office Action 4–5, ¶¶ 13–15. Since the Office Action provided direct evidence of the find() function using multiple graphical element detection techniques in the same activity, “the burden shifts to the applicant to come forward with arguments and/or evidence to rebut the prima facie case.” MPEP § 2145. The Applicant’s naked assertion that Yeh is merely “using the find() function multiple times,” when the rejection provided the Applicant with notice of much more than that, fails to rebut the burden shifted by the prima facie case. The Applicant needs to explain what it believes is incorrect about the findings given in the Office Action in order to shift the burden back to the Office.
Likewise, the Applicant’s argument for the getActiveWindow() function is yet another bare assertion that fails to grapple or even acknowledge the Office’s findings for the claim scope and Yeh’s disclosure of the function. Specifically, the Applicant’s arguments never attempt to challenge the Examiner’s interpretation of the claimed graphical element detection techniques, but this interpretation is exactly what allows the getActiveWindow() function to fall within the scope of a graphical element detection technique.
That is, working with the principles of claim construction from Phillips, the rejection finds that the broadest reasonable interpretation of “graphical element detection technique” includes the use of selectors. The rejection explains that this is a reasonable interpretation, because dependent claim 10 and the Specification both explicitly say that “selectors” are a type of graphical element detection technique. It is reasonable to look to dependent claim 10 for evidence as to the meaning of “graphical element detection technique” in its parent claim 1 because, “[b]y definition, an independent claim is broader than a claim that depends from it, so if a dependent claim reads on a particular embodiment of the claimed invention, the corresponding independent claim must cover that embodiment as well.” Littelfuse, Inc. v. Mersen USA EP Corp., 29 F. 4th 1376, 1380 (Fed. Cir. 2022); see also Phillips v. AWH Corp., 415 F. 3d 1303, 1314 (Fed. Cir. 2005) (“Quite apart from the written description and the prosecution history, the claims themselves provide substantial guidance as to the meaning of particular claim terms,” including the usage of terms across different claims). Moreover, the specification provides additional evidence confirming that selectors are within the scope of graphical element detection techniques. (See Spec. ¶¶ 3, 4, 19, 21, 23, 26, 30, and 31).
The rejection then goes on to explain that the getActiveWindow() function falls within the scope of a “selector” graphical element detection technique, because it uses the operating system’s window hierarchy to find the window. As the rejection, the getActiveWindow() function falls within the scope of a “selector” graphical element detection technique because it uses the operating system’s window hierarchy to find the window. See Spec. ¶¶ 25–29.
All of these findings were provided to the Applicant in the previous Office Action, yet the Applicant’s remarks never address them. Arguments that do not engage with the Office Action’s findings cannot possibly show the error of those findings. Accordingly, this argument is not persuasive.
Lastly, the Applicant argues that “it is unclear to Applicant how Yeh et al. is believed to disclose an RPA workflow,” because Yeh’s mention of the word “script” does not imply the specification’s definition at paragraph 44. The Examiner respectfully disagrees. As previously explained, the paragraph 44 definition says that an RPA workflow is “a custom set of steps,” where each step is an “activity,” and each activity “may include an action, such as clicking a button, reading a file, writing to a log panel, etc.” (Spec. ¶ 44). Likewise, the custom set of action commands in each of Yeh’s Sikuli scripts are activities, because they specify an action to perform in a user interface responsive to the user interface conforming to a rule for that action. And since Yeh’s Sikuli scripts comprise one or more such activities, they indeed fall within the scope of the claimed RPA workflows, exactly as they are described in the Applicant’s specification. There is no daylight between the Applicant’s interpretation of this claim limitation and Sikuli’s disclosure of it. This argument is not persuasive.
The Applicant’s remaining arguments consist of reasserting the claim 1 arguments with respect to its dependent claims and similarly recited independent claims. See Response 17. Accordingly, the Examiner’s response to those arguments is the same as the response to the arguments for claim 1.
For these reasons, the claims stand rejected under 35 U.S.C. § 102, the application is not in condition for allowance, and the Applicant’s request for a notice of allowance is respectfully denied.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1–23 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller, Sikuli: Using GUI Screenshots for Search and Automation, Proceedings of the 22nd annual ACM symposium on User interface software and technology (Oct. 4, 2009) (“Yeh”).
The Yeh reference is available at <https://doi.org/10.1145/1622176.1622213>. Those reviewing this rejection are encouraged to download the original version of the reference, as it is published with full color.
Claim 1
Yeh discloses:
A computer-implemented method for detecting graphical elements in a user interface (UI), comprising:
Yeh discloses “Sikuli Script,” which is “a scripting system that enables programmers to use screenshots of GUI elements to control them programmatically.” Yeh 184. The framework, and the scripts that implement the framework, are written as Java and Python implementations that run on “Windows and Mac OS X” including “a 3.2 GHz Windows PC.” Yeh 189. “Under the principles of inherency, if a prior art device, in its normal and usual operation, would necessarily perform the method claimed, then the method claimed will be considered to be anticipated by the prior art device.” MPEP § 2112.02 (subsection I.).
providing, by a designer application, a visual graphical element detection technique selection and configuration interface that facilitates selection of graphical element detection techniques of a plurality of different graphical element detection techniques for individual activities of a robotic process automation (RPA) workflow on a per-activity basis;
The Sikuli Script system provides “an editor to help users write visual scripts (Figure 7).” Yeh 189 ¶ 1. Using either the editor’s toolbar (or by typing the commands), the user adds find() commands that each locate a particular on-screen GUI element, and corresponding action commands that that constitute the script. Yeh 189 ¶ 1.
According to the Applicant, an RPA workflow is “a custom set of steps,” where each step is an “activity,” and each activity “may include an action, such as clicking a button, reading a file, writing to a log panel, etc.” (Spec. ¶ 44). Therefore, each claimed RPA workflow corresponds to a Sikuli script developed using the editor, while the claimed activities correspond to the actions commands in said Sikuli script.
Much like claim 1 requires, Sikuli scripts generally use the find() function on a per-action basis: the find() function “locates a particular GUI element to interact with,” Yeh 188 ¶ 1, and its corresponding action command “specif[ies] what keyword and/or mouse events [are] to be issued to the center of a region found by find().” Yeh 188 ¶ 9. Furthermore, the find() functions may be written such that they utilize several different graphical element detection techniques. See Yeh 188 ¶¶ 2–8.
receiving, by the designer application, a selection from the visual graphical element detection technique selection and configuration interface of an activity to be configured in the RPA workflow;
A developer may type an action directly into the Sikuli editor interface, in order to add it to a Sikuli script. Yeh 189. For example, in Figure 7, a user has added the click() function with the intent to have the function click a file icon that is found using the find() function.
receiving, by the designer application, selections of and/or modifications to one or more graphical element detection techniques of the plurality of different graphical element detection techniques, one or more UI descriptor attributes of the one or more different graphical element detection techniques, or a combination thereof, for the selected activity;
The developer may further type or use the toolbar to add corresponding find() function in the editor, thereby providing an instruction for the Sikuli script to find the icon that will be clicked by the click() function. Yeh 189. The editor is also able to receive modifications to find() functions that are already in the script by providing buttons inside the find() functions, which the user can click to provide a screenshot or other image to use as the search pattern for the find() command. Yeh 189 ¶ 1.
Regarding the “two or more graphical element detection techniques of the plurality of different graphical element detection techniques” alternative limitation, Examples 4 and 5 (Yeh 190) show instances where the developer has typed scripts that each use two different graphical detection techniques for a single activity.
In Example 4, the single activity is the dragDrop() command on line 3, which is condition upon the two different find() functions on line 1 that “look[] for the Interstate 10 symbol and checks if a string Houston appears nearby.” Yeh 190. As Yeh explains earlier in the reference, providing the find() function with an image pattern causes find() to use a “computer vision algorithm” to find the image pattern, whereas, when providing the find() function with a string, “OCR is used to find screen regions matching the text of the string.” Yeh 188. The two find() functions on line 1 of Example 4 thus specify two different image detection techniques for the same dragDrop() action: computer vision and OCR.
In Example 5, the two or more graphical element detection techniques used for the same click() action (line 107) include the getActiveWindow() function on line 103 (which falls within the scope of the Applicant’s “selector” graphical element detection technique) and the find() function on line 107, which uses computer vision to find an image of a button but constrained to within the result of getActiveWindow(). Yeh 190.
The getActiveWindow() function falls within the scope of a “selector” graphical element detection technique because it uses the operating system’s window hierarchy to find the window. And “selectors” are considered a graphical element detection techniques within the meaning of claim 1 because dependent claim 10 explicitly lists selectors as a type of graphical element detection technique.2
Regarding the “one or more UI descriptor attributes of the two or more graphical element detection techniques” alternative limitation, at a minimum, by prompting for this screenshot or image and receiving it from the developer, Yeh’s editor receives a selection and/or modification of one or more UI descriptor attributes of the two or more graphical element detection techniques: the UI descriptor attribute is the query image, and the query image is “of” two or more graphical element detection techniques because the find() function is implemented with “a hybrid method that uses template-matching for finding small patterns and invariant feature voting for finding large patterns.” Yeh 187. In other words, the image (and its inherent size) are the UI descriptors that are “of” the two alternative detection techniques provided in the find() function (template-matching and invariant feature voting, respectively).
Regarding the “combination thereof” alternative limitation, Example 4 is applicable here, because it uses two graphical element detection techniques (computer vision and OCR), and also uses one or more UI descriptor attributes (the Interstate 10 symbol and the “Houston” string).
and configuring the selected single activity, by the designer application, based on the received selections and/or modifications.
“Finally, the user can press the execute button (d) and the editor will be hidden and the script will be executed.” Yeh 189 ¶ 1.
Claim 2
Yeh discloses the computer-implemented method of claim 1, further comprising:
automatically configuring one or more of the plurality of different graphical element detection techniques, one or more of the UI descriptor attributes, or both, by the designer application.
One example falling within the scope of claim 2 is the editor’s code completion feature: “When the user types a command, the editor automatically displays the corresponding command template to remind the user what arguments to supply.” Yeh 189 ¶ 1.
A different example that falls within the scope of claim 2 is the find() function itself, which automatically decides whether to use the template-matching algorithm or the invariant feature voting algorithm based on the size of the of the image pattern provided for the search. Yeh 187.
Claim 3
Yeh discloses the computer-implemented method of claim 2,
wherein the automatic configuration is performed by the designer application based on an action implemented by the activity, a type of the target graphical element, a presence of one or more other graphical elements in the UI, or a combination thereof.
The find() function automatically decides whether to use the template-matching algorithm or the invariant feature voting algorithm based on the size of the of the image pattern provided for the search. Yeh 187. Thus, Yeh’s find function is at least automatically configured based on the type of the target graphical element. Claim 3 only requires the presence of one alternative limitation in the list of limitations to reach a finding of anticipation.
Claim 4
Yeh discloses the computer-implemented method of claim 1,
wherein the process of claim 1 is repeated for at least one additional activity in the RPA workflow.
“We present six example scripts to demonstrate the basic features of Sikuli Script.” Yeh 189. In other words, the process of using the Sikuli editor may be repeated six times to produce six different scripts.
Claim 5
Yeh discloses the computer-implemented method of claim 1, further comprising:
generating an RPA robot configured to implement the configured activity, by the designer application.
Scripts typed in the editor are made available for execution via “Jython.” Yeh 189 ¶ 3. Sikuli scripts “use screenshots of GUI elements directly in an automation script to programmatically control the elements with low-level keyboard and mouse input.” Yeh 187.
Claim 6
Yeh discloses the computer-implemented method of claim 5,
wherein the RPA robot is configured to use a subset of the UI descriptor attributes for at least one of the plurality of different graphical element detection techniques.
The find() function may be invoked using only an image pattern, or only a string (for the OCR functionality), rather than using the entirety of parameters provided by the Sikuli API (e.g., the constraint operators, the tuning functions, combinations of images with text, etc.). See Yeh 188.
Claim 7
Yeh discloses the computer-implemented method of claim 5, further comprising:
analyzing a UI at runtime, by the RPA robot, to identify UI element attributes;
“The find() function locates a particular GUI element to interact with. It takes a visual pattern that specifies the element’s appearance, searches the whole screen or part of the screen, and returns regions matching this pattern or false if no such region can be found.” Yeh 188 ¶ 1.
comparing the UI element attributes to the configured UI descriptor attributes for the activity, by the RPA robot;
“When created from an image, the computer vision algorithm described earlier is used to find matching screen regions. When created from a string, OCR is used to find screen regions matching the text of the string.” Yeh 188 ¶ 2.
and responsive to a UI element matching the attributes of the plurality of different graphical element detection techniques being found via an exact match or a threshold match: taking an action associated with the activity involving the UI element, by the RPA robot.
The find() function will look for either an exact match, or for one that meets a similarity threshold, depending on whether the pattern passed into find is decorated with the exact() or similar() decorator functions. Yeh 188 ¶ 3. Then, “[t]he action commands specify what keyword and/or mouse events to be issued to the center of a region found by find(),” such as clicking or typing text. Yeh 188 ¶ 9.
Claim 8
Yeh discloses the non-transitory computer-readable medium of claim 7,
wherein the UI attributes comprise images, text,
The find() function takes a Pattern object as an input, and that Pattern object “can be created from an image or a string of text.” Yeh 188 ¶ 2.
relationships between graphical elements in the UI,
“To support other types of constrained search, our visual scripting API provides a versatile set of constraint operators: left, right, above, below, nearby.” Yeh 188 ¶ 7.
a hierarchical representation of the graphical elements in the UI,
Yeh discloses two different ways to specify a hierarchical relationship as the UI attribute for comparison. The first way is to use a chain of find() commands, thereby constraining the search space of the second find() (e.g., for a specified button) to only the re-gion occupied by the result of the first find() (e.g., a parent window containing the specified button).
The second way is to simply use the inside() constraint operator, which returns a representation of the inside of a given Region. See Yeh 188 ¶ 7 (top of second column).
or a combination thereof.
“These operators can be used in combination to express a rich set of search semantics.” Yeh 188 ¶ 7.
Claim 9
Yeh discloses the non-transitory computer-readable medium of claim 7,
wherein the computer program steps of claim 7 are repeated for at least one additional activity.
“We present six example scripts to demonstrate the basic features of Sikuli Script.” Yeh 189.
Claim 10
Yeh discloses the non-transitory computer-readable medium of claim 1,
wherein types of the UI descriptors comprise two or more of
Since parent claim 1 never uses the word types, and claim 4 never refers back to the “compare” step of its parent claim, claim 4 is ambiguous as to whether the two or more “types” must be either (i) part of the comparison in claim 1, or if instead, (ii) claim 4 is listing a group of UI descriptor types that are merely available for inclusion into the plurality. For brevity, this rejection will show why Yeh anticipates the narrower interpretation, since anticipation of a narrow embodiment necessarily anticipates the broader genus that the narrow embodiment falls within. See MPEP § 2131.02.
Yeh’s disclosure of each element in the list will be provided below. As for why Yeh specifically discloses the narrow interpretation (i), this is disclosed on page 188, where Yeh explains how to chain different constraints and find() algorithms together.
a selector,
The win32gui library may be imported into a Sukuli script “to provide the function getActiveWindow(),” which “obtain[s] the handle to the active window (103).” Yeh 190.
a computer vision (CV) descriptor,
In cases where the find() command is run using a large image pattern, find() executes “an algorithm based on invariant local features such as SIFT.” Yeh 187; see also Yeh 188 (“When created from an image, the computer vision algorithm described earlier is used to find matching screen regions.”)
an image matching descriptor,
An image-based pattern object may also be tuned with the “exact()” function, to “[r]equire matches to be identical to the given search pattern pixel-by-pixel.” Yeh 188.
and an optical character recognition (OCR) descriptor.
“When [a search pattern is] created from a string, OCR is used to find screen regions matching the text of the string.” Yeh 188.
Claims 11–17
Claims 11–17 are directed to a computer program product on which substantially the same computer-implemented method of claims 1-6 and 10 are encoded, and is therefore rejected according to the same corresponding findings and rationale as provided above.
Claim 18
Yeh discloses
A computing system, comprising: memory storing computer program instructions for detecting graphical elements in a user interface (UI); and at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:
Yeh discloses “Sikuli Script,” which is “a scripting system that enables programmers to use screenshots of GUI elements to control them programmatically.” Yeh 184. The framework, and the scripts that implement the framework, are written as Java and Python implementations that run on “Windows and Mac OS X” including “a 3.2 GHz Windows PC.” Yeh 189.
provide a visual graphical element detection technique selection and configuration interface that facilitates selection of graphical element detection techniques of a plurality of different graphical element detection techniques for individual activities of a robotic process automation (RPA) workflow on a per-activity basis;
The Sikuli Script system provides “an editor to help users write visual scripts (Figure 7).” Yeh 189 ¶ 1. Using either the editor’s toolbar (or by typing the commands), the user adds find() commands that each locate a particular on-screen GUI element, and corresponding action commands that that constitute the script. Yeh 189 ¶ 1.
According to the Applicant, an RPA workflow is “a custom set of steps,” where each step is an “activity,” and each activity “may include an action, such as clicking a button, reading a file, writing to a log panel, etc.” (Spec. ¶ 44). Therefore, each claimed RPA workflow corresponds to a Sikuli script developed using the editor, while the claimed activities correspond to the actions commands in said Sikuli script.
Much like claim 1 requires, Sikuli scripts generally use the find() function on a per-action basis: the find() function “locates a particular GUI element to interact with,” Yeh 188 ¶ 1, and its corresponding action command “specif[ies] what keyword and/or mouse events [are] to be issued to the center of a region found by find().” Yeh 188 ¶ 9. Furthermore, the find() functions may be written such that they utilize several different graphical element detection techniques. See Yeh 188 ¶¶ 2–8.
receive selections of multiple graphical element detection techniques of the plurality of different graphical element detection techniques for a single activity;
A developer may type an action directly into the Sikuli editor interface, in order to add it to a Sikuli script. Yeh 189. For example, in Figure 7, a user has added the click() function with the intent to have the function click a file icon that is found using the find() function.
The developer may further type or use the toolbar to add corresponding find() function in the editor, thereby providing an instruction for the Sikuli script to find the icon that will be clicked by the click() function. Yeh 189. The editor is also able to receive modifications to find() functions that are already in the script by providing buttons inside the find() functions, which the user can click to provide a screenshot or other image to use as the search pattern for the find() command. Yeh 189 ¶ 1.
Regarding the “two or more graphical element detection techniques of the plurality of different graphical element detection techniques” alternative limitation, Examples 4 and 5 (Yeh 190) show instances where the developer has typed scripts that each use two different graphical detection techniques for a single activity.
In Example 4, the single activity is the dragDrop() command on line 3, which is condition upon the two find() functions on line 1 that “look[] for the Interstate 10 symbol and checks if a string Houston appears nearby.” Yeh 190. As Yeh explains earlier in the reference, providing the find() function with an image pattern causes find() to use a “computer vision algorithm” to find the image pattern, whereas, when providing the find() function with a string, “OCR is used to find screen regions matching the text of the string.” Yeh 188. The two find() functions on line 1 of Example 4 thus specify two different image detection techniques for the same dragDrop() action: computer vision and OCR.
In Example 5, the two or more graphical element detection techniques used for the same click() action (line 107) include the getActiveWindow() function on line 103 (which falls within the scope of the Applicant’s “selector” graphical element detection technique) and the find() function on line 107, which uses computer vision to find an image of a button but constrained to within the result of getActiveWindow(). Yeh 190.
The getActiveWindow() function falls within the scope of a “selector” graphical element detection technique because it uses the operating system’s window hierarchy to find the window.
Regarding the “one or more UI descriptor attributes of the two or more graphical element detection techniques” alternative limitation, at a minimum, by prompting for this screenshot or image and receiving it from the developer, Yeh’s editor receives a selection and/or modification of one or more UI descriptor attributes of the two or more graphical element detection techniques: the UI descriptor attribute is the query image, and the query image is “of” two or more graphical element detection techniques because the find() function is implemented with “a hybrid method that uses template-matching for finding small patterns and invariant feature voting for finding large patterns.” Yeh 187. In other words, the image (and its inherent size) are the UI descriptors that are “of” the two alternative detection techniques provided in the find() function (template-matching and invariant feature voting, respectively).
Regarding the “combination thereof” alternative limitation, Example 4 is applicable here, because it uses two graphical element detection techniques (computer vision and OCR), and also uses one or more UI descriptor attributes (the Interstate 10 symbol and the “Houston” string).
and configure the single activity based on the received graphical element detection technique selections to utilize the selected multiple graphical element detection techniques.
“Finally, the user can press the execute button (d) and the editor will be hidden and the script will be executed.” Yeh 189 ¶ 1.
Claims 19–23
The additional instructions recited in claims 19–23 cause the computing system to perform the same steps as recited in claims 2–6, and are therefore rejected according to the same findings and rationale as provided above for those claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Justin R. Blaufeld whose telephone number is (571)272-4372. The examiner can normally be reached M-F 9:00am - 4:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James K Trujillo can be reached at (571) 272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Justin R. Blaufeld
Primary Examiner
Art Unit 2151
/Justin R. Blaufeld/Primary Examiner, Art Unit 2151
1 The Applicant’s representative is reminded that Phillips and Toro are infringement cases, and that “the USPTO is not required, in the course of prosecution, to interpret claims in applications in the same manner as a court would interpret claims in an infringement suit.” MPEP § 2111 (citing In re Morris, 127 F.3d 1048, 1054-55 (Fed. Cir. 1997))
2 See Littelfuse, Inc. v. Mersen USA EP Corp., 29 F. 4th 1376, 1380 (Fed. Cir. 2022) (citing Baxalta Inc. v. Genentech, Inc., 972 F.3d 1341, 1346 (Fed. Cir. 2020) ("By definition, an independent claim is broader than a claim that depends from it, so if a dependent claim reads on a particular embodiment of the claimed invention, the corresponding independent claim must cover that embodiment as well.").
Prosecution Insights