Detailed Action
Notice of Pre-AIA or AIA status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
I. Response to Applicant’s “Formal” Request for Supervisory Review under MPEP § 707.02
On page 8 of the Response, the Applicant says that it “formally requests review of the rejections, rejections [sic], and merits of this case and assistance in expediting prosecution of the subject application by the Supervisory Patent Examiner, James K. Trujillo.”
In response, the Supervisory Patent Examiner (“SPE”) was made aware of the Applicant’s request.
The Examiner also reviewed MPEP § 707.02, and observed that it does not say anything about a separate Applicant-initiated path for making a “formal request” of SPE review (i.e., distinct from the Applicant’s existing ability to call the SPE at the number listed on every Office Action). It also does not say anything about the SPE reviewing the primary examiner’s “rejections, rejections, and merits” of third action cases pursuant to the so-called “formal request.”
Rather, MPEP § 707.02 simply outlines the USPTO’s internal procedure for ensuring that SPEs “personally check on the pendency of every application which is up for the third or subsequent Office action with a view to finally concluding its prosecution.” (Emphasis added). The words “rejection,” “merits,” “request,” or synonyms thereof do not appear anywhere in that section of the MPEP, and indeed, merely checking on pendency is quite different from a mandated substantive review.
The Examiner assumes that the Applicant’s misunderstanding of MPEP § 707.02 was merely an oversight. However, now that the Applicant’s representative has written notice of the contents of MPEP § 707.02, he is reminded of his duty under 37 C.F.R § 11.303(a) not to make known false statements to the tribunal going forward. To that end, the Examiner trusts that the Applicant’s representative will refrain from inaccurately representing the contents of that chapter, either here or in other cases.
II. Clarification of Applicant’s Interview Summary
The Applicant’s interview summary (Response 9) says that “Applicant’s representative and the Examiner discussed . . . that Yeh et al. lacks [] an interface that provides users to select which attributes to use for a given graphical element detection technique.” The Examiner wishes to clarify that the Applicant’s representative discussed why he believes Yeh lacks such an interface, while the Examiner discussed why he disagrees. (See Examiner Interview Summary for Oct. 22 Interview) (mailed Oct. 28, 2025) (recording, contemporaneously with the interview, that “[n]o agreement was reached”).
III. Rejection under 35 U.S.C. § 102
Claims 1–5, 7–13, 15–20, 22, and 23 stand rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller, Sikuli: Using GUI Screenshots for Search and Automation, Proceedings of the 22nd annual ACM symposium on User interface software and technology (Oct. 4, 2009) (“Yeh”).
In response, the Applicant amended independent claims 1, 9, and 16 to add limitations that describe the plurality of graphical element detection techniques being selected and configured by a user in a visual graphical element detection technique selection and configuration interface. The Applicant also amended dependent claims 8, 15, and 23 to require that using a subset of UI descriptor attributes is “responsive to selection by a user” in said visual interface. The rejection has been reconsidered in light of the amendments and the arguments, but is ultimately maintained. The Examiner’s responses to the Applicant’s arguments are as follows.
Starting with the Applicant’s arguments on pages 10–12 of the Response, the Applicant contends that Yeh lacks a “visual graphical element detection technique selection and configuration interface,” because Yeh’s Figures 7 and 8 merely depict a standard code editor where users manually write text-based functions, rather than an interface that allows a user to select and configure techniques (such as the checkbox and slider interface shown in Applicant’s Figure 6). This argument is unpersuasive for a few reasons.
First, as an initial matter, the entire description of the plurality of graphical element detection techniques being selected and configured by a user in a visual interface is not entitled to patentable weight because it describes design-time, pre-execution setup activity that occurs entirely before the claimed computer program performs the active execution step of comparing UI element attributes. In other words, the above claim language is merely a description of how the “compare” instruction was generated before it was ever even stored on the claimed computer readable medium, rather than what the “compare” instruction actually does. Descriptions of a computer program’s background environment do not limit its scope. See, e.g., Nazomi Communications, Inc., v. Nokia Corp., 739 F.3d 1339, 1345 (Fed Cir. 2014); Silicon Graphics, Inc. v. ATI Technologies, Inc., 607 F.3d 784, 794-95 (Fed. Cir. 2010); and Advanced Software Design Corporation v. Fiserv, Inc., 641 F.3d 1368, 1375 (Fed. Cir. 2011). Therefore, this pre-runtime selection and configuration activity does not patentably distinguish the claimed method from Yeh’s automated script execution—although Yeh’s Sikuli editor does indeed disclose what this claim language describes.
Second, even when the pre-execution configuration is given patentable weight, Yeh explicitly discloses a graphical user interface that functions exactly as the claimed visual selection and configuration interface. Specifically, with respect to the visual selection of techniques, Yeh discloses a “Script Editor” developed “to help users write visual scripts.” Yeh 189. Yeh’s toolbar (Figure 7) provides explicit, visual UI buttons that allow a user to select different graphical element input and detection strategies. For example, clicking button (a) (“Take a screenshot”) allows the user to visually capture an image pattern, which selects an image-matching or computer vision detection technique, while clicking button (c) (“Select a region”) allows the user to visually bound a specific desktop space, thereby selecting a spatial/region constraint or selector technique.
Furthermore, with respect to the visual configuration of parameters, Yeh’s preview interface shown in Figure 8 provides a direct visual configuration environment where the user can “adjust the similarity threshold and preview the results.” Yeh 189. The user interacts with a visual slider (a) and a “Number of matches” spinner (b) to directly configure the matching threshold attributes of the underlying detection algorithm. Because Yeh provides these visual tools (buttons, sliders, spinners) that allow the user to select how graphical elements are detected and visually configure their thresholds, Yeh discloses the claimed visual selection and configuration interface.
Third, notwithstanding this explicit disclosure, under the broadest reasonable interpretation, the act of typing code directly into the code editor of the Sikuli interface to programmatically write, chain, and parameterize find commands (such as utilizing the pattern classes, similarity levels, and spatial constraint operators) also falls within the broadly recited selection and configuration of graphical element detection techniques and their respective attributes. An interactive code entry area within a graphical user interface is a visual interface element that enables a user to select which detection functions are called and configure their associated parameters, thereby satisfying the functional claim language.
Moving on to pages 13–15 of the Response, the Applicant attacks the Examiner’s prior interpretations of “plurality of graphical element detection techniques,” arguing that Yeh uses a single technique at a time (e.g., a single call to find a button) and that running the same technique multiple times does not constitute utilizing a “plurality.” This argument is unpersuasive, because it is divorced from the reality of the claim language and the complete disclosure provided in the Yeh reference.
Yeh discloses using a plurality of different detection techniques to accomplish a single activity. First, Yeh explicitly discloses chaining different operators together to hone in on a single target for a single activity. For example, the script click(Region(w).inside().find(button)) (Yeh, Example 5, page 190) chains a spatial selector technique (inside()) with an image-matching technique (find()) to execute a single click() activity. The system evaluates the region attribute and the image descriptor attribute simultaneously to find the correct UI element.
Furthermore, Yeh’s core find() function operates using a “hybrid method” that automatically utilizes multiple techniques beneath the hood—specifically, “template-matching for finding small patterns and invariant feature voting” for finding large patterns. Yeh 187. Therefore, Yeh anticipates comparing UI element attributes to descriptor attributes from a plurality of techniques, in several different ways (many of which were presented in the rejection, yet never addressed in the Applicant’s remarks).
Next, on pages 15–17 of the Response, the Applicant argues that Yeh’s scripts do not constitute an “RPA workflow” and its commands do not represent “activities.” However, the Applicant’s main reasoning for this argument is that the word “workflow” never appears in Yeh, and that a script is merely code rather than structured workflow constructs designed to achieve a task.
This argument is unpersuasive, because reliance on a difference in nomenclature (“script” vs. “workflow”) ignores the functional reality of the software described. See MPEP § 2131 (explaining that “identity of terminology is not required” for a finding of anticipation under § 102).
As the Applicant notes in the Response, the specification defines a workflow broadly as a “custom set of steps developed in a workflow, defined herein as ‘activities’.” Yeh’s scripts are likewise an automated, rule-based sequence of custom steps executed to control a graphical user interface using low-level mouse and keyboard events—which is the exact functional definition of robotic process automation (RPA). For example, the recycleAll(x) script in Example 2 defines a sequence of steps to automate deleting files of multiple types by iterating through matching regions and dropping them in the recycle bin. Yeh 190. The command steps within Yeh’s scripts—such as click(), find(), and dragDrop()—map precisely to the claimed “activities” since they specify the actions involving the UI elements. Therefore, Yeh’s disclosure satisfies the broad definition of an RPA workflow and its corresponding activities. This argument does not persuade the Examiner of error.
Finally, on pages 17–19 of the Response, the Applicant argues that Yeh fails to disclose configuring a subset of UI descriptor attributes responsive to a user’s selection within a visual interface, as set forth in amended claim 8. This argument is unpersuasive.
As noted previously, Yeh’s Figures 7 and 8 provide visual controls that modify the attributes utilized by the pattern matching algorithms. When a user manipulates the visual similarity slider in Figure 8, they are visually selecting a specific subset of similarity threshold attributes for the algorithm (e.g., allowing fuzzy matches) while opting not to use the entirety of the exact() pixel-by-pixel matching attributes. Because the application of these threshold limits occurs responsively to the user interacting with the visual slider, Yeh discloses utilizing a subset of UI descriptor attributes responsive to selection in a visual interface.
In sum, the structural and behavioral features added by the Applicant’s amendments are fully disclosed by the visual script creation toolbar, preview window, and underlying hybrid image-matching algorithms of the Yeh reference under a proper broadest reasonable construction. The prior art reference continues to meet every element of the independent and dependent claims, and therefore, the rejection of claims 1–5, 7–13, 15–20, 22, and 23 under 35 U.S.C. § 102(a)(1) is maintained.
Since all claims stand rejected over the prior art, the request for a Notice of Allowance (Response 20) is respectfully denied.
Information Disclosure Statement
The information disclosure statement filed on December 10, 2025 complies with the provisions of 37 C.F.R. § 1.97, 1.98, and MPEP § 609, and therefore has been placed in the application file. The information referred to therein has been considered as to the merits.
Claim Rejections – 35 U.S.C. § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1–5, 7–13, 15–20, 22, and 23 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller, Sikuli: Using GUI Screenshots for Search and Automation, Proceedings of the 22nd annual ACM symposium on User interface software and technology (Oct. 4, 2009) (“Yeh”).
The Yeh reference is available at <https://doi.org/10.1145/1622176.1622213>. Those reviewing this rejection are encouraged to download the original version of the reference, as it is published with full color.
Claim 1
Yeh discloses:
A non-transitory computer-readable medium storing a computer program, the computer program configured to cause at least one processor to:
The Yeh reference discloses “Sikuli Script,” which is “a scripting system that enables programmers to use screenshots of GUI elements to control them programmatically.” Yeh 184. The reference also discloses “several example scripts” that use the Sikuli Script software framework. Yeh 187. The framework, and the scripts that implement the framework, are written as Java and Python that run on “Windows and Mac OS X” including “a 3.2 GHz Windows PC,” Yeh 189, and therefore constitute a “computer program configured to cause at least one processor” to perform their respective steps, within the meaning of claim 1.
Like any computer program, these scripts necessarily must be read by a computer in order for the computer to execute them, and therefore, Yeh at least inherently discloses that the foregoing program(s) are stored on a “computer-readable” medium.
compare user interface (UI) element attributes to UI descriptor attributes from a plurality of different graphical element detection techniques
Sikuli scripts use the “find()” command to invoke different versions of functions that “take[] a target pattern and returns screen regions matching the pattern” by performing the claimed comparison (which will be discussed in detail below). Yeh 187.
There are at least three different ways to interpret the claim language of “UI descriptor attributes from a plurality of different graphical element detection techniques,” and Yeh’s disclosure anticipates comparing UI element attributes to all three interpretations of UI descriptor attributes from a plurality of graphical element detection techniques:
(1) Narrowest Interpretation. Under the narrowest interpretation, there are three requirements: (i) each one of the plurality of graphical element detection techniques must be a different technique (as opposed to applying the same technique a plurality of times), (ii) each different graphical element technique has its own UI descriptor attributes that are not shared with the other detection techniques (e.g., if a first detection technique considers the size of an element, the second technique must consider something other than size), and (iii) the whole plurality of graphical element detection techniques must execute (i.e., it’s not enough for one detection technique to serve as the fallback when another one fails). This is a reasonable interpretation (albeit not necessarily the broadest reasonable interpretation) because it is consistent with at least paragraph 21 of the specification. See MPEP § 2111 (“the meaning given to a claim term . . . must be consistent with the use of the claim term in the specification and drawings.”).
Yeh discloses a first instance of this narrow interpretation with Example 5 on page 190: In order to find a button on the screen, line 103 of the Example 5 script calls the getActiveWindow() function to first find a window that might contain the button, and then line 107 calls the find() function on the content inside the window, passing a reference image of the button (within the “button” variable) as the Pattern object for find()’s matching algorithm. In Example 5, the getActiveWindow() function is a “selector” graphical element detection technique (e.g., within the meaning of dependent claim 4)—it uses a window hierarchy to find the window—and its UI descriptor attribute is that the window is whichever one the Windows API deems “active.” Likewise, this instance of the find() function falls within the scope of either a CV detection technique or an image matching detection technique, depending which version of find() executes—depending on the size of the button, find() either performs “tem-plate matching based on normalized cross-validation” (image matching) or uses “an algorithm based on invariant local features such as SIFT” (CV). Yeh 187. Thus, Example 5 uses two different graphical element detection techniques, and in each detection technique compares a different UI attribute to a different
Yeh discloses another example of the narrowest interpretation on page 188, by disclosing that multiple find() operators can be chained together to hone-in on a target graphical element, and further disclosing that each run of find() uses a different algorithm—computer vision or OCR—depending on whether it is provided with an image or text as its parameter. Yeh 188. Putting these concepts together, Yeh discloses that Sikuli framework can search for a GUI element using two different techniques and two different descriptor attributes: one find() in the chain is given text as its descriptor attribute and so it uses OCR, while another find() in the chain is given an image as its descriptor attribute, and so it uses a computer vision algorithm. Example 4 discloses a similar case, where both OCR and computer vision are used for the same dragDrop() activity. See Yeh 190.
(2) Broader Interpretation. Under a broader (but still reasonable) interpretation, the prior art must disclose an RPA process that is programmed with a plurality of different detection techniques, but some of the plurality may never run during a given execution of that RPA process, instead serving only as backup contingencies, used only when a primary detection technique fails. This interpretation is also reasonable, because it is consistent with paragraph 31 of the Applicant’s disclosure.
Yeh discloses this interpretation every time the find() function is invoked using an image pattern, even in Sikuli scripts that only call find() once. This is because Sikuli implements find() with “a hybrid method that uses template-matching for finding small patterns and invariant feature voting for finding large patterns.” Yeh 187. In other words, the image (and its inherent size) are the UI descriptors that are “from” a plurality of detection techniques (template-matching and invariant feature voting, respectively). For instance, in the case of a Sikuli script that invokes the find() function only one time on a single image pattern, find() uses the size of the image in the pattern to determine which one of the plurality of techniques to use, and applies whichever technique conforms to the threshold.
for an activity of a robotic process automation (RPA) workflow,
According to the Applicant, an RPA workflow is “a custom set of steps,” where each step is an “activity,” and each activity “may include an action, such as clicking a button, reading a file, writing to a log panel, etc.” (Spec. ¶ 44). The find() function’s comparisons are likewise performed “for” an activity of a robotic process automation (RPA) workflow, because the comparisons they perform (by matching UI elements to patterns) determine which buttons are to be clicked by the action commands. See Yeh 188 (“The find() function locates a particular GUI element to in-teract with . . . . The action commands specify what keyword and/or mouse events to be issued to the center of a region found by find().”).
In other words, the claimed RPA workflows are mapped to Yeh’s Sikuli scripts, and the claimed activity/activities are mapped to the action command(s) invoked within those scripts.
the plurality of graphical element detection techniques selected and configured by a user in a visual graphical element detection technique selection and configuration interface;
Yeh discloses the above claim language, even when given full patentable weight, for reasons that will be discussed below. However, before discussing those reasons, those reviewing this rejection should take note that the above claim language should not be entitled to patentable weight, because the above claim language describes activity that isn’t performed by the claimed computer program instruction.
Remember: the above claim language refers back to the computer program instruction of using an already-selected and already-configured graphical element detection technique to compare the UI element attributes to the UI descriptor attributes. We can confirm this by recognizing the past tense, passive language of “selected” and “configured,” rather than reciting those steps as actual steps taken by the claimed computer program. Additionally, the second sentence of paragraph 91 in the Specification expressly confirms that the selection and configuration occurs in a separate process 800 from the process 900 recited in claim 1.
In other words, the above claim language is merely a description of how the “compare” instruction was written, rather than what the “compare” instruction actually does. Descriptions of a computer program’s environment do not limit its scope. See, e.g., Nazomi Communications, Inc., v. Nokia Corp., 739 F.3d 1339, 1345 (Fed Cir. 2014); Silicon Graphics, Inc. v. ATI Technologies, Inc., 607 F.3d 784, 794-95 (Fed. Cir. 2010); and Advanced Software Design Corporation v. Fiserv, Inc., 641 F.3d 1368, 1375 (Fed. Cir. 2011).
All of the foregoing notwithstanding, Yeh discloses this claim language even when given full patentable weight. The Sikuli Script system provides “an editor to help users write visual scripts (Figure 7).” Yeh 189 ¶ 1. Using either the editor’s toolbar (or by typing the commands), the user adds find() commands that each locate a particular on-screen GUI element, and corresponding action commands that constitute the script. Yeh 189 ¶ 1. The editor is also able to receive modifications to find() functions that are already in the script by providing buttons inside the find() functions, which the user can click to provide a screenshot or other image to use as the search pattern for the find() command. Yeh 189 ¶ 1.
and responsive to a UI element matching the attributes of the plurality of graphical element detection techniques being found via an exact match or a threshold match: take an action associated with the activity involving the UI element.
The find() function will look for either an exact match, or for one that meets a similarity threshold, depending on whether the pattern passed into find is decorated with the exact() or similar() decorator functions. Yeh 188 ¶ 3. Then, “[t]he action commands specify what keyword and/or mouse events to be issued to the center of a region found by find(),” such as clicking or typing text. Yeh 188 ¶ 9.
Claim 2
Yeh discloses the non-transitory computer-readable medium of claim 1,
wherein the UI attributes comprise images, text,
The find() function takes a Pattern object as an input, and that Pattern object “can be created from an image or a string of text.” Yeh 188 ¶ 2.
relationships between graphical elements in the UI,
“To support other types of constrained search, our visual scripting API provides a versatile set of constraint operators: left, right, above, below, nearby.” Yeh 188 ¶ 7.
a hierarchical representation of the graphical elements in the UI,
Yeh discloses two different ways to specify a hierarchical relationship as the UI attribute for comparison. The first way is to use a chain of find() commands, thereby constraining the search space of the second find() (e.g., for a specified button) to only the re-gion occupied by the result of the first find() (e.g., a parent window containing the specified button).
The second way is to simply use the inside() constraint operator, which returns a representation of the inside of a given Region. See Yeh 188 ¶ 7 (top of second column).
or a combination thereof.
“These operators can be used in combination to express a rich set of search semantics.” Yeh 188 ¶ 7.
Claim 3
Yeh discloses the non-transitory computer-readable medium of claim 1,
wherein the computer program steps of claim 1 are repeated for at least one additional activity.
“We present six example scripts to demonstrate the basic features of Sikuli Script.” Yeh 189.
Claim 4
Yeh discloses the non-transitory computer-readable medium of claim 1,
wherein types of the UI descriptors comprise two or more of
Since parent claim 1 never uses the word types, and claim 4 never refers back to the “compare” step of its parent claim, claim 4 is ambiguous as to whether the two or more “types” must be either (i) part of the comparison in claim 1, or if instead, (ii) claim 4 is listing a group of UI descriptor types that are merely available for inclusion into the plurality. For brevity, this rejection will show why Yeh anticipates the narrower interpretation, since anticipation of a narrow embodiment necessarily anticipates the broader genus that the narrow embodiment falls within. See MPEP § 2131.02.
Yeh’s disclosure of each element in the list will be provided below. As for why Yeh specifically discloses the narrow interpretation (i), this is disclosed on page 188, where Yeh explains how to chain different constraints and find() algorithms together.
a selector,
The win32gui library may be imported into a Sukuli script “to provide the function getActiveWindow(),” which “obtain[s] the handle to the active window (103).” Yeh 190.
a computer vision (CV) descriptor,
In cases where the find() command is run using a large image pattern, find() executes “an algorithm based on invariant local features such as SIFT.” Yeh 187; see also Yeh 188 (“When created from an image, the computer vision algorithm described earlier is used to find matching screen regions.”)
an image matching descriptor,
An image-based pattern object may also be tuned with the “exact()” function, to “[r]equire matches to be identical to the given search pattern pixel-by-pixel.” Yeh 188.
and an optical character recognition (OCR) descriptor.
“When [a search pattern is] created from a string, OCR is used to find screen regions matching the text of the string.” Yeh 188.
Claim 5
Yeh discloses the non-transitory computer-readable medium of claim 1,
wherein the computer program is configured to automatically utilize the plurality of graphical element detection techniques.
The word “automatically” is used two different ways in the Applicant’s specification, so both will be discussed.
The first way “automatically” is used is when an RPA designer application helps decide for the RPA developer which combination of graphical element detection techniques and respective attributes to use, rather than making the RPA developer decide them manually. See Spec. ¶¶ 22 and 88. Yeh discloses by explaining that find() automatically “uses template-matching for finding small patterns and invariant feature voting for finding large patterns.” Yeh 187.
The second way “automatically” is used is with respect to the RPA workflows/robots being triggered by events or schedules, rather than manually launched. See Spec. ¶ 47. Under this interpretation, the RPA workflows automatically utilize the plurality of graphical element detection techniques that were previously defined within the workflow. Yeh discloses this version by providing examples that are triggered by computer events and schedules, rather than manually. In Example 5, the script waits until the computer provides an active window, and only performs the click() function when triggered by the active window’s presence. Yeh 190. In Example 6, line 2 causes the script to run on a schedule of every “60” units of time. Yeh 191
Claim 7
Yeh discloses the non-transitory computer-readable medium of claim 1,
wherein the computer program is or comprises an RPA robot.
Sikuli scripts “use screenshots of GUI elements directly in an automation script to programmatically control the elements with low-level keyboard and mouse input.” Yeh 187.
Claim 8
Yeh discloses the non-transitory computer-readable medium of claim 1,
wherein the computer program is configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques responsive to selection by a user in the visual graphical element detection selection and configuration interface.
The find() function may be crafted in the script editor such that it is invoked using only an image pattern, or only a string (for the OCR functionality), rather than using the entirety of parameters provided by the Sikuli API (e.g., the constraint operators, the tuning functions, combinations of images with text, etc.). See Yeh 188.
Claim 9
The method of claim 9 is nearly identical to the method that the computer program of claim 1 performs as part of its normal operation, except that it recites the additional step of “analyzing a user interface (UI) at runtime to identify UI element attributes, by a computing system.” Yeh discloses this additional step by disclosing that “[t]he find() function locates a particular GUI element to interact with. It takes a visual pattern that specifies the element’s appearance, searches the whole screen or part of the screen, and returns regions matching this pattern or false if no such region can be found.” Yeh 188 ¶ 1. Yeh discloses the remaining steps of claim 9 because “if a prior art device, in its normal and usual operation, would necessarily perform the method claimed, then the method claimed will be considered to be anticipated by the prior art device.” MPEP § 2112.02 (subsection I.).
Claims 10–13 and 15
The limitations that claims 10–13 and 15 add to claim 9 are the same as those that claims 2–5 and 8 add to claim 1, and are therefore rejected according to the same findings and rationale as provided above for those claims.
Claim 15, Second Ground of Rejection
Claim 15 is separately rejected because Yeh teaches “the method of claim 9” recited in the preamble of claim 15, while the rest of claim 15 is completely optional.
The rest of claim 15 is completely optional because the entirety of the limitations it adds to claim 9 are conditioned upon an unmet condition precedent. “The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met.” MPEP § 2111.04 (subsection II.).
In the instant case, claim 15 is a method claim, “responsive to selection by a user in the visual graphical element detection technique selection and configuration interface” is conditional statement, and “the computing system is configured to use a subset of the UI descriptor attributes for at least one of the plurality of graphical element detection techniques” is contingent upon the aforementioned conditional statement. The condition precedent for the conditional statement is never recited in the claim, that is, there is no method step in claim 15 where a user actually performs the selection in the visual graphical element detection technique selection and configuration interface.
Accordingly, there are no required elements in this claim, and the prior art does not need to disclose unrequired elements in order to reach a finding of anticipation.
Claims 16–20, 22, and 23
Claims 16–20 and 23 describe a computing system having the same program stored in substantially the same memory as corresponding claims 9–13 and 15, and are therefore rejected according to the same findings and rationale as provided above for those claims.
Claim 22 is rejected according to the same findings and rationale as provided above in the rejection of claim 7, together with the findings and rationale provided above in the rejection of claim 16.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Justin R. Blaufeld whose telephone number is (571)272-4372. The examiner can normally be reached M-F 9:00am - 4:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James K Trujillo can be reached at (571) 272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Justin R. Blaufeld
Primary Examiner
Art Unit 2151
/Justin R. Blaufeld/Primary Examiner, Art Unit 2151
Prosecution Insights