DETAILED ACTION
This action is responsive to the Request for Continued Examination filed on 01/05/2026. Claims 1, 9, 15, and 21 have been amended. Claims 2, 3, 10, and 16 had been previously canceled. Claims 1, 4-9, 11-15, and 17-21 are pending in the case. Claims 1, 9, and 15 are independent claims.
Claim Objections
Claims 1, 4-9, 11-15, and 17-21 are objected to because of the following informality:
Claim 1:
Line 11 recites “following generation of the task identifier” where “following the generation of the task identifier” was apparently intended.
Claim 9:
Line 7 recites “following generation of the task identifier” where “following the generation of the task identifier” was apparently intended.
Claim 15:
Line 10 recites “following generation of the task identifier” where “following the generation of the task identifier” was apparently intended.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 21 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention. The original specification does not appear to have sufficient support for the limitation reciting “the UI element being determined as outside of dialog or menu item groups.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-9, 11-15, and 17-21 are rejected under 35 U.S.C. § 103 as being unpatentable over Singh (US Patent Application Pub. No. 2020/0327005, hereinafter “Singh”) in view of Dwarakanath et al. (US Patent Application Pub. No. 2018/0189170, hereinafter “Dwarakanath”).
As to independent claims 1, 9, and 15, Singh shows a client computer system [¶ 02], a method [¶ 07], and a concomitant non-transitory computer readable medium [¶ 10] configured to recognize and respond to user interface (UI) elements [¶ 03], comprising:
a memory [¶ 02]; and
at least one processor coupled to the memory [¶ 02] and configured to:
receive a recognition rule comprising one or more UI element specifications [“A variety of classifiers can be used to detect error messages within UI data. For instance, in some examples, the error remediation system executes a classifier that scans the UI data, or data derived from the UI data, for keywords associated with error messages. This derived data can include text data output from an optical character recognition (OCR) and/or computer vision processes. In other examples, the classifier executes a machine learning process that operates on input including the UI data and/or data derived from the UI data. In these examples, the machine learning process can be trained to recognize text and/or objects commonly displayed at particular locations when an error message is encountered. In some examples, the error remediation system executes a single classifier for all changes in UI elements. In other examples, the error remediation system executes one or more classifiers that are tailored to specific types of monitored applications.” (¶ 47) | See also ¶¶ 03, 08, & 40-42] and a response action [e.g. a remediation (¶¶ 40-42)] from a server [e.g. the aforementioned limitations are fetched/received from a server (¶¶ 41, 45-46, 53-54, & 60)];
generate a task identifier corresponding to the received recognition rule; associate the task identifier with the recognition rule [e.g. generate a task identifier/signature and associate it with the recognition rule of a particular UI element (like an error message UI element) “so that future instances of the error can be automatically remediated” (¶ 40) | See ¶¶ 06, 09, & 40-42];
recognize, based on the one or more UI element specifications of the recognition rule and the task identifier, a UI element rendered at the client computer system, the UI element being recognized following generation of the task identifier; and execute, in response to recognition of the UI element, the response action of the recognition rule based on the task identifier [e.g. recognize, based on the previously stored one or more UI element specifications of the recognition rule/computer vision and the task identifier/signature, a live UI element rendered at the client computer system (following the generation of the original signature against which the live UI element’s signature is compared/matched) and execute a corresponding remediation upon recognizing the UI element/error message (¶¶ 40-42)];
wherein recognizing the UI element comprises executing a computer vision ("CV") process [“A variety of classifiers can be used to detect error messages within UI data. For instance, in some examples, the error remediation system executes a classifier that scans the UI data, or data derived from the UI data, for keywords associated with error messages. This derived data can include text data output from an optical character recognition (OCR) and/or computer vision processes. {…}” (¶ 47)] comprising: {…}
filtering out the UI element [see filter 204 (¶¶ 83-89)];
identifying a detail of a dialog or menu item group [see the dialog/menu detail identification features in at least ¶¶ 49, 79, 93, & 182]; and
classifying {…} using a guided heuristic [see classifier 206/1606, which may classify any computer vision-detected element using a guided heuristic (¶¶ 47, 55, 68, 90-91, 141, 163, 176-178, & 200)].
As indicated above, Singh shows almost all of the computer vision-related functionalities outlined above. Nonetheless, in lieu of simply pointing to the considerable breadth of the terms as currently recited and/or the spectrum of possible mappings their broadest reasonable interpretation would cover, it is potentially conceded that Singh does not appear to explicitly recite utilizing its existing computer vision functionalities for the intended use of “identifying a rectangle defining a border of the UI element” as currently recited. In an analogous art, Dwarakanath shows:
wherein recognizing the UI element comprises executing a computer vision ("CV") process [e.g. to “identify the set of elements using computer vision” (¶ 18) | See also ¶¶ 58 & 67)] comprising:
identifying a rectangle defining a border of the UI element [e.g. identifying a box in the shape of a rectangle defining a border of the UI element (see the “boxes” figs. 1 & 5A-5D; ¶¶ 17-18)];
filtering out the UI element [e.g. filtering out/focusing solely on the UI element associated with the identified box (¶¶ 18-19)];
identifying a detail of a dialog or menu item group [e.g. identifying a detail of a menu item group that sets the identified item apart from others (see the many detail alternatives throughout ¶¶ 17, 47, 58-61, & 86-90)]; and
classifying the rectangle using a guided heuristic [e.g. classifying/annotating the box using a guided heuristic/categorization metric based on the identification result (¶¶ 58-62 & 67)].
One of ordinary skill in the art, having the teachings of Singh and Dwarakanath before them prior to the effective filing date of the claimed invention, would have been motivated to incorporate Dwarakanath’s rectangle identification functionalities into Singh. The rationale for doing so would have been that Singh already taught its own wide-ranging computer vision techniques, and applying towards rectangles/boxes as taught by Dwarakanath would “improve automatic testing of software by reducing or eliminating the need for a tester to know, or have access to, program code underlying the user interface. Furthermore, implementations described herein conserve processing resources by reducing or eliminating errors due to changes to the underlying program code, and increase an efficiency of automatic testing by permitting dynamic adjustments to the automatic testing based on changes to the user interface” (Dwarakanath: ¶ 14). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Singh and Dwarakanath (hereinafter, the “Singh-Dwarakanath” combination) in order to obtain the invention as recited in claims 1, 9, and 15.
As to dependent claims 4 and 11, Singh-Dwarakanath further shows:
wherein to recognize the UI element based on the recognition rule further comprises to execute a machine learning (ML) process [“{…} the classifier executes a machine learning process {…}” (Singh: ¶ 47)].
As to dependent claims 5, 12, and 17, Singh-Dwarakanath further shows:
generate a numerical hash value by executing a hash function on one or more characteristics of the rendered UI element [“{…} the detection service 1510 is configured to generate a signature that can be used to identify future instances of an error message. This signature can be similar to, or the same as, a signature that would be generated by the monitor and can be generated from processed image data and/or other portions of session data. In some examples, the signature includes a representation of a hierarchy of UI elements. This UI hierarchy can include a representation of a root UI element and can continue through a line of descendants from the root UI element to the error message. For instance, in one example, the hierarchy includes a set of identifiers comprising an identifier of an application, an identifier of a window or page of the application in which the error message was displayed, an identifier of the main process that generated the error message, and/or an identifier of the error message (error code, error message text, hash generated from any combination of the error code, error text, etc.).” (Singh: ¶ 164)];
send the numerical hash value to the server [e.g. the signature (which comprises the numerical hash value) is sent to the server (Singh: ¶¶ 190 & 196)]; and
further comprising the server, wherein the server is configured to track a number of instances of the numerical hash value [e.g. the server is configured to track a number of instances of the signature/numerical hash value (Singh: ¶¶ 06, 52, 58, 74, 144, 164, & 203)].
As to dependent claims 6, 13, and 18, Singh-Dwarakanath further shows:
wherein the one or more UI element specifications describe one or more of: a size or shape of the dialog or the window; text of the dialog or the window; an icon or image of the dialog or the window; a progress bar or scroll bar of the dialog or the window; a button or other control of the dialog or the window; a menu item; or a number of instances or affected users associated with the UI element. [e.g. the one or more UI element specifications describe one or more of: a size or shape of the dialog or the window; text of the dialog or the window; an icon or image of the dialog or the window; a progress bar or scroll bar of the dialog or the window; a button or other control of the dialog or the window; a menu item; or a number of instances or affected users associated with the UI (Singh: ¶¶ 49, 79, 93, & 202-203)].
As to dependent claims 7, 14, and 20, Singh-Dwarakanath further shows:
wherein the recognition rule comprises a structured recognition rule tag including: one or more structured UI element specification tags encoding the one or more UI element specifications; and a structured response action tag encoding the response action [e.g. the recognition rule may comprise tags/IDs/metadata of the one or more UI element specifications and the response action/remediation/triage (Singh: ¶¶ 49, 69, 73, 90, 94, 131-132, & 183)].
As to dependent claims 8 and 19, Singh-Dwarakanath further shows:
wherein the UI element rendered at the client computer system comprises one or more of: an error notification; a password entry field; a delay notifier; a pop-up menu item; personal information; or a personal message [e.g. the UI element rendered at the client computer system comprises one or more of: an error notification; a password entry field; a delay notifier; a pop-up menu item; personal information; or a personal message (Singh: figs. 13-14)].
As to dependent claim 21, Singh-Dwarakanath further shows:
wherein filtering out the UI element comprises filtering out the UI element responsive to the UI element being determined as outside of dialog or menu item groups [Singh-Dwarakanath shows many examples of filtering out UI elements that may be deemed by an ordinarily-skilled artisan to be “outside” of dialog or menu item groups (Singh: ¶¶ 83-89 | Dwarakanath: ¶¶ 15-19 & 58-62). Additionally/alternatively, since the claim is a dependent of a method claim, Singh-Dwarakanath shows many instances where a UI element may also be reasonably/subjectively deemed “inside” of dialog or menu item groups (see Singh: ¶¶ 49, 79, 93, & 182 | Dwarakanath: ¶¶ 17, 47, 58-61, & 86-90), which would also meet the requirements of this method claim since the contingent condition would not have been met.].
Response to Arguments
Applicant’s arguments have been fully considered but they are not persuasive. Applicant argues:
“ {…} the cited passages at most describe detecting an error message present within live or prerecorded session data and generating an error signature unique to the error message. However, the cited passages do not describe that a recognition rule is received, a task identifier corresponding to the received recognition rule is generated, and a UI element is recognized “following generation of the task identifier.” Rather, and if anything, any generation of the signature described in Singh is subsequent detecting the error message.
Further, and even if the Office were to interpret the detection of an error message followed by matching of a generated signature to a signature stored in an error record described in Singh as allegedly teaching the claimed UI element being recognized following generation of a task identifier, such an interpretation would still not squarely teach or suggest the claimed combination. Rather, Singh states that “once a remediation to an error message is established, that remediation is automatically provided to all users who encounter the error message.” ¶[0042]. Thus, in the cited passages of Singh, the “signature” is either 1) generated after detection of the error message, or 2) provisioned to another device in response to a previously-detected error. In neither instance is a task identifier generated at a client computer system upon receipt of a recognition rule and then used to recognize a UI element, as recited in claim 1. Moreover, even if a signature were provisioned, the signature is generated in Singh based on the error, not based on a received recognition rule, and the generation occurs on a different device from the claimed client computer system. The claimed combination of a task identifier generated by a client computer system based on a recognition rule, and recognition of a UI element following generation of the task identifier, is neither taught nor suggested by Singh.”
The Office respectfully disagrees. Singh explicitly shows a single system that receives a recognition rule the first time that an error occurs, generates a signature for that original error “so that future instances of the error can be automatically remediated” (Singh: ¶ 40), and then recognizes a new/live error that matches the original error following the generation of the original error signature/identifier. See Singh: ¶¶ 40-42.
“Dwarakanath is cited as allegedly teaching certain other features of the independent claims, and does not address the deficiencies identified above. Thus, the combination of references does not and cannot teach or suggest each of the elements recited in claim 1.”
In response to Applicant’s arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 U.S.P.Q. 871 (C.C.P.A. 1981); In re Merck & Co., 800 F.2d 1091, 231 U.S.P.Q. 375 (Fed. Cir. 1986).
Therefore, the Office respectfully asserts that the cited art sufficiently teaches the limitations recited in the amended claims.
Conclusion
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALVARO R CALDERON IV whose telephone number is (571)272-1818. The examiner can normally be reached on Monday - Friday (8:30am - 5:00pm).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu, can be reached on (571) 272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALVARO R. CALDERON IV/
Examiner
Art Unit 2171
/KIEU D VU/Supervisory Patent Examiner, Art Unit 2171
Prosecution Insights