DETAILED ACTION
This action is responsive to Remarks and Claim amendments filed on February 16, 2026.
Claims 1-2, 7-8, 10-12, 15-17 and 19-20 have been amended. Claims 3-6 and 18 have been canceled. Claim 21 has been newly added.
Claims 1-2, 7-17 and 19-21 are pending and are presented to examination.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Response to amendments
The objection of claims 2-12, 15, 17-18 and 20 is withdrawn in view of applicant’s amendments.
The objection of the specification (Abstract of the Disclosure) is withdrawn in view of applicant’s amendments.
The rejection of claims 1-20 under 35 U.S.C. 101 (Abstract Idea) is withdrawn in view of applicant’s amendments.
Response to Arguments
Rejections under 35 U.S.C. 102 and 103 Applicants have argued that Lucas, Beltran, along with the remaining prior arts of record, do not teach the newly added limitations of independent claims 1, 16 and 19 (Remarks, pages 10-14). Applicants’ arguments have been fully considered and are persuasive. Therefore, the rejection is withdrawn. However, upon further consideration, a new ground of rejection is made as set forth in details below. See Mulong Xie et al. (“UIED: A Hybrid Tool for GUI Element Detection”), Ciprian Paduraru et al. (“Automated Game Testing Using Computer Vision Methods”) and Guoqing Liu et al. (“Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation”), arts being made of record as applied herein.
Claim Objections
Claims 1-2, 7-17 and 19-21 are objected to because of the following informalities: Claim 1 (and similar for claims 16 and 19) recites the limitation “processing, by one or more of the processors, the screenshot to identify one or more candidate locations of user interface elements based upon one or more image processing techniques comprising at least one of the following: a contrast/brightness adjustment, a noise/blur filtering, an edge detection, or a contour detection;” in lines 6-10. Claim 16 recites “one or more computer-readable storage media…” in line 3. Appropriate correction is required. Please amend the claim language as indicated in bold.
Dependent claims 2, 7-15, 17 and 20-21 do not overcome the deficiency of the base claim and, therefore, are rejected for the same reasons as the base claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 7-14, 16-17 and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Mulong Xie et al. (“UIED: A Hybrid Tool for GUI Element Detection”, hereinafter Xie) in view of Ciprian Paduraru et al. (“Automated Game Testing Using Computer Vision Methods”, hereinafter Paduraru).
With respect to claim 1, (currently amended) Xie teaches a method for testing a user interface [[of a video game]], the method implemented by one or more processors (Xie at p. 1655 (Abstract) discloses a method for accurate detection of graphical user interface (GUI) elements, performed by the UIED toolkit and used for GUI testing tasks. Xie at p. 1657 (section 4 “Web Implementation”) confirms that the method is implemented by one or more processors executing software (Tensorflow, Pytorch, OpenCV)), the method comprising: obtaining, by one or more of the processors, a screenshot [[of the video game]] (Xie at p. 1655 (section 1) discloses that the UIED toolkit obtains, as input, “various UI image, such as mobile app or web page screenshot, UI design drawn by Photoshop or Sketch, and even some hand-drawn UI design.” Xie thereby teaches obtaining a screenshot by the processors). processing, by one or more of the processors, the screenshot [[of the video game]] to detect one or more user interface elements comprising: (Xie at p. 1655 (Abstract) discloses processing the screenshot to detect GUI elements: “Graphical User Interface (GUI) elements detection is critical for many GUI automation and GUI testing tasks. Acquiring the accurate positions and classes of GUI elements is also the very first step to conduct GUI reverse engineering or perform GUI testing.” Xie at p. 1656 (section 3 “Our Hybrid Approach”) describes the screenshot processing pipeline that detects GUI elements such as widgets, images, and text). processing, by one or more of the processors, the screenshot to identify one or more candidate locations of user interface elements based upon one or more image processing techniques comprising at least one of the following: a contrast/brightness adjustment, a noise/blur filtering, edge detection, or contour detection (This limitation recites a Markush group of four image-processing alternatives joined by “or,” requiring “at least one of” the four to be taught. The prior-art combination teaches multiple of the four alternatives, satisfying the Markush several times over. Xie at p. 1656 (section 3 “Non-Text GUI Element Detection”) discloses identifying candidate locations of UI elements by an image-processing pipeline. Specifically, Xie discloses: (i) “the approach detects layout blocks through flood-filling algorithm combined with the Sklansky’s algorithm to acquire the blocks’ outer boundaries and produces a block map” — the “Sklansky’s algorithm” as cited at Xie reference [26] (Suzuki and Abe, “Topological structural analysis of digitized binary images by border following,” 1985) is the well-known border-following contour detection algorithm; (ii) “the method generates a binary map by a simple but efficient binarization method based on the gradient map of the input GUI” — a gradient-based edge detection step (Xie cites Canny’s seminal edge-detection paper at reference [5]); and (iii) “we detect GUI elements by connected component labelling algorithm and use Sklansky’s algorithm again to determine the elements’ boundary” — again contour boundary detection. Xie therefore expressly teaches both contour detection and gradient-based edge detection applied to the screenshot to identify candidate locations of UI elements. Examiner notes: Paduraru at p. 70 (section VI-A) further teaches: “we also applied some post-processing techniques to the rendered images, such as conversion from RGB to HSV space, computed edge detection, and smoothing, all using standard features provided by OpenCV.” Paduraru thereby independently teaches edge detection and noise/blur filtering (smoothing) applied to the video-game screenshot). processing, by one or more of the processors, the screenshot to identify text and/or icons and the corresponding locations of the text and/or icons within the screenshot based upon one or more machine learning models (Xie at p. 1656 (section 3 “GUI Text Element Detection”) discloses processing the screenshot to identify text and the corresponding locations of the text by way of a machine learning model: “We determine GUI text as a scene text and apply the state-of-the-art deep learning scene test detector EAST [34] to detect text in the GUI image. It first feeds the input image into a feature pyramid network [15] and then computes six values for each point based on the final feature map to detect text (objectness score, top/left/bottom/right offsets and rotation angle).” EAST is a deep learning (i.e., machine learning) model; the “top/left/bottom/right offsets and rotation angle” constitute the corresponding locations of the text within the screenshot. Examiner notes: Paduraru at p. 67 (section IV-B) independently discloses use of Tesseract OCR for text recognition and Yolo for icon/object detection — both machine learning models — to identify text and icons. Paduraru at p. 68 (section IV-B) recites that “the 2D bounding box coordinates of the objects of interest in the test scenario are obtained using the Yolo method,” providing the corresponding locations of the detected icons within the screenshot) and detecting, by one or more of the processors, the one or more user interface elements based upon the identified candidate locations of the user interface elements, the identified text and/or icons, and the identified locations of the text and/or icons (Xie at p. 1656 (section 3 “Our Hybrid Approach”) discloses the merging of the non-text candidate-location pipeline and the text/locations machine-learning pipeline into a final UI-element detection: “Our method divides the detection task into two part: non-text element detection and text detection. We leverage old-fashioned computer vision algorithms for non-text region extraction, and deep learning models to perform classifications and text detection.” Xie at p. 1657, Figure 1 caption is dispositive: “The overall process of our approach, where (a) is the input GUI, (b) is the text detection result by EAST, (c), (d), (e) is non-text elements detection and (f) is the merged final result.” The merged result (f) constitutes detection of the UI elements based upon both the identified candidate locations (from the image-processing pipeline (c)–(e)) and the identified text and its locations (from the EAST machine-learning model (b))). Xie is silent to disclose, however in an analogous art Paduraru teaches: testing a user interface of a video game (Paduraru at p. 65 (Abstract, section I) discloses that the computer-vision-based testing methods are directed to testing user interfaces of video games, executed by processors of a computing environment. Paduraru provides the “video game” context for the Xie method).
obtaining, a screenshot of the video game (Paduraru at p. 67 (section III) and Figure 1 (showing screenshots of the Unreal Engine 4 “ShootingGame” demo) discloses that the test framework obtains screenshots of the running video game. The combination teaches obtaining, by one or more processors, a screenshot of the video game)).
processing, the screenshot of the video game (Paduraru at p. 67 (section III “UI Testing”) teaches that the screenshot of the running video game is processed to detect in-game UI elements such as the HUD ammo display, weapon-cross icon, and menus)). performing, by one or more of the processors, one or more actions in the video game based upon the detected one or more user interface elements for testing the user interface of the video game (Xie is directed primarily to the GUI element detection task and does not itself perform in-game actions; Xie at p. 1655 (section 1) and p. 1658 (section 6) does, however, expressly identify GUI testing as the intended downstream application of the detected elements and states that the authors “are planning to utilize the UIED as the detection part for automatic GUI testing.” Paduraru at p. 67 (section III) provides concrete examples of UI testing actions performed in the video game based upon detected UI elements: “If the user shoots someone, did the score increase on the Heads-Up Display (HUD)? … Is the ammo displayed on the screen in sync with the value in the game memory? … If the user changed the weapon, is the cross icon on the screen positioned correctly?” Paduraru at p. 68 (section IV-A) recites: “if test is selected for execution, then the game testing agent will execute the specified action testAction. For this test to be considered correct, the output of the game must match the conditions specified in testExpectedBehavior.” Paduraru’s Figure 1 (left) further illustrates the game testing agent firing a weapon (an in-game action) followed by visual verification of the HUD ammo display (testing of the UI element)). It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine the hybrid computer-vision-and-machine-learning GUI element detection method of Xie with the automated video-game testing framework of Paduraru. Xie expressly identifies automated GUI testing as the intended downstream application of its detection method (Xie p. 1655 section 1; p. 1658 section 6). Paduraru expressly identifies UI testing of in-game HUD elements, menus, icons, and other UI features as a primary objective of its automated game-testing framework and identifies the need for accurate visual detection of those UI elements (Paduraru p. 67 section III). One of ordinary skill in the art would have been motivated to apply Xie’s more accurate hybrid detection scheme — which Xie reports as achieving an F1-score of 0.524, substantially superior to all five state-of-the-art baseline approaches that Xie evaluated (Xie Table 1, p. 1658) — to Paduraru’s video-game UI-testing pipeline, in order to obtain more accurate and reliable detection of in-game UI elements and thereby drive more reliable automated test execution. The combination is the use of a known technique (Xie’s hybrid GUI element detection) to improve a known device (Paduraru’s automated game testing framework) in the same way that the technique was used to improve a similar device (GUI element detection for GUI testing) and yields predictable improvement of a known testing pipeline. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398 (2007); MPEP §§ 2143(I)(A) and 2143(I)(C). With respect to claim 2 (currently amended), Xie is silent to disclose, however in an analogous art Paduraru teaches obtaining, by one or more of the processors, an instruction for testing the user interface of the video game; and wherein the one or more actions are based upon the instruction for testing the user interface of the video game (Paduraru at p. 68 (srection IV-A) discloses obtaining a test instruction in the form of a structured test specification: “Tests are specified as a set of elements that map the game state and action to be executed in that particular state, to the expected behavior that a human user would likely see” in the format “[(State, Action) → ExpectedBehavior].” Paduraru further teaches that the game testing agent’s actions are based upon the obtained test instruction: “if test is selected for execution, then the game testing agent will execute the specified action testAction” (Paduraru p. 68, section IV-A)). The combination of Xie and Paduraru for the reasons set forth above with respect to claim 1, together with Paduraru’s teaching of test-instruction-based action execution, renders claim 2 obvious. It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine the hybrid computer-vision-and-machine-learning GUI element detection method of Xie with the automated video-game testing framework of Paduraru. Xie expressly identifies automated GUI testing as the intended downstream application of its detection method (Xie p. 1655 section 1; p. 1658 section 6). Paduraru expressly identifies UI testing of in-game HUD elements, menus, icons, and other UI features as a primary objective of its automated game-testing framework and identifies the need for accurate visual detection of those UI elements (Paduraru p. 67 section III). One of ordinary skill in the art would have been motivated to apply Xie’s more accurate hybrid detection scheme — which Xie reports as achieving an F1-score of 0.524, substantially superior to all five state-of-the-art baseline approaches that Xie evaluated (Xie Table 1, p. 1658) — to Paduraru’s video-game UI-testing pipeline, in order to obtain more accurate and reliable detection of in-game UI elements and thereby drive more reliable automated test execution. The combination is the use of a known technique (Xie’s hybrid GUI element detection) to improve a known device (Paduraru’s automated game testing framework) in the same way that the technique was used to improve a similar device (GUI element detection for GUI testing) and yields predictable improvement of a known testing pipeline. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398 (2007); MPEP §§ 2143(I)(A) and 2143(I)(C). With respect to claim 7 (currently amended), Xie teaches wherein the one or more machine learning models comprises at least one of:
a text detection machine learning model, a text recognition machine learning model, an icon detection machine learning model, or an icon recognition machine learning model (Xie at p. 1656 (section 3 “GUI Text Element Detection”) teaches that the machine learning model used for text identification is EAST, a deep learning scene text detector — i.e., a text detection machine learning model, satisfying the “at least one of” Markush. Xie at p. 1656 (section 3 “Non-Text GUI Element Detection”) additionally teaches a ResNet50 classifier trained on 90,000 GUI element instances across 15 categories to classify the extracted non-text elements — i.e., an icon recognition machine learning model. Examiner notes: See also Paduraru at p. 67 (section IV-B) independently teaches Tesseract OCR (a text recognition machine learning model) and Yolo (an icon detection machine learning model). The combination teaches each of the four Markush alternatives). With respect to claim 8 (currently amended), Xie teaches wherein identifying the text and/or icons further comprises: classifying the text and/or icons (Xie at p. 1656 (section 3 “Non-Text GUI Element Detection”) discloses classifying the identified UI elements: “we train a ResNet 50 classifier on 90,000 GUI element instances with 15 categories to classify the extracted elements.” Xie thereby classifies the identified icons.
Examiner notes: See also Paduraru’s Yolo-based detection independently provides class labels for the detected objects (icons) (Paduraru p. 67, section IV-B)). With respect to claim 9 (original), Xie is silent to disclose, however in an analogous art Paduraru teaches wherein classifying the text is based upon a dictionary data structure comprising a mapping from text strings to labels (Paduraru at p. 67 (section IV-B) discloses using “Tesseract OCR from OpenCV for text recognition.” Tesseract OCR inherently uses dictionary-based language models that map observed character sequences to recognized word labels — this is a structural feature of the Tesseract OCR engine and would be understood as such by one of ordinary skill in the art. Furthermore, Paduraru at p. 67 (section III) discloses test specifications that map detected text strings (e.g., a detected ammo numeric value such as “50”) to verification labels (correct / incorrect with respect to the expected backend value), which is itself a dictionary data structure comprising a mapping from text strings to labels for the purpose of classifying the OCR-detected text. The combination of Xie and Paduraru, in view of the well-understood operation of the Tesseract OCR engine, renders claim 9 obvious. It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine the hybrid computer-vision-and-machine-learning GUI element detection method of Xie with the automated video-game testing framework of Paduraru. Xie expressly identifies automated GUI testing as the intended downstream application of its detection method (Xie p. 1655 section 1; p. 1658 section 6). Paduraru expressly identifies UI testing of in-game HUD elements, menus, icons, and other UI features as a primary objective of its automated game-testing framework and identifies the need for accurate visual detection of those UI elements (Paduraru p. 67 section III). One of ordinary skill in the art would have been motivated to apply Xie’s more accurate hybrid detection scheme — which Xie reports as achieving an F1-score of 0.524, substantially superior to all five state-of-the-art baseline approaches that Xie evaluated (Xie Table 1, p. 1658) — to Paduraru’s video-game UI-testing pipeline, in order to obtain more accurate and reliable detection of in-game UI elements and thereby drive more reliable automated test execution. The combination is the use of a known technique (Xie’s hybrid GUI element detection) to improve a known device (Paduraru’s automated game testing framework) in the same way that the technique was used to improve a similar device (GUI element detection for GUI testing) and yields predictable improvement of a known testing pipeline. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398 (2007); MPEP §§ 2143(I)(A) and 2143(I)(C). With respect to claim 10 (currently amended), Xie teaches wherein a location is represented by co-ordinates of a bounding box (Xie at p. 1656 (section 3 “GUI Text Element Detection”) teaches that EAST detects text by computing “objectness score, top/left/bottom/right offsets and rotation angle” — i.e., bounding-box coordinates for the detected text. Xie at p. 1657 (section 4 “Detection Result Export”) further teaches that the detected elements are exported with “position, size and class of elements” in JSON format — i.e., bounding-box coordinates and dimensions. Examiner notes: See also Paduraru at p. 67 (section III) likewise teaches “a given 2D bounding box” for the ammo HUD test, and Paduraru at p. 68 (§ IV-B) recites that “the 2D bounding box coordinates of the objects of interest in the test scenario are obtained using the Yolo method.”). With respect to claim 11 (currently amended), Xie teaches wherein detecting, by one or more of the processors, the one or more user interface elements based upon the identified candidate locations of the user interface elements, the identified text and/or icons, and the identified locations of the text and/or icons is based upon a proximity and/or a spatial alignment (Xie at p. 1656 (section 3 “Non-Text GUI Element Detection”) teaches detection of UI elements based upon spatial proximity. Specifically, Xie’s flood-filling algorithm operates on regions of spatially-proximate pixels of similar intensity, and Xie’s connected component labelling algorithm operates by aggregating pixels of the binary map that are spatially proximate and connected. Xie further teaches a top-down coarse-to-fine pipeline in which the screenshot is first divided into spatially-defined layout blocks (“the approach detects layout blocks … produces a block map, as shown in Figure 1 (c) where different colour regions stand for potential different layout blocks”), and then UI elements are detected within each spatial block — a detection based upon spatial alignment within the GUI layout). With respect to claim 12 (currently amended), Xie teaches grouping, by one or more of the processors, the detected user interface elements based upon the proximity and/or the spatial alignment into further user interface elements (Xie at p. 1656 (section 3 “Non-Text GUI Element Detection”) teaches hierarchical grouping of detected UI elements into further UI elements based upon their spatial proximity and alignment within the GUI layout. Specifically, Xie’s top-down approach first identifies layout blocks (e.g., the block map in Figure 1(c)) and then “segment[s] the binary map into block segments based on previously detected blocks” and detects individual GUI elements (“connected component labelling algorithm and use Sklansky’s algorithm again to determine the elements’ boundary”) within each spatially-defined block. The detected individual GUI elements are thereby grouped, based on spatial proximity and alignment, into further (parent) UI elements (the layout blocks). The export step (Xie p. 1657, section 4) preserves this grouping information). With respect to claim 13 (original), Xie is silent to disclose, however in an analogous art Paduraru teaches identifying, by one or more of the processors, a currently selected user interface element from the detected one or more user interface elements (Paduraru at p. 67 (section III) discloses identifying a currently selected user interface element. Specifically, Paduraru recites the following test scenario: “If the user changed the weapon, is the cross icon on the screen positioned correctly? E.g., see a screenshot from our demo in the middle of Fig. 1, where we perform basic cross detection … [the test] checks visually if the weapon-cross icon on the screen is the correct one for the new weapon selected.” The weapon-cross icon detected and verified by Paduraru is the user interface element corresponding to the currently selected weapon — i.e., the currently selected user interface element).
It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine the hybrid computer-vision-and-machine-learning GUI element detection method of Xie with the automated video-game testing framework of Paduraru. Xie expressly identifies automated GUI testing as the intended downstream application of its detection method (Xie p. 1655 section 1; p. 1658 section 6). Paduraru expressly identifies UI testing of in-game HUD elements, menus, icons, and other UI features as a primary objective of its automated game-testing framework and identifies the need for accurate visual detection of those UI elements (Paduraru p. 67 section III). One of ordinary skill in the art would have been motivated to apply Xie’s more accurate hybrid detection scheme — which Xie reports as achieving an F1-score of 0.524, substantially superior to all five state-of-the-art baseline approaches that Xie evaluated (Xie Table 1, p. 1658) — to Paduraru’s video-game UI-testing pipeline, in order to obtain more accurate and reliable detection of in-game UI elements and thereby drive more reliable automated test execution. The combination is the use of a known technique (Xie’s hybrid GUI element detection) to improve a known device (Paduraru’s automated game testing framework) in the same way that the technique was used to improve a similar device (GUI element detection for GUI testing) and yields predictable improvement of a known testing pipeline. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398 (2007); MPEP §§ 2143(I)(A) and 2143(I)(C). With respect to claim 14 (original), Xie teaches wherein the one or more user interface elements comprises at least one of:
a menu item, a menu bar, a grid menu, a button, or a button prompt (Xie at p. 1656 (section 3 “Our Hybrid Approach”) expressly teaches detection of buttons among the GUI elements: “UIED comprises two parts to detect UI text and graphic elements, such as button, image and input bar.” Detection of buttons satisfies the “at least one of” Markush. Examiner notes: Paduraru at p. 67 (section III) further teaches detection of HUD UI elements including the ammo display and weapon-cross icon, which constitute button-prompt-like in-game UI elements). With respect to claim 16, the claim is directed to a system that corresponds to the method recited in claim 1, respectively (see the rejection of claim 1 above; wherein Xie and Paduraru also teach such system. Xie at p. 1657 (section 4 “Web Implementation”) discloses implementation of the UIED method on processors with software stored in computer-readable storage media (using Tensorflow, Pytorch, and OpenCV). Paduraru at p. 70 (section V) discloses an architecture comprising a GameBot and GameStateChecker implemented on one or more processors with software stored in computer-readable storage media).
With respect to claim 17, the claim is directed to a system that corresponds to the method recited in claim 2, respectively (see the rejection of claim 2 above).
With respect to claim 19, the claim is directed to a non-transitory computer-readable storage medium that corresponds to the method recited in claim 1, respectively (see the rejection of claim 1 above; wherein Xie and Paduraru also teach such medium. Xie at p. 1657 (section 4) discloses the UIED toolkit as software (a web application) stored on and executed from computer-readable storage media. Paduraru at p. 70 (section V-V-A) discloses the test framework as instructions deployed on computing devices (e.g., a Flask/FlaskRest server for the GameStateChecker) and executable code (C++/C#/Python) for the GameBot). With respect to claim 20, the claim is directed to a non-transitory computer-readable storage medium that corresponds to the method recited in claim 2, respectively (see the rejection of claim 2 above). With respect to claim 21 (new), Xie teaches performing, by one or more of the processors, edge detection on the modified screenshot to generate edge detection output (Xie at p. 1656 (section 3 “Non-Text GUI Element Detection”) teaches gradient-based binarization, which generates a binary map identifying pixel-intensity transitions (edges) of the input GUI image; the binary map is the claimed edge detection output. Examiner notes: See also Paduraru at p. 70 (section VI-A) further teaches “computed edge detection … using standard features provided by OpenCV” applied to the rendered screenshot. Performing the edge detection on the smoothing-modified screenshot is the well-known conventional order, as discussed above). performing, by one or more of the processors, contour detection on the edge detection output to generate contour detection output (Xie at p. 1656 (section 3) teaches: “we detect GUI elements by connected component labelling algorithm and use Sklansky’s algorithm again to determine the elements’ boundary.” As discussed above with respect to limitation 1(b)(i), Xie’s cited “Sklansky’s algorithm” corresponds to the Suzuki/Abe border-following contour detection algorithm (Xie reference [26]), which operates on the binary edge-map output of Step 2 to generate contour boundaries — the claimed contour detection output) and generating, by one or more of the processors, bounding boxes to indicate the one or more candidate locations of the user interface elements based upon the contour detection output (Xie at p. 1656 (section 3) teaches that the detected element boundaries identify the GUI elements’ locations. Xie at p. 1657 (section 4) teaches that the detected elements are exported with “position, size and class of elements” in JSON format — i.e., as bounding boxes that indicate the candidate locations of the UI elements, derived from the contour detection output limitation above). Xie is silent to disclose, however in an analogous art, Paduraru teaches: performing, by one or more of the processors, a contrast/brightness adjustment and/or a noise/blur filtering of the screenshot to modify the screenshot for enhancing edge and/or contour detection (Paduraru at p. 70 (section VI-A) discloses applying “smoothing” as a pre/post-processing technique to the rendered video-game screenshots, together with edge detection, both using standard OpenCV features. Smoothing corresponds to the claimed noise/blur filtering. It would have been obvious to one of ordinary skill in the art at the time the invention was made to perform such smoothing on the screenshot prior to edge detection, for the express purpose of suppressing image noise that would otherwise produce spurious edges — this is the conventional order of operations in standard edge-detection methodology. Indeed, the Canny edge-detection algorithm cited by Xie at reference [5] includes Gaussian smoothing as the first step of the algorithm precisely for this purpose. Performing smoothing for the purpose of enhancing edge and/or contour detection is the well-established conventional practice in the field of computer vision). The sequence smoothing → edge detection → contour detection → bounding boxes is the conventional order of operations in standard computer-vision element detection (as exemplified by the Canny edge detector’s built-in Gaussian smoothing step and the Suzuki/Abe border-following algorithm operating on a binary edge map). It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine Paduraru’s explicitly-disclosed smoothing pre-processing step with Xie’s gradient-based edge detection, Sklansky’s contour detection, and bounding-box export, in the conventional order, to obtain the pipeline of claim 21. The combination is the use of known techniques (smoothing for noise reduction, gradient-based edge detection, border-following contour detection, bounding-box export) according to their established functions to yield a predictable result — accurate candidate-location identification of UI elements. KSR; MPEP §§ 2143(I)(A) and 2143(I)(C).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Mulong Xie et al. (“UIED: A Hybrid Tool for GUI Element Detection”, hereinafter Xie) in view of Ciprian Paduraru et al. (“Automated Game Testing Using Computer Vision Methods”, hereinafter Paduraru) and further in view of Guoqing Liu et al. (“Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation”, hereinafter Liu). With respect to claim 15 (currently amended), Xie in view of Paduraru is silent to disclose wherein performing, by one or more of the processors, the one or more actions in the video game based upon the detected one or more user interface elements for testing the user interface of the video game comprises: generating a map of the user interface of the video game by interacting with the detected one or more user interface elements (Liu at p. 237 (Abstract, section I) discloses an automated game-testing agent (“Inspector”) that operates purely on screenshot/pixel input and comprises three modules: “game space explorer, key object detector, and human-like object investigator.” Liu p. 240 (Figure 3, “decision flow of the Inspector agent”) and Liu p. 240 (section III-D) teach that the agent (i) detects key objects (UI elements) on the screenshot, (ii) performs a human-like investigation — i.e., interacts with the detected objects — and then (iii) continues exploring the game space. Liu p. 241 (section IV-B, Figures 5–8) discloses that the agent generates and outputs a coverage map of the explored game space resulting from these interactions (e.g., Figure 6 and Figure 8 each show the resulting 3D scatter plot of explored player-location coverage). The generated coverage map is generated by interacting with the detected UI elements (the key objects), and constitutes a map of the video game user interface). It would have been obvious to one of ordinary skill in the art at the time the invention was made before the effective filing date of the claimed invention to combine Liu’s pixel-based game-space exploration and coverage-map generation with the Xie/Paduraru combination as set forth above. All three references are directed to the same field of automated software testing using computer vision applied to screenshots, and Paduraru and Liu in particular are both directed to automated video-game testing using only the screenshot/pixel output of the game. Liu expressly teaches the predictable benefit of the addition: “In addition to game space exploration, our agent can also detect key objects and take human-like behaviors to interact with the objects, so as to better expose hidden bugs” (Liu p. 238, section I), and the systematic interaction produces an explicit coverage map of the explored game space (Liu Figures 5–8) that yields higher test coverage of the UI elements present in the video game. The combination is the use of a known technique (Liu’s exploration-with-coverage-map generation) to improve a known device (the Xie/Paduraru automated game UI testing pipeline) in the same way that Liu used the technique to improve automated game testing in Liu’s own framework, and yields the predictable improvement of higher UI test coverage).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANIBAL RIVERACRUZ whose telephone number is (571)270-1200. The examiner can normally be reached Monday-Friday 9:30 AM-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hyung S Sough can be reached at 5712726799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANIBAL RIVERACRUZ/Primary Examiner, Art Unit 2192
Prosecution Insights