DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending in this office action.
Claims 1, 13 and 17 are amended.
NB: claim 17 (and associated dependent claims) is a system claim, and includes a processing device". And as the specification [0031]" The processing device 62 is a hardware device for executing software instructions", claim 17 and its corresponding dependent claims are statutory.
Response to Arguments
Applicant's arguments filed 03/24/2026 have been fully considered but they are not persuasive.
In response to 101 abstract idea, the invention is directed to:
Gathering: the system in this invention captures data as screen shot and store them.
Analysis: the screenshots are analyzed; metadata are determined and is associated with each screenshot.
And finally, the screenshot as a file is displayed for vertical scrolling. The court have indicated that : Gathering and analyzing information using conventional techniques and displaying the result, TLI Communications, 823 F.3d at 612-13, 118 USPQ2d at 1747-48; may not be sufficient to show an improvement to technology.
The invention in this application is directed to bug detection (glitches), debug and fix any problems within the software[0044] and may overcome 101 abstract idea.
In response to the amended claim limitation, Brown et al discloses a screenshot capture, where each screen shot includes respective metadata. The set of screenshots are used to create a thumbnail and displayed to the user for troubleshooting: See Brown Fig.4B, Fig. 8, 11A-B. 23A-B for thumbnail visualization.
Argument:
Turning to the rejections based on Shtuchkin, Golubev and Brown, Applicant respectfully submits that the remarks from the previously submitted response are still applicable to these references, and rather than reiterate them, they are incorporated herein. Further, Applicant submits that such references, either alone or in combination, fail to teach, inter alia, creating the single image file being a newly created file comprising the sequence of screenshots separated by a set of header, each header in the set of headers comprising a web driver application program interface (API) for automating the respective user input action and change in the image displayed by the UI. A careful review of each of the references reveals that such references are silent as tot eh web driver API and its corresponding functionality as presently recited in the claims.
Response:
The issue in the argument is that arts of record do not create a single file that has metadata/header that include web browser a sequence of screenshots and associated metadata/header and displayed in a vertical column.
Shtuchkin discloses capturing screenshots of a GUI associated with an application. Each screenshot is associated with metadata. Those screenshot and associated metadata are used to create a flow with respect to their time of capture/execution. The flow is stored in a repository, and can be displayed in a user interface for a user to debug the application. Among the web browser used for this feature is :Selenium which is an open-source web application testing tool. Selenium can execute user operations in a browser and perform other operations to test functionalities of the web application . For example, a test case for testing login functionality may include operations for accessing the uniform resource locator (URL) of the web application , selecting text boxes in the web page for entering a user ID and password, and selecting a login button after entering the user ID and password. The user ID and password may be provided in the test data . Selenium may run a test script for the test case to perform these operations, including retrieving the user ID and password, from the test data and entering the information in the selected text boxes via the browser.
Shtutchin uses the same technology of selenium to edit modify and execute the test case:
[0064] For web-based applications, the automated testing utility 130 may be implemented as a plug-in for a web browser. An embodiment of an automated testing utility 130 for a web browser is Selenium IDE, Selenium RC, or Selenium webdriver developed by Selenium. In alternative embodiments, the automated testing utility 130 is for operating system level actions. An example of an operating system level automated testing utility is Macro Express sold by Insight Software Solutions.
[0065] These automated testing utilities do not require that a developer code each step in the script from scratch. Instead, the utility may include controls to record and edit specific actions or events, and additional controls that facilitate adding test specific commands to the script. For example, a first control may be used to record user clicks, and a second control may be used to insert wait/validation conditions. These controls interpret the received input and transform input into script level syntax that can then be executed by the automated testing utility 130 in a step-by-step fashion at a later time.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
the claimed invention is directed to an abstract Idea without significantly more.
Claims 1, 13 and 17 recite:
“Detect/(ing) ,by an application, a plurality of user input actions performed during use of an app”;
“Create(/ing) ,by the application, a header associated with each screenshot, each header including information associated with a respective user input action and/or information associated with a change in the image displayed by the UI during use of the app in response to a respective user input action being performed;
“Creating, by the application, a single image file, the single image file being a newly created file comprising the sequence of screenshots separated by a set of header each header in the set of headers comprising a web driver application program interface (API) for automating the respective user input action and change in the image displayed by the UI;;
that are certainly a mental process that a person can carry out mentally through observation, evaluation, judgment and/or opinion, or even with the aid of pen and paper.
Claims 1, 13 and 17 additionally recite:
“Capture(/ing), by the application, without user input, a sequence of screenshots representing images displayed by a User Interface (UI) during the use of the app, each screenshot representing an image displayed by the UI at a specific point in time ranging from before a relevant user input action is performed to after the relevant user input action is performed…”.
“store/(ing), by the application, the sequence of screenshots and headers in a single image file,.”
“Causing display, by the application, of the single image file within the user interface (UI), the caused display rendering the single image file such that the sequence of screenshots are displayed as vertically joined..”
While claims 1 and 17 additionally recites:
“By the application”, “A non-transitory computer-readable medium tangibly encoded with computer logic, that when executed, cause a processor to…” and “a processing device; and a memory device configured to store computer logic for enabling the processing device” respectively.
The additional elements “By the application”, “a processing device” , “a processor” ,”a memory device configured to…” and “non-transitory computer-readable medium” are directed to generic computer components which are recited at a high level of generality, but to nothing more than an instruction implement “to apply” the abstract idea using a generic computer. See MPEP 2106.05(f).
The additional elements “…Capture(/ing) a sequence of screenshots …” and “store/(ing) the sequence of screenshots…” and” causing display…” are directed to storing, retrieving and manipulating data, that is mere data gathering/storing/output and does nothing more than adding insignificant extra solution activity to the judicial exception, that is a mere data gathering. See MPEP 2106.05(g).
Claims 1, 13, 17 additional elements do not add meaningful limits to practicing the abstract idea, but to nothing more than an instruction to apply the abstract idea using a generic computer. Thus, the additional elements fail to integrate the judicial exception into a practical application.
Claims 1, 13, 17 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with regard to integration of the abstract idea into a practical application, the additional elements “By the application”, “a processing device” , “a processor” ,”a memory device configured to…” and “non-transitory computer-readable medium” are generic computer component used as a tool to perform the abstract idea.
With regard to the additional element “…Capture(/ing) a sequence of screenshots …” and “store/(ing) the sequence of screenshots…”, and” causing display…” the court have found and identified retrieving/storing/manipulating information as well understood, routine conventional activity in the art. See MPEP 2106.05(d).
Accordingly, the additional elements do not provide an inventive concept, thus claims 1, 13, 17 are not patent eligible.
-Dependents claims 2-12, 14-16 and 18-20:
claims 2, 14 recite “…open the single image file in response to a troubleshooting request; and receive instructions from a troubleshooter to scroll through the sequence of screenshots and headers” …”, Claim 3 recites” storing the sequence of screenshots and headers includes storing the screenshots and headers” and Claim 12 and 19 recite: “capture a first screenshot before the button press action;…and capture a second screenshot of a new screen displayed by the UI after the button press action” that are data gathering/storing, and as discussed above it fails to integrate the judicial exception into a practical application nor sufficient to amount to significantly more than the judicial exception.
Claim 4 and 15 recite “…add a highlighting feature to each of one or more screenshots….” and Claim 12 and 19 recite: “…add a highlighting feature to the first screenshot to highlight the button being pressed by the user; that is a mental process.
Claims 5, 6, 16 , 18 ,7, 8 9-11, 12 and 19- 20 recite:”…wherein each highlighting feature is….”, “wherein each header includes”, “….wherein the single image file is….”,” wherein one or more of the user input actions are….”, “….wherein the app is on….”,” wherein the non-transitory computer-readable medium is stored...”,”…wherein the user input actions include…”, “…wherein one or more of the plurality of user input actions includes…” and “…wherein the system is a cloud-based…”, that is data describing the data information used in the abstract idea and as discussed above it fails to integrate the judicial exception into a practical application nor sufficient to amount to significantly more than the judicial exception.
Claims 2-12 recite “The non-transitory computer-readable medium” and “processor” that is a generic computer components and as discussed above it fails to integrate the judicial exception into a practical application nor sufficient to amount to significantly more than the judicial exception
-Dependents claims 2-12, 14-16 and 18-20 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 8-15 and 17-20 rejected under 35 U.S.C. 103 as being unpatentable Shtuchkin et al US20170277625A1 in view of Golubev et al 20210081309A1 and Brown et al US20220374344A1.
As per claim 1, Shtuchkin discloses a non-transitory computer-readable medium tangibly encoded with computer logic, that when executed, cause a processor to:
detect , by an application, a plurality of user input actions performed during use of an app:
[0025] According to one technique, during an automated test of the web application, screenshots are taken of the web application's graphic user interface(GUI) . Screenshots are taken based on a set of criteria that triggers a screen capture immediately prior to or immediately after certain types of input and certain types of web application events” ;
capture , by the application, without user input, a sequence of screenshots representing images displayed by a User Interface (UI) during the use of the app:
[0139] Some iterations of the program flow may result in screenshot and metadata capture associated with an input step. For example, at step 304 the monitoring agent 132 may check a step in an executing script to determine the step contains an input action. The monitoring agent 132 compares the input action to screen capture and annotation criteria for that automated test to determine that the automated testing utility 130 should capture a screenshot before the action occurs”;
[0082]“In some embodiments, the screen captures occur just after a computer program responds to actions taken by a simulated user. These screen captures are taken relative to the monitoring agent 132 identifying the trigger action or event occurring in the script and waiting for the action event to actually happen in the computer program or web application”;
each screenshot representing an image displayed by the UI at a specific point in time:
[0076]“Screen captures are taken based on a set of criteria that trigger screen captures when certain types of steps are identified in the script, when certain types of actions or events occur in the web application, or a combination thereof. Thus, screen captures may be taken before they are processed by the automated testing utility 130 or after they are displayed in the GUI of the web application 128.”;
Create, by the application, a header associated with each screenshot, each header including information associated with a respective user input action and/or information associated with a change in the image displayed by the UI during use of the app in response to a respective user input action being performed:
[0139] Some iterations of the program flow may result in screenshot and metadata capture associated with an input step. For example, at step 304 the monitoring agent 132 may check a step in an executing script to determine the step contains an input action. The monitoring agent 132 compares the input action to screen capture and annotation criteria for that automated test to determine that the automated testing utility 130 should capture a screenshot before the action occurs. For example, clicking a button may be a trigger action that requires capturing a screenshot before the action occurs. At step 308, the monitoring agent 132 identifies, from the executing script, whether a DOM element (e.g., a button) is associated with the trigger action. Continuing step 308, the monitoring agent 132 compares the trigger action against the screen capture and annotation criteria to determine if location data and/or size data of the DOM element must also be captured. Continuing the example with the DOM element being a visible element such as a button, location and size metadata of the visible element is captured at step 310”;
creating, by the application, a single image file, the single image file being a newly created file comprising the sequence of screenshots separated by a header single image file;
[0136] “Assuming no location data is necessary, the monitoring agent 132 stores the new screenshot data and context metadata at step 312, and the process loops back to step 304”;
[0150] At step 314, the automated testing utility 130 completes the automated test, and at step 316, the monitoring agent 132 sends the data that was stored in each 312 step to the annotation engine 134 to generate a sequence of annotated screenshots. Each screenshot may be processed with its corresponding data to determine if an element needs to be highlighted and what contextual metadata needs to be added. Then an annotation overlay is added to the screenshot. After each screenshot is annotated, a sequence of annotated screenshots are stored as a flow.
[0151] FIGS. 4A-4E are example diagrams of a flow illustrating a simulated user adding a member to a limited team based on script 200. Annotation engine 134 collects data stored by monitoring agent 132 and annotates each screenshot accordingly to create a flow. Some screenshots may have multiple element annotations depending on de-duplication step 312 removing a screenshot but keeping location data and context data regarding that screenshot.
each header in the set of headers comprising a web driver application program interface (API) for automating the respective user input action and change in the image displayed by the UI:
[0064] For web-based applications, the automated testing utility 130 may be implemented as a plug-in for a web browser. An embodiment of an automated testing utility 130 for a web browser is Selenium IDE, Selenium RC, or Selenium webdriver developed by Selenium. In alternative embodiments, the automated testing utility 130 is for operating system level actions. An example of an operating system level automated testing utility is Macro Express sold by Insight Software Solutions.
[0065] These automated testing utilities do not require that a developer code each step in the script from scratch. Instead, the utility may include controls to record and edit specific actions or events, and additional controls that facilitate adding test specific commands to the script. For example, a first control may be used to record user clicks, and a second control may be used to insert wait/validation conditions. These controls interpret the received input and transform input into script level syntax that can then be executed by the automated testing utility 130 in a step-by-step fashion at a later time.
and store, by the application, the sequence of screenshots and headers in a single image file and causing display, by the application, of the single image file within the user interface (UI),
[0028] According to another technique, the flow repository stores multiple versions of a flow. For example, a new version of a flow may be stored in the flow repository after a feature of the web application has been changed, modified, or updated”
[0127] Each annotated screenshot may contain indications as to the automated test to which it corresponds, and further defines the step within the automated test in which the screenshot was taken. Thus, a user interface that displays the flows of annotated user interfaces may also pull the corresponding script from the script repository 142, so a user can view the script of an automated test alongside the flow for that automated test.
But not explicitly:
The specific point in time ranging from before a relevant user input action is performed to after the relevant user input action is performed;
the caused display rendering the single image file such that the sequence of screenshots are displayed as vertically joined.
Golubev discloses:
The specific point in time ranging from before a relevant user input action is performed to after the relevant user input action is performed
[0079] In some embodiments, the process of testing by the test executor 314 may include obtaining a task from a test manager 312, walking through the application 133 (e.g., by identifying and interacting with GUI elements) and obtaining test results such as captured screenshots of the GUI of the application 133 before, during, and/or after interaction with the various GUI elements”;
[0074] “The diagnostic outputs generated by screenshot analyzer 318 may also be stored in data store 124 as tags or other metadata associated with the corresponding screenshots or as separate data.”;
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Golubev into teachings of Shtuchkin for performing automated testing of a graphical user interface (GUI) of an application. generating element map data that is indicative of the identified correspondences between interactive elements. This element map data can then be used to, for example, detect errors in the GUI of the target application and/or detect when a test executor has returned to a previously accessed page in the GUI of the target application, this allow to avoid a difficulty of comparing different pages in the GUI [Golubev 0016].
But not explicitly:
the caused display rendering the single image file such that the sequence of screenshots are displayed as vertically joined.
Brown discloses:
the caused display rendering the single image file such that the sequence of screenshots are displayed as vertically joined.
[0105] “In some embodiments, the test case data entity contains data that can be used to generate a test case visualization user interface for the test case associated with the test case data entity, such as a test case visualization user interface that depicts: (i) one or more test case page images associated with the test case, and (ii) for each test case page image of the one or more test case page images, a set of test case visualizations corresponding to each test case step in the set of test case steps that relate to the particular test case page image.
Examiner interpretation:
The thumbnail is a new file, that is created using the image captured and associated context/metadata. For example, in fig.4B. the thumbnail 424 includes 3 screenshot, in a top of each screenshot is a header that includes a step number(1 of 3, 2 of 3 , 3 of 3). The thumbnail is vertically displayed in a left column and the screenshot are separated by their steps in a header. See also Fig. 8, 11A-B. 23A-B for thumbnail visualization.
Brown also discloses:
creating a single new file:
[0109]”As depicted in FIG. 4B, the test case visualization user interface 402 (corresponding to the “Documentation” navigation tab of the depicted environment) depicts test case page images and test case steps associated with a corresponding test case. In particular, a page thumbnail panel 424 of the test case visualization user interface 402 is configured to depict, for each webpage associated with the test case, a thumbnail image representation of an exemplary test case page image associated with the webpage. For example, the thumbnail image representation 424X corresponds to an exemplary test case page image for a first webpage associated with the test case, the thumbnail image representation 424Y corresponds to an exemplary test case page image for a second webpage associated with the test case, and the thumbnail image representation 424Z corresponds to an exemplary test case page image for a third webpage associated with the test case.
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Brown into teachings of Shtuchkin and Golubev for improving computational efficiency and operational reliability of test automation platforms, and operational reliability of software applications that are tested using the software application platforms. A user-friendly and intuitive test case generation techniques enhance the accuracy and reliability of test case data entities generated by test case developers, which in turn reduces the number of erroneous testing operations (e.g., erroneous automated testing operations) that are performed.[Brown 0039].
As per claim 2, the rejection of claim 1 is incorporated and furthermore Shtuchkin does not explicitly disclose:
wherein the computer logic further enables the processor to: open the single image file in response to a troubleshooting request; and receive instructions from a troubleshooter to scroll through the sequence of screenshots and headers.
Golubev discloses:
wherein the computer logic further enables the processor to: open the single image file in response to a troubleshooting request; and receive instructions from a troubleshooter to scroll through the sequence of screenshots and headers.
[0134] “For example, in response to detecting a user interaction, interactive element 1712c may dynamically expand to display results of the test scenario in the form of screen captures 1714 of the target application taken by the test executor during the various steps associated with the test scenario, as depicted in FIG. 17. Additional details on how test results can be presented in a developer GUI are described with respect to FIGS. 18-21.”;
Examiner interpretation: See also Brown fig.4B, Fig. 8, 11A-B, 23A-B for thumbnail visualization and scrolling.
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Golubev into teachings of Shtuchkin for performing automated testing of a graphical user interface (GUI) of an application. generating element map data that is indicative of the identified correspondences between interactive elements. This element map data can then be used to, for example, detect errors in the GUI of the target application and/or detect when a test executor has returned to a previously accessed page in the GUI of the target application, this allow to avoid a difficulty of comparing different pages in the GUI [Golubev 0016].
As per claim 3, the rejection of claim 2 is incorporated and furthermore Shtuchkin does not explicitly discloses:
wherein storing the sequence of screenshots and headers includes storing the screenshots and headers in a vertical column to enable vertical scrolling;
Golubev discloses:
wherein storing the sequence of screenshots and headers includes storing the screenshots and headers in a vertical column to enable vertical scrolling;
Fig. 18, fig. 19 [0136] In some embodiments, the developer GUI may enable the developer user 152 to zoom in on the screen captures to view how the GUI of the target application 132 responded to various interactions. FIG. 19 shows an example screen 1910 of the developer GUI that shows a zoomed in portion of each of the respective screen captures 1812, 1814, and 1816 of FIG. 18. Specifically, screen capture 1912 shows a zoomed in portion of screen capture 1812 depicting the GUI of the target application 132 in the first state (i.e., before any interaction). Screen capture 1914 shows a zoomed in portion of screen capture 1814 depicting the GUI of the target application 132 in the second state (i.e., after entering the number 42 but before pressing the “add” button). Screen capture 1916 shows a zoomed in portion of screen capture 1816 depicting the GUI of the target application 132 in the third state (i.e., after pressing the “add” button).
Examiner interpretation: in fig. 18 screen shot and respective metadata are displayed horizontal while in fig. 19 screenshot are displayed vertically. See also Brown Fig 4B, Fig. 8, 11A-B, 23A-B for thumbnail visualization and vertical scrolling.
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Golubev into teachings of Shtuchkin for performing automated testing of a graphical user interface (GUI) of an application. generating element map data that is indicative of the identified correspondences between interactive elements. This element map data can then be used to, for example, detect errors in the GUI of the target application and/or detect when a test executor has returned to a previously accessed page in the GUI of the target application, this allow to avoid a difficulty of comparing different pages in the GUI [Golubev 0016].
As per claim 4, the rejection of claim 1 is incorporated and furthermore Shtuchkin discloses:
wherein the computer logic further enables the processor to add a highlighting feature to each of one or more screenshots, each highlighting feature representing a respective user input action being performed:
[0107] “The annotation engine 134 takes the captured screenshots and the metadata captured in association with those screenshots to create an annotation overlay on each screenshot. Annotation engine 134 receives a set of criteria that defines how to annotate the received screenshots. Alternatively, annotation engine 134 may be hardcoded with a set of criteria that defines how to annotate the received screenshots. The set of criteria may include clicking a particular DOM element causes that element to be highlighted yellow; validating the occurrence of a particular DOM element causes that element to be highlighted purple; and loading a particular page does not cause any highlighting annotation.
As per claim 5, the rejection of claim 4 is incorporated and furthermore Shtuchkin discloses:
wherein each highlighting feature is a colored outline around a selected button or field related to the respective user input action.
[0116]” An example of an event occurring in a web application is a script awaiting the creation of “confirm password” text so the user can input the same password into a second text input box after the password is input into a first text input box. The annotated diagram gets a highlight around the “confirm password” text, because that event needs to occur before the script moves to the next step”;
As per claim 8, the rejection of claim 1 is incorporated and furthermore Shtuchkin does not explicitly disclose:
wherein one or more of the user input actions are entry actions associated with typing alphanumeric characters in a text field, and wherein, for each of the entry actions, the computer logic enables the one or more processing devices to capture a screenshot immediately before a user submits the typed alphanumeric characters or clicks on an enter button.
Golubev discloses:
wherein one or more of the user input actions are entry actions associated with typing alphanumeric characters in a text field, and wherein, for each of the entry actions, the computer logic enables the one or more processing devices to capture a screenshot immediately before a user submits the typed alphanumeric characters or clicks on an enter button:
[0135] “Screen capture 1812 shows the GUI of the target application 132 in a first state prior to entering the number, screen capture 1814 shows the GUI of the target application 132 in a second state after entering the number (“42”) but prior to pressing the “add” button, and screen capture 1816 shows the GUI of the target application 132 in a third state after pressing the “add” button”;
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Golubev into teachings of Shtuchkin for performing automated testing of a graphical user interface (GUI) of an application. generating element map data that is indicative of the identified correspondences between interactive elements. This element map data can then be used to, for example, detect errors in the GUI of the target application and/or detect when a test executor has returned to a previously accessed page in the GUI of the target application, this allow to avoid a difficulty of comparing different pages in the GUI [Golubev 0016].
As per claim 9, the rejection of claim 1 is incorporated and furthermore Shtuchkin does not explicitly disclose:
wherein the app is one of a mobile app running on a mobile device or a web application on a web browser on a computing device.
Golubev discloses:
wherein the app is one of a mobile app running on a mobile device or a web application on a web browser on a computing device.
[0026] “Interface 153 may include a GUI configured to receive user inputs and present visual outputs. The interface 153 may be accessible via a web browser, desktop application, mobile application, or over-the-top (OTT) application, or any other type of application at developer user device 150”;
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Golubev into teachings of Shtuchkin for performing automated testing of a graphical user interface (GUI) of an application. generating element map data that is indicative of the identified correspondences between interactive elements. This element map data can then be used to, for example, detect errors in the GUI of the target application and/or detect when a test executor has returned to a previously accessed page in the GUI of the target application, this allow to avoid a difficulty of comparing different pages in the GUI [Golubev 0016].
As per claim 10, the rejection of claim 9 is incorporated and furthermore Shtuchkin discloses:
wherein the non-transitory computer-readable medium is stored on a remote device in communication with the mobile device or the computing device:
[0023] “ For example, in the case of a website with interactive elements, automated testing platform 120 may be configured to test the interactive elements associated with the website as presented via one or more different web browser applications.”;
As per claim 11, the rejection of claim 1 is incorporated and furthermore Shtuchkin discloses:
wherein the user input actions include one or more of a keystroke, a text entry, a virtual button press, a mouse click, a keypad entry, an option selection, or a screen manipulation command:
[0130]”For example, as part of the automated testing process, one or more test executors 314 will crawl the target application 132 to discover and interact with various interactive elements (e.g., clicking buttons, clicking links, clicking pull-down menus, filling out forms, etc.) and will obtain results (e.g., screen captures) based on the testing.”;
As per claim 12, the rejection of claim 1 is incorporated and furthermore Shtuchkin discloses:
wherein one or more of the plurality of user input actions includes one or more button-press actions and wherein:
[0085]“crawling the GUI of the target application 132 may include detecting and interacting with one or more interactive elements in the GUI according to an automated testing scenario. The one or more interactive elements may include, for example, buttons, pull-down menus, editable text fields, etc. Interacting with the interactive elements may therefore include, for example, pressing a button, scrolling through a pull-down menu and selecting an item in the pull-down menu, entering data in the editable text field, etc.”;
in response to each button press action, the computer logic further enables the processor to: capture a first screenshot before the button press action:
[0135] “Screen capture 1812 shows the GUI of the target application 132 in a first state prior to entering the number, screen capture 1814 shows the GUI of the target application 132 in a second state after entering the number (“42”) but prior to pressing the “add” button, and screen capture 1816 shows the GUI of the target application 132 in a third state after pressing the “add” button. “;
add a highlighting feature to the first screenshot to highlight the button being pressed by the user;
[0137] “Specifically, the visual augmentation 1934 includes a highlight that surrounds a region of the GUI of the target application 132 corresponding to the “add” button. “
and capture a second screenshot of a new screen displayed by the UI after the button press action.
[0137] “This visual augmentation 1934 indicates to the developer user 152 that the “add” button is pressed resulting in the next screen capture 1916. ‘;
Claims 13, 14, 15 are the method claim corresponding to the non-transitory computer-readable medium claims 1, 2, 4 and rejected under the same rational set forth in connection with the rejection of claims 1, 2, 4 above.
Claims 17, 18, 19 are the method claim corresponding to the non-transitory computer-readable medium claims 1, 11, 12 and rejected under the same rational set forth in connection with the rejection of claims 1, 11, 12 above.
As per claim 20, the rejection of claim 17 is incorporated and furthermore Shtuchkin does not explicitly disclose:
wherein the system is a cloud-based server in communication with an end user device configured to run the app;
Golubev discloses:
wherein the system is a cloud-based server in communication with an end user device configured to run the app:
[0022] In some embodiments, certain components of automated testing platform 120 may be hosted or otherwise provided by separate cloud computing providers such as Amazon Web Services (AWS)™ or Microsoft Azure™”;
[0023] The automated testing platform 120 can be implemented to perform automated testing of a target application 132. The target application 132 may include any type of application (or app) configured to run on personal computers (e.g., for Windows™, MacOS™, etc.), applications configured to run on mobile devices (e.g., for Apple™ iOS, Android™, etc.), web applications, websites”;
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Golubev into teachings of Shtuchkin for performing automated testing of a graphical user interface (GUI) of an application. generating element map data that is indicative of the identified correspondences between interactive elements. This element map data can then be used to, for example, detect errors in the GUI of the target application and/or detect when a test executor has returned to a previously accessed page in the GUI of the target application, this allow to avoid a difficulty of comparing different pages in the GUI [Golubev 0016].
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable Shtuchkin et al US20170277625A1 in view of Golubev et al US20210081309A1 and further in view of Brown et al US20220374344A1 and Budurean et al US9934129B1.
As per claim 6, the rejection of claim 1 is incorporated and furthermore Shtuchkin does not explicitly disclose:
wherein each header includes a timestamp defining a date and time when a respective screenshot was displayed by the UI during the use of the app.
Budurean discloses:
wherein each header includes a timestamp defining a date and time when a respective screenshot was displayed by the UI during the use of the app.
Col 8 lines 20-28 “In addition, or alternatively, the metadata may define a state of the target device 166 (e.g., device type, orientation, locale, time, test execution, and any other parameters or characteristics of the target device or application) at a time when the screenshot 164 is made. For example, the metadata of one of screenshots 164 may include information about a graphical button of a GUI as well as the size, position, location, color, label, or other information about the graphical button. In some examples, the metadata of one of screenshots 164 may include information about the screen orientation, size, application state, or other information about the target device or application when that particular screenshot 164 was made”;
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Budurean into teachings of Shtuchkin, Golubev and Brown to identify, and cluster, similar screenshots generated during an automated test of an application even if the application test system executes the automated test across a broad range of different target devices and/or in a variety of different execution environments. quickly comparing the metadata between two or more screenshots to determine what differences there are if any. For example, using tree comparison techniques, developer service module 162 may determine whether the metadata of two screenshots is isomorphic, determine that the two screens are not compatible and therefore not worth bothering with a similarity score [Budurean Col 9 line 20-26].
Claim 16 is the method claim corresponding to the non-transitory computer-readable medium claim 6 and rejected under the same rational set forth in connection with the rejection of claim 6 above.
Claim 7 rejected under 35 U.S.C. 103 as being unpatentable Shtuchkin et al US20170277625A1 in view of Golubev et al US20210081309A1 and further in view of Brown et al US20220374344A1 and Levi et al US20160139914A1.
As per claim 7, the rejection of claim 1 is incorporated and furthermore Shtuchkin does not explicitly disclose:
wherein the single image file is a jpg file in which image data is compressed.
Levi discloses:
wherein the single image file is a jpg file in which image data is compressed.
[0057] Screen shot 220 may be an image (e.g., a GIF, JPG, PNG or the like) that shows a part of the UI that is displayed to the user in conjunction with the related screen state. Each reduced stage 2 properties file 224 may be a subset of a stage 2 properties file (e.g., 214) for the corresponding language.:
[0058] Translation package creation module 210 may then, for each screen state, package together screen shot 220, the at least one reduced stage 2 properties file 224 (e.g., one for each supported language), and perhaps a reduced stage 1 properties file 222. As explained above, in alternate embodiments, module 210 may package one TP per screen state or one TP per screen state and up to one per supported language. Translation package 218 may take the form of any file or folder that may contain multiple files (e.g., a .zip file, a .rar file, a digital folder or the like).
It would have obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of cited references. One of ordinary skill in the art before the effective filling date of the claimed invention would have been motivated to incorporate the teachings of Levi into teachings of Shtuchkin, Golubev and Brown for each of a multiple screen states, a screen shot and at least one reduced properties file that includes language specific elements displayable in such a screen shot. The present disclosure describes receiving user action data (e.g., provided as a result of testing) and recreating the user's actions by emulating the environment by which the user interacted with the software product and analyzing the user action data. During such recreation, the multiple screen states may be detected and the reduced properties files and screen shots are generated to optimize storage size [Levi 0045]..
Pertinent arts:
US20140074452A1:
A first and second screenshots of a first and second screens produced by an application may be captured and stored in a model. Transition information related to a transition from the first to the second screen may be obtained and stored in the model. A session may be recorded by recording screenshots and related events.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRAHIM BOURZIK whose telephone number is (571)270-7155. The examiner can normally be reached Monday-Friday (8-4:30).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wei Y Mui can be reached at 571-270-2738. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRAHIM BOURZIK/ Examiner, Art Unit 2191
/WEI Y MUI/ Supervisory Patent Examiner, Art Unit 2191
Prosecution Insights