DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 10, 2026 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-6, 8-9, 11-12, 14-15, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Bergman et al. (US PGPUB 2011/0276944; hereinafter “Bergman”) in view of Liu et al. (US PGPUB 2024/0303184; hereinafter “Liu”), Shi et al. (US PGPUB 2022/0413997; hereinafter “Shi”), Cobb, JR. et al. (US PGPUB 2016/0132421; hereinafter “Cobb”) and Lundberg et al. (US PGPUB 2013/0268260; hereinafter “Lundberg”).
Claim 1: (Currently Amended)
Bergman teaches a method for automated testing of software, the method comprising:
obtaining a test script for testing the software, the test script comprising a sequence of actions that are sequentially performed within a user interface of the software for automated software testing ([0107] “the software testing application 550 includes … a script 570, and a NL test 571.” [0109] “The script 570 is comprised of a hierarchy of test objects with associated actions that are broken down to primitive actions (e.g., a click is a mouse down followed by a mouse up action).” [0020] “The recorded sequence of steps can then be replayed and/or viewed.” [0126] “The natural language text both describes a location of a control displayed on the web page and instructs the software testing application to perform a UI event on the control.”);
executing the software based on the test script ([0121] “The script 570 provides the test action and test object to the replay engine 564. The replay engine generates and provides the user action to the browser 538.”);
determining that the test script includes an action that cannot be completed ([0122] “FIG. 6D shows a computer system executing the software testing application during a replay and recovery operation in accordance with an example embodiment. The replay operation proceeds as in the description of FIG. 6C, but the user action provided by the replay engine 564 to the browser 538 fails.”),
the action that cannot be completed comprising an action to be performed on a user interface element of the user interface that is not present on the user interface when executing the software based on the test script ([0068] “The record replay engine 100 transmits the text to the textual semantic engine 170 which, in turn, generates and transmits the UI control description to the semantic matching engine 190.” [0092] “FIG. 4 shows a method or algorithm for the semantic matching engine.” [0093] “According to block 400, find elements that match the description. Elements are filtered by context (context objects are the type of element being matched), and identify context objects by the textual semantic engine. Recursively match the object (elements, object description). If a match is not found…”);
inputting, into a natural language processing system, a query ([0122] “The replay engine then initiates a recovery operation, in which the NL test 571 provides the text step description to the NL semantic engine 566.” [0091] “The semantic matching engine 190 has input as layout hierarchy and UI control description and output as a unique reference to the UI control in the object hierarchy… This engine is responsible for finding a UI control that corresponds to a description of the control.”),
comprising a natural language instruction to select an element of one or more elements that are present when attempting to complete the action that is the most closely related to the user interface element that is not present ([0038] “The grammar for natural language tests includes, but is not limited to, one or more of the following:” [0039] “(1) Nouns that represent basic and common GUI controls (e.g., buttons, text fields, sliders, scroll bars, etc.).” [0040] “(2) Verbs that are applicable to controls (e.g., click, press, drag, type, enter, etc.).” [0042] “(4) Spatial Relations--prepositions supported: in, inside, next to, to the left of, to the right of, above, below, under, over, adjacent, etc.” [0043] “(5) Ordinals that are used to select the correct object among those that match a description.” [0093] “If a match is not found, then match by adjacent elements that match the keyword. Search an object of a different type (e.g., a label) that matches the description and search the objects around it. Next, return the ordinal element according to the ordinal value identified by the textual semantic engine (if no ordinal value is mentioned it will return the first one). More complex implementation can use soft decision making (i.e., rating each match using some metric and sorting the returned elements by their rate).” [0075] “The original text description itself can also be an identifier and can be used in replay if other identifiers fail to uniquely identify an object.”);
receiving, from the natural language processing system in response to the query, an identified element of the one or more elements that is most closely related to the user interface element that is not present; continuing the executing of the software by performing the action on the identified element on the user interface, wherein performing the action causes the user interface to transition to an updated user interface ([0093] “If a match is not found, then match by adjacent elements that match the keyword. Search an object of a different type (e.g., a label) that matches the description and search the objects around it. Next, return the ordinal element according to the ordinal value identified by the textual semantic engine (if no ordinal value is mentioned it will return the first one). More complex implementation can use soft decision making (i.e., rating each match using some metric and sorting the returned elements by their rate).” [0122] “the user action provided by the replay engine 564 to the browser 538 fails. The replay engine then initiates a recovery operation, in which the NL test 571 provides the text step description to the NL semantic engine 566, which in turn provides the action and object description to the semantic matching engine 568. The semantic engine 562 provides the semantic hierarchy to the semantic matching engine.”);
determining that the updated user interface includes a user interface element associated with at least one of the action of the test script and a next action of the test script ([0122] “The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538,” wherein the above language recites an alternative limitation and further wherein the disclosure regarding “[finding] the test object” in Bergman teaches the claimed “user interface element associated with … the action of the test script”.);
updating the test script, wherein updating the test script includes adding a new action to the test script for performing the action on the identified element based on a determination that the updated user interface includes the element associated with the action ([0122] “the semantic matching engine, which finds the test object... The step is re-recorded by the record engine 560 and the script 507 is updated.”); and
executing the software based on the updated test script ([0122] “FIG. 6D shows a computer system executing the software testing application during a replay and recovery operation… The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538. The step is re-recorded by the record engine 560 and the script 507 is updated,” wherein Fig. 6D shows that the step of updating the script, using the ‘Record Engine 560’ and the ‘Script 570’, is followed by the step of using the ‘Replay Engine 564’ to continue executing the software based on the updated test script, i.e. [0121] “The script 570 provides the test action and test object to the replay engine 564. The replay engine generates and provides the user action to the browser 538.”).
With further regard to Claim 1, Bergman does not teach the following, however, Liu teaches:
training a machine learning model to select an element from a set of elements that most closely relates to a specified action in a test script, the machine learning model trained on labeled training data that includes observational data of user interactions with a prior version of a software under test ([0040] “machine learning models may be trained in a supervised… manner with previously-generated test cases… In situations where the previously generated test cases are associated with a range of possible inputs for inputting into compiled/interpreted/scripted source code, machine learning models may be trained to recognize both the various of possible inputs associated with previous corpuses of source code, as well as changes made to previous corpuses of source code, where test cases are input into, and the expected results… in situations where graphical user interface interactions or key test scripts are used in testing previous corpuses of source code, historical graphical user interface interactions or historical key test scripts, as well as the previous corpuses of source code themselves, the results of testing, etc. are used in training machine learning models for testing of new corpuses of source code,” wherein “supervised” training necessarily includes the use of “labeled training data”.) and
an identification of a function of the software under test that correspond to the user interactions ([0032] “associated with the corpus of source code, hypothetical mouse movements (or other graphical user interface interactions, one or more key test scripts used in testing of the corpus of source code (as well as one or more mappings mapping the one or more key test scripts to one or more features of the original corpus of source code), or anything of the like, as one of skill in the art would understand.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Bergman with the training of a machine learning model as taught by Liu “So, the machine learning models trained in such a way will be capable of testing an updated corpus of source code” (Liu [0040]).
With further regard to Claim 1, Bergman in view of Liu does not teach the following, however, Shi teaches:
the machine learning model trained on user interface elements that correspond to actions of test scripts ([0006] “(ii) receiving a machine learning model trained to determine positions of groups of user interface elements and their corresponding constituent user interface elements in an image of a graphical user interface.” [0020] “automated locating of user interface elements during graphical user interface testing… If an element is not found, a scrolling technique is applied to generate an expanded virtual GUI of one or more groups of user interface elements, and the machine learning algorithm refreshes the index values using the expanded virtual GUI.” [0048] “a scrolling technique is utilized to generate a virtual mapping of the group of GUI elements of which the expected index value is expected to belong to, with the virtual mapping scrolled in one or more directions to reveal other GUI elements of the group, and a virtual mapping of the given group expanded to reveal the other GUI elements not previously visible. The GUI is then indexed again, accounting for the virtual mapping, and the mapping table is updated.”); and
inputting, into a natural language processing system comprising the trained machine learning model, a query ([0048] “GUI test execution mod 308 includes a set of machine language instructions for causing a computer processor to simulate a plurality of user inputs into the first GUI, where the set of machine language instructions locates GUI elements of the first GUI for simulated user interaction by querying the mapping table stored in mapping table datastore mod 307.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Bergman in view of Liu with the further training of a machine learning model as taught by Shi since “it overcomes the additional burden of text locator translation problems in multilingual environments, and is more efficient and easier to use” (Shi [0070]).
With further regard to Claim 1, Bergman in view of Liu and Shi does not teach the following, however, Cobb teaches:
responsive to determining that the action cannot be completed because the respective user interface element is not present, identifying elements of the user interface of the software which are present when attempting to complete the action during the execution of the software ([0040] “FIG. 6 illustrates an example method of adapting automated test scripts.” [0046] “the k-limited local crawl 612 is typically initiated when an exact match 604 is not found for an element referred to in a current test step of the sequence…The crawling process may explore the application up to depth k in either a breadth-first manner or a depth-first manner…a breadth-first exploration may work as follows. It enumerates all elements available on the current screen status…,” wherein “enumerates all elements” is equivalent to “identifying elements”.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Bergman in view of Liu and Shi with the identification of user interface elements following a test script failure as taught by Cobb in order “to locate missing elements by a variety of methods thus allowing the test sequence to progress to the following sequential step of the script” (Cobb [0034]).
With further regard to Claim 1, Bergman in view of Liu, Shi and Cobb does not teach the following, however, Lundberg teaches:
inputting, into a natural language processing system comprising the trained machine learning model and via an application programming interface (API) of the natural language processing system ([0114] “NLI core 702 may, in order to process user inputs or to create responses, require access to one or more external data sources or external services; such access is provided by one or more connection APIs 706, which manage interactions between NLI core 702 (and possibly also input interpretation module 707 and output generation module 708) and external data sources and services.”), and
a query comprising a natural language instruction structured as a question to the natural language processing system ([0098] “Requests 410 to natural language interaction engine 420 may be made using any of a number of user interface means known in the art, including but not limited to use of text-based requests 412 (for instance, generated by typing a question or command into a text entry field in a user interface, such as on a mobile device application, on a consumer device, on a web site, or in an email or other message).”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Bergman in view of Liu, Shi and Cobb with the use of an API and a query structured as a question as taught by Lundberg in order “to efficiently develop, test, deploy, and maintain useful natural language interaction applications” (Lundberg [0008]).
Claim 2:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches the method of claim 1, and Bergman teaches:
further comprising determining that the updated user interface includes the user interface element associated with a consecutive action of the test script ([0020] “Automated software testing can use a record/replay paradigm in which the software tester or quality assurance (QA) specialist records a sequence of steps that test a particular functionality of the application. The recorded sequence of steps can then be replayed and/or viewed.” [0122] “The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538,” see also Fig. 8 showing a sequence of consecutive test actions within “Tool 820”.).
Claim 3:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches the method of claim 2, and Bergman further teaches:
wherein updating the test script includes updating the test script to include the identified element based on a determination that the updated user interface includes the user interface element associated with the consecutive action of the test script ([0122] “The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538. The step is re-recorded by the record engine 560 and the script 507 is updated,” wherein since “the semantic matching engine… finds the test object” is the “determination that the updated user interface includes the user interface element.”).
Claim 5:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches the method of claim 1, and Bergman further teaches:
wherein determining that the test script includes the action that cannot be completed includes determining that the action refers to a first user interface element that is not one of the one or more elements of the user interface of the software ([0068] “The record replay engine 100 transmits the text to the textual semantic engine 170 which, in turn, generates and transmits the UI control description to the semantic matching engine 190.” [0092] “FIG. 4 shows a method or algorithm for the semantic matching engine.” [0093] “According to block 400, find elements that match the description. Elements are filtered by context (context objects are the type of element being matched), and identify context objects by the textual semantic engine. Recursively match the object (elements, object description). If a match is not found…”).
Claim 6:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches all the limitations of claim 1 as described above. Bergman in view of Shi, Cobb and Lundberg does not teach the following, however, Liu teaches:
wherein the test script is obtained by training a machine learning model with observational data of interactions of users with a prior version of the software ([0027] “Interactive development environment… then automatically generates one or more new test cases for testing of the updated corpus of source code.” [0031]-[0032] “Test cases stored in original source code module 212 may also include… one or more key test scripts used in testing of the corpus of source code.” [0039] “one or more machine learning models associated with debugging module 247 are involved in the generation of one or more new tests cases for testing of the updated corpus of source code.” [0040] “in situations where graphical user interface interactions or key test scripts are used in testing previous corpuses of source code, historical graphical user interface interactions or historical key test scripts, as well as the previous corpuses of source code themselves, the results of testing, etc. are used in training machine learning models for testing of new corpuses of source code.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Bergman in view of Shi, Cobb and Lundberg with the training of a machine learning model as taught by Liu for purposes “of automatically generating and managing test cases for the new versions of source code” (Liu [0026]).
Claims 8-9 and 11-12:
With regard to Claims 8-9 and 11-12, these claims are equivalent in scope to Claims 1, 3 and 5-6 rejected above, merely having a different independent claim type, and as such Claims 8-9 and 11-12 are rejected under the same grounds and for the same reasons as discussed above with regard to Claims 1, 3 and 5-6.
Claim 14: (Currently Amended)
Bergman teaches a system having a memory, computer readable instructions, and a processing system for executing the computer readable instructions, the computer readable instructions controlling the processing system to perform operations comprising ([0132] “the methods illustrated herein and data and instructions associated therewith are stored in respective storage devices, which are implemented as one or more computer-readable or computer-usable storage media or mediums.” See also Claim 10 of Bergman: “A tangible computer readable storage medium having instructions for causing a computer to execute a method”):
executing software based on a test script ([0121] “The script 570 provides the test action and test object to the replay engine 564. The replay engine generates and provides the user action to the browser 538.”),
the test script comprising a sequence of actions that are sequentially performed within a user interface of the software for automated software testing ([0107] “the software testing application 550 includes … a script 570, and a NL test 571.” [0109] “The script 570 is comprised of a hierarchy of test objects with associated actions that are broken down to primitive actions (e.g., a click is a mouse down followed by a mouse up action).” [0020] “The recorded sequence of steps can then be replayed and/or viewed.” [0126] “The natural language text both describes a location of a control displayed on the web page and instructs the software testing application to perform a UI event on the control.”); and
determining that the test script includes an action that cannot be completed ([0122] “FIG. 6D shows a computer system executing the software testing application during a replay and recovery operation in accordance with an example embodiment. The replay operation proceeds as in the description of FIG. 6C, but the user action provided by the replay engine 564 to the browser 538 fails.”),
the action that cannot be completed comprising an action to be performed on a user interface element of the user interface that is not present on the user interface when executing the software based on the test script ([0068] “The record replay engine 100 transmits the text to the textual semantic engine 170 which, in turn, generates and transmits the UI control description to the semantic matching engine 190.” [0092] “FIG. 4 shows a method or algorithm for the semantic matching engine.” [0093] “According to block 400, find elements that match the description. Elements are filtered by context (context objects are the type of element being matched), and identify context objects by the textual semantic engine. Recursively match the object (elements, object description). If a match is not found…”);
inputting, into a natural language processing system, a first query ([0122] “The replay engine then initiates a recovery operation, in which the NL test 571 provides the text step description to the NL semantic engine 566.” [0091] “The semantic matching engine 190 has input as layout hierarchy and UI control description and output as a unique reference to the UI control in the object hierarchy… This engine is responsible for finding a UI control that corresponds to a description of the control.”),
comprising a natural language instruction to select an element of one or more elements that are present when attempting to complete the action that is the most closely related to the user interface element that is not present ([0093] “If a match is not found, then match by adjacent elements that match the keyword. Search an object of a different type (e.g., a label) that matches the description and search the objects around it. Next, return the ordinal element according to the ordinal value identified by the textual semantic engine (if no ordinal value is mentioned it will return the first one). More complex implementation can use soft decision making (i.e., rating each match using some metric and sorting the returned elements by their rate).” [0075] “The original text description itself can also be an identifier and can be used in replay if other identifiers fail to uniquely identify an object.”);
receiving, from the natural language processing system in response to the first query, an identified element of the one or more elements that is most closely related to the user interface element that is not present; continuing the executing of the software by performing the action on the identified element on the user interface ([0093] “If a match is not found, then match by adjacent elements that match the keyword. Search an object of a different type (e.g., a label) that matches the description and search the objects around it. Next, return the ordinal element according to the ordinal value identified by the textual semantic engine (if no ordinal value is mentioned it will return the first one). More complex implementation can use soft decision making (i.e., rating each match using some metric and sorting the returned elements by their rate).” [0122] “ the user action provided by the replay engine 564 to the browser 538 fails. The replay engine then initiates a recovery operation, in which the NL test 571 provides the text step description to the NL semantic engine 566, which in turn provides the action and object description to the semantic matching engine 568. The semantic engine 562 provides the semantic hierarchy to the semantic matching engine.”);
determining that performing the action on the first identified element resulted in completion of the action ([0122] “The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538.”);
updating the test script ([0122] “the semantic matching engine, which finds the test object... The step is re-recorded by the record engine 560 and the script 507 is updated.”); and
executing the software based on the updated test script ([0122] “FIG. 6D shows a computer system executing the software testing application during a replay and recovery operation… The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538. The step is re-recorded by the record engine 560 and the script 507 is updated,” wherein Fig. 6D shows that the step of updating the script, using the ‘Record Engine 560’ and the ‘Script 570’, is followed by the step of using the ‘Replay Engine 564’ to continue executing the software based on the updated test script, i.e. [0121] “The script 570 provides the test action and test object to the replay engine 564. The replay engine generates and provides the user action to the browser 538.”).
With further regard to Claim 14, Bergman does not teach the following, however, Liu teaches:
training a machine learning model to select an element from a set of elements that most closely relates to a specified action in a test script, the machine learning model trained on labeled training data that includes observational data of user interactions with a prior version of a software under test ([0040] “machine learning models may be trained in a supervised… manner with previously-generated test cases… In situations where the previously generated test cases are associated with a range of possible inputs for inputting into compiled/interpreted/scripted source code, machine learning models may be trained to recognize both the various of possible inputs associated with previous corpuses of source code, as well as changes made to previous corpuses of source code, where test cases are input into, and the expected results… in situations where graphical user interface interactions or key test scripts are used in testing previous corpuses of source code, historical graphical user interface interactions or historical key test scripts, as well as the previous corpuses of source code themselves, the results of testing, etc. are used in training machine learning models for testing of new corpuses of source code,” wherein “supervised” training necessarily includes the use of “labeled training data”.) and
an identification of a function of the software under test that correspond to the user interactions ([0032] “associated with the corpus of source code, hypothetical mouse movements (or other graphical user interface interactions, one or more key test scripts used in testing of the corpus of source code (as well as one or more mappings mapping the one or more key test scripts to one or more features of the original corpus of source code), or anything of the like, as one of skill in the art would understand.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as disclosed by Bergman with the training of a machine learning model as taught by Liu “So, the machine learning models trained in such a way will be capable of testing an updated corpus of source code” (Liu [0040]).
With further regard to Claim 14, Bergman in view of Liu does not teach the following, however, Shi teaches:
the machine learning model trained on user interface elements that correspond to actions of test scripts ([0006] “(ii) receiving a machine learning model trained to determine positions of groups of user interface elements and their corresponding constituent user interface elements in an image of a graphical user interface.” [0020] “automated locating of user interface elements during graphical user interface testing… If an element is not found, a scrolling technique is applied to generate an expanded virtual GUI of one or more groups of user interface elements, and the machine learning algorithm refreshes the index values using the expanded virtual GUI.” [0048] “a scrolling technique is utilized to generate a virtual mapping of the group of GUI elements of which the expected index value is expected to belong to, with the virtual mapping scrolled in one or more directions to reveal other GUI elements of the group, and a virtual mapping of the given group expanded to reveal the other GUI elements not previously visible. The GUI is then indexed again, accounting for the virtual mapping, and the mapping table is updated.”); and
inputting, into a natural language processing system comprising the trained machine learning model, a first query ([0048] “GUI test execution mod 308 includes a set of machine language instructions for causing a computer processor to simulate a plurality of user inputs into the first GUI, where the set of machine language instructions locates GUI elements of the first GUI for simulated user interaction by querying the mapping table stored in mapping table datastore mod 307.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as disclosed by Bergman in view of Liu with the further training of a machine learning model as taught by Shi since “it overcomes the additional burden of text locator translation problems in multilingual environments, and is more efficient and easier to use” (Shi [0070]).
With further regard to Claim 14, Bergman in view of Liu and Shi does not teach the following, however, Cobb teaches:
responsive to determining that the action cannot be completed because the respective user interface element is not present, identifying elements of the user interface of the software which are present when attempting to complete the action during the execution of the software ([0040] “FIG. 6 illustrates an example method of adapting automated test scripts.” [0046] “the k-limited local crawl 612 is typically initiated when an exact match 604 is not found for an element referred to in a current test step of the sequence…The crawling process may explore the application up to depth k in either a breadth-first manner or a depth-first manner…a breadth-first exploration may work as follows. It enumerates all elements available on the current screen status…,” wherein “enumerates all elements” is equivalent to “identifying elements”.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as disclosed by Bergman in view of Liu and Shi with the identification of user interface elements following a test script failure as taught by Cobb in order “to locate missing elements by a variety of methods thus allowing the test sequence to progress to the following sequential step of the script” (Cobb [0034]).
With further regard to Claim 14, Bergman in view of Liu, Shi and Cobb does not teach the following, however, Lundberg teaches:
inputting, into a natural language processing system comprising the trained machine learning model and via an application programming interface (API) of the natural language processing system ([0114] “NLI core 702 may, in order to process user inputs or to create responses, require access to one or more external data sources or external services; such access is provided by one or more connection APIs 706, which manage interactions between NLI core 702 (and possibly also input interpretation module 707 and output generation module 708) and external data sources and services.”), and
a first query comprising a natural language instruction structured as a question to the natural language processing system ([0098] “Requests 410 to natural language interaction engine 420 may be made using any of a number of user interface means known in the art, including but not limited to use of text-based requests 412 (for instance, generated by typing a question or command into a text entry field in a user interface, such as on a mobile device application, on a consumer device, on a web site, or in an email or other message).”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as disclosed by Bergman in view of Liu, Shi and Cobb with the use of an API and a query structured as a question as taught by Lundberg in order “to efficiently develop, test, deploy, and maintain useful natural language interaction applications” (Lundberg [0008]).
Claim 15:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches the system of claim 14, and Bergman further teaches wherein the operations further comprise:
determining that performing the action on the first identified element did not result in completion of the action ([0122] “FIG. 6D shows a computer system executing the software testing application during a replay and recovery operation in accordance with an example embodiment. The replay operation proceeds as in the description of FIG. 6C, but the user action provided by the replay engine 564 to the browser 538 fails.”);
inputting, into the natural language processing system, a second query comprising the action and the one or more elements without the first identified element ([0075] “Furthermore, the description can include a collection of identifiers in which their intersection leads to the correct UI control. The original text description itself can also be an identifier and can be used in replay if other identifiers fail to uniquely identify an object.”);
receiving, from the natural language processing system in response to the second query, a second identified element of the one or more elements; continuing the executing of the software by performing the action on the second identified element on the user interface ([0122] “the NL semantic engine 566, which in turn provides the action and object description to the semantic matching engine 568. The semantic engine 562 provides the semantic hierarchy to the semantic matching engine.”);
determining that performing the action on the second identified element resulted in completion of the action ([0122] “The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538.”); and
updating the test script ([0122] “the semantic matching engine, which finds the test object... The step is re-recorded by the record engine 560 and the script 507 is updated.”).
Claim 17:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches the system of claim 14, and Bergman further teaches wherein the operations further comprise:
determining that performing the action on the first identified element did not result in completion of the action ([0122] “FIG. 6D shows a computer system executing the software testing application during a replay and recovery operation in accordance with an example embodiment. The replay operation proceeds as in the description of FIG. 6C, but the user action provided by the replay engine 564 to the browser 538 fails.”);
identifying one or more user interface elements of an updated user interface of the software; inputting, into the natural language processing system, a second query comprising the action and the one or more user interface elements from the updated user interface ([0122] “The replay engine then initiates a recovery operation, in which the NL test 571 provides the text step description to the NL semantic engine 566.”);
receiving, from the natural language processing system in response to the second query, a second identified element; continuing the executing of the software by performing the action on the second identified element on the updated user interface ([0122] “the NL semantic engine 566, which in turn provides the action and object description to the semantic matching engine 568. The semantic engine 562 provides the semantic hierarchy to the semantic matching engine.”);
determining that performing the action on the second identified element resulted in completion of the action ([0122] “The semantic engine 562 provides the semantic hierarchy to the semantic matching engine, which finds the test object and provides the test action and test object to the browser 538.”); and
updating the test script ([0122] “the semantic matching engine, which finds the test object... The step is re-recorded by the record engine 560 and the script 507 is updated.”).
Claim 19:
With regard to Claim 19, this claim is equivalent in scope to Claim 6 rejected above, merely having a different independent claim type, and as such Claim 19 is rejected under the same grounds and for the same reasons as discussed above with regard to Claim 6.
Claim 7, 16 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bergman in view of Liu, Shi, Cobb and Lundberg as applied to claims 1, 15 and 17 above, and further in view of Avery et al. (US PGPUB 2016/0179658; hereinafter “Avery”).
Claim 7
Bergman in view of Liu, Shi, Cobb and Lundberg teaches all the limitations of claim 1 as described above. Bergman in view of Liu, Shi, Cobb and Lundberg does not teach the following, however, Avery teaches further comprising:
determining that the updated user interface does not include an element associated with at least one of the action and a consecutive action of the test script; and transmitting an error notification ([0060] “the abstraction layer can identify the location of the GUI code corresponding to the specific implementation of the queried-for GUI element… a corresponding object or other piece of code corresponding to the GUI element can be wrapped in the response with a label corresponding to the type of GUI element sought for in the method call… Further, in instances where the method call results in no matching GUI elements being found, the method can return an empty set, error, or other message.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Bergman in view of Liu, Shi, Cobb and Lundberg with the error notification as taught by Avery in order to “communicate the condition to the testing system” (Avery [0060]).
Claim 16:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches all the limitations of claim 15 as described above. Bergman in view of Liu, Shi, Cobb and Lundberg does not teach the following, however, Avery teaches further comprising:
determining that performing the action on the second identified element did not result in completion of the action; and transmitting an error notification to a user ([0060] “the abstraction layer can identify the location of the GUI code corresponding to the specific implementation of the queried-for GUI element… a corresponding object or other piece of code corresponding to the GUI element can be wrapped in the response with a label corresponding to the type of GUI element sought for in the method call… Further, in instances where the method call results in no matching GUI elements being found, the method can return an empty set, error, or other message.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system as disclosed by Bergman in view of Liu, Shi, Cobb and Lundberg with the error notification as taught by Avery in order to “communicate the condition to the testing system” (Avery [0060]).
Claim 18:
With regard to Claim 18, this claim is equivalent in scope to Claim 16 rejected above, merely having a different independent claim type, and as such Claim 18 is rejected under the same grounds and for the same reasons as discussed above with regard to Claim 16.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Bergman in view of Liu, Shi, Cobb and Lundberg as applied to claim 8 above, and further in view of Splaine et al. (US PGPUB 2008/0086627; hereinafter “Splaine”).
Claim 13:
Bergman in view of Liu, Shi, Cobb and Lundberg teaches all the limitations of claim 8 as described above. Bergman in view of Liu, Shi, Cobb and Lundberg does not teach the following, however, Splaine teaches:
wherein the determining that performing the action on the identified element resulted in completion of the action is based on detection of an event associated with the action in a testing log ([0076] “At lines 720, the function associated with the ‘COMBOBOX’ GUI element type is called. The called function performs the action specified by the read line of the test case… If the function returns a result indicating that the action was performed successfully, then, at lines 722 a ‘pass’ result is logged (e.g., is logged to the test log 106 of FIG. 1. If the function returns a result indicating that the action was not performed successfully, then, at lines 724 a ‘fail’ result is logged.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Bergman in view of Liu, Shi, Cobb and Lundberg with the testing log as taught by Splaine in order “to verify that the software operates properly” (Splaine [0003]).
Response to Arguments
Applicant's arguments, see Pages 9-14 of the Remarks filed April 10, 2026 with respect to the rejections under 35 U.S.C. 103 of Claims 1-3, 5-9 and 11-19 have been fully considered but they are not persuasive.
With respect to the Applicant’s argument, Pages 10-13 that the newly amended claim language of Claims 1, 8 and 14 is not taught by the previously cited prior art, this argument has been fully considered but is moot in view of the newly cited Shi et al. (US PGPUB 2022/0413997) reference, as well as newly cited teachings from the previously cited Liu et al. (US PGPUB 2024/0303184) reference, as discussed above in the respective rejections.
With respect to the Applicant’s further arguments, Page 14 of the Remarks, that the features of the dependent claims are not taught by the cited prior art, the Office respectfully disagrees. These arguments rely upon the arguments as presented in relation to Claims 1, 8 and 14 discussed above, and as such the Office directs the Applicant to the responses above regarding these arguments.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure is as follows:
Ciminelli et al. (US Patent 12,596,535) discloses systems and methods for editing user interfaces from free text and sketches, wherein a textual input in a natural language or a sketch may be received and analyzed to determine at least one change to a portion of the design of the user interface.
Chen et al. (“Improving Crowd-Supported GUI Testing with Structural Guidance,” 2020) discusses two techniques to increase crowd testers’ testing coverage, interactive event-flow graphs and GUI-level guidance, wherein the graphs track and aggregate tester interactions to find new navigation paths and the guidance helps testers avoid only exploring common paths.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Joanne G. Macasiano whose telephone number is (571)270-7749. The examiner can normally be reached Monday to Thursday, 10:30 AM to 6:00 PM Eastern Standard Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bradley Teets can be reached at (571) 272-3338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOANNE G MACASIANO/Examiner, Art Unit 2197
Prosecution Insights