Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This is the initial office action based on the application filed on August 01st, 2024, which claims 1-20 are presented for examination.
Status of Claims
3. Claims 1-20 are pending, of which claims, of which claim 1, 10 and 19 are in independent form.
Priority
4. No priority has been considered for this application.
The Office's Note:
5. The Office has cited particular paragraphs / columns and line numbers in the reference(s) applied to the claims above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to specific limitations within the individual claim(s), other passages and figures may apply as well. It is respectfully requested from the Applicant in preparing responses, to fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the cited passages as taught by the prior art or relied upon by the Examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
6. Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1, claim 10 and claim 19 recite “ accessing a video comprising a plurality of video frames of a user using an application under test (AUT); using an artificial intelligence (AI) to determine a user intention associated with the AUT from at least one of the plurality of video frames; generating a test script comprising a set of operations that perform the user intention; and writing the test script to a data storage.” as drafted, are functions that, under its broadest reasonable interpretation, recite the abstract idea of a mental process. The limitations encompass a human mind carrying out the function through observation, evaluation judgment and /or opinion, or even with the aid of pen and paper. Thus, this limitation recites and falls within the “Mental Processes” grouping of abstract ideas under Prong 1.
Under Prong 2, this judicial exception is not integrated into a practical application. The additional elements ““memory”, and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “generating a test script comprising a set of operations that perform the user intention; and writing the test script to a data storage” do nothing more than add insignificant extra solution activity to the judicial exception of merely gathering, displaying, updating, transmitting and storing data/information. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application and the claim is therefore directed to the judicial exception. See MPEP 2106.05(g).
Under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of ““memory,” and “processor” are recited at a high-level of generality such that it amounts no more than mere instructions to apply the exception using generic computer, and/or mere computer components, and “writing the test script to a data storage”, the courts have identified merely gathering, displaying, updating, transmitting and storing data/information on a display is well-understood, routine and conventional activity. See MPEP 2106.05(d). The recitation of generic computer instruction and computer components to apply the judicial exception, and merely displaying data do not amount to significantly more, thus, cannot provide an inventive concept. Accordingly, the claims are not patent eligible under 35 USC 101.
In conclusion, claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. Claim 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang (US 20220083458 – hereinafter Zhang) and further in view of Luzon (US 20220261336– hereinafter Luzon).
Claim 1 is rejected, Zhang teaches a method, comprising:
accessing a video comprising a plurality of video frames of a user using an application under test (AUT) (Shang, US 20220083458, fig. 1 and para [0019-0021], The electronic device 102 includes a user input device 110, which can be used by a user at the electronic device 102 to make selections of UI elements in the GUI 108 presented by the program under test 104. Para [0022-0024], the electronic device 102 further includes a video recorder 118, which can record video frames of the GUI 108 as a user interacts with the GUI 108. Para [0025].) ;
using an artificial intelligence (AI) to determine a user intention associated with the AUT from at least one of the plurality of video frames(Shang, para [0025-0026], The interactive area identification engine 106 outputs UI element test objects 124, which represent respective UI elements identified by the interactive area identification engine 106. Para [0027-0028], The test script 128 can include information regarding a collection (e.g., a sequence) of user actions to be performed on respective UI elements of the GUI 108. These user actions perform with respect to corresponding UI elements can include the collection of user actions that led to a detected issue (or multiple detected issues) of the program under test.);
generating a test script comprising a set of operations that perform the user intention(Shang, para [0025-0026], The interactive area identification engine 106 outputs UI element test objects 124, which represent respective UI elements identified by the interactive area identification engine 106. The UI element test objects 124 are provided to a test script generator 126, which produces a test script 128 for testing the program under test 104. Para [0027-0028], The test script 128 includes program code. The test script includes UI test objects that represent respective UI elements of the GUI 108, along with representations of user actions that are to be performed on UI elements represented by the UI element test objects 124. The information relating to the user actions to be performed on the UI elements are derived from the event data 114. Note that because event timestamps 116 and video timestamps 122 are associated with respective events and video frames that include images of UI elements, the interactive area identification engine 106 can correlate, based on the timestamps, events corresponding to user actions with respective UI elements identified by the interactive area identification engine 106. Information of the correlated events 125 is also provided by the interactive area identification engine 106 to the test script generator 126, which uses the information of the correlated events 125 along with the UI element test objects 124 to build the test script 128. Para [0013-0015], generate a test script.); and
writing the test script to a data storage (Shang, fig. 1 and para [0029-0035], The test script 128 is provided to a replay engine 130, which can include a simulation program that executes the test script 128. In some examples, the test script 128 can include a compiled code that can be executed by the replay engine 130. In other examples, test script 128 can include interpreted code that can be interpreted and executed by the replay engine 130. Executing the test script 128 by the replay engine 130 recreates (simulates) user actions described in the test script 128, which are applied to the program under test 104 in an effort to debug issues associated with the program under test 104. Fig. 3, para [0057-0059], storage medium.).
The Office would like to use prior art Luzon to back up Zhang to further teach limitation
artificial intelligence (Luzon, US 20220261336, para [0003], functional testing of an application based on evaluation of contents of a user interface of the application using artificial intelligence. According to one embodiment, a method for performing functional testing on an Application Under Test (AUT) can comprise building a model defining each of a plurality of object classifications for objects of a user interface of the AUT based on a graphical appearance of the objects. Para [0006-0011], train and retrain. Fig. 3 and para [0049], artificial intelligence.).
It would have obvious to one having ordinary skill in the art before the effecting filing date of the claimed invention to combine the teachings of cited references. Thus, one of ordinary skill in the art before the effecting filing date of the claimed invention would have been motivated to incorporate Luzon into Zhang to enable building a model defining the object classifications for the objects of the user interface of an application under test based on a graphical appearance of the objects, thus ensuring simple and efficient functional testing process.as suggested by Luzon (See abstract and summary).
Claim 2 is rejected for the reasons set forth hereinabove for claim 1, Zhang and Luzon teach the method of claim 1, wherein using the AI to determine the user intention associated with the AUT from at least one of the plurality of video frames further comprises providing a prompt to the AI, the prompt comprising the at least one of the plurality of video frames, a domain associated with the AUT, a plurality of prior videos, each prior video comprising a prior user interacting with a prior application under test to accomplish a prior user intention, and a request to determine the user intention(Luzon, Para [0006-0011], train and retrain. Fig. 3 and para [0049], artificial intelligence. Zhang, para [0034-0035], The interactive area identification engine 106 attempts to identify (at 208) an area (or multiple areas) that changed in the first and second sets of video frames. In other words, the interactive area identification engine 106 seeks to identify a given area in the selected range of frames that has a first state (a first groups of pixels with first values in the given area) in the first set of video frames before the event timestamp, and a different second state (a second group of pixels having different values in the given area) in the second set of video frames after the event timestamp. The given area that changed from a first state before the event to a different second state after the event is likely a user interactive area that was changed due to a user action (a user input, a user click, a user key press, a user touch, etc.) made in GUI 108. Zhang, para [0025-0028], The test script 128 can include information regarding a collection (e.g., a sequence) of user actions to be performed on respective UI elements of the GUI 108. These user actions perform with respect to corresponding UI elements can include the collection of user actions that led to a detected issue (or multiple detected issues) of the program under test. Luzon, para [0053-0054], Once the training data 330 and validation data 335 has been prepared, the model 350 can be trained and validated by the model training and validation functions 345. This can be accomplished using any of a variety of available machine learning algorithms. For example, a Deep Neural Network (DNN) can be used for object detection such as the Single Shot multibox Detector (SSD) architecture publish by Google. Once trained and validated, the model can be used by the object identification engine 320 to identify objects in the user interface 315 of the AUT 310 based on their graphical or visual appearance. Tests executed by the test functions 305 can be based on the objects identified by the object identification engine 320. Results of the tests can be provided in one or more printed, displayed, or saved test reports 355.).
Claim 3 is rejected for the reasons set forth hereinabove for claim 1, Zhang and Luzon teach the method of claim 1, wherein generating the test script comprising the set of operations that perform the user intention further comprises generating the test script to comprise variations of at least one of a user action and an object of a graphical user interface (GUI) generated by the AUT receiving the user action (Shang, para [0013-0015], The system identifies an area of the GUI that corresponds to a respective user action of the user actions. The identifying of the area of the GUI uses a set of first video frames (where a “set” of video frames can include just a single video frame or multiple video frames)) before an event corresponding to the respective user action, and a set of second video frames after the event corresponding to the respective user action. The system identifies, based on the identified area, a test object representing a user interface (UI) element, generates a test script for testing the program, the test script including the test object. Para [0026-0028], The interactive area identification engine 106 outputs UI element test objects 124, which represent respective UI elements identified by the interactive area identification engine 106. The UI element test objects 124 are provided to a test script generator 126, which produces a test script 128 for testing the program under test 104. ).
Claim 4 is rejected for the reasons set forth hereinabove for claim 1, Zhang and Luzon teach the method of claim 1, wherein using the AI to determine the user intention associated with the AUT from at least one of the plurality of video frames further comprises detecting a user action performed on an object of a graphical user interface (GUI) generated by the AUT (Luzon, Para [0006-0011], train and retrain. Fig. 3 and para [0049], artificial intelligence. Shang, para [0013-0015], The system identifies an area of the GUI that corresponds to a respective user action of the user actions. The identifying of the area of the GUI uses a set of first video frames (where a “set” of video frames can include just a single video frame or multiple video frames)) before an event corresponding to the respective user action, and a set of second video frames after the event corresponding to the respective user action. The system identifies, based on the identified area, a test object representing a user interface (UI) element, generates a test script for testing the program, the test script including the test object. Para [0026-0028], The interactive area identification engine 106 outputs UI element test objects 124, which represent respective UI elements identified by the interactive area identification engine 106. The UI element test objects 124 are provided to a test script generator 126, which produces a test script 128 for testing the program under test 104. ).
Claim 5 is rejected for the reasons set forth hereinabove for claim 1, Zhang and Luzon teach the method of claim 1, wherein using the AI to determine the user intention associated with the AUT from at least one of the plurality of video frames further comprises providing a first frame of the plurality of video frames showing a user action on an object and a second frame of the plurality of video frames, and wherein the second frame of the plurality of video frames shows a result of the user action (Luzon, Para [0006-0011], train and retrain. Fig. 3 and para [0049], artificial intelligence. Zhang, para [0034-0035], The interactive area identification engine 106 attempts to identify (at 208) an area (or multiple areas) that changed in the first and second sets of video frames. In other words, the interactive area identification engine 106 seeks to identify a given area in the selected range of frames that has a first state (a first groups of pixels with first values in the given area) in the first set of video frames before the event timestamp, and a different second state (a second group of pixels having different values in the given area) in the second set of video frames after the event timestamp. The given area that changed from a first state before the event to a different second state after the event is likely a user interactive area that was changed due to a user action (a user input, a user click, a user key press, a user touch, etc.) made in GUI 108.).
Claim 6 is rejected for the reasons set forth hereinabove for claim 1, Zhang and Luzon teach the method of claim 1, wherein the AI comprises a neural network trained to receive the video and determine therefrom the user intention (Zhang, para [0025-0028], The test script 128 can include information regarding a collection (e.g., a sequence) of user actions to be performed on respective UI elements of the GUI 108. These user actions perform with respect to corresponding UI elements can include the collection of user actions that led to a detected issue (or multiple detected issues) of the program under test. Luzon, para [0053-0054], Once the training data 330 and validation data 335 has been prepared, the model 350 can be trained and validated by the model training and validation functions 345. This can be accomplished using any of a variety of available machine learning algorithms. For example, a Deep Neural Network (DNN) can be used for object detection such as the Single Shot multibox Detector (SSD) architecture publish by Google. Once trained and validated, the model can be used by the object identification engine 320 to identify objects in the user interface 315 of the AUT 310 based on their graphical or visual appearance. Tests executed by the test functions 305 can be based on the objects identified by the object identification engine 320. Results of the tests can be provided in one or more printed, displayed, or saved test reports 355.).
Claim 7 is rejected for the reasons set forth hereinabove for claim 6, Zhang and Luzon teach the method of claim 6, wherein the neural network is trained to determine the user intention, comprising(Luzon, para [0053-0054]):
collecting a set of prior videos, each prior video comprising video frames of a set of prior users using a prior application under test (Zhang, para [0023-0025], Video frames can refer to a continual sequence of image frames. The video recorder 118 outputs video frames 120, along with metadata including video timestamps 122 that indicate times at which respective video frames were acquired. In some examples, the video timestamps 122 are part of the video frames 120, while in other examples, the video timestamps 122 can include metadata separate from but associated with the video frames 120. Luzon, para [0061], building the model can comprise receiving 505 a set of images. Each image of the set of images can comprise an image of a user interface of a plurality of user interfaces and can represent the one or more objects of the user interface. For example, the plurality of user interfaces can comprise interfaces of one or more previously tested application or any other interface used to train the model. );
applying one or more transformations to each of the prior videos, including selecting a different object to perform a prior user action, altering the prior user action, and obtaining a different prior result to create a modified set of prior videos(Zhang, para [0034-0035], The interactive area identification engine 106 attempts to identify (at 208) an area (or multiple areas) that changed in the first and second sets of video frames. In other words, the interactive area identification engine 106 seeks to identify a given area in the selected range of frames that has a first state (a first groups of pixels with first values in the given area) in the first set of video frames before the event timestamp, and a different second state (a second group of pixels having different values in the given area) in the second set of video frames after the event timestamp. The given area that changed from a first state before the event to a different second state after the event is likely a user interactive area that was changed due to a user action (a user input, a user click, a user key press, a user touch, etc.) made in GUI 108. Para [0036-0039], the identification of features of a UI element uses a Scale-Invariant Feature Transform (SIFT) routine from the Open Source Computer Vision Library (OpenCV). The SIFT routine can be used to identify features in an image. In other examples, other types of feature identification techniques can be employed. Luzon, para [0061-0062], Each image of the set of images can comprise an image of a user interface of a plurality of user interfaces and can represent the one or more objects of the user interface. For example, the plurality of user interfaces can comprise interfaces of one or more previously tested application or any other interface used to train the model. Each object in each image of the set of images can be tagged 510. An exemplary process for tagging each object will be described below with reference to FIG. 6);
creating a first training set comprising the collected set of prior videos, the modified set of prior videos, and a set of videos absent the user intention (Zhang, para [0047-0050], In alternative examples, a classifier, such as a deep learning neural network classifier, can be used to identify areas containing UI elements such as menu, toolbar, scrollbar, and other UI elements. The classifier can be trained to recognize certain types of UI elements in the video frames. The areas containing UI elements identified by the classifier can be included in UI element test objects.);
training the neural network in a first stage of training using the first training set( Zhang, para [0047-0050], In alternative examples, a classifier, such as a deep learning neural network classifier, can be used to identify areas containing UI elements such as menu, toolbar, scrollbar, and other UI elements. The classifier can be trained to recognize certain types of UI elements in the video frames. The areas containing UI elements identified by the classifier can be included in UI element test objects.);
creating a second training set for a second stage of training comprising the first training set and members of the set of videos absent the user intention incorrectly detected as having the user intention after the first stage of training (Luzon, para [0006-0011], Building the model can comprise receiving a set of images. Each image of the set of images can comprise an image of a user interface of a plurality of user interfaces and can represent the one or more objects of the user interface. Each object in each image of the set of images can be tagged. Each image of the set of images can be assigned to either a training data set of the model or a validation data set of the model. Assigning each image to either the training data set or the validation data set can further comprise balancing the training data set and the validation data set. The model can then be trained based on the training data set and validated based on the validation data set. Fig. 4 and para [0060], the model can be retrained 420 based on results of identifying the one or more object in the image of the user interface of the AUT.); and
training the neural network in the second stage of training using the second training set(Luzon, para [0006-0010], the model can be retrained based on results of identifying the one or more object in the image of the user interface of the AUT. Fig. 4 and para [0060], the model can be retrained 420 based on results of identifying the one or more object in the image of the user interface of the AUT.).
Claim 8 is rejected for the reasons set forth hereinabove for claim 7, Zhang and Luzon teach the method of claim 7, wherein training the neural network further comprises training the neural network on the set of prior videos comprising videos of the set of prior users using the prior application under test within a common domain as the AUT(Luzon, para [0006-0007], The instruction can further cause the processor to retrain the model based on results of identifying the one or more object in the image of the user interface of the AUT. Para [0010-0011], to perform functional testing on an Application Under Test (AUT) by building a model defining each of a plurality of object classifications for objects of a user interface of the AUT based on a graphical appearance of the objects, identifying one or more objects in an image of the user interface of the AUT based on the plurality of object classifications defined in the model and the graphical appearance of each of the one or more objects in the image of the user interface of the AUT, and executing a test script defining one or more functional tests on the AUT. Executing the test script can comprise performing the one or more functional tests on the AUT based on the identified one or more objects in the image of the user interface of the AUT. The instruction can further cause the processor to retrain the model based on results of identifying the one or more object in the image of the user interface of the AUT. Para [0060], the model can be retrained 420 based on results of identifying the one or more object in the image of the user interface of the AUT. ).
Claim 9 is rejected for the reasons set forth hereinabove for claim 1, Zhang and Luzon teach the method of claim 1, further comprising executing, test script to test the AUT (Zhang, para [0029-0035], The test script 128 is provided to a replay engine 130, which can include a simulation program that executes the test script 128. In some examples, the test script 128 can include a compiled code that can be executed by the replay engine 130. In other examples, test script 128 can include interpreted code that can be interpreted and executed by the replay engine 130. Executing the test script 128 by the replay engine 130 recreates (simulates) user actions described in the test script 128, which are applied to the program under test 104 in an effort to debug issues associated with the program under test 104.).
As per claim 10, this is the system claim to method claim 1. Therefore, it is rejected for the same reasons as above.
As per claim 11, this is the system claim to method claim 2. Therefore, it is rejected for the same reasons as above.
As per claim 12, this is the system claim to method claim 3. Therefore, it is rejected for the same reasons as above.
As per claim 13, this is the system claim to method claim 4. Therefore, it is rejected for the same reasons as above.
As per claim 14, this is the system claim to method claim 5. Therefore, it is rejected for the same reasons as above.
As per claim 15, this is the system claim to method claim 6. Therefore, it is rejected for the same reasons as above.
As per claim 16, this is the system claim to method claim 7. Therefore, it is rejected for the same reasons as above.
As per claim 17, this is the system claim to method claim 8. Therefore, it is rejected for the same reasons as above.
As per claim 18, this is the system claim to method claim 9. Therefore, it is rejected for the same reasons as above.
As per claim 19, this is the device claim to method claim 1. Therefore, it is rejected for the same reasons as above.
Claim 20 is rejected for the reasons set forth hereinabove for claim 19, Zhang and Luzon teach the test script generating device of claim 19, wherein the instructions to cause the at least one microprocessor to provide the at least one of the plurality of video frames to the AI and receive therefrom the user intention associated with the AUT, further comprise instructions to cause the at least one microprocessor to execute the AI and provide the at least one of the plurality of video frames to the AI executing thereon (Luzon, Para [0006-0011], train and retrain. Fig. 3 and para [0049], artificial intelligence. Zhang, para [0025-0028], The test script 128 can include information regarding a collection (e.g., a sequence) of user actions to be performed on respective UI elements of the GUI 108. These user actions perform with respect to corresponding UI elements can include the collection of user actions that led to a detected issue (or multiple detected issues) of the program under test. Zhang, para [0029-0033], The test script 128 is provided to a replay engine 130, which can include a simulation program that executes the test script 128. In some examples, the test script 128 can include a compiled code that can be executed by the replay engine 130. In other examples, test script 128 can include interpreted code that can be interpreted and executed by the replay engine 130. Executing the test script 128 by the replay engine 130 recreates (simulates) user actions described in the test script 128, which are applied to the program under test 104 in an effort to debug issues associated with the program under test 104. Zhang, para [0034-0035], The interactive area identification engine 106 attempts to identify (at 208) an area (or multiple areas) that changed in the first and second sets of video frames. In other words, the interactive area identification engine 106 seeks to identify a given area in the selected range of frames that has a first state (a first groups of pixels with first values in the given area) in the first set of video frames before the event timestamp, and a different second state (a second group of pixels having different values in the given area) in the second set of video frames after the event timestamp. The given area that changed from a first state before the event to a different second state after the event is likely a user interactive area that was changed due to a user action (a user input, a user click, a user key press, a user touch, etc.) made in GUI 108. Luzon, para [0053-0054], Once the training data 330 and validation data 335 has been prepared, the model 350 can be trained and validated by the model training and validation functions 345. This can be accomplished using any of a variety of available machine learning algorithms. For example, a Deep Neural Network (DNN) can be used for object detection such as the Single Shot multibox Detector (SSD) architecture publish by Google. Once trained and validated, the model can be used by the object identification engine 320 to identify objects in the user interface 315 of the AUT 310 based on their graphical or visual appearance. Tests executed by the test functions 305 can be based on the objects identified by the object identification engine 320. Results of the tests can be provided in one or more printed, displayed, or saved test reports 355.).
Inquiry
8. Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY KHUONG THANH NGUYEN whose telephone number is (571)270-7139. The examiner can normally be reached Monday - Friday 0800-1630.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lewis Bullock can be reached at 5712723759. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY KHUONG T NGUYEN/ Primary Examiner, Art Unit 2199
Prosecution Insights