DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This Office Action is responsive to the communications filed on 21 February 2025. Claims 1, 4-16 and 18-22 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4-7, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hunt et al. (Hereinafter, US 2017/0040037 A1) in view of Banne (US 2020/0371818 A1), Elges (US 2020/0192790 A1), Siddall et al. (Hereinafter, Siddall, US 10,885,903 B1), and further in view of Lau et al. (Hereinafter, Lau, US 20120/317547 A1).
Per claim 1, Hunt discloses a method of automatically generating a video (Abstract; paragraph [0008], “…Embodiments of the present invention provide systems, methods, and computer program products for automatic production of training videos by using universal identifiers (UID) associated with a software product user interface (UI) ...“), the method comprising:
receiving, with a processor of a computer system (e.g., host system 400 as shown in fig. 4; paragraph [0036] ), two or more test scripts wherein each test script comprises a plurality of user interactions (Abstract, “… Test scripts are created based on matching the universal identifiers to task instructions ... “; paragraph [0003], “ … creating one or more test scripts based on matching the one or more universal identifiers to one or more task instructions … “; paragraph [0024], “ Step 208 creates a test script that executes UI elements of software product 126 contained in UI properties file(s) 122, and references task instructions contained in task properties file(s) 124… “);
generating, with the processor, a step action tree (e.g., step 206 as shown in Fig. 2; paragraph [0023], “Step 206 creates task instructions and associated UIDs in task properties file(s) 124 based on UIDs contained in UI properties file(s) 122. The UID reference associates a task step to a UI element used by software product 126. It should be noted when introducing alternate natural languages, the task instruction text is updated without affecting an assigned UID in task properties file(s) 124.”; Examiner’s Note: Examiner is interpreting the task instructions and associated UIDs in task properties file(s) 124 based on UIDs contained in UI properties file(s) 122 to be the claimed step action tree. ) comprising the plurality of user actions of each of the two or more scripts wherein each test script comprises a plurality of user interactions (e.g., task_1 320, task_2 322, task_3 324 and task_n; paragraph [0003]; paragraph [0017],” In one embodiment of the present invention, task properties file 124 can be a combination of a plurality of task properties files. Task properties file 124 contains, in part, one or more UIDs matching UIDs contained in UI properties file(s) 122 and predetermined instructional text for a plurality of UI elements.”; paragraph [0031], ” Further depicted on UI text 308 scale are task_1 320, task_2 322, task_3 324 and task_n which represent instructional task text associated with the UID in task properties file(s) 124 and where ‘n’ indicates one to many possible tasks.”); but does not expressly disclose:
wherein the step action tree includes branches representing deviations between the test scripts;
displaying, via a graphical user interface, the plurality of actions in the step action tree;
receiving, with the processor, via the graphical user interface, a selection of one or more a branch of the step action tree;
in response to the selection of the one or more actions:
automatically generating, with the processor, one or more of mouse movements and text entry in a graphical user interface performing the selected one or more actions;
generating, with the processor, a first video clip comprising a recording of the one or more of mouse movements and text entry in the graphical user interface performing the selected one or more actions;
generating text associated with the selected one or more actions using a machine learning algorithm; and
associating, with the processor, the first video clip and the text generated using the machine learning algorithm with the selected one or more actions.
Elges discloses:
displaying, via a graphical user interface(e.g., test application user interface 112 as shown in Fig. 1; paragraph [0024], “…Computing architecture 100 comprises computing device 102A, which is displaying a test application user interface 112 for no-code testing of custom ERP applications, metadata storage sub-environment 120, network and processing sub-environment 114, and computing device 102B, which is displaying error user interface 128 for highlighting errors detected with relation to corresponding custom ERP application modifications ... “; paragraph [0025], “The test application user interface 112 displayed on computing device 102A comprises no-code elements for modifying an existing ERP application framework, and testing elements of the ERP application framework that correspond to the modification ...”), the plurality of actions(e.g., ‘Add tax’ , ‘Apply discount’, ‘Add produce type’, ‘Add sales region’ 106 as shown Fig. 1) in the step action tree(e.g., ‘Add action’ 104 as shown Fig. 1; paragraph [0006], “… The test application user interface may display a plurality of steps and/or actions associated with the ERP application that may be added, modified and/or deleted from the ERP application and tested by the test application prior to rolling the modified software out in the actual application ... “); and
receiving, with the processor, via the graphical user interface, a selection of one or more a branch of the step action tree (paragraph [0030], “ In the specific example shown regarding FIG. 1, a user has opened the no-code test application, selected the “add tax” user interface element 108, and confirmed that selection via fly-out window 110. According to some examples, the user may have interacted with a specific column in the ERP application that the user would like to add the tax too, and the “add tax” user interface element 108 may have then been presented to the user… “; paragraph s [0037-0038]; Examiner’s Note: Fig. 3 illustrates receiving, with the processor, via the graphical user interface, a selection of one or more actions in a branch of the step action tree.).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use Elges’ automated testing with Hunt’s approach for creation and maintenance of video-based training documentation for auto-executing instructions provided in a video on a computing platform in order to make upgrading and/or fixing software elements easier and less time consuming as suggested by Elges (Abstract; paragraph [0003]).
Banne discloses:
in response to the selection of the one or more actions(e.g., Extract User interaction (UI) events 114 as shown in Fig. 1; paragraph [0070], “The action extraction engine 114 in certain example embodiments extracts actions that can be taken by a user as prompted by the audio and/or video input, e.g., as facilitated by tools included in imagine processing libraries. The actions that may be extracted may include, for example, typing on a keyboard, mouse movements, mouse events, switching of windows, etc. …. “ ) :
automatically generating, with the processor, one or more of mouse movements (e.g., RPA Commands 116 as shown in Fig. 1; paragraph [0071], “ … The action extraction engine 114 may interface with program logic portions (e.g., image processing libraries) to aid in this extraction and identification. FIG. 2, for example, shows a first image processing module 216a for tracking an area around a mouse pointer, a second image processing module 216b for performing frame subtraction, and further image processing modules may be implemented in the above-identified and/or other respects. “) and text entry in a graphical user interface performing the selected one or more actions (e.g., Step 112 as shown in Fig. 1; paragraph [0061]);
generating, with the processor, a first video clip comprising a recording of the one or more of mouse movements and text entry in the graphical user interface performing the selected one or more actions (e.g., RPA bot 118 as shown in Fig. 1; paragraph [0076-0077]).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use Banne’s techniques with Hunt and Elges’ approach for creation and maintenance of video-based training documentation for auto-executing instructions provided in a video on a computing platform in order to reduce the manual burden on the user as suggested by Banne (Abstract; paragraph [0007]).
Siddall disclose:
generating text associated with the selected one or more actions using a machine learning algorithm (e.g., Steps (2)-(3) as shown in Fig. 3; Abstract; column 7, lines 47-67 and column 8, lines 1-32; Examiner’s Note: Siddall uses a machine learning or AI algorithm to identify context keywords or actions corresponding to items on the video data. ); and
associating, with the processor, the first video clip and the text generated using the machine learning algorithm with the one or more actions (e.g., Steps (4)-(5) as shown in Fig. 3; Abstract; column 8, lines 33-67 and column 9, lines 1-34; Examiner’s Note: Siddall uses am machine learning or AI algorithm to output text associated with a video, i.e., closed captioning information, textual content streams, etc. ).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the textual transcription device of Siddall with Hunt, Elges, and Banne’s approach for creation and maintenance of video-based training documentation for the purpose of addressing the inefficiencies with regard to processing video content to generate textual transcriptions as suggested by Siddall.
Lau discloses wherein the step action tree includes branches representing deviations between the test scripts(e.g., step 205 as shown in Fig. 2; paragraph[0042], “As shown in FIG. 2, automatic identification of subroutines from test scripts 200 may include a step 205 of identifying at least one subroutine from at least one test script, such that the subroutine includes at least one instruction class, wherein the instruction class includes at least a instruction class type and a subroutine object type, and the test script includes at least one test script action, and the test script action includes at least a test script action type and a test script object type ….”; paragraph [0052]; paragraph [0053]; Examiner’s Note: For example, subroutine instructions 730 as shown in Fig. 7 illustrates branches representing deviations from the login script. ).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the subroutines of Lau with Hunt, Elges, Banne, and Siddall’s approach for a more efficient system and method for automatic identification of subroutines from test scripts as suggested by Lau.
Per claim 4, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 1, wherein a description of the selected one or more actions is used as an input to the machine learning algorithm and the text associated with the selected one or more actions is an output of the machine learning algorithm (Siddall, Abstract; Step 408 as shown in Fig. 4; column 10, lines 41-67 and column 11, lines 1-4).
Per claim 5, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 4, further comprising, generating a narration audio clip for the selected one or more actions using the text generated using the machine learning algorithm (Siddall, e.g., Steps (4)-(5) as shown in Fig. 3; Abstract; column 8, lines 33-67 and column 9, lines 1-34; Examiner’s Note: Siddall uses a machine learning or AI algorithm to output text associated with a video, i.e., closed captioning information, textual content streams, etc.).
Per claim 6, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 5, wherein the narration audio clip is generated by processing the text generated using the machine learning algorithm for the first action using a text-to-speech engine(Hunt, paragraph [0019], “In one embodiment of the present invention, speech generator 130 is a text-to-speech converter used to produce an audio channel. Speech generator 130 also marks sections of the audio channel using UIDs as defined in task properties file(s) 124. “).
Per claim 7, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 6, further comprising appending the narration audio clip to the first video clip (Hunt, e.g., Fig. 3 illustrates appending the narration audio clip 304 to the first video clip 302; paragraph [0033]).
Per claim 19, Hunt discloses a user device (e.g., Host computer system 400 as shown in Fig. 4; paragraph [0036] ) comprising:
a processor (e.g., processors 404 as shown in Fig. 4; paragraph [0037]); and
a computer-readable storage medium storing computer-readable instructions (e.g., memory 406 as shown in Fig. 4; paragraph [0038-0039]) which, when executed by the processor, cause the processor to execute a method for automatically generating a video( Abstract; paragraph [0008], “…Embodiments of the present invention provide systems, methods, and computer program products for automatic production of training videos by using universal identifiers (UID) associated with a software product user interface (UI) ...“), the method comprising:
receiving two or more test scripts, wherein each test script comprises a plurality of user interactions (Abstract, “… Test scripts are created based on matching the universal identifiers to task instructions ... “; paragraph [0003], “ … creating one or more test scripts based on matching the one or more universal identifiers to one or more task instructions … “; paragraph [0024], “ Step 208 creates a test script that executes UI elements of software product 126 contained in UI properties file(s) 122, and references task instructions contained in task properties file(s) 124… “);
generating a step action tree (e.g., step 206 as shown in Fig. 2; paragraph [0023], “Step 206 creates task instructions and associated UIDs in task properties file(s) 124 based on UIDs contained in UI properties file(s) 122. The UID reference associates a task step to a UI element used by software product 126. It should be noted when introducing alternate natural languages, the task instruction text is updated without affecting an assigned UID in task properties file(s) 124.”; Examiner’s Note: Examiner is interpreting the task instructions and associated UIDs in task properties file(s) 124 based on UIDs contained in UI properties file(s) 122 to be the claimed step action tree. ) comprising the plurality of user actions of the two or more scripts of each of the two or more test scripts (e.g., task_1 320, task_2 322, task_3 324 and task_n; paragraph [0003]; paragraph [0017],” In one embodiment of the present invention, task properties file 124 can be a combination of a plurality of task properties files. Task properties file 124 contains, in part, one or more UIDs matching UIDs contained in UI properties file(s) 122 and predetermined instructional text for a plurality of UI elements.”; paragraph [0031], ” Further depicted on UI text 308 scale are task_1 320, task_2 322, task_3 324 and task_n which represent instructional task text associated with the UID in task properties file(s) 124 and where ‘n’ indicates one to many possible tasks.”); but does not expressly disclose:
wherein the step action tree includes branches representing deviations between the test scripts;
displaying, via a graphical user interface, the plurality of actions in the step action tree;
receiving via the graphical user interface, a selection of one or more actions of the plurality of actions in the step action tree;
automatically generating, with the processor, one or more of mouse movements and text entry in a graphical user interface performing the selected one or more actions;
generating, with the processor, a first video clip comprising a recording of] the one or more of mouse movements and text entry in the graphical user interface performing the selected one or more actions;
generating text associated with the selected one or more actions using a machine learning algorithm; and
associating the first video clip and the text generated using the machine learning algorithm with the selected one or more actions.
Elges discloses:
displaying, via a graphical user interface(e.g., test application user interface 112 as shown in Fig. 1; paragraph [0024], “…Computing architecture 100 comprises computing device 102A, which is displaying a test application user interface 112 for no-code testing of custom ERP applications, metadata storage sub-environment 120, network and processing sub-environment 114, and computing device 102B, which is displaying error user interface 128 for highlighting errors detected with relation to corresponding custom ERP application modifications ... “; paragraph [0025], “The test application user interface 112 displayed on computing device 102A comprises no-code elements for modifying an existing ERP application framework, and testing elements of the ERP application framework that correspond to the modification ...”), the plurality of actions(e.g., ‘Add tax’ , ‘Apply discount’, ‘Add produce type’, ‘Add sales region’ 106 as shown Fig. 1) in the step action tree(e.g., ‘Add action’ 104 as shown Fig. 1; paragraph [0006], “… The test application user interface may display a plurality of steps and/or actions associated with the ERP application that may be added, modified and/or deleted from the ERP application and tested by the test application prior to rolling the modified software out in the actual application ... “); and
receiving via the graphical user interface, a selection of one or more a branch of the step action tree (paragraph [0030], “ In the specific example shown regarding FIG. 1, a user has opened the no-code test application, selected the “add tax” user interface element 108, and confirmed that selection via fly-out window 110. According to some examples, the user may have interacted with a specific column in the ERP application that the user would like to add the tax too, and the “add tax” user interface element 108 may have then been presented to the user… “; paragraph s [0037-0038]; Examiner’s Note: Fig. 3 illustrates receiving, with the processor, via the graphical user interface, a selection of one or more actions in a branch of the step action tree.).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use Elges’ automated testing with Hunt’s approach for creation and maintenance of video-based training documentation for auto-executing instructions provided in a video on a computing platform in order to make upgrading and/or fixing software elements easier and less time consuming as suggested by Elges (Abstract; paragraph [0003]).
Banne discloses:
in response to the selection of theThe action extraction engine 114 in certain example embodiments extracts actions that can be taken by a user as prompted by the audio and/or video input, e.g., as facilitated by tools included in imagine processing libraries. The actions that may be extracted may include, for example, typing on a keyboard, mouse movements, mouse events, switching of windows, etc. …. “ ) :
automatically generating, with the processor, one or more of mouse movements (e.g., RPA Commands 116 as shown in Fig. 1; paragraph [0071], “ … The action extraction engine 114 may interface with program logic portions (e.g., image processing libraries) to aid in this extraction and identification. FIG. 2, for example, shows a first image processing module 216a for tracking an area around a mouse pointer, a second image processing module 216b for performing frame subtraction, and further image processing modules may be implemented in the above-identified and/or other respects. “) and text entry in a graphical user interface performing the selected one or more actions (e.g., Step 112 as shown in Fig. 1; paragraph [0061]);
generating a first video clip comprising a recording of the one or more of mouse movements and text entry in the graphical user interface performing the selected one or more actions (e.g., RPA bot 118 as shown in Fig. 1; paragraph [0076-0077]).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use Banne’s techniques with Hunt’s approach for creation and maintenance of video-based training documentation for auto-executing instructions provided in a video on a computing platform in order to reduce the manual burden on the user as suggested by Banne (Abstract; paragraph [0007]).
Siddall disclose:
generating text and audio associated with the first action using a machine learning algorithm (e.g., Steps (2)-(3) as shown in Fig. 3; Abstract; column 3, lines 52-51; column 7, lines 47-67 and column 8, lines 1-32; Examiner’s Note: Siddall uses a machine learning or AI algorithm to identify context keywords or actions corresponding to items on the video data. ); and
associating the first video clip and the text generated using the machine learning algorithm with the selected one or more actions (e.g., Steps (4)-(5) as shown in Fig. 3; Abstract; column 8, lines 33-67 and column 9, lines 1-34; Examiner’s Note: Siddall uses am machine learning or AI algorithm to output text associated with a video, i.e., closed captioning information, textual content streams, etc. ).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the textual transcription device of Siddall with Hunt and Banne’s approach for creation and maintenance of video-based training documentation for the purpose of addressing the inefficiencies with regard to processing video content to generate textual transcriptions as suggested by Siddall.
Lau discloses wherein the step action tree includes branches representing deviations between the test scripts(e.g., step 205 as shown in Fig. 2; paragraph[0042], “As shown in FIG. 2, automatic identification of subroutines from test scripts 200 may include a step 205 of identifying at least one subroutine from at least one test script, such that the subroutine includes at least one instruction class, wherein the instruction class includes at least a instruction class type and a subroutine object type, and the test script includes at least one test script action, and the test script action includes at least a test script action type and a test script object type ….”; paragraph [0052]; paragraph [0053]; Examiner’s Note: For example, subroutine instructions 730 as shown in Fig. 7 illustrates branches representing deviations from the login script. ).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the subroutines of Lau with Hunt, Elges, Banne, and Siddall’s approach for a more efficient system and method for automatic identification of subroutines from test scripts as suggested by Lau.
Per claim 20, Hunt discloses computer program product comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith(e.g., memory 406 as shown in Fig. 4; paragraph [0038-0039]), the computer-readable program code configured, when executed by a processor, to execute a method for automatically generating a video(Abstract; paragraph [0008], “…Embodiments of the present invention provide systems, methods, and computer program products for automatic production of training videos by using universal identifiers (UID) associated with a software product user interface (UI) ...“), the method comprising:
receiving two or more test scripts, wherein each test script comprises a plurality of user interactions (Abstract, “… Test scripts are created based on matching the universal identifiers to task instructions ... “; paragraph [0003], “ … creating one or more test scripts based on matching the one or more universal identifiers to one or more task instructions … “; paragraph [0024], “ Step 208 creates a test script that executes UI elements of software product 126 contained in UI properties file(s) 122, and references task instructions contained in task properties file(s) 124… “);
generating a step action tree (e.g., step 206 as shown in Fig. 2; paragraph [0023], “Step 206 creates task instructions and associated UIDs in task properties file(s) 124 based on UIDs contained in UI properties file(s) 122. The UID reference associates a task step to a UI element used by software product 126. It should be noted when introducing alternate natural languages, the task instruction text is updated without affecting an assigned UID in task properties file(s) 124.”; Examiner’s Note: Examiner is interpreting the task instructions and associated UIDs in task properties file(s) 124 based on UIDs contained in UI properties file(s) 122 to be the claimed step action tree. ) comprising the plurality of user actions of the two or more scripts of each of the two or more test scripts In one embodiment of the present invention, task properties file 124 can be a combination of a plurality of task properties files. Task properties file 124 contains, in part, one or more UIDs matching UIDs contained in UI properties file(s) 122 and predetermined instructional text for a plurality of UI elements.”; paragraph [0031], ” Further depicted on UI text 308 scale are task_1 320, task_2 322, task_3 324 and task_n which represent instructional task text associated with the UID in task properties file(s) 124 and where ‘n’ indicates one to many possible tasks.”); but does not expressly disclose:
wherein the step action tree includes branches representing deviations between the test scripts;
displaying, via a graphical user interface, the plurality of actions in the step action tree;
receiving via the graphical user interface, a selection of one or more a branch of the step action tree;
automatically generating, with the processor, one or more of mouse movements and text entry in a graphical user interface performing the selected one or more actions;
generating, with the processor, a first video clip comprising a recording of] the one or more of mouse movements and text entry in the graphical user interface performing the selected one or more actions;
generating text associated with the selected one or more actions using a machine learning algorithm; and
associating the first video clip and the text generated using the machine learning algorithm with the selected one or more actions.
Elges discloses:
displaying, via a graphical user interface(e.g., test application user interface 112 as shown in Fig. 1; paragraph [0024], “…Computing architecture 100 comprises computing device 102A, which is displaying a test application user interface 112 for no-code testing of custom ERP applications, metadata storage sub-environment 120, network and processing sub-environment 114, and computing device 102B, which is displaying error user interface 128 for highlighting errors detected with relation to corresponding custom ERP application modifications ... “; paragraph [0025], “The test application user interface 112 displayed on computing device 102A comprises no-code elements for modifying an existing ERP application framework, and testing elements of the ERP application framework that correspond to the modification ...”), the plurality of actions(e.g., ‘Add tax’ , ‘Apply discount’, ‘Add produce type’, ‘Add sales region’ 106 as shown Fig. 1) in the step action tree(e.g., ‘Add action’ 104 as shown Fig. 1; paragraph [0006], “… The test application user interface may display a plurality of steps and/or actions associated with the ERP application that may be added, modified and/or deleted from the ERP application and tested by the test application prior to rolling the modified software out in the actual application ... “); and
receiving via the graphical user interface, a selection of one or more a branch of the step action tree (paragraph [0030], “ In the specific example shown regarding FIG. 1, a user has opened the no-code test application, selected the “add tax” user interface element 108, and confirmed that selection via fly-out window 110. According to some examples, the user may have interacted with a specific column in the ERP application that the user would like to add the tax too, and the “add tax” user interface element 108 may have then been presented to the user… “; paragraph s [0037-0038]; Examiner’s Note: Fig. 3 illustrates receiving, with the processor, via the graphical user interface, a selection of one or more actions in a branch of the step action tree.).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use Elges’ automated testing with Hunt’s approach for creation and maintenance of video-based training documentation for auto-executing instructions provided in a video on a computing platform in order to make upgrading and/or fixing software elements easier and less time consuming as suggested by Elges (Abstract; paragraph [0003]).
Banne discloses:
in response to the selection of theThe action extraction engine 114 in certain example embodiments extracts actions that can be taken by a user as prompted by the audio and/or video input, e.g., as facilitated by tools included in imagine processing libraries. The actions that may be extracted may include, for example, typing on a keyboard, mouse movements, mouse events, switching of windows, etc. …. “ ) :
automatically generating, with the processor, one or more of mouse movements (e.g., RPA Commands 116 as shown in Fig. 1; paragraph [0071], “ … The action extraction engine 114 may interface with program logic portions (e.g., image processing libraries) to aid in this extraction and identification. FIG. 2, for example, shows a first image processing module 216a for tracking an area around a mouse pointer, a second image processing module 216b for performing frame subtraction, and further image processing modules may be implemented in the above-identified and/or other respects. “) and text entry in a graphical user interface performing the selected one or more actions (e.g., Step 112 as shown in Fig. 1; paragraph [0061]);
generating a first video clip comprising a recording of the one or more of mouse movements and text entry in the graphical user interface performing the selected one or more actions (e.g., RPA bot 118 as shown in Fig. 1; paragraph [0076-0077]).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use Banne’s techniques with Hunt’s approach for creation and maintenance of video-based training documentation for auto-executing instructions provided in a video on a computing platform in order to reduce the manual burden on the user as suggested by Banne (Abstract; paragraph [0007]).
Siddall disclose:
generating text and audio associated with the first action using a machine learning algorithm (e.g., Steps (2)-(3) as shown in Fig. 3; Abstract; column 3, lines 52-51; column 7, lines 47-67 and column 8, lines 1-32; Examiner’s Note: Siddall uses a machine learning or AI algorithm to identify context keywords or actions corresponding to items on the video data. ); and
associating the first video clip and the text generated using the machine learning algorithm with the selected one or more actions (e.g., Steps (4)-(5) as shown in Fig. 3; Abstract; column 8, lines 33-67 and column 9, lines 1-34; Examiner’s Note: Siddall uses am machine learning or AI algorithm to output text associated with a video, i.e., closed captioning information, textual content streams, etc. ).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the textual transcription device of Siddall with Hunt and Banne’s approach for creation and maintenance of video-based training documentation for the purpose of addressing the inefficiencies with regard to processing video content to generate textual transcriptions as suggested by Siddall.
Lau discloses wherein the step action tree includes branches representing deviations between the test scripts(e.g., step 205 as shown in Fig. 2; paragraph[0042], “As shown in FIG. 2, automatic identification of subroutines from test scripts 200 may include a step 205 of identifying at least one subroutine from at least one test script, such that the subroutine includes at least one instruction class, wherein the instruction class includes at least a instruction class type and a subroutine object type, and the test script includes at least one test script action, and the test script action includes at least a test script action type and a test script object type ….”; paragraph [0052]; paragraph [0053]; Examiner’s Note: For example, subroutine instructions 730 as shown in Fig. 7 illustrates branches representing deviations from the login script. ).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the subroutines of Lau with Hunt, Elges, Banne, and Siddall’s approach for a more efficient system and method for automatic identification of subroutines from test scripts as suggested by Lau.
Claims 8-10 and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Hunt et al. (Hereinafter, US 2017/0040037 A1) in view of Banne (US 2020/0371818 A1), Elges (US 2020/0192790 A1), Siddall et al. (Hereinafter, Siddall, US 10,885,903 B1), Lau et al. (Hereinafter, Lau, US 20120/317547 A1), and further in view of Panuganty (US 2020/0034764 A1).
Per claim 8, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 1 but do not expressly disclose the method as further comprising, prior to generating the text receiving a selection of two or more languages.
Panuganty discloses prior to generating text for the first action, receiving a selection of two or more languages (paragraph [0175], “ …The audible output description can be in any suitable language, such as a default language (e.g., English) and/or a user-defined language (e.g., French, German, Mandarin), where audio generation module 1704 includes machine-learning algorithms corresponding to the selected language. In some implementations, the audible output can be customized via alternate or additional user-defined settings, such as a gender-voice setting, output pace setting, verbal tone, etc. ... “ ).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the narrated analytics playlist device of Panuganty with Hunt, Elges, Banne, Siddall, and Lau’s approach for creation and maintenance of video-based training documentation for the purpose of making the analysis of large volumes of data easier and more meaningful as suggested by Panuganty.
Per claim 9, Hunt, Elges, Banne, Siddall, Lau, and Panuganty disclose the method of claim 8, wherein generating text comprises generating a text string in each of the two or more languages (Hunt, paragraph [0016], “… In support of alternate natural languages, software product 126 can reference settings such as user locale or other attributes to display appropriate UI text. “).
Per claim 10, Hunt, Elges, Banne, Siddall, Lau, and Panuganty disclose the method of claim 9, further comprising generating a separate video clip for the one or more actions for each of the In support of alternate natural languages, software product 126 can reference settings such as user locale or other attributes to display appropriate UI text. “).
Per claim 21, Hunt, Elges, Banne, Siddall, and Lau disclose the computer program product of claim 20, but do not expressly disclose wherein a description of the first action is used as an input to the machine learning algorithm and the text associated with the first action is an output of the machine learning algorithm.
However, Panuganty discloses wherein the text is generated using a machine learning algorithm (e.g., Step 2114 as shown in Fig. 21; paragraph [0203], “At 2114, and in response to receiving the script, one or more implementations augment the script to generate narrated analytics playlists. In various implementations, the personalized analytics system applies a computational algorithm to the script to identify what components and/or visualizations to include in a playlist that help explain the various insights. One or more implementations augment the script with narrative description(s) using various types of machine-learning algorithms, such as grammar-based algorithms, language pattern algorithms, syntactic algorithms, etc. In turn, the textual description generated by these machine-learning algorithms can be converted into an audible output, such as through the use of various text-to-speech algorithms. “; paragraph [0203]; Examiner’s Note: Panuganty disclose using machine language to generate a textual description from a script .).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the narrated analytics playlist device of Panuganty with Hunt, Elges, Banne, Siddall, and Lau’s approach for creation and maintenance of video-based training documentation for the purpose of making the analysis of large volumes of data easier and more meaningful as suggested by Panuganty.
Per claim 22, Hunt, Elges, Banne, Siddall, and Lau disclose the computer program product of claim 20, but do not expressly disclose wherein the audio is a narration audio clip for the one or more actions.
Panuganty discloses wherein the audio is a narration audio clip for the one or more actions (Panuganty, paragraph [0175], “Audio generation module 1704 converts the descriptions generated by vocabulary and SSML generation module 1606 of FIG. 16 into an audible form. One or more implementations include text-to-speech algorithms to generate audible output. In scenarios in which the statically bundled content includes multiple narrative descriptions, the audio generation module 1704 selects one of the narrative descriptions, such as the verbose narrative description, to convert into audible output…”).
It would have been obvious for a person of ordinary skill in the art before the
effective filing date of the claimed invention to use the narrated analytics playlist device of Panuganty with Hunt, Banne, and Siddall’s approach for creation and maintenance of video-based training documentation for the purpose of making the analysis of large volumes of data easier and more meaningful as suggested by Panuganty.
Claims 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Hunt et al. (Hereinafter, US 2017/0040037 A1) in view of Banne (US 2020/0371818 A1), Elges (US 2020/0192790 A1), Siddall et al. (Hereinafter, Siddall, US 10,885,903 B1), Lau et al. (Hereinafter, Lau, US 20120/317547 A1), and further in view of Kim et al. (Hereinafter, Kim, US 11,004,471 B1).
Per claim 11, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 1, but do not expressly disclose further comprising determining a plurality of preceding actions based on the selection of the one or more actions.
Kim discloses determining a plurality of preceding actions based on the selection of the first action (e.g., Step 404 as shown in Fig. 4; Abstract; column 9,lines 16-25; Kim discloses a plurality of preceding actions navigating back to the latest video clip. ).
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to use the video clip editing device of Kim with Hunt, Elges, Banne, Siddall, and Lau’s approach for creation and maintenance of video-based training documentation for the purpose of providing expanded video editing capabilities as suggested by Kim.
Per claim 12, Hunt, Banne, Siddall, Lau, and Kim disclose the method of claim 11, further comprising generating a video clip for each of the plurality of preceding actions (Hunt, e.g., Step 408 as shown in Fig. 4; column 9, lines 25--27).
Per claim 13, Hunt, Elges, Banne, Siddall, Lau, and Kim disclose the method of claim 12, further comprising generating a video file comprising each video clip for each of the plurality of preceding actions and the first video clip (Hunt, e.g., Step 218 as shown in Fig. 2; paragraph [0031]).
Claims 14 is rejected under 35 U.S.C. 103 as being unpatentable over Hunt et al. (Hereinafter, US 2017/0040037 A1) in view of Elges (US 2020/0192790 A1), Banne (US 2020/0371818 A1), Siddall et al. (Hereinafter, Siddall, US 10,885,903 B1), Lau et al. (Hereinafter, Lau, US 20120/317547 A1), and further in view of Boyle et al. (Hereinafter, Boyle, US 2016/0133295 A1).
Per claim 14, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 1, but do not expressly disclose the method further comprising appending music to the first video clip
Boyle discloses appending music to the first video clip (paragraph [0025], “In some versions of the editing software a music clip library is available and music clips may be appended to the video clips. The music clips may be stored on the user's device or may be accessible through the Internet.”; See claims 5 and 11).
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to use the editing systems of Boyle with Hunt, Banne, and Siddall’s approach for creation and maintenance of video-based training documentation for the purpose of making it easier to identify highlights in video recordings as suggested by Boyle.
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Hunt et al. (Hereinafter, US 2017/0040037 A1) in view of Elges (US 2020/0192790 A1), Banne (US 2020/0371818 A1), Siddall et al. (Hereinafter, Siddall, US 10,885,903 B1), Lau et al. (Hereinafter, Lau, US 20120/317547 A1), and further in view Coste et al. (Hereinafter, Coste, US 2004/0046792 A1).
Per claim 15, Hunt, Elges, Banne, Siddall, and Lau disclose the method of claim 1, but do not expressly disclose wherein generating the step action tree comprises recording, with a developer application, a user interacting with graphical user interface elements of a client application.
Coste discloses wherein generating the step action tree comprises recording, with a developer application (e.g., capture tool 102 as shown in Fig. 1; paragraph [0047], “The capture tool 102 facilitates the production of application graphical user interface (GUI) simulations by capturing screen images and other relevant data, such as controls and events from the target application and operating system, while a course developer is performing a desired set of steps for a particular task. In addition to capturing an image of the screen of the target application, the capture tool 102 captures controls that appear on the screen, including various properties and data associated with the controls.”), a user interacting with graphical user interface elements of a client application (paragraph [0051], “In addition to capturing the controls on a screen, the capture tool 102 may capture the events that occur while the course developer is interacting with the application. This event information may include various mouse and keyboard events that occur between screen captures.”; paragraph [0052]).
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to use the system and methods of Coste with Hunt, Elges, Banne, Siddall, and Lau for creation and maintenance of video-based training documentation for the purpose of reducing the time and expense required to build highly interactive training simulations as suggested by Coste.
Per claim 16, Hunt, Elges, Banne, Siddall, Lau, and Coste disclose the method of claim 15, further comprising automatically detecting, with the developer application, each interactive graphical user interface element of the client application prior to recording the user interacting with the one or more graphical user interface elements (Coste, paragraph [0007]; paragraph [0088]).
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to use the system and methods of Coste with Hunt, Elges, Banne, Siddall, and Lau for creation and maintenance of video-based training documentation for the purpose of reducing the time and expense required to build highly interactive training simulations as suggested by Coste.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Hunt et al. (Hereinafter, US 2017/0040037 A1) in view of Elges (US 2020/0192790 A1), Banne (US 2020/0371818 A1), Siddall et al. (Hereinafter, Siddall, US 10,885,903 B1), Coste et al. (Hereinafter, Coste, US 2004/0046792 A1), and further in view of Liu (US 2017/0046053 A1).
Per claim 18, Hunt, Elges, Banne, Siddall, and Coste disclose the method of claim 16, but do not expressly disclose wherein the first video clip comprises a zoomed-in view of the graphical user interface performing the one or more actions
Liu discloses wherein the first video clip comprises a zoomed-in view of the graphical user interface performing the one or more actions
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to use the video enhanced browsing of Liu with the Hunt, Elges, Banne, Siddall, and Coste approach for creation and maintenance of video-based training documentation for the purpose of reducing the amount of time browsing as suggested by Liu.
Response to Arguments
Applicant's arguments filed 17 September 2025 have been fully considered but they are not persuasive.
First, the applicant argues that “no cited reference, whether considered alone or in combination with any other cited reference, discloses, teaches, or suggests, receiving, via a graphical user interface, a selection of one or more actions in a branch of the step action tree as recited in claim 1.”
Examiner disagrees since Elges discloses receiving via the graphical user interface, a selection of one or more actions in a branch of the step action tree (paragraph [0030], “ In the specific example shown regarding FIG. 1, a user has opened the no-code test application, selected the “add tax” user interface element 108, and confirmed that selection via fly-out window 110. According to some examples, the user may have interacted with a specific column in the ERP application that the user would like to add the tax too, and the “add tax” user interface element 108 may have then been presented to the user… “; paragraph s [0037-0038]. Fig. 3 illustrates receiving, via the graphical user interface, a selection of one or more actions in a branch of the step action tree.
Second, the Examiner disagrees since Lau discloses wherein the step action tree includes branches representing deviations between the test scripts(e.g., step 205 as shown in Fig. 2; paragraph[0042], “As shown in FIG. 2, automatic identification of subroutines from test scripts 200 may include a step 205 of identifying at least one subroutine from at least one test script, such that the subroutine includes at least one instruction class, wherein the instruction class includes at least a instruction class type and a subroutine object type, and the test script includes at least one test script action, and the test script action includes at least a test script action type and a test script object type ….”; paragraph [0052]; paragraph [0053). For example, subroutine instructions 730 as shown in Fig. 7 illustrates branches representing deviations from the login script.
Furthermore, Hunt, Banne, Lau, and Siddall were not relied upon to disclose receiving, via a graphical user interface, a selection of one or more actions in a branch of the step action tree as recited in claim 1. For this reason, independent claim 1 is not allowable and should not proceed to allowance. Additionally independent claims 19 and 20 recite the same or similar features as those of claim 1 and should not be allowed for the same reasons. Furthermore, the dependent claims are each allowable based on
its dependence on one of the rejected independent claims.
For at least the reasons above, Examiner maintains the rejection of Claims 1, 4-16 and 18-22.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DARRIN HOPE whose telephone number is (571)270-5079. The examiner can normally be reached Mon-Thr - 6:45-4:15, Fri - 6:45-3:15, Alt. Fri Off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen S Hong can be reached at (571)272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DARRIN HOPE
Examiner
Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178