DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This Office Action is in response to the application filed on May 22,2024.
3. Claims 1-20 are pending, all the pending claims are examined.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
4. Claims 1-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bradfield (US 20220276882 A1).
Bradfield is directed to artificial intelligence based systems and methods for autonomously generating customer service help guides with integrated graphical components and for autonomously error-checking knowledge base support resources.
As per claim 1, Bradfield discloses a computing apparatus (see various computing devices in Fig. 5) , comprising:
one or more computer readable storage media (e.g., memory 1006, Fig. 5);
one or more processors operatively coupled with the one or more computer readable storage media (Processor(s) 1004 may interface with memory 1006 via the computer bus to execute an operating system (OS), Fig. 5) and program instructions stored on the one or more computer readable storage media that, when executed by the one or more processors, ([0233]For example, in various aspects, any one or more of the products or tools, as described herein, may implement or execute computing instructions for gathering data or information from various networked sources);
direct the computing apparatus to at least:
identify a sequence of steps to accomplish a task in an application ([0080] The action recorder can record the location of the action locus using various identifying information, including the URL for a webpage, the text of a hyperlink or the shape of a clickable symbol, as well as the approximate quadrant of the GUI. [0104] Recorded action steps may include action loci information if applicable, i.e. the precise or general location on the GUI where the action should be executed or initiated, as well as other identifying features indicating where an action should take place, e.g. the text string to be clicked, the button to be checked, the box for entering text, the link to be selected, etc.)
generate a script for a browser automation tool to perform an interaction with an instance of the application based on the sequence of steps ([0161] Once a terminal page and/or state has been associated with a task title label or labels, the series of steps that led to the terminal page and/or state is automatically reverse-engineered to create an action script which is then associated with the task title label. That is, the series of steps required to arrive at particular page and/or state is automatically recorded as an action script during the initial mapping process performed by the automated web browser or user interface);
cause the browser automation tool to execute the script ([0097] TOOL 1 automatically runs a test of each recorded action script by executing the actions listed in the action script in their specified order. The text execution is done on the device, operating system, and/or web browser specified in the device interface label of the help base article (i.e. if possible, using the same version of the device/operating system/browser in the label). TOOL 1 may use an already existing software simulator to execute the action script, where the simulator simulates the behavior of a particular device, operating system, web browser type, etc., e.g., including specific versions of those such devices, operating systems or web browsers);
capture a video of the interaction with an instance of the application based on the script ([0234] For example, each of servers 1062 and 1064 may host video and/or text information comprising “help” content for a specific product or service. Server 1062 may represent, for example, a YOUTUBE server that hosts video content comprising “help” content for a specific product or service (e.g., a video teaching users how to use or interact with a specific service or product); and
cause display of the video in a user interface of the application ([0122] TOOL 2 then daisy-chains (i.e., puts in sequence) these GUI screenshots together to produce an animated sequence or video covering the task from start to completion (see, e.g., FIGS. 2A through 2O, GUIs 2000 through 2140). In particular, FIGS. 2A-2O illustrate example GUIs in accordance with various aspects disclosed herein).
As per claim 2, Bradfield further discloses that the computing apparatus of claim 1, wherein the program instructions further direct the computing apparatus to prompt a generative artificial intelligence (AI) model to generate a list of graphical user interface (GUI) elements based on the sequence of steps ([0145] As a further option, the Show Me How button can be delivered via a web browser (e.g. Microsoft Edge, Google Chrome, Apple Safari, Mozilla Firefox, etc.) plug-in or extension. First, the user must download and install or otherwise enable the Show Me How browser extension/plug-in. Then, when the user visits a website which has help guides available, the user will have the available option of clicking or tapping the extension/plug-in GUI button within the browser's GUI to activate TOOL 3. Clicking the GUI button launches a pop-up window, or other GUI interface, where the user can access the help guides available for the website being visited, e.g. the user can type in their customer support inquiry and see a list of help guides available, or they can search the help guide database for a topic, etc. Once the user finds the customer service or help task they would like to execute, they can select/click/tap the desired task to initiate the execution of the task).
As per claim 3, Bradfield further discloses that the computing apparatus of claim 2, wherein to generate the script for the browser automation tool to perform the interaction with the instance of the application, the program instructions direct the computing apparatus to generate the script for the browser automation tool to interact with the instance of the application based on the list of GUI elements ([0123] Using the registered action script as directions, TOOL 2 has an automated graphics overlay tool which overlays animated graphics over the animated sequence to highlight any relevant action loci and clarify actions to be taken. These graphics may include blinking halos, blinking or non-blinking text, enlarged call-out icons, animated arrows, or other circular or oval marks.. [0159] Additionally, or alternatively, an automated computing instruction or application is used to map all possible action link(s) or path(s) originating from the home page or initial state of a website, mobile app, or software application. This application can be an automated web browser or automated user interface that simulates a mobile or desktop operating system user. This automated application can be pre-programmed or otherwise configured to fill in common user parameters such as username, password, contact information, or the like.
As per claim 4, Bradfield further discloses that the computing apparatus of claim 1, wherein to capture the video of the interaction with the instance of the application, the program instructions direct the computing apparatus to capture the video of the interaction with the instance of the application in a window of a browser application ([0133] The help guide video window can also be made available using an internet web browser (e.g., Microsoft Edge, Google Chrome, Apple Safari, Mozilla Firefox, etc.) plug-in or extension. Once a web browser user has added the help guides plug-in, the web browser will provide a GUI button that will activate the plug-in, launching a separate GUI window which will show a list of help guide videos available that are applicable to the help base or website that the user is viewing).
As per claim 5, Bradfield further discloses that the computing apparatus of claim 1, wherein to generate the script for the browser automation tool to perform the interaction with the instance of the application, the program instructions direct the computing apparatus to prompt a large language model to generate the script for the browser automation tool to perform the interaction with the instance of the application based on the sequence of steps ([0035] The AI based computer method may further comprise receiving, by the one or more processors, a request from a client device of a user for assistance with a given user task, wherein the client device is a device corresponding to the computing device. The AI based computer method may further comprise rendering, by the one or more processors in response to the request, the one or more graphic-and-text displays associated with a relevant action script on a GUI of the client device of the user to display one or more user directives to visually indicate how to complete the given user task. The relevant action script may be determined by the AI model).
As per claim 6, Bradfield further discloses that the computing apparatus of claim 1, wherein to identify the sequence of steps to accomplish the task in the application, the program instructions direct the computing apparatus to prompt a large language model to generate the sequence of steps based on a knowledge base of the application ([0018] The computing instructions, when executed, may further cause the one or more processors to store the one or more action scripts in the one or more memories, wherein each action script is comprised of one or more user directives for completing one or more user tasks on a computing device and by a software application (app) configured to render a GUI on a display screen of the computing device, and wherein each action script corresponds to at least one user task of the one or more user tasks and is adapted to operate on the computing device, the software app, and the GUI. [0225] For example, FIG. 5 illustrates an example artificial intelligence based system 1000 for autonomously generating help bases and related graphic components, in accordance with various aspects disclosed herein. The products/tools as described herein may be implemented via artificial intelligence based system 1000, which provides a computing platform or otherwise computing ecosystem for the implementation of the products/tools described herein).
As per claim 7, Bradfield further discloses that the computing apparatus of claim 1, further comprising causing display of screenshots from the video in the user interface of the application ([0118] TOOL 2 creates autonomously-generated animated screenshot help guides (each one a “help guide”) derived from existing help base material, as well as from customer support help data created or generated by TOOLS 1, 4, 5 and 8 described herein. Also see [0121]).
As per claim 8, Bradfield further discloses that the computing apparatus of claim 1, wherein to identify the sequence of steps to accomplish the task in the application, the program instructions further direct the computing apparatus to receive user input comprising a request for information to accomplish the task in the application ([0120] TOOL 2 accesses the action scripts database which has been populated as described for TOOL 1, and then executes the sequence of recorded actions recorded in a given action script. While executing those actions, TOOL 2 also simultaneously records a screenshot from the GUI of each step within the action script. 0123] Using the registered action script as directions, TOOL 2 has an automated graphics overlay tool which overlays animated graphics over the animated sequence to highlight any relevant action loci and clarify actions to be taken. These graphics may include blinking halos, blinking or non-blinking text, enlarged call-out icons, animated arrows, or other circular or oval marks).
As per method claims 9-14, these method claims include limitations that are similar to that of computing apparatus claims 1-6, respectively. The method claims are also rejected under similar citations given to the apparatus claims, respectively.
As per computer readable storage media claims 15-20, these storage media claims include limitations that are similar to that of computing apparatus claims 1-6, respectively. The storage media claims are also rejected under similar citations given to the apparatus claims, respectively.
Conclusion
5. Any inquiry concerning this communication or earlier communications from the examiner should be directed to TADESSE HAILU whose telephone number is (571)272-4051; and the email address is Tadesse.hailu@USPTO.GOV. The examiner can normally be reached Monday- Friday 9:30-5:30 (Eastern time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bashore, William L. can be reached (571) 272-4088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TADESSE HAILU/Primary Examiner, Art Unit 2174