Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are currently pending for examination.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 11-20 are rejected under 35 U.S.C. 101 because they do not fall within at least one of the four categories of patent eligible subject matter
As per claim 11, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
Software per se, when expressed solely as code or instructions without any structural or hardware limitations, does not have a physical embodiment and therefore does not qualify as a machine, manufacture, or composition of matter. Although the claims recite a computing device, they are directed toward the ‘embodied agent simulator’ rather than the ‘computing device’- which is not tied to any physical hardware. Thus claim 11 lacks the necessary physical articles or hardware components to constitute a machine within the meaning of 35 USC 101.
As per claim 12, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 13, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 14, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 15, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 16, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 17, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 18, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 19, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
As per claim 20, the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because an “embodied agent simulator for simulating interaction with an end user” is considered to be software per se.
It is advised that the applicant amend claims 11-20 to directly tie the agent simulator to a tangible medium or hardware in order to meet the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Rogers (US 8850315 B1) in view of Fisher (US 20150339589 A1) in further view of King (US 20190026132 A1).
As per claim 1, Rogers discloses:
A method for interacting with an artificial agent and digital content on an end user display device, the method comprising: simulating the artificial agent in an agent virtual space, displaying the artificial agent and the artificial agent virtual space to an end user on an end user display device ("a first-time user of a user interface for a web-based application (e.g., a sales portal) is presented with informative materials (e.g., a virtual agent presenting a tutorial with a coordinated focus indicator) indicating the functions and tools of the web-based application, which are superimposed on the user interface of the web-based application.", col.1, lines 38-44 ; "The electronic device provides the user interface for presentation to a respective user, the user interface includes a plurality of displayed user interface objects for initiating performance of corresponding functions of an online application. The electronic device determines whether to superimpose a respective portion of the informative materials on the user interface, where the informative materials are distinct from the user interface.", col.1, lines 49-56 ; Examiner Note: the virtual agent presenting a tutorial equates to an artificial agent)
Rogers discloses the above limitations of claim 1, but does not explicitly disclose applying an attention model to the end user’s attention.
However, Fisher discloses:
receiving images of the end user from a camera; tracking the end user’s attention; applying an attention model to the end user’s attention and providing an output to the artificial agent (“Any existing commercial and/or custom-built gaze tracker apparatus may be utilized in order to obtain gaze direction pattern of a human trainer executing a task. The gaze pattern may be the task dependent and highly indicative the overt attention of the human performing the task. The gaze pattern (saccades, fixations, and/or smooth pursuit) may be converted into a dynamic heat-map of attention (also referred to as “importance map”). In one or more implementations the attention map may be obtained using live image in real time and/or recorded video.", 0066 ; "The method may comprise: for a given visual scene: determining a feature within a portion of a digital image of the visual scene, the determination being based on an analysis of a saliency map associated with the task, the saliency map being representative of one or more areas of preferential attention by a human trainer; and executing the task based on an association between with the feature and the task. The saliency map may be determined by a learning process of the robotic apparatus. The association between with the feature and the task may be determined by the learning process. The learning process may have been previously trained to execute the task using gaze of the human trainer.", 0024 ; Examiner Note: the determining of a feature based upon an analysis of the saliency map requires that the system was provided the output of the learning process, i.e., attention model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to combine the teachings of Rogers with the of Fisher in order to improve the accuracy of the user attention data through the use of IR light (Fisher, [0092]).
Rogers in view of Fisher may disclose the above limitations of claim 1, but they do not explicitly disclose the output of the attention model being considered in the simulating of the embodied interaction between an artificial agent and end user, nor the agent space being a virtual space.
However, King discloses: simulating embodied interaction between the artificial agent and the end user based on the output of the attention model. ("In some embodiments, one or more sensing, detecting or data capturing devices, such as a camera, touch sensor, heat sensor, or sonar, may be used to detect the motion of the user.", 0010 ; "Systems and methods according to specific embodiments described herein provide a method of interaction between a user and live help where the two parties communicate with each other by visually referencing the physical locations of various features of the device through which the two parties communicate and, in some embodiments, of surrounding devices or systems.", 0008 ; "In some embodiments, one or more sensing, detecting or data capturing devices, such as a camera, touch sensor, heat sensor, or sonar, may be used to detect the motion of the user.", 0010 ; Examiner Note: the interpretation of visual referencing by the program necessitates an attention model be applied, the live help agent equates to an artificial agent)
simulating the artificial agent in an agent virtual space ("In some embodiments, a virtual reality environment is presented at the agent's location, allowing the agent to indicate actions relating to various objects in the user's environment. In some embodiments, augmented reality environments and mixed reality environments may be presented at the agent's location. In some embodiments, the agent may be using one or more devices that provide virtual reality, augmented reality, or mixed reality environments, such as a headset, glasses, or controller.", 0038)
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Rogers in view of Fisher with those of King to increase the effectiveness of the artificial agent in aiding the user by allowing the agent to indicate devices which are either on or off of the user interface (King, [0039]).
As per claim 2, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 1.
Furthermore, King discloses:
the attention model is also applied to the artificial agent’s attention ("In some embodiments, one or more sensing, detecting or data capturing devices, such as a camera, touch sensor, heat sensor, or sonar, may be used to detect the motion of the user. In some embodiments, such devices may detect the motion of the agent or helper.", 0010)
The system of Rogers in view of Fisher in further view of King would provide the motion of the agent (King, [0010]) as input to the learning process of Fisher (Fisher, [0024]).
As per claim 3, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 1.
Furthermore, King discloses:
the embodied agent simulator is further programmed to receiving images of the user’s space from the camera. ("In some embodiments, the placement of the agent's pointing device is displayed on the user's screen as though the agent is pointing to a feature in the user environment.", 0011 ; "In some embodiments, when live help is being provided, the user GUI shows a live video image of a live agent, including the agent's hand 310 pointing to user GUI elements or to one or more external devices 320 in a user's environment.", 0048 ; Examiner Note: it is necessary that images of the user’s environment are received in order for the agent to point to a feature in the environment)
As per claim 4, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 1.
Furthermore, King discloses:
the method further comprising representing digital content to be perceivable by the artificial agent by simulating the digital content in the agent virtual space, the digital content being visible to the end user. ("In some embodiments, when live help is being provided, the user GUI shows a live video image of a live agent, including the agent's hand 310 pointing to user GUI elements or to one or more external devices 320 in a user's environment.", 0033 ; "In some embodiments, a virtual reality environment is presented at the agent's location, allowing the agent to indicate actions relating to various objects in the user's environment.", 0038)
As per claim 5, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 1.
Furthermore, Fisher discloses:
tracking the end user’s attention comprises one or more of tracking a gaze of the end user, tracking an eye movement of the end user, and tracking the eye movement of the end user on the display ("Any existing commercial and/or custom-built gaze tracker apparatus may be utilized in order to obtain gaze direction pattern of a human trainer executing a task. The gaze pattern may be the task dependent and highly indicative the overt attention of the human performing the task. The gaze pattern (saccades, fixations, and/or smooth pursuit) may be converted into a dynamic heat-map of attention (also referred to as “importance map”). In one or more implementations the attention map may be obtained using live image in real time and/or recorded video.", 0066)
As per claim 6, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 1.
Furthermore, Fisher discloses:
the attention model includes a salience factor ("The method may comprise: for a given visual scene: determining a feature within a portion of a digital image of the visual scene, the determination being based on an analysis of a saliency map associated with the task, the saliency map being representative of one or more areas of preferential attention by a human trainer", 0024 ; "The saliency map acquired iteratively or cumulatively, as described here, may comprise a statistical description of salience at a given location (with respect to sensory input) at a given time. Examples of such a statistical description may comprise a probability distribution, a confidence interval, a mean, and/or a standard deviation of the salience as a function of position and/or time.", 0093)
Furthermore, Rogers discloses:
objects that are jointly attended by the artificial agent and the end user increase in salience ("In FIG. 3C, the informative materials include video content including a virtual agent 338 and a focus indicator 340 coordinated with the video content. For example, the video content includes a tutorial or walkthrough for first-time users of the web-based application that describes the functions and tools provided by the web-based application. In this example, focus indicator 340 is coordinated with the video content in such a way that when virtual agent 340 points to, describes or speaks of a user interface element of the user interface, focus indicator 340 draws the first-time user's attention to the corresponding user interface element of the user interface displayed on touch screen 215 (e.g., focus indicator 340 points to the referenced user interface element).", col.10, lines 9-23)
The system of Rogers in view of Fisher in further view of King would provide the focus indications of the user and of the virtual agent as inputs to the learning process (Fisher, [0024]) which would generate a saliency map for each input (Fisher, [0066]).
As per claim 7, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 6.
Furthermore, Fisher discloses:
the salience factor includes a weighted salience gaze factor based on the end users and the artificial agent’s salience gaze map. ("Gaze directions falling within the area 604 over successive frames 600 may be weighted by the kernel to obtain saliency distribution associated with that portion of the frame.", 0090 ; Examiner Note: the weighted saliency distributions associated with individual areas equate to weighted salience gaze factors)
The combination of Rogers in view of Fisher in further view of King would provide a system which would generate weighted salient gaze factors based on both the end user’s and the artificial agent’s salience map (King, [0038]).
As per claim 8, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 1.
Furthermore, Fisher discloses:
the attention model is also applied to the end user’s eye movement, actions of the end user, objects in the end user’s environment and actions in the end user’s environment (“During training, the eye tracking data (e.g., 106 in FIG. 1) and the sensory information (e.g., 116, 118 in FIG. 1) may be utilized in order to determine a saliency map, sensory context, and/or action between actions by the trainer and one or more salient objects determined on from the saliency map.”, 0076 ; "Additional equipment may be used to facilitate gaze detection as well as record, for example, the driver head position, as required for reliable extraction of the saliency map and the context data.", 0092 ; Examiner Note: changing driver head position equates to an action)
Fisher may disclose the providing of user eye movement and action information to an attention model, but does not explicitly disclose the providing of the objects and actions in the end user’s environment to the attention model.
However, King discloses:
the attention model is also applied to the end user’s eye movement, actions of the end user, objects in the end user’s environment and actions in the end user’s environment ("In some embodiments, one or more physical features outside the user's display, at the user's location or surrounding area, are identifiable at the live help location", 0009 ; " In some embodiments, a virtual reality environment is presented at the agent's location, allowing the agent to indicate actions relating to various objects in the user's environment.", 0038 ; “According to specific embodiments, systems and methods as described herein store or acquire relative spatial information for parts of a user's computing device and optionally also for objects at a user's location”, 0053)
As per claim 9, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 1.
Furthermore, Fisher discloses:
attention model is also applied to end user inputs ("The driver ‘A’ may issue a signal (e.g. touch the screen, press a button, and/or click a mouse in the appropriate location) based on identifying a condition where issuing an alert may appropriate. Such a condition may be, for example, a misdirection of gaze of the driver ‘B’, and/or a mismatch between the saliency map predicted by component 550 and the current saliency map of the driver ‘B’. In some implementations, expert driver ‘A’ may rate (score) the output signals 552 according to their appropriateness. Those skilled in the arts will appreciate that the teaching signal, as provided by the expert driver ‘A’, may in some cases be used not only to train the algorithm for computation of the mismatch between the predicted saliency and the current saliency, but also provide an additional teaching input to the saliency predictor in component 550.", 0119 ; Examiner Note: the touching of the screen or pressing of the button equate to end user inputs)
As per claim 10, Rogers in view of Fisher in further view of King fully discloses the limitations of claim 9.
Furthermore, Fisher discloses:
the end user inputs comprise one or more of auditory or textual inputs, verbal inputs, touch screen or end user mouse movements, visual inputs, wherein the visual inputs are gestures or facial expressions, wherein the gestures comprise movement of one of the arm, hand or finger. ("The driver ‘A’ may issue a signal (e.g. touch the screen, press a button, and/or click a mouse in the appropriate location) based on identifying a condition where issuing an alert may appropriate. Such a condition may be, for example, a misdirection of gaze of the driver ‘B’, and/or a mismatch between the saliency map predicted by component 550 and the current saliency map of the driver ‘B’. In some implementations, expert driver ‘A’ may rate (score) the output signals 552 according to their appropriateness. Those skilled in the arts will appreciate that the teaching signal, as provided by the expert driver ‘A’, may in some cases be used not only to train the algorithm for computation of the mismatch between the predicted saliency and the current saliency, but also provide an additional teaching input to the saliency predictor in component 550.", 0119)
As per claim 11, it is an embodied agent simulator (King, [0038] ; “In some embodiments, a live agent may have the capability to physically point to or perform gestures at a specific location of the user's screen”, 0033) claim with substantially the same limitations as claim 1, and as such, it is rejected for substantially the same reasons.
As per claim 12, it is an embodied agent simulator claim with substantially the same limitations as claim 2, and as such, it is rejected for substantially the same reasons.
As per claim 13, it is an embodied agent simulator claim with substantially the same limitations as claim 3, and as such, it is rejected for substantially the same reasons.
As per claim 14, it is an embodied agent simulator claim with substantially the same limitations as claim 4, and as such, it is rejected for substantially the same reasons.
As per claim 15, it is an embodied agent simulator claim with substantially the same limitations as claim 5, and as such, it is rejected for substantially the same reasons.
As per claim 16, it is an embodied agent simulator claim with substantially the same limitations as claim 6, and as such, it is rejected for substantially the same reasons.
As per claim 17, it is an embodied agent simulator claim with substantially the same limitations as claim 7, and as such, it is rejected for substantially the same reasons.
As per claim 18, it is an embodied agent simulator claim with substantially the same limitations as claim 8, and as such, it is rejected for substantially the same reasons.
As per claim 19, it is an embodied agent simulator claim with substantially the same limitations as claim 9, and as such, it is rejected for substantially the same reasons.
As per claim 20, it is an embodied agent simulator claim with substantially the same limitations as claim 10, and as such, it is rejected for substantially the same reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Page (US 20180225874 A1) – discloses contact center augmented reality system including a supervisor device, a location tracker, a state monitor, and an augmenter. The supervisor device may include a display to augment a supervisor view of the contact center. The location tracker may monitor agent locations in the contact center and may monitor the supervisor device location in the contact center.
Anderson (US 20130212501 A1) – discloses a method including receiving a statement from a user, observing user behavior, determining a user context based on the behavior, processing the user statement and user context to generate a reply to the user, and presenting the reply to the user on a user interface.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROSS MICHAEL VINCENT whose telephone number is (703)756-1408. The examiner can normally be reached Mon-Fri 8:30AM-5:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, April Blair can be reached at (571) 270-1014. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.M.V./
Examiner, Art Unit 2196
/HIREN P PATEL/Primary Examiner, Art Unit 2196
Prosecution Insights