DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of as a continuation of U.S. Patent Application Number 13/852,126 filed on 03/28/2013, which claims priority to, and the benefit of U.S. Provisional Application No. 61/618593 filed on 03/30/2012 is acknowledged.
Specification
The substitute specification filed 04/01/2021 has been entered.
Drawings
The drawings were received on 03/31/2022. These drawings are acceptable.
Claim Interpretation
The examiner notes the following guidance regarding the interpretation of claim language for determining the broadest reasonable interpretation (BRI), in light of the specification.
MPEP notes:
MPEP 2111: CLAIMS MUST BE GIVEN THEIR BROADEST REASONABLE INTERPRETATION IN LIGHT OF THE SPECIFICATION
“During patent examination, the pending claims must be ‘given their broadest reasonable interpretation consistent with the specification.’ The Federal Circuit’s en banc decision in Phillips v. AWH Corp., 415 F.3d 1303, 1316, 75 USPQ2d 1321, 1329 (Fed. Cir. 2005) expressly recognized that the USPTO employs the "broadest reasonable interpretation" standard:
The Patent and Trademark Office ("PTO") determines the scope of claims in patent applications not solely on the basis of the claim language, but upon giving claims their broadest reasonable construction "in light of the specification as it would be interpreted by one of ordinary skill in the art." In re Am. Acad. of Sci. Tech. Ctr., 367 F.3d 1359, 1364[, 70 USPQ2d 1827, 1830] (Fed. Cir. 2004)…” (emphasis added)
2111.01 (I). THE WORDS OF A CLAIM MUST BE GIVEN THEIR "PLAIN MEANING" UNLESS SUCH MEANING IS INCONSISTENT WITH THE SPECIFICATION
“Under a broadest reasonable interpretation (BRI), words of the claim must be given their plain meaning, unless such meaning is inconsistent with the specification. The plain meaning of a term means the ordinary and customary meaning given to the term by those of ordinary skill in the art at the relevant time. The ordinary and customary meaning of a term may be evidenced by a variety of sources, including the words of the claims themselves, the specification, drawings, and prior art. However, the best source for determining the meaning of a claim term is the specification - the greatest clarity is obtained when the specification serves as a glossary for the claim terms…
The presumption that a term is given its ordinary and customary meaning may be rebutted by the applicant by clearly setting forth a different definition of the term in the specification. In re Morris, 127 F.3d 1048, 1054, 44 USPQ2d 1023, 1028 (Fed. Cir. 1997) (the USPTO looks to the ordinary use of the claim terms taking into account definitions or other "enlightenment" contained in the written description); But c.f. In re Am. Acad. of Sci. Tech. Ctr., 367 F.3d 1359, 1369, 70 USPQ2d 1827, 1834 (Fed. Cir. 2004) ("We have cautioned against reading limitations into a claim from the preferred embodiment described in the specification, even if it is the only embodiment described, absent clear disclaimer in the specification."). When the specification sets a clear path to the claim language, the scope of the claims is more easily determined and the public notice function of the claims is best served.” (emphasis added)
2111.01 (II). IT IS IMPROPER TO IMPORT CLAIM LIMITATIONS FROM THE SPECIFICATION
“Though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim. For example, a particular embodiment appearing in the written description may not be read into a claim when the claim language is broader than the embodiment." Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875, 69 USPQ2d 1865, 1868 (Fed. Cir. 2004). See also Liebel-Flarsheim Co. v. Medrad Inc., 358 F.3d 898, 906, 69 USPQ2d 1801, 1807 (Fed. Cir. 2004) (discussing recent cases wherein the court expressly rejected the contention that if a patent describes only a single embodiment, the claims of the patent must be construed as being limited to that embodiment)…” (emphasis added)
The following claim terms are considered under BRI in light of the specification, where the specification provides no definition that limits the use of the plain meaning given/documented by the examiner:
Avatar: an avatar is any representation of a person in a cartoon-like/digitalized image or other type of character having human characteristics of an human the avatar is based on. The avatars are animated autonomously to interact with other user, other than the human the avatar characteristics are based on. Avatar interact with others during a deployment session and allows interaction when the human subject modeled by the avatar is unable to communicate in person and communicates using the digital avatar to others.
Specification notes some examples of avatars as models, corresponding to human subject or human subject’s personality, that can interact with a third party using a digital interface, in:
[0066] … For example, a base model avatar may only accept text input communication. Upgraded versions of the avatar may include automatic speech recognition capabilities, where an avatar interacts with spoken data input from an avatar user. Additionally, paid upgrades may improve resolution of a depicted avatar. A base model avatar may be cartoonish in character, while upgraded versions become more photorealistic and include more detailed modeling of facial characteristics.
[0068] In some embodiments, the interactive personality simulation system may comprise a "call center" using live actors to impersonate celebrities/legends. The live actor provides the "personality" characteristics in place of target person 208, and a camera 210 may capture the actor's performance. In one embodiment, the live actor, whose avatar 246 is presented to one or more users 232, receives interactive data (such as text input, live voice, live video, and/or other information) and responds thereto.
[0077] Avatars may be stored and implemented using a variety of different protocols. In one example, an avatar is implemented using computer-generated imagery (CGI). In the example CGI implementation, a human subject on whom the avatar is based is observed interacting with another person or an avatar or a device or object, such as in the form of an interview…
[0081] A further implementation displays an avatar as an animated talking head. A human subject is observed during one or more interactions, where photos of the human subject (e.g., the subject's face) are captured as well as an audio response stream…
[0090] FIG. 7 is a block diagram depicting an avatar engine that develops an avatar model and implements an avatar for interaction with a third person…
Contemporaneous: existing or occurring in the same period of time;
Examiner interprets that any avatar or virtual agent is considered related to contemporaneous events/things used for training/developing the avatar or virtual agent to function autonomously in a computing environment (e.g. capture using initial training data or video records or other recorded data used to render the avatar or virtual agent).
Response to Arguments
Applicant's arguments filed 05/06/2026 have been fully considered.
Regarding the applicant remarks directed to the rejection under 35 USC 112, the amended limitations remove the problematic language, thus render the applicant remarks moot. The rejection noted in the previous in the office action is no longer applicable and the rejection has been withdrawn.
Regarding the applicant remarks directed to the rejection under 35 USC 103, the remarks , the examiner remarks are provided below.
First, the applicant argues that the cited prior art fails, Xu et al. (US Pub. No. 2012/0130717, hereinafter ‘Xu’), to disclose limitations newly amended claim limitations. The examiner has not previously rejected the noted amended limitation and refers to the current office action. See current office action below.
Second, the applicant argues that the cited prior art fails to disclose limitations requiring the human subject to different from the user because the Xu does not contemplate the avatar subject is unavailable.
Examiner respectfully disagrees. MPEP 2111 requires that claim limitation be given there broadest reasonable interpretation (BRI) in light of the specification. In addition, although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993) and that though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim.
Examiner notes that an avatar that is presented virtual when the human subject is unavailable to communicate in person is within the scope of the claimed invention, as disclosed by the cited teaches in the Xu reference. Kuhn also teaches the use of an autonomous dealer avatar, considered within the scope of the claim limitation noted in pgs. 13-14 of remarks. The claims do not limit this interpretation, thus the rejection of claims is maintained.
Examiner further notes that the claims regarding the teaches of the prior art reference are not restricted by the claim limitations. For example:
Xu teaches in paragraph 0050, are not analogous to the claimed teaches for transmitting details of the monitored interaction (in absence of the human subject) and another user how is not the human subject. Examiner notes that as should above, the claim limitations relate the human characteristics as avatar data used to render the claimed avatar as a rendered human subject and the other user interacting with the avatar is considered an interactions that is monitored/transmitted for facilitating autonomous communication between an avatar and a user in a game application or social networking environment as disclosed by the Xu and Kuhn references. The applicant claims includes the use of the avatar interaction data as part of the data used in rendering the avatar and does not exclude the other user from the interaction as claimed in the limitations “receiving, at a server hosting a rendering engine, video data depicting a human subject, the video data captured by a camera of a first client device; processing, by the server, the video data to determine a set of human characteristics representative of the human subject, the set of human characteristics including physical appearance characteristics, voice characteristics, and personality characteristics; maintaining, in memory of the server, avatar data including a simulated human memory, the avatar data being based on the set of human characteristics, the simulated human memory comprising details of a previous interaction with an avatar, the avatar being a computer-generated autonomous representation of the human subject based on the avatar data” The claimed invention is rendered an obvious combination for generating an avatar based on avatar data from user interactions with the avatar and an human subject (e.g. a dealer in a game is an human subject for generating the virtual/avatar dealer). Applicant remarks appear to disavow the scope or their own claimed invention that clearly include the use of artificial intelligence techniques in generating and deploying an avatar, of an human subject, that learns (e.g. updates memory and transmits interaction details) for interactions with others (i.e. another person other than the avatar’s human subject).
The remarks that the applicant gives appear to support that the teaches of the Xu and Kuhn references are within the scope of the claimed inventions.
The applicant appears to be arguing against their own claim limitations as the claim limitation provide support for making the prior art combination.
Examiner reminds the applicant that claim limitations must be given BRI and limitations not recited in the claim limitations cannot be used to limited the scope of the claim to a preferred embodiment, see MPEP 2111.
The applicant’s allegations are considered mere allegation of patentability as the remarks fails to be recited by claim limitations.
Xu does teach rendering an avatar that can adapt its behavior in Xu [0019] and [0024]. The secondary reference Kuhn (US Pat. No. 9,202,171, hereinafter ‘Kuhn’) also teaches the use of an avatar/virtual dealer for interacting with players as other users during a game. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant argues the examiner has not made a prima facie case of obviousness because one of ordinary skill would not be motivated to combine the Xu and Kuhn references . Specifically, the applicant alleges that Kuhn teaches the rendering of an avatar autonomously while Xu discloses capturing elements of the human subject to render an avatar autonomously.
Examiner disagrees.
The guidance for documenting a prima face case of obviousness are specified in MPEP 2141 The MPEP section provides a clear framework for the objective analysis used for determining obviousness under 35 U.S.C. 103 stated in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966) as follows:
(A) Determining the scope and content of the prior art;
(B) Ascertaining the differences between the claimed invention and the prior art; and
(C) Resolving the level of ordinary skill in the pertinent art.,”
where the examiner has identified all the required elements in both the pervious and current office actions. Therefore, a prima facie case has been made of record by the examiner. When a prima facie case of obviousness is established, the burden shifts to the appellant to come forward with arguments and/or evidence to rebut the prima facie case, see MPEP 2145. The applicant has provided no objective evidence to conclude that the proposed modification and combinations recited in the secondary references would change the principle operation of the primary reference.
In addition, both Xu and Kuhn are in the same field of endeavor generating virtual agents (e.g. avatars) based on a human subject using information processing and machine learning techniques. Specifically, Xu states in 0019:
“This disclosure describes an architecture and techniques for providing an expressive avatar for various applications. For instance, the techniques described below may allow a user to represent himself or herself as an avatar in some applications, such as chat applications, game applications, social network applications, and the like. Furthermore, the techniques may enable the avatar to express a range of emotional states with realistic facial expressions, lip synchronization, and head movements to communicate in a more interactive manner with another user.” (emphasis added)
The Kuhn (US Pat. No. 9,202,171, hereinafter ‘Kuhn’) states that it teaches techniques to generates a virtual dealer that can be animated for presentation as an hologram or video, considered an avatar, for a game application in abstract:
“Virtual game dealers based on artificial intelligence are described. In one implementation, an artificial intelligence (AI) engine tracks player attributes and game states of a given electronic game, such as a multiplayer electronic card game hosted by a virtual dealer. The virtual dealer may be embodied as a video, hologram, or robot. The AI engine selects speech and gestures for the virtual dealer based on the game states and player attributes… Supported by the AI engine, the virtual dealer may personalize dialogue, provide information, and perform card and chip tricks. The AI engine may also animate a virtual player and select interactions between the virtual dealer and the virtual player based on game states and attributes of the human players.”
And in Kuhn 5:40-56:
“The output of the AI engine 102 is a signal representing intelligent reactions 406 for the virtual dealer 106, to be projected by the virtual dealer projector (projection engine) 636. The virtual dealer projector 636 may break the task of generating the virtual dealer 106 into several components. Thus, the virtual dealer projector 636 may have an emotion engine 638, a speech engine 640, a gesture engine 642, a character engine 644, and an accessories engine 646. The virtual dealer projector 636 also includes a video generator 518 and an audio generator 520. The video generator 518 may include a video synthesizer 648 to animate a video dealer 50 image or avatar and/or may include stored video segments 650 or "clips" that can be executed to generate virtual dealer behavior…” (emphasis added)
The examiner highlights the cited section as further support that the Xu and Kuhn reference are in the same field of endeavor as required by MPEP 2141.
Applicant's arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references.
The rejections made in the previous office action has been maintained.
Applicant argues that the prior art fails to teach elements in the amended claim. Examiner notes that the amended claim have not been previously examined by the examiner. See current office action for rejection of the amended claim limitations under 35 USC 103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 41-43, 45-47, 49-51, 53-55, and 57-59 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US Pub. No. 2012/0130717, hereinafter ‘Xu’) in view of Kuhn (US Pat. No. 9,202,171, hereinafter ‘Kuhn’) and in further view of Deng et al. (NPL: “Computer Facial Animation: A Survey”, hereinafter ‘Deng’) and Listermann (US 20130018882, hereinafter ‘List’).
Regarding independent claim 41 limitations, Xu teaches: a method, performed by one or more computing systems of enhancing the anthropomorphism of an interactive avatar through recall simulation, the method comprising: (Xu teaches in n 0075-0076: FIG. 9 is a block diagram showing an example server usable with the environment of FIG. 1. The server 112 may be configured as any suitable system capable of services, which includes, but is not limited to, implementing the avatarbased service [computing systems of enhancing the anthropomorphism of an interactive avatar through recall simulation] 110 for online services, such as providing avatars in instant-messaging programs. In one example configuration, the server 114 comprises at least one processor 900, a memory 902, and a communication connection(s) 904… Turning to the contents of the memory 902 in more detail, the memory 902 may store an operating system 906, and the avatar application 116…; And 0082: The server 114 [a computing systems] may also include additional removable storage 914 and/or non-removable storage 916. Any memory described herein may include volatile memory ( such as RAM), nonvolatile memory, removable memory, and/or non-removable memory, implemented in any method or technology for storage of information, such as computer-readable storage media, computer-readable instructions, data structures, applications, program modules, emails, and/or other content. Also, any of the processors described herein may include onboard memory in addition to or instead of the memory shown in the figures….)
a request to initiate an avatar development session that includes an exchange of communications with a participating human subject; (in [0037] FIG. 2 is a flowchart showing an example process 200 [a request to initiate an avatar development session] of high-level functions performed by the avatar-based service 110 and/or the avatar application 116. The process 200 may be divided into five phases, an initial phase to create a personalized avatar [a request to initiate an avatar development session that includes an exchange of communications with a participating human subject] comprising an animated representation of an individual 202, a second phase to create and train an animated model 204 [an exchange of communications with a participating human subject], a third phase to provide animated synthesis based on speech input and the animated model 206 [an exchange of communications with a participating human subject], … [0038] The first phase is to create a personalized avatar comprising an animated representation of an individual 202 [a request to initiate an avatar development session that includes an exchange of communications with a participating human subject]… [0049] The avatar application 116 receives speech and motion data to create animated models 400 [a request to initiate an avatar development session that includes an exchange of communications with a participating human subject]…. Each of the different upper body parts may be modeled using same [an exchange of communications with a participating human subject] or different observation data.)
receiving, at the server during the avatar development session, video data depicting the human subject, the video data captured by a camera of a first client device; (in [0019] This disclosure describes an architecture and techniques for providing an expressive avatar for various applications. For instance, the techniques described below may allow a user to represent himself or herself as an avatar in some applications, such as chat applications, game applications, social network applications, and the like. Furthermore, the techniques may enable the avatar to express a range of emotional states with realistic facial expressions, lip synchronization, and head movements to communicate in a more interactive manner with another user... Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user, which may result in a more fruitful and interactive communication…. )
processing, by the server during the avatar development session, the video data to determine a set of human characteristics representative of the human subject, the set of human characteristics including physical appearance characteristics, voice characteristics, and personality characteristics; (Xu in 0050-0054: …For example, the process may apply and track about 60 or more facial markers to capture facial features when expressing facial expressions. Multiple cameras [the video data captured by a camera of a first client device] may record the movement to a computer [claimed receiving, at a server hosting a rendering engine, video data depicting a human subject]. The performance capture [claimed processing, by the server during the avatar development session, the video data to determine a set of human characteristics representative of the human subject, the set of human characteristics including physical appearance characteristics, voice characteristics, and personality characteristics]may use a higher resolution to detect and to track subtle facial expressions [claimed the set of human characteristics including physical appearance characteristics,], such as small movements of the eyes and lips [claimed voice characteristics]… The markers may be placed on each side of the shoulder and in the back. Implementations of the data include using a live video feed or a recorded video stored in the database 118… The avatar application 116 processes the speech [voice characteristics] and observations to identify the relationships between the speech, facial expressions [claimed the set of human characteristics including physical appearance characteristics,], head and shoulder movements [claimed the set of human characteristics including physical appearance characteristics]. The avatar application 116 uses the relationships to create one or more animated models for the different upper body parts… In an implementation, the one or more animated models learn and train from the observations [processing, by the server during the avatar development session, the video data to determine a set of human characteristics representative of the human subject, the set of human characteristics including physical appearance characteristics, voice characteristics, and personality characteristics] of the speech and motion data to generate probabilistic motions of the upper body parts… Returning to FIG. 4, at 402, the avatar application 116 extracts features based on speech signals of the data [claimed voice characteristics]. The avatar application 116 extracts segmented speech phoneme and prosody features from the data. The speech phoneme is further segmented into some or all of the following: individual phones, diphones, half-phones, syllables, morphemes, words, phrases, and sentences to determine speech characteristics.
Fig. 1 and Fig. 9 depicts the server system for performing claimed functions, in 0075-0076: FIG. 9 is a block diagram showing an example server usable with the environment of FIG. 1. The server 112 [by the server during the avatar development session] may be configured as any suitable system capable of services, which includes, but is not limited to, implementing the avatarbased service [by the server during the avatar development session] 110 for online services, such as providing avatars in instant-messaging programs. In one example configuration, the server 114 comprises at least one processor 900, a memory 902, and a communication connection(s) 904… Turning to the contents of the memory 902 in more detail, the memory 902 may store an operating system 906, and the avatar application 116…)
maintaining, in memory of the server during the presentation session, avatar development session, avatar data including a simulated human memory, the avatar data being based on the set of human characteristics, the simulated human memory comprising details of the exchange of communications with the human subject; (Xu teaches simulated memory as trained models for creating animated models, in 0053: The avatar application 116 [claimed maintaining, in memory of the server during the avatar development session, avatar data including a simulated human memory, the avatar data being based on the set of human characteristics, the simulated human memory comprising details of the exchange of communications with the human subject] processes the speech and observations to identify the relationships between the speech, facial expressions, head and shoulder movements [claimed … the avatar data being based on the set of human characteristics, the simulated human memory comprising details of the exchange of communications with the human subject…]... The animated model may perform similar to a probabilistic trainable model, such as Hidden Markov Models (HMM) or Artificial Neural Networks (ANN). For example, HMMs are often used for modeling as training is automatic and the HMMs are simple and computationally feasible to use. In an implementation, the one or more animated models [claimed maintaining, in memory of the server during the avatar development session, avatar data including a simulated human memory, …] learn and train from the observations of the speech and motion data [claimed the avatar data being based on the set of human characteristics, the simulated human memory comprising details of the exchange of communications with the human subject] to generate probabilistic motions of the upper body parts.; Additionally, capturing observations for generating personalized avatar models, in 0043-0053: FIG. 3 is a flowchart showing an illustrative process of creating a personalized avatar comprising an animated representation of an individual 202 [claimed the simulated human memory comprising details of a previous interaction with an avatar, the avatar being a computer- generated autonomous representation of the human subject based on the avatar data] ( discussed at a high level above)… At 300, the avatar application 116 receives a frontal view image of the user 102 as viewed on the computing device 106. Images for the frontal view may start from a top of a head down to a shoulder in some instances, while in other instances these images may include an entire view of a user from head to toe. The images may be photographs or taken from sequences of video, and in color or in black or white. In some instances, the applications for the avatar 104 focus primarily on movements of upper body parts, from the top of the head down to the shoulder. Some possible applications with the upper body parts are to use the personalized avatar 104 as a virtual news anchor, a virtual assistant, a virtual weather person, and as icons in services or programs. Other applications may focus on a larger or different size of avatar, such as a head-to-toe version of the created avatar… The personalized avatar represents dimensions of the user's features as close as possible without any enlargement of any feature. In an implementation, the avatar application 116 may The avatar application 116 receives speech and motion data [claimed maintaining, in memory of the server during the avatar development session, avatar data including a simulated human memory,] to create animated models 400. The speech and motion data may be collected using motion capture and/or performance capture, which records movement of the upper body parts and translates the movement onto the animated models [claimed maintaining, in memory of the server during the avatar development session, avatar data including a simulated human memory, the avatar data being based on the set of human characteristics, the simulated human memory comprising details of the exchange of communications with the human subject]. The upper body parts include but are not limited to one or more of overall face, a chin, a mouth, a tongue, a lip, a nose, eyes, eyebrows, a forehead, cheeks, a head, and a shoulder. Each of the different upper body parts may be modeled using same or different observation data…)
terminating the avatar development session; based on the human subject being unavailable to communicate, initiating a presentation session between an avatar and a user different from the human subject, the avatar being a computer-generated autonomous representation of the human subject based on the avatar data; (in [0019] This disclosure describes an architecture and techniques for providing an expressive avatar for various applications. [terminating the avatar development session; based on the human subject being unavailable to communicate, initiating a presentation session between an avatar and a user different from the human subject, the avatar being a computer-generated autonomous representation of the human subject based on the avatar data] For instance, the techniques described below may allow a user to represent himself or herself as an avatar [the avatar being a computer-generated autonomous representation of the human subject based on the avatar data] in some applications, such as chat applications, game applications, social network applications, and the like [terminating the avatar development session; based on the human subject being unavailable to communicate, initiating a presentation session between an avatar and a user different from the human subject the application same that allows the deployed avatar to interact autonomously with another user]…. Furthermore, the techniques may enable the avatar [based on the human subject being unavailable to communicate in person but as a virtual avatar] to express a range of emotional states with realistic facial expressions, lip synchronization, and head movements to communicate in a more interactive manner with another user [terminating the avatar development session; based on the human subject being unavailable to communicate, initiating a presentation session between an avatar and a user different from the human subject the application same that allows the deployed avatar to interact autonomously with another user]… [0024] A variety of applications may use the expressive avatar. The expressive avatar may be referred to as a digital avatar [the avatar being a computer-generated autonomous representation of the human subject based on the avatar data;], a cartoon character, or a computer-generated character that exhibits human characteristics. The various applications using the avatar include but are not limited to, instant-messaging programs, social networks, video or online games, cartoons, television programs, movies, videos, virtual worlds, and the like…)
receiving, at the server during the presentation session, a request from the user different from the human subject to present the avatar at a second client device; (in [0019] This disclosure describes an architecture and techniques for providing an expressive avatar for various applications. For instance, the techniques described below may allow a user to represent himself or herself as an avatar in some applications, such as chat applications, game applications, social network applications, and the like. Furthermore, the techniques may enable the avatar to express a range of emotional states with realistic facial expressions, lip synchronization, and head movements to communicate in a more interactive manner with another user… By animating the avatar in this manner, the other user that views [receiving, at the server during the presentation session, a request from the user different from the human subject to present the avatar at a second client device as a request to view and interact with the animation of the avatar of the human subject] the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user, which may result in a more fruitful and interactive communication. [0024] A variety of applications may use the expressive avatar. The expressive avatar may be referred to as a digital avatar, a cartoon character, or a computer-generated character that exhibits human characteristics. The various applications using the avatar [receiving, at the server during the presentation session, a request from the user different from the human subject to present the avatar at a second client device] include but are not limited to, instant-messaging programs, social networks, video or online games, cartoons, television programs, movies, videos, virtual worlds, and the like… And in Xu teaches in 0035: … For discussion purposes, the processes are described with reference to the computing environment 100 shown in FIG. 1. However, the processes may be performed using different environments and devices. Moreover, the environments and devices [including claimed a second client device] described herein may be used to perform different processes. And including processes to render the avatar as claimed request to present in 0042:…)
processing, by the rendering engine of the server and in response to the request, animation data corresponding to the avatar data to generate a first set of rendered frames; (Xu teaches in 0035: … For discussion purposes, the processes are described with reference to the computing environment 100 shown in FIG. 1. However, the processes may be performed using different environments and devices. Moreover, the environments and devices [including claimed a second client device] described herein may be used to perform different processes. And including processes to render the avatar as claimed request to present in 0042: … This phase includes combining the personalized avatar generated 202 with the mapping of a number of points (e.g., about 92 points, etc.) to the face to generate a 2D cartoon avatar [processing, by the rendering engine of the server and in response to the request, animation data corresponding to the avatar data to generate a first set of rendered frames]. The 2D cartoon avatar is a low resolution, which allows rendering [by the rendering engine of the server] of this avatar to occur on many computing devices...)
transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar to a user different from the human subject at a first-time in a display of the second client device; (Xu teaches in 0019: This disclosure describes an architecture and techniques for providing an expressive avatar for various applications [transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar … at a first-time in a display of the second client device]. For instance, the techniques described below may allow a user to represent himself or herself as an avatar in some applications, such as chat applications, game applications, social network applications, and the like… For example, the user, through the avatar, may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking. By animating the avatar [the first set of rendered frames for presenting the avatar … at a first-time in a display of the second client device] in this manner, the other user [a user different from the human subject at a first-time in a display of the second client device] that views the avatar [transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar to a user different from the human subject at a first-time in a display of the second client device] is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [presenting the avatar to a user different from the human subject at a first-time in a display of the second client device], which may result in a more fruitful and interactive communication.
monitoring, by the second client device during the presentation session, an interaction between the avatar and the user different from the human subject, the interaction comprising the avatar responding autonomously, based on the avatar data, to input from the user different from the human subject; (Xu teaches in 0019: … For example, the user, through the avatar [the interaction comprising the avatar responding autonomously, based on the avatar data] may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking. By animating the avatar in this manner, the other user [the user different from the human subject] that views the avatar is more likely to respond accordingly based on the avatar's visual appearance [monitoring, by the second client device during the presentation session, an interaction between the avatar and the user different from the human subject, …]. Stated otherwise, the expressive avatar may be able to represent the user's mood [the avatar] to the other user [a user different from the human subject], which may result in a more fruitful and interactive communication [monitoring, by the second client device during the presentation session, an interaction between the avatar and the user different from the human subject, the interaction comprising the avatar responding autonomously, based on the avatar data, to input from the user different from the human subject.)
transmitting, from the second client device to the server during the presentation session, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject; (Xu teaches learning as claimed updating of the stimulated memory, in 0053: …In an implementation, the one or more animated models learn and train [claimed transmitting, from the second client device to the server during the presentation session, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject] from the observations [claimed with details related to the interaction to establish an updated simulated human memory related to the user] of the speech and motion data to generate probabilistic motions of the upper body parts; And observation of the user including the inputted observations of the avatar interacting with user other than human subject in in 0019: … For example, the user, through the avatar [transmitting, from the second client device to the server during the presentation session, details related to the interaction] may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking. By animating the avatar in this manner, the other user [to establish an updated simulated human memory related to the user different from the human subject] that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [details related to the interaction to establish an updated simulated human memory related to the user different from the human subject], which may result in a more fruitful and interactive communication [transmitting, from the second client device to the server, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject].) And in 0022-0024: The avatar application receives real-time speech input and synthesizes an animated sequence of motion of the upper body parts by applying the animated model [transmitting, from the second client device to the server during the presentation session, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject]… The various applications using the avatar include but are not limited to, instant-messaging programs, social networks, video or online games, cartoons, television programs, movies, videos, virtual worlds, and the like. For example, an instant-messaging program displays an avatar representative of a user in a small window. Through text-to-speech technology, the avatar speaks the text as the user types the text being used at a chat window. In particular, the user is able to share their mood, temperament, or disposition with the other user [transmitting, from the second client device to the server during the presentation session, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject], by having the avatar exhibit facial expressions synchronized with head/shoulder movements representative of the emotional state of the user…. And in claimed simulated memory in 0056: At 406, the avatar application 116 trains the one or more animated models by using the extracted features from the speech 402, motion trajectories transformed from the motion data 404, and speech and motion data 400. The avatar application 116 trains the animated models using the extracted features, such as sentences, phrases, words, phonemes, and transformed motion trajectories on a new coordinate motion. In particular, the animated model may generate a set of motion trajectories, referred to as probabilistic motion sequences of the upper body parts based on the extracted features of the speech...
and transmitting, by the server to the second client device during the presentation session, a second set of rendered frames for presenting the avatar at memory. (Xu teaches providing the avatar in real-time, in 0077-0079: The avatar application 116 provides access to avatar-based service 110. It receives real-time speech input. The avatar application 116 further provides a display of the application on the user interface [and transmitting, by the server to the second client device during the presentation session, a second set of rendered frames for presenting the avatar at a second-time in the display of the second client device,… ], and interacts with the other modules to provide the real-time animation of the avatar in 2D… The training model module 908 receives the speech and motion data, builds, and trains the animated model. The training model module 908 computes relationships between speech and upper body parts motion by constructing the one or more animated models [the updated simulated human memory] for the different upper body parts…. The synthesis module 910 synthesizes an animated sequence of motion of upper body parts by applying the animated model in response [the updated simulated human memory] to the real-time speech input… The synthesis module 910 provides an output of speech corresponding to the real-time speech input [claimed the second set of rendered frames comprising a sequence of at least one phoneme and at least one viseme that substantially convey a reference to the monitored interaction between the avatar and the user different from the human subject based on the updated simulated human memory], and constructs a real-time animation based on the output of speech synchronized to the animation sequence of motions of the one or more upper body parts [output speech synchronized to animation as claimed a sequence of at least one phoneme… that substantially convey a reference to the monitored interaction between the avatar and the user different from the human subject based on the updated simulated human memory].; And the animation model for generating an user avatar to interact with another user, in 0019: … For instance, the techniques described below may allow a user to represent himself or herself as an avatar in some applications [and transmitting, by the server to the second client device during the presentation session, a second set of rendered frames for presenting the avatar atframes comprising a sequence of at least one phoneme and at least the interaction over time using an user application], such as chat applications, game applications, social network applications, and the like. Furthermore, the techniques may enable the avatar to express a range of emotional states with realistic facial expressions, lip synchronization [at least one viseme as data capturing position and movement of the mouth and face when speaking], and head movements to communicate in a more interactive manner with another user. In some instances, the expressed emotional states may correspond to emotional states being expressed by the user. For example, the user, through the avatar, may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking. By animating the avatar in this manner, the other user that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [the user different from the human subject], which may result in a more fruitful and interactive communication [the monitored interaction between the avatar and the user different from the human subject based on the updated simulated human memory]...; in [0054] Returning to FIG. 4, at 402, the avatar application 116 extracts features based on speech signals of the data. The avatar application 116 extracts segmented speech phoneme and prosody features from the data [the second set of rendered frames comprising a sequence of at least one phoneme and at least one viseme]. The speech phoneme is further segmented into some or all of the following: individual phones, diphones, half-phones, syllables, morphemes, words, phrases, and sentences to determine speech characteristics… The avatar application 116 uses the extracted features of speech to provide probabilistic motions of the upper body parts.)
Xu teaches the avatar that is displayed, during claimed deployment session, using an user interface device to process speech inputs in real-time in a first and second time as noted above. One of ordinary skill in the art would understand that user interact with avatar in real-time using speech input in a sequence in close a first and second event interaction time (e.g. at a first and second time), where the human subject is unable to communicate in person but interacts virtually (e.g. via an avatar animation). Xu does not expressly disclose user interaction with avatar in real-time using speech input in a sequence including a first and second time period of an interaction sequence for presenting an avatar as claimed in transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar to a user different from the human subject at a first-time in a display of the second client device,… transmitting, by the server to the second client device during the presentation session, a second set of rendered frames for presenting the avatar at a second-time in the display of the second client device,…
Kuhn expressly teaches user interaction with avatar in real-time using speech input in a sequence including a first and second time period of an interaction sequence for presenting an avatar as claimed in transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar to a user different from the human subject at a first-time in a display of the second client device,… transmitting, by the server to the second client device during the presentation session, a second set of rendered frames for presenting the avatar at a second-time in the display of the second client device,… in 3:11-41: Rather than presenting the exemplary virtual game dealers 106 described herein as electronic game tables 100 that display or project a virtual dealer 106 [the first set of rendered frames for presenting the avatar], the electronic game table 100, the underlying artificial intelligence 102, and the projected virtual dealer 106 [transmitting, by the server to the second client device during the presentation session, …] will be referred to collectively herein 15 as "the virtual dealer" 106. A relatively elementary implementation of the virtual dealer 106 may engage in appropriately intelligent dialogue with human players [a user different from the human subject at a first-time in a display of the second client device] while a more sophisticated version of the virtual dealer 106 constitutes an emulated human agent that has learning capability and displays a full range of emulated human reactions, gestures, speech, etc. In one implementation, the virtual dealer 106 uses custom greetings [transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar to a user different from the human subject at a first-time in a display of the second client device,], e.g., using surnames or nicknames, cites sport fan affiliations, and may use geographical monikers and comments specific to a player's region or country. Such speech may be a simple as "Hello Joe" to "It's probably raining in Seattle today, Jane." The virtual dealer 106 may make personal references to a birthday or other events, such as custom celebrations. The virtual dealer 106 may make game-oriented comments, such as "nice hit, Dave", "great bet, Johmiie" to consoling comments, such as "better luck next time, Peggy". The virtual dealer 106 can also engage in random dialogue from the current game session [transmitting, by the server to the second client device during the presentation session, a second set of rendered frames for presenting the avatar at a second-time in the display of the second client device], from previous sessions of the same game [transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar to a user different from the human subject at a first-time in a display of the second client device], or previous sessions from different locations: "hope you're as lucky here 35 as you were at the MGM." In one implementation, the virtual dealer 106 can add visual emotional affectation and associated gestures to the verbal speech. The virtual dealer 106 can also provide additional entertainment functions, such as performing chip tricks, acting as bartender, and telling stories, 40 etc. And in 4:2-5: … For example, the AI engine 102 may store a record of a player's verbal input or a player's user interface input as data to consider when interacting with that player in the future. Thus, the virtual dealer's interactions with the player may become more sophisticated over time [transmitting, by the server to the second client device during the presentation session, the first set of rendered frames for presenting the avatar at a first-time in a display of the second client device,… transmitting, by the server to the second client device during the presentation session, a second set of rendered frames for presenting the avatar at a second-time in the display of the second client device].)
While Xu teaches applying the trained artificial intelligence model for simulating the avatar memory and interacting with other users other than the avatar. Examiner interprets applying the avatar based on an artificial intelligence model (e.g. animated model) to improve communication interactions with another human user as captured exchange data that is also within the scope of the claim limitations: transmitting, by the server to the second client device during the presentation session, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject; and transmitting, by the server to the second client device during the presentation session, … convey a reference to the monitored interaction between the avatar and the user different from the human subject based on the updated simulated human memory.
Additionally, Kuhn teaches applying the avatar based on an artificial intelligence model to improve communication interactions with another human user as within the scope of the claim limitations: transmitting, from the second client device to the server during the presentation session, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject; and transmitting, by the server to the second client device during the presentation session, … convey a reference to the monitored interaction between the avatar and the user different from the human subject based on the updated simulated human memory (in 5:34-56: The AI engine 102 uses artificial intelligence to associate [transmitting, from the second client device to the server during the presentation session, details related to the interaction to establish an updated simulated human memory related to the user different from the human subject] game states and player [the user different from the human subject] attributes with possible virtual dealer 35 gestures 504 and speech content 506. Associations can be saved as knowledge 408, enabling the AI engine 102 to learn [transmitting, from the second client device to the server details related to the interaction to establish an updated simulated human memory] and to create more sophisticated interactions with a given player [details related to the interaction to establish an updated simulated human memory related to the user different from the human subject] over time. The output of the AI engine 102 is a signal representing 40 intelligent reactions 406 [transmitting, by the server to the second client device during the presentation session] for the virtual dealer 106, to be projected by the virtual dealer projector (projection engine) 636. The virtual dealer projector 636 may break the task of generating the virtual dealer 106 into several components. Thus, the virtual dealer projector 636 may have an emotion engine 638, a speech engine 640, a gesture engine 642, a character engine 644, and an accessories engine 646. The virtual dealer projector 636 also includes a video generator 518 and an audio generator 520. The video generator 518 may include a video synthesizer 648 to animate a video dealer image or avatar and/or may include stored video segments 650 or "clips" that can be executed to generate virtual dealer behavior…)
the simulated human memory comprising details of the exchange of communications with the human subject, in 3:57-4:6: The AI engine 102 considers current stimuli 402, such as identities and attributes of one or more players [the simulated human memory comprising details of the exchange of communications with the human subject], in light of a database of possible responses 404. Through various techniques of machine intelligence, to be described below, the AI engine 102 determines an intelligent reaction 406 to the current stimuli 402. The AI engine 102 may find associations between different manifestations of the current stimuli 402, and may learn associations made between stimuli 402 and intelligent reactions 406, which can then be stored as knowledge 408 for future decision-making [the simulated human memory comprising details of the exchange of communications with the human subject]. The knowledge 408 may also include a memory record of past stimuli 402, which becomes information to consider during future decision-making. For example, the AI engine 102 may store a record of a player's verbal input or a player's user interface input as data to consider when interacting with that player in the future [the simulated human memory comprising details of the exchange of communications with the human subject,]. Thus, the virtual dealer's interactions with the player may become more sophisticated over time
terminate the avatar development session; and establish, during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject, (in5:49-53: … The video generator 518 may include a video synthesizer 648 to animate a video dealer image or avatar [terminate the avatar development session] and/or may include stored video segments 650 or “clips” that can be executed to generate virtual dealer behavior [terminate the avatar development session]… 6:32-42: In one implementation, behavioral logic of the virtual dealer 106 is driven by the AI engine 102 [during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject the avatar’s AI engine drives the behavior autonomously without use of the human subject through its avatar], which selects a current emotion most likely to fit current circumstances, to be executed by the emotion engine 638. In one implementation, the virtual dealer establishes current circumstances by assessing the state of the game 608 [during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject]. When the game is going badly for one or more player [a user different from the human subject], overly cheerful emotions to be displayed by the virtual dealer 106 are moderated. A big win triggers energetic emotions for the virtual dealer 106 to display, i.e., facial affect and gestures associated with surprise and congratulations …)
The Xu and Kuhn references would have been recognized by those of ordinary skill in the art as useful for applicant’s purpose in developing information processing methods for computer-generated interactive virtual agent systems.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for generating an interactive avatar based on capture user characteristics with stimulated memory captured using machine learning techniques as disclosed by Xu with the method of information using machine learning techniques to building and training interactive avatar based on human characteristics/traits for dynamically interacting with other user through dialogue at different occasions as disclosed by Kuhn.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods of Xu and Kuhn in order using machine learning techniques in order to provide an improvement to enable player attribute learning system for supporting an artificial intelligence (Al) virtual dealer avatar (Kuhn, Abstract).
While Xu and Kuhn teaches rendering an avatar based on learned human characteristics/simulated memory for generating speech and expressions when communication interactively with user. Xu and Kuhn do not expressly teach the speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme that substantially convey a reference to the monitored interaction based on the updated simulated human memory.
Deng does expressly teach speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme that substantially convey a reference to the monitored interaction based on the updated simulated human memory. (Deng teaches in Pg. 15-19: Sec. 10: … In linguistics literature, speech co-articulation [speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme] is defined as follows: phonemes are not pronounced as an independent sequence of sounds, but rather that the sound of a particular phoneme is affected by adjacent phonemes. Visual speech co-articulation is analogous…. Previous research efforts in visual speech animation generation can be roughly classified into two different categories: viseme-driven approaches and data-driven approaches. Viseme-driven approaches require animators to design key mouth shapes for phonemes (termed as visemes) [speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme] in order to generate novel speech animations… Viseme is defined as a basic visual unit that corresponds to the phoneme in speech [speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme]. Viseme-driven approaches typically require animators to design visemes (key mouth shapes), and then empirical smooth functions [82, 83, 14, 84, 85, 86] or co-articulation rules [speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme that substantially convey a reference to the monitored interaction based on the updated simulated human memory] [87, 88, 89] are used to synthesize novel speech animations.).. Learning-based approaches model speech co-articulations as implicit functions in statistical models [speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme that substantially convey a reference to the monitored interaction based on the updated simulated human memory]… Blanz et al. [106] reanimate 2D faces in images and video by reconstructing 3D face model using the morphable face model framework [47] and learning an expression and viseme space from scanned 3D faces. This approach ad-dresses both speech and expressions. Deng et al. [91, 92] proposed an expressive speech animation system that learn speech co-articulation models and expression eigen-spaces from recorded facial motion capture data [speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme that substantially convey a reference to the monitored interaction based on the updated simulated human memory]…; Depicted in Fig. 1.8
PNG
media_image1.png
627
832
media_image1.png
Greyscale
Fig. 1.8. Sketched general pipeline of data-driven speech animation generation approaches. The sample-based approaches go with the blue path, and the learning based approaches go with the red path [speech in a set of rendered frames comprising a sequence of at least one phoneme and at least one viseme that substantially convey a reference to the monitored interaction based on the updated simulated human memory].
The Xu, Kuhn, and Deng references would have been recognized by those of ordinary skill in the art as useful for applicant’s purpose in developing information processing methods for computer-generated interactive virtual agent systems.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art disclosing the method of information using machine learning techniques to generate avatar speech animations using captured frames based on captured human characteristics/traits using motion recording device and learning-based approaches as disclosed by Deng with the teaches for generating an interactive avatar based on capture user characteristics with stimulated memory captured using machine learning techniques as collectively disclosed by Xu and Kuhn.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods of Xu, Kuhn, and Deng in order using implement learning-based approaches for modeling speech co-articulations for avatar speech animation tasks (Deng, Pg. 19 and Fig. 1.8 ); Doing so allows developing automated learning base approaches for facial modeling and animation that create realistic animations that operate in real-time (Deng, Pg. 1 Sec. Introduction and Pg. 19).
While the Xu, Kuhn and Deng references disclose the use of an autonomous avatar interacts with other users without requiring the human subject to communicating in person with another party. These references do not expressly disclose death as an event that would make the human subject unable to communicate in person and deploy their avatar to autonomously interact with another subject or other avatars digitally.
Additionally List teaches death as an events that would make the human subject unable to communicate in person and deploy their avatar to autonomously interact with another subject or other avatars as claimed establish[ing], during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject, (in [0054] Members may also build a LifePage for loved ones, for lost loved ones or people of interest. In other words, the use of an in depth profiling may be provided by family or loved ones to create a profile for a person deceased [during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject] or unable to provide a profile… And in [0131] System provides the ability to query and receive profile information and other information associated with a person. System provides ability to create customized database and interactive interface. System provides support for creating custom audio and video. As an example, a daughter may visit the model representing the deceased father [during a period in which the human subject is unavailable to communicate] for interacting with the father [a presentation session between an avatar and a user different from the human subject]. As a non-limiting example, daughter may receive advice from the model of the father [an avatar]. Furthermore, the daughter may graphically interact with the father's model via a number of applications…)
The List, Xu, Kuhn, and Deng references would have been recognized by those of ordinary skill in the art as useful for applicant’s purpose in developing information processing methods for computer-generated interactive virtual agent systems
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art disclosing the method of developing information retrieval and processing techniques for modeling in computer-enabled social system as disclosed by List with the teaches for generating an interactive avatar based on capture user characteristics with stimulated memory captured using machine learning techniques as collectively disclosed by Deng, Kuhn and Xu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods of List, Deng, Kuhn and Xu as noted above; Doing so allows developing autonomous avatar system that provides capability for viewing and interacting with information associated with a person's life experiences, (List, 0249).
Regarding claim 49, Xu in combination with Kuhn, Deng and List teaches:
A non-transitory computer-readable storage medium, the computer readable storage medium including instructions that when executed by a an avatar server hosting rendering engine, cause the avatar server to; (Xu teaches in 0075-0084: FIG. 9 is a block diagram showing an example server usable with the environment of FIG. 1… Also, any of the processors described herein may include onboard memory in addition to or instead of the memory shown in the figures. The memory may include storage media such as, but not limited to, random access memory (RAM), read only memory (ROM), flash memory, optical storage, magnetic disk storage or other magnetic storage devices, or any other medium… Various instructions, methods, techniques, applications, and modules described herein may be implemented as computer-executable instructions that are executable by one or more computers, servers, or telecommunication devices. Generally, program modules include routines, programs, objects, components, data structures, etc. for performing particular tasks or implementing particular abstract data types…)
establish an avatar development session including an exchange of communications with a participating human subject, wherein the avatar server is configured to: (in [0037] FIG. 2 is a flowchart showing an example process 200 [establish an avatar development session] of high-level functions performed by the avatar-based service 110 and/or the avatar application 116. The process 200 may be divided into five phases, an initial phase to create a personalized avatar [a establish an avatar development session including an exchange of communications with a participating human subject] comprising an animated representation of an individual 202, a second phase to create and train an animated model 204 [an exchange of communications with a participating human subject, wherein the avatar server is configured to], a third phase to provide animated synthesis based on speech input and the animated model 206 [an exchange of communications with a participating human subject, wherein the avatar server is configured to], … [0038] The first phase is to create a personalized avatar comprising an animated representation of an individual 202 [establish an avatar development session including an exchange of communications with a participating human subject]… [0049] The avatar application 116 receives speech and motion data to create animated models 400 [establish an avatar development session including an exchange of communications with a participating human subject]…. Each of the different upper body parts may be modeled using same [an exchange of communications with a participating human subject] or different observation data.)
and terminate the avatar development session; and establish, during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject, the avatar being a computer-generated autonomous representation of the human subject based on the avatar data, wherein the avatar server is configured to: (in [0019] This disclosure describes an architecture and techniques for providing an expressive avatar for various applications. [terminate the avatar development session; and establish, during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject, the avatar being a computer-generated autonomous representation of the human subject based on the avatar data, wherein the avatar server is configured to] For instance, the techniques described below may allow a user to represent himself or herself as an avatar [the avatar being a computer-generated autonomous representation of the human subject based on the avatar data, wherein the avatar server is configured to] in some applications, such as chat applications, game applications, social network applications, and the like [terminate the avatar development session; and establish, during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject, the application same that allows the deployed avatar to interact autonomously with another user]…. Furthermore, the techniques may enable the avatar [during a period in which the human subject is unavailable to communicate in person but as a virtual avatar] to express a range of emotional states with realistic facial expressions, lip synchronization, and head movements to communicate in a more interactive manner with another user [terminate the avatar development session; and establish, during a period in which the human subject is unavailable to communicate, a presentation session between an avatar and a user different from the human subject, the application same that allows the deployed avatar to interact autonomously with another user]… [0024] A variety of applications may use the expressive avatar. The expressive avatar may be referred to as a digital avatar [a computer-generated autonomous representation of the human subject based on the avatar data], a cartoon character, or a computer-generated character that exhibits human characteristics. The various applications using the avatar include but are not limited to, instant-messaging programs, social networks, video or online games, cartoons, television programs, movies, videos, virtual worlds, and the like…)
receive, from the second client device, details related to an interaction between the avatar and the user different from the human subject, the interaction comprising the avatar responding autonomously, based on the avatar data, to input from the user different from the human subject; (Processes may be performed using different devices including claimed second device, in 0035: … For discussion purposes, the processes are described with reference to the computing environment 100 shown in FIG. 1. However, the processes may be performed using different environments and devices. Moreover, the environments and devices [including claimed a second client device] described herein may be used to perform different processes. And including processes to render the avatar as claimed request to present in 0042: … This phase includes combining the personalized avatar generated 202 with the mapping of a number of points (e.g., about 92 points, etc.) to the face to generate a 2D cartoon avatar. The 2D cartoon avatar is a low resolution, which allows rendering of this avatar to occur on many computing devices [receive, from the second client device,]; And Xu teaches in 0019: … For example, the user, through the avatar [the human subject], may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking. By animating the avatar in this manner [the interaction comprising the avatar responding autonomously, based on the avatar data, to input from the user different from the human subject], the other user [receive, from the second client device, details related to an interaction between the avatar and the user different from the human subject,] that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [the user different from the human subject], which may result in a more fruitful and interactive communication [receive, from the second client device, details related to an interaction between the avatar and the user different from the human subject, the interaction comprising the avatar responding autonomously, based on the avatar data, to input from the user different from the human subject].)
update the simulated human memory with details related to the interaction to establish an updated simulated human memory related to the user; (Xu teaches learning as claimed updating of the stimulated memory, in 0053: …In an implementation, the one or more animated models learn and train [claimed update the simulated human memory with details related to the interaction to establish an updated simulated human memory related to the user] from the observations [claimed with details related to the interaction to establish an updated simulated human memory related to the user] of the speech and motion data to generate probabilistic motions of the upper body parts; And observation of the user including the inputted observations in 0056: …The animated model trains by observing and learning the extracted speech synchronized to the motion trajectories of the upper body parts. The avatar application 116 stores the trained animated models in the database 118 to be accessible upon receiving real-time speech input… & 00074: … At 808 is a neutral emotional state without expressing any emotions. At 810 and 812, the avatar may be in a happy mood with the facial expressions changing slightly and the lips opening wider. The avatar may display this happy emotional state in response to the application 116 detecting that the user's inputted text matches [claimed … details related to the interaction to establish an updated simulated human memory related to the user] a predetermined phrase associated with this "happy" emotional state. As such, when the user provides a "happy" input, the avatar correspondingly displays this happy emotional state; And in in 0053-0056: The avatar application 116 processes the speech and observations to identify the relationships between the speech, facial expressions, head and shoulder movements. The avatar application 116 uses the relationships to create one or more animated models for the different upper body parts… In an implementation, the one or more animated models learn [an updated simulated human memory related to the user] and train from the observations of the speech [details related to the interaction to establish an updated simulated human memory related to the user] and motion data [details related to the interaction to establish an updated simulated human memory related to the user] to generate probabilistic motions of the upper body parts … Thus, prosody may reflect various features of a speaker, based on the tone and inflection. In an implementation, the duration information extracted may be used to scale and synchronize motions modeled by the one or more animated models to the real-time speech input... The motion trajectories represent the upper body parts… The animated model trains by observing and learning [transmitting, from the second client device to the server, details related to the interaction to establish an updated simulated human memory related to the user] the extracted speech synchronized to the motion trajectories of the upper body parts. The avatar application 116 stores the trained animated models in the database 118 to be accessible upon receiving real-time speech input.)
The remaining claim limitations in claim 49 are similar to claim 41 limitations and are rejected under the same rationale.
Regarding claim 57, Xu in combination with Kuhn, Deng and List teaches:
A server comprising: one or more processors; and memory storing computer-executable instructions that, when executed by the one or more processors, cause the server to (Xu teaches in 0075-0084: FIG. 9 is a block diagram showing an example server usable with the environment of FIG. 1… Also, any of the processors described herein may include onboard memory in addition to or instead of the memory shown in the figures. The memory may include storage media such as, but not limited to, random access memory (RAM), read only memory (ROM), flash memory, optical storage, magnetic disk storage or other magnetic storage devices, or any other medium… Various instructions, methods, techniques, applications, and modules described herein may be implemented as computer-executable instructions that are executable by one or more computers, servers, or telecommunication devices. Generally, program modules include routines, programs, objects, components, data structures, etc. for performing particular tasks or implementing particular abstract data types…)
monitor an interaction between the avatar and a user different from the human subject, the interaction comprising the avatar delivering autonomous remarks, based on the avatar data, responsive to input from the user different from the human subject; (Xu teaches in 0019: … For example, the user, through the avatar [an interaction between the avatar and a user different from the human subject, the interaction comprising the avatar delivering autonomous remarks] may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking [the interaction comprising the avatar delivering autonomous remarks]. By animating the avatar in this manner, the other user [responsive to input from the user different from the human subject] that views the avatar is more likely to respond accordingly based on the avatar's visual appearance [monitor an interaction between the avatar and a user different from the human subject, … , based on the avatar data, …]. Stated otherwise, the expressive avatar may be able to represent the user's mood [the avatar] to the other user [the user different from the human subject], which may result in a more fruitful and interactive communication [monitor an interaction between the avatar and a user different from the human subject, the interaction comprising the avatar delivering autonomous remarks, based on the avatar data, responsive to input from the user different from the human subject.)
update the simulated human memory with details related to the interaction to establish an updated simulated human memory related to the user different from the human subject; (Xu teaches learning as claimed updating of the stimulated memory, in 0053: …In an implementation, the one or more animated models learn and train [claimed update the simulated human memory with details related to the interaction to establish an updated simulated human memory related to the user different from the human subject] from the observations [claimed details related to the interaction to establish an updated simulated human memory related to the user different from the human subject] of the speech and motion data to generate probabilistic motions of the upper body parts; And observation of the user including the inputted observations of the avatar interacting with user other than human subject in in 0019: … For example, the user, through the avatar may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking. By animating the avatar in this manner, the other user [to establish an updated simulated human memory related to the user different from the human subject] that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [the user different from the human subject], which may result in a more fruitful and interactive communication. And in 0022-0024: The avatar application receives real-time speech input and synthesizes an animated sequence of motion of the upper body parts by applying the animated model [details related to the interaction to establish an updated simulated human memory related to the user different from the human subject]… The various applications using the avatar include but are not limited to, instant-messaging programs, social networks, video or online games, cartoons, television programs, movies, videos, virtual worlds, and the like. For example, an instant-messaging program displays an avatar representative of a user in a small window. Through text-to-speech technology, the avatar speaks the text as the user types the text being used at a chat window. In particular, the user is able to share their mood, temperament, or disposition with the other user, by having the avatar exhibit facial expressions synchronized with head/shoulder movements representative of the emotional state of the user…. And in claimed simulated memory in 0056: At 406, the avatar application 116 trains the one or more animated models by using the extracted features from the speech 402, motion trajectories transformed from the motion data 404, and speech and motion data 400. The avatar application 116 trains the animated models using the extracted features, such as sentences, phrases, words, phonemes, and transformed motion trajectories on a new coordinate motion. In particular, the animated model may generate a set of motion trajectories, referred to as probabilistic motion sequences of the upper body parts based on the extracted features of the speech...
generate a second set of rendered frames based at least on the updated simulated human memory related to the user different from the human subject; and present the avatar at a second-time in the webpage, the presenting including the second set of rendered frames depicting an animation of the avatar referring to the monitored interaction. (Xu teaches providing the avatar in real-time, in 0077-0079: The avatar application 116 provides access to avatar-based service 110. It receives real-time speech input. The avatar application 116 further provides a display of the application on the user interface [generate a second set of rendered frames based at least on the updated simulated human memory related to the user different from the human subject; and present the avatar at a second-time in the webpage,], and interacts with the other modules to provide the real-time animation of the avatar in 2D… The training model module 908 receives the speech and motion data, builds, and trains the animated model. The training model module 908 computes relationships between speech and upper body parts motion by constructing the one or more animated models [the updated simulated human memory] for the different upper body parts…. The synthesis module 910 synthesizes an animated sequence of motion of upper body parts by applying the animated model in response [the updated simulated human memory] to the real-time speech input… The synthesis module 910 provides an output of speech corresponding to the real-time speech input [claimed the presenting including the second set of rendered frames depicting an animation of the avatar referring to the monitored interaction], and constructs a real-time animation based on the output of speech synchronized to the animation sequence of motions of the one or more upper body parts.; And the animation model for generating an user avatar to interact with another user, in 0019: … For instance, the techniques described below may allow a user to represent himself or herself as an avatar in some applications [the presenting including the second set of rendered frames depicting an animation of the avatar referring to the monitored interaction], such as chat applications, game applications, social network applications, and the like. Furthermore, the techniques may enable the avatar to express a range of emotional states with realistic facial expressions, lip synchronization, and head movements to communicate in a more interactive manner with another user. In some instances, the expressed emotional states may correspond to emotional states being expressed by the user. For example, the user, through the avatar, may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking. By animating the avatar in this manner, the other user that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [the user different from the human subject], which may result in a more fruitful and interactive communication [generate a second set of rendered frames based at least on the updated simulated human memory related to the user different from the human subject].)
Xu teaches the avatar that is displayed on a user interface device to process speech inputs in real-time in a first and second time as noted above. One of ordinary skill in the art would understand that user interact with avatar in real-time using speech input in a sequence in close a first and second event interaction time (e.g. at a first and second time). Xu does not expressly disclose user interaction with avatar in real-time using speech input in a sequence including a first and second time period of an interaction sequence for presenting an avatar as claimed in present, at a first time in a webpage for display at a user interface of the second client device …, the first set of rendered frames depicting an animation of the avatar … present the avatar at a second-time in the webpage, the presenting including the second set of rendered frames depicting an animation of the avatar referring to the monitored interaction,…
Kuhn expressly teaches user interaction with avatar in real-time using speech input in a sequence including a first and second time period of an interaction sequence for presenting an avatar as claimed in present, at a first time in a webpage for display at a user interface of the second client device …, the first set of rendered frames depicting an animation of the avatar … present the avatar at a second-time in the webpage, the presenting including the second set of rendered frames depicting an animation of the avatar referring to the monitored interaction,… in 3:11-41: Rather than presenting the exemplary virtual game dealers 106 described herein as electronic game tables 100 that display or project a virtual dealer 106 [the first set of rendered frames depicting an animation of the avatar], the electronic game table 100, the underlying artificial intelligence 102, and the projected virtual dealer 106 [present, at a first time in a webpage for display at a user interface of the second client device] will be referred to collectively herein 15 as "the virtual dealer" 106 [present, at a first time in a webpage for display at a user interface of the second client device and independent of contemporaneous input from the human subject ]. A relatively elementary implementation of the virtual dealer 106 may engage in appropriately intelligent dialogue with human players [a user different from the human subject,] while a more sophisticated version of the virtual dealer 106 constitutes an emulated human agent that has learning capability and displays a full range of emulated human reactions, gestures, speech, etc. In one implementation, the virtual dealer 106 uses custom greetings [present, at a first time in a webpage for display at a user interface of the second client device …, the first set of rendered frames depicting an animation of the avatar], e.g., using surnames or nicknames, cites sport fan affiliations, and may use geographical monikers and comments specific to a player's region or country. Such speech may be a simple as "Hello Joe" to "It's probably raining in Seattle today, Jane." The virtual dealer 106 may make personal references to a birthday or other events, such as custom celebrations. The virtual dealer 106 may make game-oriented comments, such as "nice hit, Dave", "great bet, Johmiie" to consoling comments, such as "better luck next time, Peggy". The virtual dealer 106 can also engage in random dialogue from the current game session [present the avatar at a second-time in the webpage, the presenting including the second set of rendered frames depicting an animation of the avatar referring to the monitored interaction], from previous sessions of the same game [present, at a first time in a webpage for display at a user interface of the second client device …, the first set of rendered frames depicting an animation of the avatar], or previous sessions from different locations: "hope you're as lucky here 35 as you were at the MGM." In one implementation, the virtual dealer 106 can add visual emotional affectation and associated gestures to the verbal speech. The virtual dealer 106 can also provide additional entertainment functions, such as performing chip tricks, acting as bartender, and telling stories, 40 etc. And in 4:2-5: … For example, the AI engine 102 may store a record of a player's verbal input or a player's user interface input as data to consider when interacting with that player in the future. Thus, the virtual dealer's interactions with the player may become more sophisticated over time [present, at a first time in a webpage for display at a user interface of the second client device …, the first set of rendered frames depicting an animation of the avatar … present the avatar at a second-time in the webpage, the presenting including the second set of rendered frames depicting an animation of the avatar referring to the monitored interaction].)
The Xu and Kuhn references would have been recognized by those of ordinary skill in the art as useful for applicant’s purpose in developing information processing methods for computer-generated interactive virtual agent systems.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for generating an interactive avatar based on capture user characteristics with stimulated memory captured using machine learning techniques as disclosed by Xu with the method of information using machine learning techniques to building and training interactive avatar based on human characteristics/traits for dynamically interacting with other user through dialogue at different occasions as disclosed by Kuhn.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods of Xu and Kuhn in order using machine learning techniques in order to provide an improvement to enable player attribute learning system for supporting an artificial intelligence (Al) virtual dealer avatar (Kuhn, Abstract).
The remaining claim limitations in claim 57 are similar to those examined claims 41 and 49 and are rejected under the same rationale.
Regarding claim 42, the rejection of claim 41 is incorporated and Xu in combination with Kuhn, Deng and List teaches the method of claim 41, in which the interaction between the avatar and the user different from the human subject is a voice or text exchange. (Xu teaches in 0032: In the illustrated example, the computing device 106 may render a user interface (UI) 114 on a display of the computing device 106. The UI 114 facilitates access to the avatar-based service 110 providing real-time networked communication systems. In one implementation, the UI 114 is a browser-based UI that presents a page received from an avatar application 116. For example, the user 102 employs the UI 114 when submitting text or speech input [claimed the interaction between the avatar … is a voice or text exchange] to an instant-messaging program while also displaying the avatar 104…; And in 0053: …In an implementation, the one or more animated models learn and train from the observations of the speech [the interaction between the avatar and the user different from the human subject is a voice or text exchange] and motion data to generate probabilistic motions of the upper body parts; And observation of the user including the inputted observations of the avatar interacting with user other than human subject [the interaction between the avatar and the user different from the human subject]…; And in 0019: … For example, the user, through the avatar [transmitting, from the second client device to the server, details related to the interaction] may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking [the interaction between the avatar and the user different from the human subject is a voice or text exchange]. By animating the avatar in this manner, the other user [the interaction between the avatar and the user different from the human subject] that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [the interaction between the avatar and the user different from the human subject], which may result in a more fruitful and interactive communication [the interaction between the avatar and the user different from the human subject].)
Additionally Kuhn teaches the interaction between the avatar and the user different from the human subject is a voice or text exchange (in 3:16-41: A relatively elementary implementation of the virtual dealer 106 [the avatar] may engage in appropriately intelligent dialogue [the interaction between the avatar and the user different from the human subject is a voice or text exchange] with human players [the user different from the human subject] while a more sophisticated version of the virtual dealer 106 constitutes an emulated human agent that has learning capability and displays a full range of emulated human reactions, gestures, speech, etc. In one implementation, the virtual dealer 106 uses custom greetings, e.g., using surnames or nicknames, … In one implementation, the virtual dealer 106 uses custom greetings, e.g., using surnames or nicknames, cites sport fan affiliations, and may use geographical monikers and comments specific to a player's region or country. Such speech may be a simple as "Hello Joe" to "It's probably raining in Seattle today, Jane." The virtual dealer 106 may make personal references to a birthday or other events, such as custom celebrations. The virtual dealer 106 may make game-oriented comments, such as "nice hit, Dave", "great bet, Johmiie" to consoling comments, such as "better luck next time, Peggy". The virtual dealer 106 can also engage in random dialogue [the interaction between the avatar and the user different from the human subject is a voice or text exchange] from the current game session, from previous sessions of the same game, or previous sessions from different locations: "hope you're as lucky here as you were at the MGM." In one implementation, the virtual dealer 106 can add visual emotional affectation and associated gestures to the verbal speech. The virtual dealer 106 can also provide additional entertainment functions, such as performing chip tricks, acting as bartender, and telling stories [the interaction between the avatar and the user different from the human subject is a voice or text exchange], etc.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Xu and Kuhn for the same reasons disclosed above.
Regarding claim 43, the rejection of claim 41 is incorporated and Xu in combination with Kuhn, Deng and List teaches the method of claim 41, further comprising adjusting the avatar data based on real world event data. (Xu teaches in 0001: An avatar is a representation of a person in a cartoon-like image or other type of character having human characteristics. Computer graphics present the avatar as two-dimensional icons or three-dimensional models, depending on an application scenario [claimed comprising adjusting the avatar data based on real world event data] or a computing device that provides an output. Computer graphics and animations create moving images of the avatar on a display of the computing device. Applications using avatars include social networks, instant-messaging programs, videos, games, and the like. In some applications, the avatars are animated by using a sequence of multiple images that are replayed repeatedly. In another example, such as instant-messaging programs, an avatar represents a user and speaks aloud as the user inputs text in a chat window [claimed comprising adjusting the avatar data based on real world event data].; And in 0019: …Furthermore, the techniques may enable the avatar to express a range of emotional states with realistic facial expressions, lip synchronization, and head movements to communicate in a more interactive manner with another user [claimed comprising adjusting the avatar data based on real world event data]. In some instances, the expressed emotional states may correspond to emotional states being expressed by the user. For example, the user, through the avatar, may express feelings of happiness while inputting text into an application, in response, the avatar's lips may turn up at the corners to show the mouth of the avatar smiling while speaking. By animating the avatar in this manner, the other user that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user, which may result in a more fruitful and interactive communication.)
Regarding claim 45, the rejection of claim 41 is incorporated and Xu in combination with Kuhn, Deng and List teaches the method of claim 41, further comprising adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject. (Xu teaches in 0059-0062: In an implementation, the emotional state to be expressed may be one of happiness [claimed comprising adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user]. The animated sequence of motion of the upper body parts may include exhibiting a facial expression of wide open eyes or raised eyebrows, lip movements turned up at the corners in a smiling manner, a head nodding or shaking in an up and down movement, and/or shoulders in an upright position to represent body motions of being happy. The one or more predetermined phrases may include "I graduated," "I am engaged," "I am pregnant," and "I got hired." The happy occasion phrases may be related to milestones of life in some instances. In another implementation, the emotional state that may also be expressed is sadness [claimed further comprising adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user]. The animated sequence of motion of the upper body parts may include exhibiting facial expressions of eyes looking down, lip movements turned down at the corners in a frown, nostrils flared, the head bowed down, and/or the shoulders in a slouch position, to represent body motions of sadness. One or more predetermined phrases may include "I lost my parent," "I am getting a divorce," "I am sick," and "I have cancer." The sad occasion phrases tend to be related to disappointments associated with death, illness, divorce, abuse, and the like... , the avatar application 116 or avatar-based service 110 receives real-time speech input 600. Real-time speech input indicates receiving the input to generate a real-time based animated synthesis for facial expressions, lip-synchronization, and head/shoulder movements..; And the use of the input to process the adjustments in the facial appearances, And in 0019: … For example, the user, through the avatar [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject] may express feelings of happiness while inputting text into an application, in response, the avatar's lips may tum up at the comers to show the mouth of the avatar smiling while speaking [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject]. By animating the avatar in this manner, the other user [the avatar was presented to the user different from the human subject] that views the avatar is more likely to respond accordingly based on the avatar's visual appearance. Stated otherwise, the expressive avatar may be able to represent the user's mood to the other user [the avatar was presented to the user different from the human subject], which may result in a more fruitful and interactive communication [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject].)
Additionally, Kuhn teaches the use that adjust base on a current condition or requester data that is after an initial time, as claimed time lapse, Kuhn teaches in col. 2 lines 36-51: This disclosure describes virtual game dealers that are based on artificial intelligence. In one implementation, as shown in FIG. 1, an electronic game table 100 keeps a real time inventory of game states and player attributes for one player or for multiple players [the user different from the human subject] at the electronic game table 100. The electronic game table 100 then applies artificial intelligence techniques, e.g., via an artificial intelligence (AI) engine 102 hosted by a computing device 104, to analyze and prioritize a complex of player-game circumstances and to respond appropriately. The electronic game table 100 [the avatar was presented to the user] interacts with the one or more players [the user different from the human subject] by projecting a virtual dealer 106 [the avatar was presented to the user] that responds to and interacts with players in a playerdealer relationship that aims to be socially appropriate and dynamically responsive in real time [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user]. The electronic game table platform of FIG. 1 is described in greater detail below with respect to FIGS. 10-12. The virtual dealer 106 (also referred to as a "virtual host")…; And in col. 3 lines 1-41: … Real time game flow and game state information is also fed to the AI engine 102. In another implementation, the AI engine 102 also relies on perceptual input, i.e., machine vision via cameras 110, and 5 natural language processing via input from microphones 112 to emulate authentically human behavior, including emotional reactions [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics]… The virtual dealer 106 can also engage in random dialogue from the current game session [… based on a time elapsed since the avatar was presented to the user], from previous sessions of the same game [….based on a time elapsed since the avatar was presented to the user], or previous sessions from different locations: "hope you're as lucky here as you were at the MGM." In one implementation, the virtual dealer 106 can add visual emotional affectation and associated gestures to the verbal speech. The virtual dealer 106 can also provide additional entertainment functions, such as performing chip tricks, acting as bartender, and telling stories, etc…
Kuhn teaches adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject (in 3:16-41: A relatively elementary implementation of the virtual dealer 106 [the avatar] may engage in appropriately intelligent dialogue with human players [the user different from the human subject] while a more sophisticated version of the virtual dealer 106 constitutes an emulated human agent that has learning capability and displays a full range of emulated human reactions, gestures, speech, etc. In one implementation, the virtual dealer 106 uses custom greetings [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject], e.g., using surnames or nicknames, … In one implementation, the virtual dealer 106 uses custom greetings, e.g., using surnames or nicknames, cites sport fan affiliations, and may use geographical monikers and comments specific to a player's region or country. Such speech may be a simple as "Hello Joe" to "It's probably raining in Seattle today, Jane." The virtual dealer 106 may make personal references to a birthday or other events, such as custom celebrations. The virtual dealer 106 may make game-oriented comments, such as "nice hit, Dave", "great bet, Johmiie" to consoling comments, such as "better luck next time, Peggy". The virtual dealer 106 can also engage in random dialogue [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject] from the current game session, from previous sessions of the same game, or previous sessions from different locations [adjusting at least one of the physical appearance characteristics, voice characteristics, and personality characteristics based on a time elapsed since the avatar was presented to the user different from the human subject]: "hope you're as lucky here as you were at the MGM." In one implementation, the virtual dealer 106 can add visual emotional affectation and associated gestures to the verbal speech. The virtual dealer 106 can also provide additional entertainment functions, such as performing chip tricks, acting as bartender, and telling stories, etc.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Xu and Kuhn for the same reasons disclosed above.
Regarding claim 46, the rejection of claim 41 is incorporated and Xu in combination with Kuhn, Deng and List teaches the method of claim 41, further comprising adjusting the physical appearance characteristics based on an image of the human subject. (Xu teaches adjusting emotion based on an image as depicted in Fig. 3, in 0043-0045: FIG. 3 is a flowchart showing an illustrative process of creating a personalized avatar comprising an animated representation of an individual 202 ( discussed at a high level above). At 300, the avatar application 116 receives a frontal view image of the user 102 as viewed on the computing device 106. Images for the frontal view may start from a top of a head down to a shoulder in some instances, while in other instances these images may include an entire view of a user from head to toe. The images may be photographs or taken from sequences of video [adjusting the physical appearance characteristics based on an image of the human subject], and in color or in black or white…. For example, the avatar application 116 receives a frontal view image of an individual having a large chin. The avatar application 116 may exaggerate [claimed adjusting the physical appearance characteristics based on an image of the human subject] the chin by depicting a large pointed chin based on doubling to tripling the dimensions of the chin…; Where the avatar is rendered based on characteristics of the human subject characteristics, in 0001: An avatar is a representation of a person in a cartoon-like image or other type of character having human characteristics [claimed adjusting the physical appearance characteristics based on an image of the human subject]. Computer graphics present the avatar as two-dimensional icons or three-dimensional models, depending on an application scenario or a computing device that provides an output…the avatars are animated by using a sequence of multiple images [claimed adjusting the physical appearance characteristics based on an image of the human subject] that are replayed repeatedly…)
Regarding claim 47, the rejection of claim 41 is incorporated and Xu in combination with Kuhn, Deng and List teaches the method of claim 41, further comprising dynamically adjusting the avatar based on user input. (Xu teaches in 0059-0062: In an implementation, the emotional state to be expressed may be one of happiness [claimed dynamically adjusting the avatar based on user input]. The animated sequence of motion of the upper body parts may include exhibiting a facial expression of wide open eyes or raised eyebrows, lip movements turned up at the corners in a smiling manner, a head nodding or shaking in an up and down movement, and/or shoulders in an upright position to represent body motions of being happy. The one or more predetermined phrases may include "I graduated," "I am engaged," "I am pregnant," and "I got hired." The happy occasion phrases may be related to milestones of life in some instances. In another implementation, the emotional state that may also be expressed is sadness [claimed dynamically adjusting the avatar based on user input]. The animated sequence of motion of the upper body parts may include exhibiting facial expressions of eyes looking down, lip movements turned down at the corners in a frown, nostrils flared, the head bowed down, and/or the shoulders in a slouch position, to represent body motions of sadness. One or more predetermined phrases may include "I lost my parent," "I am getting a divorce," "I am sick," and "I have cancer." The sad occasion phrases tend to be related to disappointments associated with death, illness, divorce, abuse, and the like... , the avatar application 116 or avatar-based service 110 receives real-time speech input 600. Real-time speech input indicates receiving the input to generate a real-time based animated synthesis for facial expressions, lip-synchronization, and head/shoulder movements..; And the use of the input to process the adjustments in the facial appearances, in 0053: The avatar application 116 processes the speech and observations [claimed dynamically adjusting the avatar based on user input] to identify the relationships between the speech, facial expressions, head and shoulder movements. The avatar application 116 uses the relationships to create one or more animated models for the different upper body parts…)
Additionally, Kuhn teaches the use that adjust base on a current condition or requester data that is after an initial time, as claimed time lapse, in 2:45-51: …The electronic game table 100 then applies artificial intelligence techniques, e.g., via an artificial intelligence (AI) engine 102 hosted by a computing device 104, to analyze and prioritize a complex of player-game circumstances and to respond appropriately. The electronic game table 100 interacts with the one or more players by projecting a virtual dealer 106 that responds to and interacts with players [user input] in a playerdealer relationship that aims to be socially appropriate and dynamically responsive in real time [claimed dynamically adjusting the avatar based on user input].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Xu and Kuhn for the same reasons disclosed above.
Regarding claims 50-51 and 53-55, the claim limitations are similar to limitations in claims 42-43 and 45-47 respectively and thus rejected under the same rationale.
Regarding claims 58-59, the claim limitations are similar to limitations in claims 42-43 respectively and thus rejected under the same rationale.
Claims 44, 52, and 60 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US Pub. No. 2012/0130717, hereinafter ‘Xu’) over Kuhn et al. (US Pat. No. 9,202,171, hereinafter ‘Kuhn’) in further view of Deng et al. (NPL: “Computer Facial Animation: A Survey”, hereinafter ‘Deng’) in further view of Listermann (US 20130018882, hereinafter ‘List’) and Hauk et al. (US Pub. No. 20030126068, hereinafter ‘Hauk’).
Regarding claim 44, the rejection of claim 43 is incorporated. While Xu in combination with Kuhn Deng and List teaches the use of machine learning models to generate avatar knowledge for performing actions. Xu, Kuhn, List and Deng do not expressly teach the model for performing the activities recited in claim 44 limitations.
Hauk does expressly claim 44, directed to the use of a model for performing the activities recited in claim 44 limitations, in which the real world event data includes trading prices of a tradable commodity. (Hauk teaches in 0005-0006: More recently, computer systems and networks have begun replacing the actual exchanges or trading floors, and now provide a live auction market for stocks and commodities that serves at present hundreds of active participants who monitor price changes in stocks and commodities on their computers and react by executing orders via their computer at their off-site location… The off-site trader is provided with the current bid and offer of a specific stock or commodity and the trading sessions' highest and lowest prices, as well as other information. Oftentimes the previous day's closing price is displayed as well. By observing the prices of different stocks and commodities as displayed on a computer monitor in a spreadsheet or the like, off-site traders [claimed in which the real world event data includes trading prices of a tradable commodity] can actively participate in the market activity via a remote computer or terminal from virtually anywhere... Graphical representations of real or imagined environments on a two dimensional monitor, such as that of a computer, is one way to simulate action. Virtual reality takes such simulation further. Virtual reality is the simulation of a real or imagined environment that can be experienced visually in the three dimensions of width, height, and depth and that may additionally provide an interactive experience visually in full real-time motion with sound and possibly with tangible, perceptible feedback… Virtual reality simulations may include graphical metaphors [claimed in which the real world event data includes trading prices of a tradable commodity] that represent a real person in a cyberspace system. These graphical metaphors are sometimes referred to as "avatars." Upon entering such a system, an off-site trader can be defined by or can choose from a number of avatars [claimed in which the real world event data includes trading prices of a tradable commodity]. Sophisticated 3D avatars even change shape depending on what they are doing (e.g., walking, sitting, etc.).)
The Xu, Kuhn, Deng, List and Hauk references would have been recognized by those of ordinary skill in the art as useful for applicant’s purpose in developing information processing methods for computer-generated interactive virtual agent systems.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for generating an interactive avatar based on capture user characteristics with stimulated memory captured for performing real life actions in a virtual environment as disclosed by Hauk with the method of information using machine learning techniques to building and training interactive computer-generated model rendered as avatars based on human characteristics/traits as collectively disclosed by List, Deng, Kuhn, and Xu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods of Xu, Kuhn, Deng, List and Hauk in order enable virtual trading floor for simulating and buying and selling in a financial market using an avatar trader (Hauk, Abstract & 0005-0006); Doing so allows for computer generated representations of a person as an avatar interacting in a virtual environment as an offsite trader (Hauk, 0005-0006).
Regarding claim 52, the rejection of claim 51 is incorporated; and the limitations are similar to limitations in claim 44 and thus rejected under the same rationale.
Regarding claim 60, the rejection of claim 57 is incorporated; and the limitations are similar to limitations in claim 44 and thus rejected under the same rationale.
Claims 48 and 56 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (US Pub. No. 2012/0130717, hereinafter ‘Xu’) over Kuhn et al. (US Pat. No. 9,202,171, hereinafter ‘Kuhn’) in further view of Deng et al. (NPL: “Computer Facial Animation: A Survey”, hereinafter ‘Deng’) in further view of Listermann (US 20130018882, hereinafter ‘List’) and Wood (MPL: “Skill Acquisition Through Program-Level Imitation in a Real-Time Domain”).
Regarding claim 48, the rejection of claim 41 is incorporated and Xu in combination with Kuhn Deng and List teaches the method of claim 41, further comprising dynamically adjusting the avatar based on a … Markov decision process. (Xu teaches in 0053: The avatar application 116 processes the speech and observations to identify the relationships between the speech, facial expressions, head and shoulder movements. The avatar application 116 uses the relationships to create one or more animated models for the different upper body parts. The animated model may perform similar to a probabilistic trainable model, such as Hidden Markov Models (HMM) [claimed dynamically adjusting the avatar based on a … Markov decision process] or Artificial Neural Networks (ANN). For example, HMMs [claimed dynamically adjusting the avatar based on a … Markov decision process] are often used for modeling as training is automatic and the HMMs [claimed dynamically adjusting the avatar based on a … Markov decision process]are simple and computationally feasible to use. In an implementation, the one or more animated models learn and train from the observations of the speech and motion data to generate probabilistic motions of the upper body parts.)
While Xu, Kuhn, Deng and List teaches the use of machine learning techniques and models to acquire knowledge from observations to build autonomous agents such as avatars. Xu, Kuhn, Deng and List do not expressly teach the use of a partially observable Markov decision process.
Wood teaches expressly teach the use of a partially observable Markov decision process. (Wood in pg. 283: Sec.B & C: We describe the process of imitation learning using two agents: the expert demonstrates the completion of some task, and the imitator learns while observing the expert acting, then attempts to act in kind … The human’s actions (which guide the expert) are forcibly mapped via the bot controller to an avatar, while COIL’s action repertoire is restricted by and to the methods available in the bot controller. … The only true prior knowledge built into COIL is that the expert is to be imitated, no matter how intrinsically “rewarding” that may or may not be. This all assumes the environment is fully observable, and if this is not the case then we venture into the realm of partially observable Markov decision processes (POMDPs) [claimed dynamically adjusting the avatar based on a partially observable Markov decision process] [54], and the complexity rises by an order of magnitude. Conversely, we have shown COIL to operate in real-time in a partially observable9 environment.)
The Xu, Kuhn, Deng, List and Wood references would have been recognized by those of ordinary skill in the art as useful for applicant’s purpose in developing information processing methods for computer-generated interactive virtual agent systems.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for generating an interactive avatar based on capture user characteristics with stimulated memory captured for performing real life actions in a virtual environment as disclosed by Wood with the method of information using machine learning techniques to building and training interactive computer-generated model rendered as avatars based on human characteristics/traits as collectively disclosed by Xu, Kuhn, Deng and List.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods of Xu, Kuhn, Deng, List and Hauk in order enable virtual trading floor for simulating and buying and selling in a financial market using an avatar trader (Hauk, Abstract & 0005-0006); Doing so allows for computer generated representations of a person as an avatar interacting in a virtual environment as an offsite trader (Hauk, 0005-0006).
Regarding claim 56, the rejection of claim 49 is incorporated; and the limitations are similar to limitations in claim 48 and thus rejected under the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure, listed below:
Kalaboukis et al. (US 20100103075): teaches in [0117] 6. Example application--adding life caching: Virtual entities representative of absent or dead relatives can be added to the field of view of a user of a wearable reality overlay device. Artificial intelligence (AI) technologies can be used to simulate the actions of such virtual persons as viewed in a wearable device if such persons can be "life cached." A user may be enabled to have a conversation with a virtual representation of a dead, famous, absent or other person, based on their cached life.
Ranathunga et al. (NPL: Identifying events taking place in Second Life virtual environments): teaches multipurpose online virtual worlds can be considered as a medium that facilitates the next generation of human social interactions… For example, many universities are considering the possibility of conducting virtual lectures in their virtual universities in Second Life. Some universities and private institutions are using Second Life simulations to train people for tasks for which real-life training could be too expensive or dangerous… The co-existence of individuals (known as ‘‘avatars’’) and organizations has created virtual communities inside Second Life, … Apart from these general properties of entities, avatars and objects can have properties specific to them, depending on the virtual world or the selected simulation. For example, an avatar may have dynamic properties such as animation (denoting the animation it is currently playing), and if in a gaming simulation, health (denoting the current health level). As for objects, they can have a size property, and an owner to whom the object belongs…
Hodge et al. (US 20090299932): teaches typically, the avatar is an object representing an embodiment of a user. In Yahoo! Messenger, for example, users may create avatars having a unique human appearance to provide a graphical representation of the user to other users.
Crutchfield, Jr. et al. (US 9117316): teaches a computing system may be configured to apply the science of identity theory to simulated entities (e.g. avatars) in a simulated environment that accurately imitates human identity goal seeking behavior. The computing system may include a behavior library database and configured to select behavior from the behavior library database to create an automated entity such as the avatar that emulates a selected entity behavior such as human behavior as it engages in interaction with other entities, including humans.
Molyneaux et al. (US 20130342527): teaches an avatar is a posable, virtual representation of a subject--e.g., a human subject. At the present time, avatars are used commonly in video games to represent players, but many other applications are possible as well.
Kastanis et al. (NPL: “Reinforcement learning utilizes proxemics: An avatar learns to manipulate the position of people in immersive virtual reality”): teaches techniques for implementing a machine learning (ML) agent represented by a virtual character (or avatar) in an IVE can learn to manipulate the behavior of people so that they perform a task that they had not previously been instructed to carry out.
Williams et al. (MPL: “Partially observable Markov decision processes for spoken dialog systems”): teaches the use of partially observable Markov decision processes for spoken dialog systems.
Gaggioli et al. (NPL: “Avatars in clinical psychology: A framework for the clinical use of virtual humans”): teaches capturing human behaviors for generating avatar activity in virtual reality applications. For the modeling of behaviors, the goal is to build intelligent autonomous virtual humans with adaptation, perception and memory.
Lee et al. (NPL: “Precomputing Avatar Behavior From Human Motion Data”) teaches a pre-computation method that allows avatars to be animated and controlled interactively from a large collection of human motion data at minimal runtime cost. Our method tabulates the utility of taking different actions in any given state so that an appropriate sequence of actions can be found efficiently using table lookup. Our approach is based on dynamic programming which allows us to produce a control policy (or behavior) for a given state action model.
Ofek et al. (US 20100218094): teaches using interactive avatars to as displayed cognitive agents for interacting with user.
Blattner et al (US 7636755): teaches avatar interacting with the user; and using the personality of the human represented avatar.
Karkanias et al. (US 20090309891): teaches avatar interacting with the user and simulating the personality and physical characteristics.
Borenstein et al. (NPL: “Evolving Imitating Agents and the Emergence of a Neural Mirror System”) teaches the use of partially observable hidden Markov model in imitation learning of individual agents to learn actions from observed demonstrations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWATOSIN ALABI whose telephone number is (571)272-0516. The examiner can normally be reached Monday-Friday, 8:00am-5:00pm EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWATOSIN ALABI/Primary Examiner, Art Unit 2129
Prosecution Insights