DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. Claim 19 is directed to an abstract idea. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The rationale for this determination is explained below:
Claim 15, and it’s related dependent claims 16-20, is rejected under 35 U.S.C 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. Claim 15 is directed to an abstract idea that a computer-readable storage medium having a computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising: generating a virtual representative based on visual data of a first user; generating an emovector of a second user of a virtual environment; generating an input of a machine learning model based on the emovector of the second user; and controlling the virtual representative based on an output of the machine learning model.
The following analysis of facts of this particular patent application follows the rationale suggested in the "Federal Register Notice: 2019 Revised Patent Subject Matter Eligibility Guidance " (OG Notices: January 7, 2019, available from the US PTO website at https://www.govinfo.gov/content/pkg/FR-2019-01-07/pdf/2018-28282.pdf).
The Guidelines states:
Limitations that were found not to be enough to qualify as ‘‘significantly
more’’ when recited in a claim with a judicial exception include (P6):
• An additional element merely recites the words ‘‘apply it’’ (or an
equivalent) with the judicial exception, or merely includes instructions to
implement an abstract idea on a computer, or merely uses a computer as
a tool to perform an abstract idea;
• an additional element adds insignificant extra-solution activity to
the judicial exception;
• an additional element does no more than generally link the use of a judicial
exception to a particular technological environment or field of use.
In the instant case, at least one embodiment of the claimed invention is merely a computer-readable storage medium having a computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising: generating a virtual representative based on visual data of a first user; generating an emovector of a second user of a virtual environment; generating an input of a machine learning model based on the emovector of the second user; and controlling the virtual representative based on an output of the machine learning model.
Claim 15, and it’s related dependent claims 16-20, is rejected under §35 U.S.C. 101 as not falling within one of the four statutory categories of invention because the claimed invention is directed to computer program per se. See MPEP 2106(1). A claim directed toward a non-transitory computer readable medium having the program encoded thereon establishes a sufficient functional relationship between the program and a computer so as to remove it from the realm of “program per se”. MPEP 2111.05(111). Hence, adding the limitation of “A non-transitory” before “computer-readable storage medium” would resolve this issue.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-11, 14-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Miller (US 20150356781 A1) in view of Zalewski, etc. (US 20080215972 A1).
Regarding claim 1, Miller teaches that a method (See Miller: Figs. 1-4 and 10-11, and [0217], “As illustrated in FIGS. 1-4, an augmented reality system may include a light field generation subsystem operable to render virtual content (e.g., virtual objects, virtual tools, and other virtual constructs, for instance applications, features, characters, text, digits, and other symbols) in a field of view of a user. The augmented reality system may optionally also include an audio subsystem. As illustrated in FIG. 1, the light field generation subsystem (e.g., comprising both an optical sub-system 100 and a processing sub-system 102) may include multiple instances of personal augmented reality systems, for example a respective personal augmented reality system for each user”; and [0276], “FIG. 10 illustrates an example method 2100 of interacting with the passable world model. At 2102, the user's individual AR system may detect a location and orientation of the user within the world. In one or more embodiments, the location may be derived by a topological map of the system, as will be described in further detail below. In other embodiments, the location may be derived by GPS or any other localization tool. It should be appreciated that the passable world may be constantly accessed by the individual AR system”) comprising:
generating a virtual representative based on visual data of a first user (See Miller: Figs. 1-4 and 10-11, and [0048], “In one or more embodiments, the captured data pertains to an interaction of the user with one or more totems of the head-mounted augmented reality display system. In one or more embodiments, the animated avatar is displayed to another user of another head-mounted augmented reality display system”; and [0304], “In some implementations, in order to render an avatar that properly mimics the user, the user may train the AR system, for example by moving through a desired or prescribed set of movements. In response, the AR system may generate an avatar sequence in which an avatar replicates the movements, for example, by animating the avatar. Thus, the AR system captures or receives images of a user, and generates animations of an avatar based on movements of the user in the captured images. The user may be instrumented, for example, by wearing one or more sensors. In one or more embodiments, the AR system knows where the pose of the user's head, eyes, and/or hands based on data captured by various sensors of his/her individual AR system”);
generating an emovector of a second user of a virtual environment (See Miller: Figs. 1-4 and 10-11, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that the emotion state is mapped to the emovector);
generating an input of a machine learning model based on the emovector of the second user (See Miller: Figs. 1-4 and 20, and [0309], “Based on captured set of data pertaining to the user (e.g., movement, emotions, direction of movement, speed of movement, physical attributes, movement of body parts relative to the head, etc.) a pose of the sensors (e.g., sensors of the individual AR system) relative to the user may be determined. The pose (e.g., position and orientation) allow the system to determine a point of view from which the movement/set of data was captured such that it can be translated/transformed accurately. Based on this information, the AR system may determine a set of parameters related to the user's movement (e.g., through vectors) and animate a desired avatar with the calculated movement”; [0386], “Referring now to FIG. 20, an example method 1800 of selecting an appropriate light map is provided. At 1802, the user's individual AR system captures an image of the ambient surrounding through the user's FOV cameras. Next, the system selects at least one parameter of the captured image data to compare against the library of light maps. For example, the system may compare a color palette of the captured image against the library of light maps. At 1804, the system compares the parameter of the captured image against the parameters of the light maps, determines a closest approximation of the parameter (1806) and selects a light map having the closest approximation (1808). The system selects the closest approximation, and renders the virtual object based on the selected light map, at 1810”; and [0387], “Alternatively, or additionally, a selection technique utilizing artificial neural networks may be used. The AR system may use a neural network trained on the set or library of light maps. The neural network uses the selection criteria data as input, and produces a light map selection as output. After the neural network is trained on the library, the AR system presents the real world data from the user's camera to the neural network, and the neural network selects the light map with the least error from the library, either instantly or in real-time”. Note that various parameters for the neural network may be mapped to the input of a machine learning model, but a secondary art will be used to address this explicitly); and
controlling the virtual representative based on an output of the machine learning model (See Miller: Figs. 1-4 and 20, and [0306], “In one or more embodiments, the passable world may also contain information about various avatars inhabiting a space. It should be appreciated that every user may be rendered as an avatar in one embodiment. Or, a user operating an individual AR system from a remote location can create an avatar and digitally occupy a particular space as well. In either case, since the passable world is not a static data structure, but rather constantly receives information, avatar rendering and remote presence of users into a space may be based on the user's interaction with the user's individual AR system. Thus, rather than constantly updating an avatar's movement based on captured keyframes, as captured by cameras, avatars may be rendered based on a user's interaction with his/her individual augmented reality device. Advantageously, this reduces the need for individual AR systems to retrieve data from the cloud, and instead allows the system to perform a large number of computation tasks involved in avatar animation on the individual AR system itself”; and [0574], “Next in complexity, an avatar may resemble a physical appearance of the associated user, and may include updating of the avatar based on information collected from the associated user in real-time. For example, an image of a first user's face may have been captured or pre-scanned for use in generating the avatar. The avatar may have a face that appears either as realistic representation (e.g., photographic) or as a recognizable representation (e.g., drawn, cartoonish or caricature) The body of the avatar may, for example, be drawn, cartoonish or caricature, and may even be out of portion with the head of the avatar”).
However, Miller fails to explicitly disclose that generating an input of a machine learning model.
However, Zalewski teaches that generating an input of a machine learning model (See Zalewski: Figs. 1A-F and Figs. 2A-F, and [0084], “In some embodiments, the avatars 14 may express emotion through animation, facial change, sound, particle or chat bubble change to communicate a specific emotion. Such expressions of emotion by the avatar (sometimes called "emotes") may be pre-programmed and may be triggered by user commands. In particular embodiments of the invention, emotions expressed by the user during interaction with the virtual world may be mapped to emotion exhibited by the user's avatar. In certain embodiments, the user may select an emotional state that can be projected by the avatar. By way of example avatar emotes may be selected from a menu presented to the user by the apparatus 102. If, for example, the user selects "happy", the user's avatar may be shown with a smile on its face. If the user selects "sad", the avatar may be shown with a frown. Such menu-drive emotions may be somewhat awkward for a user to implement quickly. Therefore, in certain embodiments of the apparatus 102 may be configured to detect an emotional state of the user in real time and then appropriately change the features of the user's avatar to reflect that state. Such real time tracking of user emotional state can be particularly useful, e.g., for mapping user emotional state onto an avatar during video communication in which an image of the user's avatar is presented to a real device”; [0085], “Once the user's emotional state has been determined various combinations of body language and facial features indicative of the emotional state may be reflected in emotes exhibited by animation of the avatar (e.g., a raised first combined with bared teeth to indicate anger)”; and [0094], “In some embodiments, generating the email may involve tracking an emotional state of the source, e.g., as described above, and mapping the emotional state to the theme. For example, a serene or calm emotional state may be mapped to a theme characterized by fixed camera position or relatively slow virtual camera movement. An agitated or excited emotional state may be mapped to a theme characterized by jarring camera movement, extreme close-ups, harsh camera angles, and the like”. Note that mapping the emotional states to the avatar features in real time via system processing is mapped to “generating an input pf the machine learning model).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Miller to have generating an input of a machine learning model as taught by Zalewski in order to provide the popular virtual gaming environments with more number of experiences with respect to game action of other games (See Zalewski: Figs. 2A-F, and [0082], “Therefore, in certain embodiments of the apparatus 102 may be configured to detect an emotional state of the user in real time and then appropriately change the features of the user's avatar to reflect that state. Such real time tracking of user emotional state can be particularly useful, e.g., for mapping user emotional state onto an avatar during video communication in which an image of the user's avatar is presented to a real device”). Miller teaches a method and system that may generate an alternative visualization of a data set based on a specification of a selected first visualization of the data set and parameters related to the data set; while Zalewski teaches a system and method that may detect the user emotional states and incorporate these real time emotional states into the avatar features via the system processing to provide more game experiences for the users. Therefore, it is obvious to one of ordinary skill in the art to modify Miller by Zalewski to input the emotional states into the neural network to generate real-time avatar features to represent the user emotional changes. The motivation to modify Miller by Zalewski is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. Further, Miller teaches that the method of claim 1, wherein the virtual representative comprises a model or representation of the first user in the virtual environment, wherein the virtual representative shares audio, behavioral, structural, visual, or psychological features with the first user (See Miller: Fig. 4, and [0222], “The optical apparatus 460 in the form of a WRAP apparatus 410 or multiple depth plane 3D display system may, for instance, project images into each eye of a user, either directly or indirectly. When the number and radial placement of the virtual depth planes is comparable to the depth resolution of the human vision system as a function of radial distance, a discrete set of projected depth planes mimics the psycho-physical effect that is produced by a real, continuous, three dimensional object or scene. In one or more embodiments, the system 400 may comprise a frame 470 that may be customized for each AR user. Additional components of the system 400 may include electronics 430 (as will be discussed in further detail below) to connect various electrical and electronic subparts of the AR system to each other”; and [0614], “The approach described herein provides a very complex artificial intelligence (AI) property by performing deterministic acts with completely deterministic globally visible mechanisms for transitioning from one state to another. These actions are implicitly map-able to a behavior that a user cares about. Constant insight through monitoring of these global values of an overall state of the system is required, which allows the insertion of other states or changes to the current state. As a further example, an autonomous navigation definition or object may be responsive to a distance to a neighbor. The autonomous navigation definition or object may define a gradient around a neighbor, for example with a steep gradient on a front portion and a shallow gradient on a back portion. This creates an automatic behavior for the associated virtual object. For example, as the virtual object moves, it may for instance tend to move toward the shallow gradient rather than the steep gradient, if defined as such”), and wherein the virtual environment includes a virtual reality world or an augmented reality overlay of a real world (See Miller: Figs. 1-4, and [0217], “As illustrated in FIGS. 1-4, an augmented reality system may include a light field generation subsystem operable to render virtual content (e.g., virtual objects, virtual tools, and other virtual constructs, for instance applications, features, characters, text, digits, and other symbols) in a field of view of a user. The augmented reality system may optionally also include an audio subsystem. As illustrated in FIG. 1, the light field generation subsystem (e.g., comprising both an optical sub-system 100 and a processing sub-system 102) may include multiple instances of personal augmented reality systems, for example a respective personal augmented reality system for each user”).
Regarding claim 3, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. Further, Miller teaches that the method of claim 1, wherein the emovector of the second user includes time-synced facial expression data or body movement data associated with a timespan of a word spoken by the second user, and wherein the emovector of the second user represents at least one of: a presence of the second user, an audio or text input of the second user, a facial expression of the second user, a body movement of the second user, an absence of audio or text input of the second user, an absence of facial expressions of the second user, or an absence of body movements of the second user (See Miller: Figs. 1-4, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that real time facial expression mapping during talking is mapped to time-synced).
Regarding claim 4, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. Further, Miller and Zalewski teach that the method of claim 1, wherein generating the emovector of the second user comprises:
generating audio data and visual data of the second user (See Miller: Fig. 2, and [0229], “As illustrated in FIG. 2, the audio transducers may integrate with the visual, for example each audio transducers supported from a common frame with the visual components. Alternatively, the audio transducers may be distinct from the frame that carries the visual components. For example, the audio transducers may be part of a belt pack, such as the ones shown in FIGS. 1 (102a, 102b) and 2 (102)”);
transcribing and syncing the audio data with the visual data of the second user (See Miller: Fig. 2, and [0253], “In many implementations, the personal (or individual) AR system(s) worn by the user(s) may include one or more sensors, transducers, or other components. The sensors, transducers, or other components may be categorized into two general categories, i) those that detect aspects of the user who wears the sensor(s) (e.g., denominated herein as inward facing sensors), and ii) those that detect conditions in the ambient environment in which the user is located (e.g., denominated herein as outward facing sensors). These sensors may take a large variety of forms. For example, the sensor(s) may include one or more image sensors, for instance digital still or moving image cameras. Also for example, the sensor(s) may include one or more audio sensors or microphones. Other sensors may detect position, movement, temperature, heart rate, perspiration, etc.”; and [0576], “The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”);
quantizing the visual data of the second user (See Zalewski: Figs. 1A-F, and [0081], “As mentioned above, communicating between the real and virtual communication devices may involve video communication. According to a particular embodiment, an image of the avatar may be displayed with the real communication device during the video communication. The system that generates the virtual world may facilitate lip-synching of the avatar image to real or synthesized speech generated by the user associated with the avatar. For example, the user may record a voice message to be sent to the real device as part of a video message. The system may generate a video message of the avatar speaking the voice message in which the avatar's lip movements are synchronized to the user's speech within the message. Alternatively, the user may enter text of the message into a virtual device. The system may then synthesize speech for the avatar from the text and then generate a video image of the avatar in which the avatar's lip movements are synchronized to the synthesized speech. In other embodiments, the user may record a sound and video message, e.g., using the video image capture device 116 and audio signal capture device 118”. Note that recording the audio video signals is mapped to quantizing the audio video signals); and
generating the emovector of the second user based on the quantized visual data of the second user (See Miller: Figs. 1-4 and 10-11, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that the emotion state is mapped to the emovector).
Regarding claim 7, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. Further, Miller teaches that the method of claim 1, wherein controlling the virtual representative involves outputting at least one of: an audio or text output of the virtual representative, a facial expression of the virtual representative, or a body movement of the virtual representative (See Miller: Figs. 1-2, and [0306], “In one or more embodiments, the passable world may also contain information about various avatars inhabiting a space. It should be appreciated that every user may be rendered as an avatar in one embodiment. Or, a user operating an individual AR system from a remote location can create an avatar and digitally occupy a particular space as well. In either case, since the passable world is not a static data structure, but rather constantly receives information, avatar rendering and remote presence of users into a space may be based on the user's interaction with the user's individual AR system. Thus, rather than constantly updating an avatar's movement based on captured keyframes, as captured by cameras, avatars may be rendered based on a user's interaction with his/her individual augmented reality device. Advantageously, this reduces the need for individual AR systems to retrieve data from the cloud, and instead allows the system to perform a large number of computation tasks involved in avatar animation on the individual AR system itself”).
Regarding claim 8, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. Further, Miller and Zalewski teach that a system (See Miller: Figs. 1-4 and 10-11, and [0217], “As illustrated in FIGS. 1-4, an augmented reality system may include a light field generation subsystem operable to render virtual content (e.g., virtual objects, virtual tools, and other virtual constructs, for instance applications, features, characters, text, digits, and other symbols) in a field of view of a user. The augmented reality system may optionally also include an audio subsystem. As illustrated in FIG. 1, the light field generation subsystem (e.g., comprising both an optical sub-system 100 and a processing sub-system 102) may include multiple instances of personal augmented reality systems, for example a respective personal augmented reality system for each user”; and [0276], “FIG. 10 illustrates an example method 2100 of interacting with the passable world model. At 2102, the user's individual AR system may detect a location and orientation of the user within the world. In one or more embodiments, the location may be derived by a topological map of the system, as will be described in further detail below. In other embodiments, the location may be derived by GPS or any other localization tool. It should be appreciated that the passable world may be constantly accessed by the individual AR system”), comprising:
a processor (See Miller: Figs. 1-4, and [0231], “The computation component 102 may include one or more processors, for example, one or more micro-controllers, microprocessors, graphical processing units, digital signal processors, application specific integrated circuits (ASICs), programmable gate arrays, programmable logic circuits, or other circuits either embodying logic or capable of executing logic embodied in instructions encoded in software or firmware. The computation component 102 may include one or more non-transitory computer- or processor-readable media, for example volatile and/or nonvolatile memory, for instance read only memory (ROM), random access memory (RAM), static RAM, dynamic RAM, Flash memory, EEPROM, etc.”); and
memory or storage comprising an algorithm or computer instructions, which when executed by the processor, performs an operation (See Miller: Figs. 1-4, and [0231], “The computation component 102 may include one or more processors, for example, one or more micro-controllers, microprocessors, graphical processing units, digital signal processors, application specific integrated circuits (ASICs), programmable gate arrays, programmable logic circuits, or other circuits either embodying logic or capable of executing logic embodied in instructions encoded in software or firmware. The computation component 102 may include one or more non-transitory computer- or processor-readable media, for example volatile and/or nonvolatile memory, for instance read only memory (ROM), random access memory (RAM), static RAM, dynamic RAM, Flash memory, EEPROM, etc.”) comprising:
generate a virtual representative based on visual data of a first user (See Miller: Figs. 1-4 and 10-11, and [0048], “In one or more embodiments, the captured data pertains to an interaction of the user with one or more totems of the head-mounted augmented reality display system. In one or more embodiments, the animated avatar is displayed to another user of another head-mounted augmented reality display system”; and [0304], “In some implementations, in order to render an avatar that properly mimics the user, the user may train the AR system, for example by moving through a desired or prescribed set of movements. In response, the AR system may generate an avatar sequence in which an avatar replicates the movements, for example, by animating the avatar. Thus, the AR system captures or receives images of a user, and generates animations of an avatar based on movements of the user in the captured images. The user may be instrumented, for example, by wearing one or more sensors. In one or more embodiments, the AR system knows where the pose of the user's head, eyes, and/or hands based on data captured by various sensors of his/her individual AR system”);
generate an emovector of a second user of a virtual environment (See Miller: Figs. 1-4 and 10-11, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that the emotion state is mapped to the emovector);
generate an input of a machine learning model (See Zalewski: Figs. 1A-F and Figs. 2A-F, and [0084], “In some embodiments, the avatars 14 may express emotion through animation, facial change, sound, particle or chat bubble change to communicate a specific emotion. Such expressions of emotion by the avatar (sometimes called "emotes") may be pre-programmed and may be triggered by user commands. In particular embodiments of the invention, emotions expressed by the user during interaction with the virtual world may be mapped to emotion exhibited by the user's avatar. In certain embodiments, the user may select an emotional state that can be projected by the avatar. By way of example avatar emotes may be selected from a menu presented to the user by the apparatus 102. If, for example, the user selects "happy", the user's avatar may be shown with a smile on its face. If the user selects "sad", the avatar may be shown with a frown. Such menu-drive emotions may be somewhat awkward for a user to implement quickly. Therefore, in certain embodiments of the apparatus 102 may be configured to detect an emotional state of the user in real time and then appropriately change the features of the user's avatar to reflect that state. Such real time tracking of user emotional state can be particularly useful, e.g., for mapping user emotional state onto an avatar during video communication in which an image of the user's avatar is presented to a real device”; [0085], “Once the user's emotional state has been determined various combinations of body language and facial features indicative of the emotional state may be reflected in emotes exhibited by animation of the avatar (e.g., a raised first combined with bared teeth to indicate anger)”; and [0094], “In some embodiments, generating the email may involve tracking an emotional state of the source, e.g., as described above, and mapping the emotional state to the theme. For example, a serene or calm emotional state may be mapped to a theme characterized by fixed camera position or relatively slow virtual camera movement. An agitated or excited emotional state may be mapped to a theme characterized by jarring camera movement, extreme close-ups, harsh camera angles, and the like”. Note that mapping the emotional states to the avatar features in real time via system processing is mapped to “generating an input pf the machine learning model) based on the emovector of the second user (See Miller: Figs. 1-4 and 20, and [0309], “Based on captured set of data pertaining to the user (e.g., movement, emotions, direction of movement, speed of movement, physical attributes, movement of body parts relative to the head, etc.) a pose of the sensors (e.g., sensors of the individual AR system) relative to the user may be determined. The pose (e.g., position and orientation) allow the system to determine a point of view from which the movement/set of data was captured such that it can be translated/transformed accurately. Based on this information, the AR system may determine a set of parameters related to the user's movement (e.g., through vectors) and animate a desired avatar with the calculated movement”; [0386], “Referring now to FIG. 20, an example method 1800 of selecting an appropriate light map is provided. At 1802, the user's individual AR system captures an image of the ambient surrounding through the user's FOV cameras. Next, the system selects at least one parameter of the captured image data to compare against the library of light maps. For example, the system may compare a color palette of the captured image against the library of light maps. At 1804, the system compares the parameter of the captured image against the parameters of the light maps, determines a closest approximation of the parameter (1806) and selects a light map having the closest approximation (1808). The system selects the closest approximation, and renders the virtual object based on the selected light map, at 1810”; and [0387], “Alternatively, or additionally, a selection technique utilizing artificial neural networks may be used. The AR system may use a neural network trained on the set or library of light maps. The neural network uses the selection criteria data as input, and produces a light map selection as output. After the neural network is trained on the library, the AR system presents the real world data from the user's camera to the neural network, and the neural network selects the light map with the least error from the library, either instantly or in real-time”. Note that various parameters for the neural network may be mapped to the input of a machine learning model, but a secondary art will be used to address this explicitly); and
control the virtual representative based on an output of the machine learning model (See Miller: Figs. 1-4 and 20, and [0306], “In one or more embodiments, the passable world may also contain information about various avatars inhabiting a space. It should be appreciated that every user may be rendered as an avatar in one embodiment. Or, a user operating an individual AR system from a remote location can create an avatar and digitally occupy a particular space as well. In either case, since the passable world is not a static data structure, but rather constantly receives information, avatar rendering and remote presence of users into a space may be based on the user's interaction with the user's individual AR system. Thus, rather than constantly updating an avatar's movement based on captured keyframes, as captured by cameras, avatars may be rendered based on a user's interaction with his/her individual augmented reality device. Advantageously, this reduces the need for individual AR systems to retrieve data from the cloud, and instead allows the system to perform a large number of computation tasks involved in avatar animation on the individual AR system itself”; and [0574], “Next in complexity, an avatar may resemble a physical appearance of the associated user, and may include updating of the avatar based on information collected from the associated user in real-time. For example, an image of a first user's face may have been captured or pre-scanned for use in generating the avatar. The avatar may have a face that appears either as realistic representation (e.g., photographic) or as a recognizable representation (e.g., drawn, cartoonish or caricature) The body of the avatar may, for example, be drawn, cartoonish or caricature, and may even be out of portion with the head of the avatar”).
Regarding claim 9, Miller and Zalewski teach all the features with respect to claim 8 as outlined above. Further, Miller teaches that the system of claim 8, wherein the virtual representative comprises a model or representation of the first user in the virtual environment, wherein the virtual representative shares audio, behavioral, structural, visual, or psychological features with the first user (See Miller: Fig. 4, and [0222], “The optical apparatus 460 in the form of a WRAP apparatus 410 or multiple depth plane 3D display system may, for instance, project images into each eye of a user, either directly or indirectly. When the number and radial placement of the virtual depth planes is comparable to the depth resolution of the human vision system as a function of radial distance, a discrete set of projected depth planes mimics the psycho-physical effect that is produced by a real, continuous, three dimensional object or scene. In one or more embodiments, the system 400 may comprise a frame 470 that may be customized for each AR user. Additional components of the system 400 may include electronics 430 (as will be discussed in further detail below) to connect various electrical and electronic subparts of the AR system to each other”; and [0614], “The approach described herein provides a very complex artificial intelligence (AI) property by performing deterministic acts with completely deterministic globally visible mechanisms for transitioning from one state to another. These actions are implicitly map-able to a behavior that a user cares about. Constant insight through monitoring of these global values of an overall state of the system is required, which allows the insertion of other states or changes to the current state. As a further example, an autonomous navigation definition or object may be responsive to a distance to a neighbor. The autonomous navigation definition or object may define a gradient around a neighbor, for example with a steep gradient on a front portion and a shallow gradient on a back portion. This creates an automatic behavior for the associated virtual object. For example, as the virtual object moves, it may for instance tend to move toward the shallow gradient rather than the steep gradient, if defined as such”), and wherein the virtual environment includes a virtual reality world or an augmented reality overlay of a real world (See Miller: Figs. 1-4, and [0217], “As illustrated in FIGS. 1-4, an augmented reality system may include a light field generation subsystem operable to render virtual content (e.g., virtual objects, virtual tools, and other virtual constructs, for instance applications, features, characters, text, digits, and other symbols) in a field of view of a user. The augmented reality system may optionally also include an audio subsystem. As illustrated in FIG. 1, the light field generation subsystem (e.g., comprising both an optical sub-system 100 and a processing sub-system 102) may include multiple instances of personal augmented reality systems, for example a respective personal augmented reality system for each user”).
Regarding claim 10, Miller and Zalewski teach all the features with respect to claim 8 as outlined above. Further, Miller teaches that the system of claim 8, wherein the emovector of the second user includes time-synced facial expression data or body movement data associated with a timespan of a word spoken by the second user, and wherein the emovector of the second user represents at least one of: a presence of the second user, an audio or text input of the second user, a facial expression of the second user, a body movement of the second user, an absence of audio or text input of the second user, an absence of facial expressions of the second user, or an absence of body movements of the second user (See Miller: Figs. 1-4, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that real time facial expression mapping during talking is mapped to time-synced).
Regarding claim 11, Miller and Zalewski teach all the features with respect to claim 8 as outlined above. Further, Miller and Zalewski teach that the system of claim 8, wherein generating the emovector of the second user comprises:
generating audio data and visual data of the second user (See Miller: Fig. 2, and [0229], “As illustrated in FIG. 2, the audio transducers may integrate with the visual, for example each audio transducers supported from a common frame with the visual components. Alternatively, the audio transducers may be distinct from the frame that carries the visual components. For example, the audio transducers may be part of a belt pack, such as the ones shown in FIGS. 1 (102a, 102b) and 2 (102)”);
transcribing and syncing the audio data with the visual data of the second user (See Miller: Fig. 2, and [0253], “In many implementations, the personal (or individual) AR system(s) worn by the user(s) may include one or more sensors, transducers, or other components. The sensors, transducers, or other components may be categorized into two general categories, i) those that detect aspects of the user who wears the sensor(s) (e.g., denominated herein as inward facing sensors), and ii) those that detect conditions in the ambient environment in which the user is located (e.g., denominated herein as outward facing sensors). These sensors may take a large variety of forms. For example, the sensor(s) may include one or more image sensors, for instance digital still or moving image cameras. Also for example, the sensor(s) may include one or more audio sensors or microphones. Other sensors may detect position, movement, temperature, heart rate, perspiration, etc.”; and [0576], “The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”);
quantizing the visual data of the second user (See Zalewski: Figs. 1A-F, and [0081], “As mentioned above, communicating between the real and virtual communication devices may involve video communication. According to a particular embodiment, an image of the avatar may be displayed with the real communication device during the video communication. The system that generates the virtual world may facilitate lip-synching of the avatar image to real or synthesized speech generated by the user associated with the avatar. For example, the user may record a voice message to be sent to the real device as part of a video message. The system may generate a video message of the avatar speaking the voice message in which the avatar's lip movements are synchronized to the user's speech within the message. Alternatively, the user may enter text of the message into a virtual device. The system may then synthesize speech for the avatar from the text and then generate a video image of the avatar in which the avatar's lip movements are synchronized to the synthesized speech. In other embodiments, the user may record a sound and video message, e.g., using the video image capture device 116 and audio signal capture device 118”. Note that recording the audio video signals is mapped to quantizing the audio video signals); and
generating the emovector of the second user based on the quantized visual data of the second user (See Miller: Figs. 1-4 and 10-11, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that the emotion state is mapped to the emovector).
Regarding claim 14, Miller and Zalewski teach all the features with respect to claim 8 as outlined above. Further, Miller teaches that the system of claim 8, wherein controlling the virtual representative involves outputting at least one of: an audio or text output of the virtual representative, a facial expression of the virtual representative, or a body movement of the virtual representative (See Miller: Figs. 1-2, and [0306], “In one or more embodiments, the passable world may also contain information about various avatars inhabiting a space. It should be appreciated that every user may be rendered as an avatar in one embodiment. Or, a user operating an individual AR system from a remote location can create an avatar and digitally occupy a particular space as well. In either case, since the passable world is not a static data structure, but rather constantly receives information, avatar rendering and remote presence of users into a space may be based on the user's interaction with the user's individual AR system. Thus, rather than constantly updating an avatar's movement based on captured keyframes, as captured by cameras, avatars may be rendered based on a user's interaction with his/her individual augmented reality device. Advantageously, this reduces the need for individual AR systems to retrieve data from the cloud, and instead allows the system to perform a large number of computation tasks involved in avatar animation on the individual AR system itself”).
Regarding claim 15, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. Further, Miller and Zalewski teach that a computer-readable storage medium having a computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation (See Miller: Figs. 1-4 and 10-11, and [0217], “As illustrated in FIGS. 1-4, an augmented reality system may include a light field generation subsystem operable to render virtual content (e.g., virtual objects, virtual tools, and other virtual constructs, for instance applications, features, characters, text, digits, and other symbols) in a field of view of a user. The augmented reality system may optionally also include an audio subsystem. As illustrated in FIG. 1, the light field generation subsystem (e.g., comprising both an optical sub-system 100 and a processing sub-system 102) may include multiple instances of personal augmented reality systems, for example a respective personal augmented reality system for each user”; and [0276], “FIG. 10 illustrates an example method 2100 of interacting with the passable world model. At 2102, the user's individual AR system may detect a location and orientation of the user within the world. In one or more embodiments, the location may be derived by a topological map of the system, as will be described in further detail below. In other embodiments, the location may be derived by GPS or any other localization tool. It should be appreciated that the passable world may be constantly accessed by the individual AR system”) comprising:
generating a virtual representative based on visual data of a first user (See Miller: Figs. 1-4 and 10-11, and [0048], “In one or more embodiments, the captured data pertains to an interaction of the user with one or more totems of the head-mounted augmented reality display system. In one or more embodiments, the animated avatar is displayed to another user of another head-mounted augmented reality display system”; and [0304], “In some implementations, in order to render an avatar that properly mimics the user, the user may train the AR system, for example by moving through a desired or prescribed set of movements. In response, the AR system may generate an avatar sequence in which an avatar replicates the movements, for example, by animating the avatar. Thus, the AR system captures or receives images of a user, and generates animations of an avatar based on movements of the user in the captured images. The user may be instrumented, for example, by wearing one or more sensors. In one or more embodiments, the AR system knows where the pose of the user's head, eyes, and/or hands based on data captured by various sensors of his/her individual AR system”);
generating an emovector of a second user of a virtual environment (See Miller: Figs. 1-4 and 10-11, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that the emotion state is mapped to the emovector);
generating an input of a machine learning model (See Zalewski: Figs. 1A-F and Figs. 2A-F, and [0084], “In some embodiments, the avatars 14 may express emotion through animation, facial change, sound, particle or chat bubble change to communicate a specific emotion. Such expressions of emotion by the avatar (sometimes called "emotes") may be pre-programmed and may be triggered by user commands. In particular embodiments of the invention, emotions expressed by the user during interaction with the virtual world may be mapped to emotion exhibited by the user's avatar. In certain embodiments, the user may select an emotional state that can be projected by the avatar. By way of example avatar emotes may be selected from a menu presented to the user by the apparatus 102. If, for example, the user selects "happy", the user's avatar may be shown with a smile on its face. If the user selects "sad", the avatar may be shown with a frown. Such menu-drive emotions may be somewhat awkward for a user to implement quickly. Therefore, in certain embodiments of the apparatus 102 may be configured to detect an emotional state of the user in real time and then appropriately change the features of the user's avatar to reflect that state. Such real time tracking of user emotional state can be particularly useful, e.g., for mapping user emotional state onto an avatar during video communication in which an image of the user's avatar is presented to a real device”; [0085], “Once the user's emotional state has been determined various combinations of body language and facial features indicative of the emotional state may be reflected in emotes exhibited by animation of the avatar (e.g., a raised first combined with bared teeth to indicate anger)”; and [0094], “In some embodiments, generating the email may involve tracking an emotional state of the source, e.g., as described above, and mapping the emotional state to the theme. For example, a serene or calm emotional state may be mapped to a theme characterized by fixed camera position or relatively slow virtual camera movement. An agitated or excited emotional state may be mapped to a theme characterized by jarring camera movement, extreme close-ups, harsh camera angles, and the like”. Note that mapping the emotional states to the avatar features in real time via system processing is mapped to “generating an input pf the machine learning model) based on the emovector of the second user (See Miller: Figs. 1-4 and 20, and [0309], “Based on captured set of data pertaining to the user (e.g., movement, emotions, direction of movement, speed of movement, physical attributes, movement of body parts relative to the head, etc.) a pose of the sensors (e.g., sensors of the individual AR system) relative to the user may be determined. The pose (e.g., position and orientation) allow the system to determine a point of view from which the movement/set of data was captured such that it can be translated/transformed accurately. Based on this information, the AR system may determine a set of parameters related to the user's movement (e.g., through vectors) and animate a desired avatar with the calculated movement”; [0386], “Referring now to FIG. 20, an example method 1800 of selecting an appropriate light map is provided. At 1802, the user's individual AR system captures an image of the ambient surrounding through the user's FOV cameras. Next, the system selects at least one parameter of the captured image data to compare against the library of light maps. For example, the system may compare a color palette of the captured image against the library of light maps. At 1804, the system compares the parameter of the captured image against the parameters of the light maps, determines a closest approximation of the parameter (1806) and selects a light map having the closest approximation (1808). The system selects the closest approximation, and renders the virtual object based on the selected light map, at 1810”; and [0387], “Alternatively, or additionally, a selection technique utilizing artificial neural networks may be used. The AR system may use a neural network trained on the set or library of light maps. The neural network uses the selection criteria data as input, and produces a light map selection as output. After the neural network is trained on the library, the AR system presents the real world data from the user's camera to the neural network, and the neural network selects the light map with the least error from the library, either instantly or in real-time”. Note that various parameters for the neural network may be mapped to the input of a machine learning model, but a secondary art will be used to address this explicitly); and
controlling the virtual representative based on an output of the machine learning model (See Miller: Figs. 1-4 and 20, and [0306], “In one or more embodiments, the passable world may also contain information about various avatars inhabiting a space. It should be appreciated that every user may be rendered as an avatar in one embodiment. Or, a user operating an individual AR system from a remote location can create an avatar and digitally occupy a particular space as well. In either case, since the passable world is not a static data structure, but rather constantly receives information, avatar rendering and remote presence of users into a space may be based on the user's interaction with the user's individual AR system. Thus, rather than constantly updating an avatar's movement based on captured keyframes, as captured by cameras, avatars may be rendered based on a user's interaction with his/her individual augmented reality device. Advantageously, this reduces the need for individual AR systems to retrieve data from the cloud, and instead allows the system to perform a large number of computation tasks involved in avatar animation on the individual AR system itself”; and [0574], “Next in complexity, an avatar may resemble a physical appearance of the associated user, and may include updating of the avatar based on information collected from the associated user in real-time. For example, an image of a first user's face may have been captured or pre-scanned for use in generating the avatar. The avatar may have a face that appears either as realistic representation (e.g., photographic) or as a recognizable representation (e.g., drawn, cartoonish or caricature) The body of the avatar may, for example, be drawn, cartoonish or caricature, and may even be out of portion with the head of the avatar”).
Regarding claim 16, Miller and Zalewski teach all the features with respect to claim 15 as outlined above. Further, Miller teaches that the computer-readable storage medium of claim 15, wherein the virtual representative comprises a model or representation of the first user in the virtual environment, wherein the virtual representative shares audio, behavioral, structural, visual, or psychological features with the first user (See Miller: Fig. 4, and [0222], “The optical apparatus 460 in the form of a WRAP apparatus 410 or multiple depth plane 3D display system may, for instance, project images into each eye of a user, either directly or indirectly. When the number and radial placement of the virtual depth planes is comparable to the depth resolution of the human vision system as a function of radial distance, a discrete set of projected depth planes mimics the psycho-physical effect that is produced by a real, continuous, three dimensional object or scene. In one or more embodiments, the system 400 may comprise a frame 470 that may be customized for each AR user. Additional components of the system 400 may include electronics 430 (as will be discussed in further detail below) to connect various electrical and electronic subparts of the AR system to each other”; and [0614], “The approach described herein provides a very complex artificial intelligence (AI) property by performing deterministic acts with completely deterministic globally visible mechanisms for transitioning from one state to another. These actions are implicitly map-able to a behavior that a user cares about. Constant insight through monitoring of these global values of an overall state of the system is required, which allows the insertion of other states or changes to the current state. As a further example, an autonomous navigation definition or object may be responsive to a distance to a neighbor. The autonomous navigation definition or object may define a gradient around a neighbor, for example with a steep gradient on a front portion and a shallow gradient on a back portion. This creates an automatic behavior for the associated virtual object. For example, as the virtual object moves, it may for instance tend to move toward the shallow gradient rather than the steep gradient, if defined as such”), and wherein the virtual environment includes a virtual reality world or an augmented reality overlay of a real world (See Miller: Figs. 1-4, and [0217], “As illustrated in FIGS. 1-4, an augmented reality system may include a light field generation subsystem operable to render virtual content (e.g., virtual objects, virtual tools, and other virtual constructs, for instance applications, features, characters, text, digits, and other symbols) in a field of view of a user. The augmented reality system may optionally also include an audio subsystem. As illustrated in FIG. 1, the light field generation subsystem (e.g., comprising both an optical sub-system 100 and a processing sub-system 102) may include multiple instances of personal augmented reality systems, for example a respective personal augmented reality system for each user”).
Regarding claim 17, Miller and Zalewski teach all the features with respect to claim 15 as outlined above. Further, Miller teaches that the computer-readable storage medium of claim 15, wherein the emovector of the second user includes time-synced facial expression data or body movement data associated with a timespan of a word spoken by the second user, and wherein the emovector of the second user represents at least one of: a presence of the second user, an audio or text input of the second user, a facial expression of the second user, a body movement of the second user, an absence of audio or text input of the second user, an absence of facial expressions of the second user, or an absence of body movements of the second user (See Miller: Figs. 1-4, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that real time facial expression mapping during talking is mapped to time-synced).
Regarding claim 18, Miller and Zalewski teach all the features with respect to claim 15 as outlined above. Further, Miller and Zalewski teach that the computer-readable storage medium of claim 15, wherein generating the emovector of the second user comprises:
generating audio data and visual data of the second user (See Miller: Fig. 2, and [0229], “As illustrated in FIG. 2, the audio transducers may integrate with the visual, for example each audio transducers supported from a common frame with the visual components. Alternatively, the audio transducers may be distinct from the frame that carries the visual components. For example, the audio transducers may be part of a belt pack, such as the ones shown in FIGS. 1 (102a, 102b) and 2 (102)”);
transcribing and syncing the audio data with the visual data of the second user (See Miller: Fig. 2, and [0253], “In many implementations, the personal (or individual) AR system(s) worn by the user(s) may include one or more sensors, transducers, or other components. The sensors, transducers, or other components may be categorized into two general categories, i) those that detect aspects of the user who wears the sensor(s) (e.g., denominated herein as inward facing sensors), and ii) those that detect conditions in the ambient environment in which the user is located (e.g., denominated herein as outward facing sensors). These sensors may take a large variety of forms. For example, the sensor(s) may include one or more image sensors, for instance digital still or moving image cameras. Also for example, the sensor(s) may include one or more audio sensors or microphones. Other sensors may detect position, movement, temperature, heart rate, perspiration, etc.”; and [0576], “The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”);
quantizing the visual data of the second user (See Zalewski: Figs. 1A-F, and [0081], “As mentioned above, communicating between the real and virtual communication devices may involve video communication. According to a particular embodiment, an image of the avatar may be displayed with the real communication device during the video communication. The system that generates the virtual world may facilitate lip-synching of the avatar image to real or synthesized speech generated by the user associated with the avatar. For example, the user may record a voice message to be sent to the real device as part of a video message. The system may generate a video message of the avatar speaking the voice message in which the avatar's lip movements are synchronized to the user's speech within the message. Alternatively, the user may enter text of the message into a virtual device. The system may then synthesize speech for the avatar from the text and then generate a video image of the avatar in which the avatar's lip movements are synchronized to the synthesized speech. In other embodiments, the user may record a sound and video message, e.g., using the video image capture device 116 and audio signal capture device 118”. Note that recording the audio video signals is mapped to quantizing the audio video signals); and
generating the emovector of the second user based on the quantized visual data of the second user (See Miller: Figs. 1-4 and 10-11, and [0576], “Voice may be passed through to appear to be emanating from the avatar. In some implementations in which the avatar has a realistic face, the facial images may also be passed through. Where the avatar does not have a realistic face, the AR system may discern facial expressions from the images and/or inflections in voice from the sound. The AR system may update facial expressions of the avatar based on the discerned facial expressions and/or inflections in voice. For example, the AR system may determine an emotion state (e.g., happy, sad, angry, content, frustrated, satisfied) of the first user based on the facial expressions and/or inflections. The AR system may select a facial expression to render on the avatar based on the determined emotion state of the first user. For example, the AR system may select from a number of animation or graphical representations of emotion. Thus, the AR system may employ real time texture mapping to render emotional state of a user on an avatar that represents the user”. Note that the emotion state is mapped to the emovector).
Regarding claim 20, Miller and Zalewski teach all the features with respect to claim 15 as outlined above. Further, Miller teaches that the computer-readable storage medium of claim 15, wherein controlling the virtual representative involves outputting at least one of: an audio or text output of the virtual representative, a facial expression of the virtual representative, or a body movement of the virtual representative (See Miller: Figs. 1-2, and [0306], “In one or more embodiments, the passable world may also contain information about various avatars inhabiting a space. It should be appreciated that every user may be rendered as an avatar in one embodiment. Or, a user operating an individual AR system from a remote location can create an avatar and digitally occupy a particular space as well. In either case, since the passable world is not a static data structure, but rather constantly receives information, avatar rendering and remote presence of users into a space may be based on the user's interaction with the user's individual AR system. Thus, rather than constantly updating an avatar's movement based on captured keyframes, as captured by cameras, avatars may be rendered based on a user's interaction with his/her individual augmented reality device. Advantageously, this reduces the need for individual AR systems to retrieve data from the cloud, and instead allows the system to perform a large number of computation tasks involved in avatar animation on the individual AR system itself”).
Claims 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Miller (US 20150356781 A1) in view of Zalewski, etc. (US 20080215972 A1), further in view of Wang, etc. (US 20180357286 A1).
Regarding claim 5, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. However, Miller, modified by Zalewski, fails to explicitly disclose that the method of claim 1, wherein the machine learning model represents a language learning model trained to generate conversational language with the second user, wherein the input of the machine learning model includes an input emovector word comprising an input emovector or a combination of input text and the input emovector, and wherein the output of the machine learning model includes a probability distribution of an output emovector or a combination of output text and the output emovector.
However, Wang teaches that the method of claim 1, wherein the machine learning model represents a language learning model trained to generate conversational language with the second user, wherein the input of the machine learning model includes an input emovector word comprising an input emovector or a combination of input text and the input emovector, and wherein the output of the machine learning model includes a probability distribution of an output emovector or a combination of output text and the output emovector (See Wang: Fig. 1-3, and [0066], “The response determination module 316 is configured to determine one or more response(s) 336 via a response model 328 and the user text/query 324. In one embodiment, the response determination module 316 and the response model 328 are implemented as a supervised machine learning algorithm, where the response model 328 is developed using labeled training data to formulate the response(s) 336 to the user text/query 324. In conjunction with the response determination module 316, the response model 328 may be implemented as a support vector machine (SVM), a Bayesian classifier (naïve or otherwise), a k-nearest neighbor classifier, or any other such supervised machine learning implementation or combination thereof. One example of a machine learning-based model that may be implemented by the response determination module 316 and/or the response model 328 is discussed in Prakash”; [0068], “The emotion determination module 318 is configured to determine one or more emotion(s) 334 via the emotion model 326 when provided with the user data 322 and/or user text/query 324 as input. As discussed above, the user data 322 may include various types of user data, such as one or more images, a video, audio data (e.g., the prosody of the user's voice), historical GPS location information, a current GPS location, biometric information, and other such user data or combination of user data. As also explained above, the type of user data that is used as signals to the emotion model 326 is data that the user 108 has authorized the emotional intelligence determination server 106 to use. The types of received user data 322 may further correspond to the types of crowdsourced labeled data 332 (e.g., retrieved from the labeled emotion database 118). In instances where the type of received user data 322 does not correspond to a type of crowdsourced labeled data (e.g., the user data 322 includes video data and the crowdsourced labeled data does not include video data), the non-corresponding user data may be stored in the unlabeled emotion database 116. Thereafter, when the emotional intelligence determination server 106 has a predetermined amount of the non-corresponding user data type, the emotional intelligence determination server 106 may then invoke the crowdsource training module 314 to disseminate the non-corresponding user data type as the untrained emotion data 330, along with various emotion labels, to be labeled by one or more of the human operators 124. In this manner, the emotional intelligence determination server 106 can be configured to manage instances where an unknown user data type is received”; [0087], “In one embodiment, selecting a response in either Operation 618 or Operation 620 is dependent on the ranking of the responses according to whether the emotions associated with the responses are more similar to the determined user emotion 234 or the determined chatbot emotion 236. As explained above, when a set of emotions are determined for a set of response(s) 238, the determined emotions are compared with the user emotion 234 and/or the chatbot emotion 236, and the response ranking module 224 ranks those responses higher where their corresponding emotions are determined to be similar to as the user emotion 234 and/or the chatbot emotion 236. In performing this ranking, the response ranking module 224 may first rank the response(s) 238 according to whether their corresponding emotions are similar to the user emotion 234 and/or chatbot emotion 236, and then according to the probability associated with such emotions. Additionally, and/or alternatively, probabilities of the emotions that are determined as being similar to the user emotion 234 are multiplied with a first predetermined weighting factor and probabilities of the emotions that are determined as being similar to the chatbot emotion 236 are multiplied with second predetermined weight factor. Where the emotion of a selected response is not similar to the user emotion 234 and/or chatbot emotion 236, there may be no weighting factor applied. Furthermore, the first and second weighting factors may be different, such that the first weighting factor is greater than the second weighting factor (or vice versa). In this manner, the weighting of the probabilities of the emotions associated with the response(s) 238 may affect the re-ranking of such response(s) 238”; [0093], “Although the foregoing example employs a predetermined threshold to the user historical data 232, alternative embodiments may employ more sophisticated means for analyzing the user historical data 232. For example, the data 218 may include an emotional chatbot emotion model that, having been trained via the crowdsource training module 314 and communicated to the client device 104, determines an emotion to assign to the emotional chatbot 220 given the user historical data 232. Like the emotion determination model 318 and the emotion model 326, this emotional chatbot emotion model may also be implemented as a supervised machine learning algorithm that is configured with labeled data in the form of labeled user historical data (e.g., prior user queries and/or their associated emotions). Thus, the emotional assignment module 222 may leverage this model in determining an emotional state for the emotional chatbot 220 alternatively, and/or in addition to, the predetermined threshold approach”; and Fig. 8, and [0111], “The I/O components 850 may include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I/O components 850 that are included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I/O components 850 may include many other components that are not shown in FIG. 8. The I/O components 850 are grouped according to functionality merely for simplifying the following discussion and the grouping is in no way limiting. In various example embodiments, the I/O components 850 may include output components 852 and input components 854. The output components 852 may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components 854 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and/or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like”. Note that the ML with text query is mapped to the language learning model, various inputs such as audio, visual, expression, etc., is mapped to the emovector input, and the various outputs is mapped to the emovector output).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Miller to have the method of claim 1, wherein the machine learning model represents a language learning model trained to generate conversational language with the second user, wherein the input of the machine learning model includes an input emovector word comprising an input emovector or a combination of input text and the input emovector, and wherein the output of the machine learning model includes a probability distribution of an output emovector or a combination of output text and the output emovector as taught by Wang in order to provide higher probabilities of the current and correct emotional state of the user (See Wang: Figs. 1-3, and [0025], “The obtained user data and/or information each represent a signal for an emotion model maintained by the emotional intelligence determination server 106. As explained with reference to FIG. 3, the emotional model may be implemented within a supervised machine learning algorithm such that the supervised machine learning algorithm generates one or more probabilities for one or more potential emotions associated with the user 108. In this way, the more robust information provided to the emotional intelligence determination server 106, the more likely that the emotional intelligence determination server 106 provides higher probabilities of the current (and correct) emotional state of the user 108”). Miller teaches a method and system that may generate an alternative visualization of a data set based on a specification of a selected first visualization of the data set and parameters related to the data set; while Wang teaches a system and method that may input the emovectors into the language learning model to generate emovector output with the probability of the correct emotional state of the user. Therefore, it is obvious to one of ordinary skill in the art to modify Miller by Wang to input the emotional states into the language learning model to generate higher or correct emotional state of the user. The motivation to modify Miller by Wang is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 12, Miller and Zalewski teach all the features with respect to claim 8 as outlined above. Further, Wang teaches that the system of claim 8, wherein the machine learning model represents a language learning model trained to generate conversational language with the second user, wherein the input of the machine learning model includes an input emovector word comprising an input emovector or a combination of input text and the input emovector, and wherein the output of the machine learning model includes a probability distribution of an output emovector or a combination of output text and the output emovector (See Wang: Fig. 1-3, and [0066], “The response determination module 316 is configured to determine one or more response(s) 336 via a response model 328 and the user text/query 324. In one embodiment, the response determination module 316 and the response model 328 are implemented as a supervised machine learning algorithm, where the response model 328 is developed using labeled training data to formulate the response(s) 336 to the user text/query 324. In conjunction with the response determination module 316, the response model 328 may be implemented as a support vector machine (SVM), a Bayesian classifier (naïve or otherwise), a k-nearest neighbor classifier, or any other such supervised machine learning implementation or combination thereof. One example of a machine learning-based model that may be implemented by the response determination module 316 and/or the response model 328 is discussed in Prakash”; [0068], “The emotion determination module 318 is configured to determine one or more emotion(s) 334 via the emotion model 326 when provided with the user data 322 and/or user text/query 324 as input. As discussed above, the user data 322 may include various types of user data, such as one or more images, a video, audio data (e.g., the prosody of the user's voice), historical GPS location information, a current GPS location, biometric information, and other such user data or combination of user data. As also explained above, the type of user data that is used as signals to the emotion model 326 is data that the user 108 has authorized the emotional intelligence determination server 106 to use. The types of received user data 322 may further correspond to the types of crowdsourced labeled data 332 (e.g., retrieved from the labeled emotion database 118). In instances where the type of received user data 322 does not correspond to a type of crowdsourced labeled data (e.g., the user data 322 includes video data and the crowdsourced labeled data does not include video data), the non-corresponding user data may be stored in the unlabeled emotion database 116. Thereafter, when the emotional intelligence determination server 106 has a predetermined amount of the non-corresponding user data type, the emotional intelligence determination server 106 may then invoke the crowdsource training module 314 to disseminate the non-corresponding user data type as the untrained emotion data 330, along with various emotion labels, to be labeled by one or more of the human operators 124. In this manner, the emotional intelligence determination server 106 can be configured to manage instances where an unknown user data type is received”; [0087], “In one embodiment, selecting a response in either Operation 618 or Operation 620 is dependent on the ranking of the responses according to whether the emotions associated with the responses are more similar to the determined user emotion 234 or the determined chatbot emotion 236. As explained above, when a set of emotions are determined for a set of response(s) 238, the determined emotions are compared with the user emotion 234 and/or the chatbot emotion 236, and the response ranking module 224 ranks those responses higher where their corresponding emotions are determined to be similar to as the user emotion 234 and/or the chatbot emotion 236. In performing this ranking, the response ranking module 224 may first rank the response(s) 238 according to whether their corresponding emotions are similar to the user emotion 234 and/or chatbot emotion 236, and then according to the probability associated with such emotions. Additionally, and/or alternatively, probabilities of the emotions that are determined as being similar to the user emotion 234 are multiplied with a first predetermined weighting factor and probabilities of the emotions that are determined as being similar to the chatbot emotion 236 are multiplied with second predetermined weight factor. Where the emotion of a selected response is not similar to the user emotion 234 and/or chatbot emotion 236, there may be no weighting factor applied. Furthermore, the first and second weighting factors may be different, such that the first weighting factor is greater than the second weighting factor (or vice versa). In this manner, the weighting of the probabilities of the emotions associated with the response(s) 238 may affect the re-ranking of such response(s) 238”; [0093], “Although the foregoing example employs a predetermined threshold to the user historical data 232, alternative embodiments may employ more sophisticated means for analyzing the user historical data 232. For example, the data 218 may include an emotional chatbot emotion model that, having been trained via the crowdsource training module 314 and communicated to the client device 104, determines an emotion to assign to the emotional chatbot 220 given the user historical data 232. Like the emotion determination model 318 and the emotion model 326, this emotional chatbot emotion model may also be implemented as a supervised machine learning algorithm that is configured with labeled data in the form of labeled user historical data (e.g., prior user queries and/or their associated emotions). Thus, the emotional assignment module 222 may leverage this model in determining an emotional state for the emotional chatbot 220 alternatively, and/or in addition to, the predetermined threshold approach”; and Fig. 8, and [0111], “The I/O components 850 may include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I/O components 850 that are included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I/O components 850 may include many other components that are not shown in FIG. 8. The I/O components 850 are grouped according to functionality merely for simplifying the following discussion and the grouping is in no way limiting. In various example embodiments, the I/O components 850 may include output components 852 and input components 854. The output components 852 may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components 854 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and/or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like”. Note that the ML with text query is mapped to the language learning model, various inputs such as audio, visual, expression, etc., is mapped to the emovector input, and the various outputs is mapped to the emovector output).
Claims 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Miller (US 20150356781 A1) in view of Zalewski, etc. (US 20080215972 A1), further in view of Feinson, etc. (US 20190236464 A1).
Regarding claim 6, Miller and Zalewski teach all the features with respect to claim 1 as outlined above. However, Miller, modified by Zalewski, fails to explicitly disclose that the method of claim 1, wherein generating the input of the machine learning model comprises: mapping a first emovector to an embedded space shared with known emovectors of the machine learning model; determining distances between the first emovector and the known emovectors; mapping the first emovector to a second emovector of the known emovectors, wherein the second emovector represents a shortest distance of the distances between the first emovector and the known emovectors; and generating an emovector word based on the second emovector.
However, Feinson teaches that the method of claim 1, wherein generating the input of the machine learning model comprises:
mapping a first emovector to an embedded space shared with known emovectors of the machine learning model (See Feinson: Figs. 2A-C and Fig. 3, and [0197], “In step 304, the set of affective values of the artificial intelligence entity may be continuously updated based on the growth or decay factors during a time period. It is noted that the set of affective attributes are associated with the set of affective values of the artificial intelligence entity. The affective values (associated with the affective attributes) of the artificial intelligence entity may be continuously updated based on the growth or decay factors associated with the affective attributes. For example, as illustrated in FIGS. 2A-2C, the affective values 201, 202, and 212 (e.g., 207a-207f, 208a-208f, and 218a-218f) may be continuously updated based on one or more growth or decay factors associated with affective attributes A, B, and C. In some embodiments, the continuous updating of the affective values of the artificial intelligence entity may include periodic updating of the affective values of the artificial intelligence entity based on the one more growth or decay factors”. Note that the affective attributes are mapped to the shared embedding space, and the affective values are updated and fed into the AI model);
determining distances between the first emovector and the known emovectors (See Feinson: Figs. 2A-C and Fig. 3, and [0185], “In this way, the AI entity can “think” and learn from its current database of information (e.g., a sufficiently rich graph) without necessarily obtaining that information through new conversations or other sources of information. Such attribute prediction helps facilitate induction of grounding, affect/emotion, ontology, latent factors (inferred new nodes that previously didn't exist), and conversational outputs, etc. As an example, the graph data and structure can be used for graph neural networks that learn information about graph entries from other entries. In some embodiments, the similarities in word association may be used (e.g., by the AI entity or other system) to transfer information from pre-trained word embeddings to the graph, which would then later go into graph embeddings. As such, although pre-trained word embeddings may not encode the graph structure or the types of metadata encoded in the AI entity's full ontology-affect graph, the graph embeddings (generated from the graph or portions thereof) may include such graph structure, metadata, or information other than the information in the word embeddings. As an example, a graph embedding network may be configured to use similarity measures on word embeddings in the graph to retrieve words close to the label of a given node (e.g., within a certain tolerance defined by a hyperparameter or learned weight of the network), or, symbolically walk the graph to find similar entries like those that share a class. Then the AI entity may use either a neural network or similarity measures to find words that are close to known graph nodes, those words can be inserted in the graph and connected to the given node that started the query. In one use case, a search for new connections may be performed where a new word retrieved from a conversational input overlaps, in the pre-trained word embedding space, with the words for visceral or affect data associated with a given node (e.g., the new words are deemed to be similar to the given node based on recognized visceral or affect words). In some embodiments asymmetries across nodes' contexts may indicate knowledge gaps and trigger the AI to acquire new equalizing information. In one use case, a disproportionately asymmetrical context association (such as nodes sharing strong class similarity but one is lacking visceral similarity) shared between nodes, or, an asymmetrical history of method acquisition between similar nodes (node a deduction, node b induction) may trigger the system to attempt through induction/sub-symbolic, deduction/symbolic, or conversational means to equalize the asymmetries. In some embodiments, logically or probabilistically contradicting information within the ontology-affect graph or between the graph and external inputs may also trigger deconfliction mechanisms using such aforementioned knowledge acquisition techniques”; and [0188], “In some embodiments, template variable selection (e.g., a process of filling in predicates, subjects, verbs, and other unknowns in the selected template) may be performed. An example of using learned vectors is as follows: An input to the AI entity may be “I need a new friend, who should I go to lunch with?” A candidate response template is selected, such as “Good question, I'm [positive emotion] to help. You should go meet [Person A]. [He or she] is [adjective].” The selection of the response template may trigger a query of the Euclidean or cosine similarity distance between the interlocutor's graph embedding and all the other entities in the graph. Such functions (e.g., distance measures) represent certain measures of similarity between concepts. Once the foregoing query is received, the query may be used to fill the template, along with a symbolic query for the emotional disposition of the AI entity towards the interlocutor. If pressed for an explanation for the AI entity's response reason, the AI entity can walk the graph to retrieve and output symbolic similarities between the two individuals in this case, or a simply symbolic realization of the vector calculation in the general case. In this way, although post hoc, this pattern mirrors similar post hoc rationalization of intuitive decisions in human reasoning and demonstrates the possibility of mixing symbolic and sub symbolic vector space reasoning. Other possible queries exist with more extensive symbolic and sub-symbolic functions that may be more tied together”. Note that the similarity comparison is measured in distance);
mapping the first emovector to a second emovector of the known emovectors, wherein the second emovector represents a shortest distance of the distances between the first emovector and the known emovectors (See Feinson: Figs. 2A-C and Fig. 3, and [0152], “Our brains designate various elements of knowledge to different levels of importance. The statement “a tiger is dangerous” may be more important than “the grass is green” because, from a typical human prospective, the former statement contains a higher absolute emotional content than the latter statement. Thus, the highest score represents the closest fit for the theme of the conversation, with a degree of novelty and emotional content, and, depending on the certainty levels of the knowledge (explicit or inferred), the response might be “I think that means they can smell well””; and [0188], “In some embodiments, template variable selection (e.g., a process of filling in predicates, subjects, verbs, and other unknowns in the selected template) may be performed. An example of using learned vectors is as follows: An input to the AI entity may be “I need a new friend, who should I go to lunch with?” A candidate response template is selected, such as “Good question, I'm [positive emotion] to help. You should go meet [Person A]. [He or she] is [adjective].” The selection of the response template may trigger a query of the Euclidean or cosine similarity distance between the interlocutor's graph embedding and all the other entities in the graph. Such functions (e.g., distance measures) represent certain measures of similarity between concepts. Once the foregoing query is received, the query may be used to fill the template, along with a symbolic query for the emotional disposition of the AI entity towards the interlocutor. If pressed for an explanation for the AI entity's response reason, the AI entity can walk the graph to retrieve and output symbolic similarities between the two individuals in this case, or a simply symbolic realization of the vector calculation in the general case. In this way, although post hoc, this pattern mirrors similar post hoc rationalization of intuitive decisions in human reasoning and demonstrates the possibility of mixing symbolic and sub symbolic vector space reasoning. Other possible queries exist with more extensive symbolic and sub-symbolic functions that may be more tied together”. Note that selection based on similarity is mapped the closest distance matching); and
generating an emovector word based on the second emovector (See Feinson: Figs. 2A-C and Fig. 3, and [0030], “In some embodiments, once an AI system achieves sentience (or a reasonable facsimile thereof), its behavior and thinking processes may be further optimized. As an example, such optimization may include modifying the system to reflect more human-like behaviors, increased learning efficiency, more nuanced emotional qualities, or other aspects”; [0178], “In some embodiments, the vector may be provided as an input to the consumption network, and the consumption network may generate one or more outputs based on the vector. The architecture of the consumption network's input layer (e.g., an embedding input layer) may be configured based on the specific architecture and hyper-parameters of the upstream graph embedding network to enable the consumption network to properly process the embedding vectors. In some embodiments, the vector may be used as weights (e.g., frozen or learnable weights) for other input encodings (e.g., as one hot representation). In some cases, if the vectors are used as learnable weights, the updated vectors may be transferred as a weight vector back to the graph embedding network and further fine-tuned by the graph embedding network, before being transferred back downstream (e.g., to the consumption network or other consumption networks)”; and [0179], “In some embodiments, such a consumption network (that consumes the vectors generated by the upstream embedding network) may include a sequence neural network. As an example, the sequence neural network may be trained to maintain or output a dense vector representation of a conversation history, current state, or other such memory-related data. In one use case, with respect to a conversation between the AI entity and another entity (e.g., a human user, another artificial intelligent entity, etc.), the AI entity may rely on a long short-term memory (LSTM) network (or other sequence neural network) to consume the relevant ontology-affect graph embedding vectors (and/or BERT or other pre-trained word embeddings) representing the other entity's inputs (e.g., words, phrases, sentences, or other input provided by the other entity) to link or build up the vectors into a temporal structure (e.g., along with emotional and sentiment information of the AI entity or the other entity). Based on the conversation history information of the LSTM network, the LSTM network may output a vector representing the conversation state (also referred to as “conversation state vector” herein) (e.g., analogous to a person's memory of a conversation). As an example, the LSTM network may output the conversation state vector in response to the subsequent input provided by the other entity or other automated triggers (e.g., a similar conversation or context, the other entity's name or other identifier being brought up in a subsequent conversation, etc.). This vector can then be stored in the ontology-affect graph as a conversation memory node or consumed by other neural networks”. Note that the conversational tyle output is mapped to the emovector output word).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Miller to have the method of claim 1, wherein generating the input of the machine learning model comprises: mapping a first emovector to an embedded space shared with known emovectors of the machine learning model; determining distances between the first emovector and the known emovectors; mapping the first emovector to a second emovector of the known emovectors, wherein the second emovector represents a shortest distance of the distances between the first emovector and the known emovectors; and generating an emovector word based on the second emovector as taught by Feinson in order to enable utilizing a procedure to construct hypothesis to test based on rules (See Feinson: Fig. 1, and [0127], “B) Inference Questions. Complex questions requiring backward chaining (goal-directed reasoning) by the system, utilize a procedure where the process constructs a hypothesis and works backward through its rules to test against that hypothesis”). Miller teaches a method and system that may generate an alternative visualization of a data set based on a specification of a selected first visualization of the data set and parameters related to the data set; while Feinson teaches a system and method that may map the emovector to known emovectors and select the closest emovector to generate the emovector output word. Therefore, it is obvious to one of ordinary skill in the art to modify Miller by Feinson to select the most similar emovector as input to the machine learning model to generate the output emovector word. The motivation to modify Miller by Feinson is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 13, Miller and Zalewski teach all the features with respect to claim 8 as outlined above. Further, Feinson teaches that the system of claim 8, wherein generating the input of the machine learning model comprises:
mapping a first emovector to an embedded space shared with known emovectors of the machine learning model (See Feinson: Figs. 2A-C and Fig. 3, and [0197], “In step 304, the set of affective values of the artificial intelligence entity may be continuously updated based on the growth or decay factors during a time period. It is noted that the set of affective attributes are associated with the set of affective values of the artificial intelligence entity. The affective values (associated with the affective attributes) of the artificial intelligence entity may be continuously updated based on the growth or decay factors associated with the affective attributes. For example, as illustrated in FIGS. 2A-2C, the affective values 201, 202, and 212 (e.g., 207a-207f, 208a-208f, and 218a-218f) may be continuously updated based on one or more growth or decay factors associated with affective attributes A, B, and C. In some embodiments, the continuous updating of the affective values of the artificial intelligence entity may include periodic updating of the affective values of the artificial intelligence entity based on the one more growth or decay factors”. Note that the affective attributes are mapped to the shared embedding space, and the affective values are updated and fed into the AI model);
determining distances between the first emovector and the known emovectors (See Feinson: Figs. 2A-C and Fig. 3, and [0185], “In this way, the AI entity can “think” and learn from its current database of information (e.g., a sufficiently rich graph) without necessarily obtaining that information through new conversations or other sources of information. Such attribute prediction helps facilitate induction of grounding, affect/emotion, ontology, latent factors (inferred new nodes that previously didn't exist), and conversational outputs, etc. As an example, the graph data and structure can be used for graph neural networks that learn information about graph entries from other entries. In some embodiments, the similarities in word association may be used (e.g., by the AI entity or other system) to transfer information from pre-trained word embeddings to the graph, which would then later go into graph embeddings. As such, although pre-trained word embeddings may not encode the graph structure or the types of metadata encoded in the AI entity's full ontology-affect graph, the graph embeddings (generated from the graph or portions thereof) may include such graph structure, metadata, or information other than the information in the word embeddings. As an example, a graph embedding network may be configured to use similarity measures on word embeddings in the graph to retrieve words close to the label of a given node (e.g., within a certain tolerance defined by a hyperparameter or learned weight of the network), or, symbolically walk the graph to find similar entries like those that share a class. Then the AI entity may use either a neural network or similarity measures to find words that are close to known graph nodes, those words can be inserted in the graph and connected to the given node that started the query. In one use case, a search for new connections may be performed where a new word retrieved from a conversational input overlaps, in the pre-trained word embedding space, with the words for visceral or affect data associated with a given node (e.g., the new words are deemed to be similar to the given node based on recognized visceral or affect words). In some embodiments asymmetries across nodes' contexts may indicate knowledge gaps and trigger the AI to acquire new equalizing information. In one use case, a disproportionately asymmetrical context association (such as nodes sharing strong class similarity but one is lacking visceral similarity) shared between nodes, or, an asymmetrical history of method acquisition between similar nodes (node a deduction, node b induction) may trigger the system to attempt through induction/sub-symbolic, deduction/symbolic, or conversational means to equalize the asymmetries. In some embodiments, logically or probabilistically contradicting information within the ontology-affect graph or between the graph and external inputs may also trigger deconfliction mechanisms using such aforementioned knowledge acquisition techniques”; and [0188], “In some embodiments, template variable selection (e.g., a process of filling in predicates, subjects, verbs, and other unknowns in the selected template) may be performed. An example of using learned vectors is as follows: An input to the AI entity may be “I need a new friend, who should I go to lunch with?” A candidate response template is selected, such as “Good question, I'm [positive emotion] to help. You should go meet [Person A]. [He or she] is [adjective].” The selection of the response template may trigger a query of the Euclidean or cosine similarity distance between the interlocutor's graph embedding and all the other entities in the graph. Such functions (e.g., distance measures) represent certain measures of similarity between concepts. Once the foregoing query is received, the query may be used to fill the template, along with a symbolic query for the emotional disposition of the AI entity towards the interlocutor. If pressed for an explanation for the AI entity's response reason, the AI entity can walk the graph to retrieve and output symbolic similarities between the two individuals in this case, or a simply symbolic realization of the vector calculation in the general case. In this way, although post hoc, this pattern mirrors similar post hoc rationalization of intuitive decisions in human reasoning and demonstrates the possibility of mixing symbolic and sub symbolic vector space reasoning. Other possible queries exist with more extensive symbolic and sub-symbolic functions that may be more tied together”. Note that the similarity comparison is measured in distance);
mapping the first emovector to a second emovector of the known emovectors, wherein the second emovector represents a shortest distance of the distances between the first emovector and the known emovectors (See Feinson: Figs. 2A-C and Fig. 3, and [0152], “Our brains designate various elements of knowledge to different levels of importance. The statement “a tiger is dangerous” may be more important than “the grass is green” because, from a typical human prospective, the former statement contains a higher absolute emotional content than the latter statement. Thus, the highest score represents the closest fit for the theme of the conversation, with a degree of novelty and emotional content, and, depending on the certainty levels of the knowledge (explicit or inferred), the response might be “I think that means they can smell well””; and [0188], “In some embodiments, template variable selection (e.g., a process of filling in predicates, subjects, verbs, and other unknowns in the selected template) may be performed. An example of using learned vectors is as follows: An input to the AI entity may be “I need a new friend, who should I go to lunch with?” A candidate response template is selected, such as “Good question, I'm [positive emotion] to help. You should go meet [Person A]. [He or she] is [adjective].” The selection of the response template may trigger a query of the Euclidean or cosine similarity distance between the interlocutor's graph embedding and all the other entities in the graph. Such functions (e.g., distance measures) represent certain measures of similarity between concepts. Once the foregoing query is received, the query may be used to fill the template, along with a symbolic query for the emotional disposition of the AI entity towards the interlocutor. If pressed for an explanation for the AI entity's response reason, the AI entity can walk the graph to retrieve and output symbolic similarities between the two individuals in this case, or a simply symbolic realization of the vector calculation in the general case. In this way, although post hoc, this pattern mirrors similar post hoc rationalization of intuitive decisions in human reasoning and demonstrates the possibility of mixing symbolic and sub symbolic vector space reasoning. Other possible queries exist with more extensive symbolic and sub-symbolic functions that may be more tied together”. Note that selection based on similarity is mapped the closest distance matching); and
generating an emovector word based on the second emovector (See Feinson: Figs. 2A-C and Fig. 3, and [0030], “In some embodiments, once an AI system achieves sentience (or a reasonable facsimile thereof), its behavior and thinking processes may be further optimized. As an example, such optimization may include modifying the system to reflect more human-like behaviors, increased learning efficiency, more nuanced emotional qualities, or other aspects”; [0178], “In some embodiments, the vector may be provided as an input to the consumption network, and the consumption network may generate one or more outputs based on the vector. The architecture of the consumption network's input layer (e.g., an embedding input layer) may be configured based on the specific architecture and hyper-parameters of the upstream graph embedding network to enable the consumption network to properly process the embedding vectors. In some embodiments, the vector may be used as weights (e.g., frozen or learnable weights) for other input encodings (e.g., as one hot representation). In some cases, if the vectors are used as learnable weights, the updated vectors may be transferred as a weight vector back to the graph embedding network and further fine-tuned by the graph embedding network, before being transferred back downstream (e.g., to the consumption network or other consumption networks)”; and [0179], “In some embodiments, such a consumption network (that consumes the vectors generated by the upstream embedding network) may include a sequence neural network. As an example, the sequence neural network may be trained to maintain or output a dense vector representation of a conversation history, current state, or other such memory-related data. In one use case, with respect to a conversation between the AI entity and another entity (e.g., a human user, another artificial intelligent entity, etc.), the AI entity may rely on a long short-term memory (LSTM) network (or other sequence neural network) to consume the relevant ontology-affect graph embedding vectors (and/or BERT or other pre-trained word embeddings) representing the other entity's inputs (e.g., words, phrases, sentences, or other input provided by the other entity) to link or build up the vectors into a temporal structure (e.g., along with emotional and sentiment information of the AI entity or the other entity). Based on the conversation history information of the LSTM network, the LSTM network may output a vector representing the conversation state (also referred to as “conversation state vector” herein) (e.g., analogous to a person's memory of a conversation). As an example, the LSTM network may output the conversation state vector in response to the subsequent input provided by the other entity or other automated triggers (e.g., a similar conversation or context, the other entity's name or other identifier being brought up in a subsequent conversation, etc.). This vector can then be stored in the ontology-affect graph as a conversation memory node or consumed by other neural networks”. Note that the conversational tyle output is mapped to the emovector output word).
Regarding claim 19, Miller and Zalewski teach all the features with respect to claim 15 as outlined above. Further, Feinson teaches that the computer-readable storage medium of claim 15, wherein generating the input of the machine learning model comprises:
mapping a first emovector to an embedded space shared with known emovectors of the machine learning model (See Feinson: Figs. 2A-C and Fig. 3, and [0197], “In step 304, the set of affective values of the artificial intelligence entity may be continuously updated based on the growth or decay factors during a time period. It is noted that the set of affective attributes are associated with the set of affective values of the artificial intelligence entity. The affective values (associated with the affective attributes) of the artificial intelligence entity may be continuously updated based on the growth or decay factors associated with the affective attributes. For example, as illustrated in FIGS. 2A-2C, the affective values 201, 202, and 212 (e.g., 207a-207f, 208a-208f, and 218a-218f) may be continuously updated based on one or more growth or decay factors associated with affective attributes A, B, and C. In some embodiments, the continuous updating of the affective values of the artificial intelligence entity may include periodic updating of the affective values of the artificial intelligence entity based on the one more growth or decay factors”. Note that the affective attributes are mapped to the shared embedding space, and the affective values are updated and fed into the AI model);
determining distances between the first emovector and the known emovectors (See Feinson: Figs. 2A-C and Fig. 3, and [0185], “In this way, the AI entity can “think” and learn from its current database of information (e.g., a sufficiently rich graph) without necessarily obtaining that information through new conversations or other sources of information. Such attribute prediction helps facilitate induction of grounding, affect/emotion, ontology, latent factors (inferred new nodes that previously didn't exist), and conversational outputs, etc. As an example, the graph data and structure can be used for graph neural networks that learn information about graph entries from other entries. In some embodiments, the similarities in word association may be used (e.g., by the AI entity or other system) to transfer information from pre-trained word embeddings to the graph, which would then later go into graph embeddings. As such, although pre-trained word embeddings may not encode the graph structure or the types of metadata encoded in the AI entity's full ontology-affect graph, the graph embeddings (generated from the graph or portions thereof) may include such graph structure, metadata, or information other than the information in the word embeddings. As an example, a graph embedding network may be configured to use similarity measures on word embeddings in the graph to retrieve words close to the label of a given node (e.g., within a certain tolerance defined by a hyperparameter or learned weight of the network), or, symbolically walk the graph to find similar entries like those that share a class. Then the AI entity may use either a neural network or similarity measures to find words that are close to known graph nodes, those words can be inserted in the graph and connected to the given node that started the query. In one use case, a search for new connections may be performed where a new word retrieved from a conversational input overlaps, in the pre-trained word embedding space, with the words for visceral or affect data associated with a given node (e.g., the new words are deemed to be similar to the given node based on recognized visceral or affect words). In some embodiments asymmetries across nodes' contexts may indicate knowledge gaps and trigger the AI to acquire new equalizing information. In one use case, a disproportionately asymmetrical context association (such as nodes sharing strong class similarity but one is lacking visceral similarity) shared between nodes, or, an asymmetrical history of method acquisition between similar nodes (node a deduction, node b induction) may trigger the system to attempt through induction/sub-symbolic, deduction/symbolic, or conversational means to equalize the asymmetries. In some embodiments, logically or probabilistically contradicting information within the ontology-affect graph or between the graph and external inputs may also trigger deconfliction mechanisms using such aforementioned knowledge acquisition techniques”; and [0188], “In some embodiments, template variable selection (e.g., a process of filling in predicates, subjects, verbs, and other unknowns in the selected template) may be performed. An example of using learned vectors is as follows: An input to the AI entity may be “I need a new friend, who should I go to lunch with?” A candidate response template is selected, such as “Good question, I'm [positive emotion] to help. You should go meet [Person A]. [He or she] is [adjective].” The selection of the response template may trigger a query of the Euclidean or cosine similarity distance between the interlocutor's graph embedding and all the other entities in the graph. Such functions (e.g., distance measures) represent certain measures of similarity between concepts. Once the foregoing query is received, the query may be used to fill the template, along with a symbolic query for the emotional disposition of the AI entity towards the interlocutor. If pressed for an explanation for the AI entity's response reason, the AI entity can walk the graph to retrieve and output symbolic similarities between the two individuals in this case, or a simply symbolic realization of the vector calculation in the general case. In this way, although post hoc, this pattern mirrors similar post hoc rationalization of intuitive decisions in human reasoning and demonstrates the possibility of mixing symbolic and sub symbolic vector space reasoning. Other possible queries exist with more extensive symbolic and sub-symbolic functions that may be more tied together”. Note that the similarity comparison is measured in distance);
mapping the first emovector to a second emovector of the known emovectors, wherein the second emovector represents a shortest distance of the distances between the first emovector and the known emovectors (See Feinson: Figs. 2A-C and Fig. 3, and [0152], “Our brains designate various elements of knowledge to different levels of importance. The statement “a tiger is dangerous” may be more important than “the grass is green” because, from a typical human prospective, the former statement contains a higher absolute emotional content than the latter statement. Thus, the highest score represents the closest fit for the theme of the conversation, with a degree of novelty and emotional content, and, depending on the certainty levels of the knowledge (explicit or inferred), the response might be “I think that means they can smell well””; and [0188], “In some embodiments, template variable selection (e.g., a process of filling in predicates, subjects, verbs, and other unknowns in the selected template) may be performed. An example of using learned vectors is as follows: An input to the AI entity may be “I need a new friend, who should I go to lunch with?” A candidate response template is selected, such as “Good question, I'm [positive emotion] to help. You should go meet [Person A]. [He or she] is [adjective].” The selection of the response template may trigger a query of the Euclidean or cosine similarity distance between the interlocutor's graph embedding and all the other entities in the graph. Such functions (e.g., distance measures) represent certain measures of similarity between concepts. Once the foregoing query is received, the query may be used to fill the template, along with a symbolic query for the emotional disposition of the AI entity towards the interlocutor. If pressed for an explanation for the AI entity's response reason, the AI entity can walk the graph to retrieve and output symbolic similarities between the two individuals in this case, or a simply symbolic realization of the vector calculation in the general case. In this way, although post hoc, this pattern mirrors similar post hoc rationalization of intuitive decisions in human reasoning and demonstrates the possibility of mixing symbolic and sub symbolic vector space reasoning. Other possible queries exist with more extensive symbolic and sub-symbolic functions that may be more tied together”. Note that selection based on similarity is mapped the closest distance matching); and
generating an emovector word based on the second emovector (See Feinson: Figs. 2A-C and Fig. 3, and [0030], “In some embodiments, once an AI system achieves sentience (or a reasonable facsimile thereof), its behavior and thinking processes may be further optimized. As an example, such optimization may include modifying the system to reflect more human-like behaviors, increased learning efficiency, more nuanced emotional qualities, or other aspects”; [0178], “In some embodiments, the vector may be provided as an input to the consumption network, and the consumption network may generate one or more outputs based on the vector. The architecture of the consumption network's input layer (e.g., an embedding input layer) may be configured based on the specific architecture and hyper-parameters of the upstream graph embedding network to enable the consumption network to properly process the embedding vectors. In some embodiments, the vector may be used as weights (e.g., frozen or learnable weights) for other input encodings (e.g., as one hot representation). In some cases, if the vectors are used as learnable weights, the updated vectors may be transferred as a weight vector back to the graph embedding network and further fine-tuned by the graph embedding network, before being transferred back downstream (e.g., to the consumption network or other consumption networks)”; and [0179], “In some embodiments, such a consumption network (that consumes the vectors generated by the upstream embedding network) may include a sequence neural network. As an example, the sequence neural network may be trained to maintain or output a dense vector representation of a conversation history, current state, or other such memory-related data. In one use case, with respect to a conversation between the AI entity and another entity (e.g., a human user, another artificial intelligent entity, etc.), the AI entity may rely on a long short-term memory (LSTM) network (or other sequence neural network) to consume the relevant ontology-affect graph embedding vectors (and/or BERT or other pre-trained word embeddings) representing the other entity's inputs (e.g., words, phrases, sentences, or other input provided by the other entity) to link or build up the vectors into a temporal structure (e.g., along with emotional and sentiment information of the AI entity or the other entity). Based on the conversation history information of the LSTM network, the LSTM network may output a vector representing the conversation state (also referred to as “conversation state vector” herein) (e.g., analogous to a person's memory of a conversation). As an example, the LSTM network may output the conversation state vector in response to the subsequent input provided by the other entity or other automated triggers (e.g., a similar conversation or context, the other entity's name or other identifier being brought up in a subsequent conversation, etc.). This vector can then be stored in the ontology-affect graph as a conversation memory node or consumed by other neural networks”. Note that the conversational tyle output is mapped to the emovector output word).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights