Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4/20/2026 has been entered.
Response to Amendment
The amendment filed on 4/20/2026 has been entered and made of record. Claims 1, 7, 13, 16, 19, 21 and 29-30 are amended. Claims 2 and 22 are cancelled. Claims 1, 3-21 and 23-30 are pending.
Response to Arguments
Applicant’s arguments with respect to the rejections of independent claims 1, 21 and 29-30 have been fully considered but they are moot because the arguments do not apply to the references being used in the current rejection.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“means for processing image data… means for processing audio data…. means for generating feature data… means for modifying the face data…” in claim 30.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8, 10-21, and 23-30 are rejected under 35 U.S.C. 103 as being unpatentable over Tsou (US 2018/0176168) in view of Jiang et al. (CN107369449A) and Bendale et al. (US 2021/0201549), further in view of Shin et al. (US 2020/0090393 A1) and Scarasso et al. (US 9,761,222 B1).
As to Claim 1, Tsou teaches A device comprising: a memory configured to store image data; and one or more processors configured to:
process the image data corresponding to a user's face to generate face data (Tsou discloses “As another example, the initialization process may include capturing images of the recipient 132 experiencing various emotional states. As an additional example, the initialization process may include the recipient 132 identifying a new emotional state, identifiers of the new emotional state, and/or providing a visual image associated with the new emotional state” in [0027]; “an image may be captured of the facial expression of the recipient of the electronic message depicting the real time response of the recipient” in [0037]);
process audio data associated with a second participant engaged in a conversation with the user (Tsou discloses “The system 100 may include an electronic device 110 via which a user may generate a message to be sent out over a network 120 to one or more recipients 132…The outgoing message may take any form. For example, the outgoing message may include text, images, videos, audio, etc.” in [0023]; “In these and other embodiments, the analysis of the image and/or video may be combined with voice and/or speech analysis for factors such as loudness, sentiment, tone, and/or silent time” in [0039]).
Tsou is silent on background noise. The combination of Jiang further teaches following limitations:
wherein the audio data includes user speech and background noise of an environment (Jiang discloses “However, simple recording will record background noise, environmental noise, echo, etc. at the same time, and it is inevitable that non-real voice will also be recorded” in [0004]; “Through the effective speech recognition device provided by the embodiment of the present invention, while the recording device records the speech data of the sound source object, the facial image data of the sound source object is obtained through the camera device. Combined with the open mouth image, the effective speech in the ASR recognition result of the speech data is identified, and the background noise, environmental noise, and speech content of non-sound source objects in the ASR recognition result can be accurately filtered out, effectively improving the application value of the ASR recognition result.” in [0064]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tsou with the teaching of Jiang so as to filter the background noise to accurately identify the speech content from the collected sound source data (Jiang, [0064]).
Tsou and Jiang don’t directly use claim language semantic context. The combination of Bendale further teaches following limitations:
determine a semantical context based on a type of relationship between the user and the second participant, the type of relationship determined based on a contact list stored on the device (Tsou discloses “the device 110 may predict an expected emotional state of the recipient 132 based on the content of the outgoing message… For example, certain words, phrases, images, videos, the relationship of the user and the recipient 132, etc. may affect the predicted emotional state of the recipient 132. As additional examples, factors such as a ratio of positive to negative words, the voice and tone of the speech (for video), the response time or times of silence (for video), facial expression (for images or video), the topic and reason of the communication (e.g., the context in which the communication occurs such as personal, school, customer service, business inquiry, compliant), etc.” in [0034];” For example, one API may facilitate the identification of a limited number of potential emotional states (e.g., angry or calm, sad or happy, excited or bored, etc.), associated facial expressions with the emotional states, and/or visual images associated with the facial expressions” in [0036]; “the notification setup region 530 may include a recipient watch list field 536 such that certain recipients may be particularly alerted, such as those previously targeted for cyberbullying, etc.” in [0096]; “For example, a school administrator or system administrator may enter information into various fields to set various parameters associated with a user and/or a recipient profile associated with the school, business, or other organization. Such fields may include a default score field 542, a special relationships field 544, any of the pieces of information identified in FIG. 4, etc… The special relationships field 544 may set various relationships among various users and/or recipients. For example, a user may have one or more close friends that they frequently send messages to. As another example, a user's parents or siblings may be identified in the special relationships field.” in [0097]; see also user relationship in [0043], identifying the context of any keywords in [0062]. Bendale further discloses “In particular embodiments, analysis may be performed on text input, audio input, expression input, and video inputs to identify characteristics of a particular semantic context… In particular embodiments, each of the plurality of semantic contexts may be indicative of an expression” in [0030]);
generate feature data based on the processed audio data, the processed image data, and the semantical context (Bendale discloses “Features 602 may be extracted from input data 502 to be fed into a machine-learning model 506. In particular embodiments, video features 602a are extracted from video input 504a and text features 602c are extracted from text input 504c to be fed into a text to video learning model 506b. In particular embodiments, video features 602a are extracted from video input 504a and audio frequency features 602b are extracted from audio input 504b to be fed into an audio to video learning model 506d. In particular embodiments, audio frequency features 602b are extracted from audio input 504b and expressions features 602d are extracted from expressions input 504d to be fed into audio to expression learning model 506c.” in [0051].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tsou and Jiang with the teaching of Bendale so as to perform an action on a digital avatar based on the one or more identified semantic context (Bendale, [0032]).
Tsou, Jiang and Bendale don’t teach modifying facial feature data. The combination of Shin further teaches following limitations:
modify the face data based on the feature data to generate adjusted face data, wherein the adjusted face data corresponds to an avatar facial expression that is based on the semantical context, and wherein, to modify the face data, the one or more processors are configured to: modify the face data based on the feature data; merge the face data with facial expression data corresponding to the feature data; combine an encoded representation of the face data with the feature data to generate the adjusted face data; or a combination thereof (Shin discloses “According to the embodiment, the robot 100 may generate an avatar character by synthesizing a facial expression landmark point image generated in correspondence with recognized emotion information on the face image data of the user as augmented reality” in [0210]; “Alternatively, the robot 100 may first generate the animation character based on face information of the user. Such an animation character may also be generated by reflecting the detected facial expression landmark points of the user. For example, in the example of a user having a large nose, animation character having a large nose may be created. Additionally, the robot 100 may change the facial expression landmark points of the generated animation character to correspond to the recognized emotion information, thereby generating an avatar character expressing the specific emotion of the user” in [0211]; “Alternatively, the robot 100 may generate the avatar character by changing facial expression landmark points of the preset animation character to correspond to the recognized emotion information” in [0212], see also [0185]);
generate a visual representation of an avatar based on the adjusted face data (Tsou, [0040, 0044]. Bendale, [0032-0033]. Shin, [0210-0212].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tsou, Jiang and Bendale with the teaching of Shin so as to quickly and easily generate an avatar character by reflecting user characteristic for recognizing only the facial expression landmark points in the previously generated animation character (Shin, [0212]).
In response to the new limitation the semantical context used to predict a type of the conversation between the user and the second participant, Tsou discloses “The banned words field 521 may trigger an automatic block of a message and the alert words field 525 may trigger an alert if the word is used in a message… such that if an image of the banned/alert category is included in an outgoing message, the outgoing message may be banned or trigger an alert” in [0095]. Scarasso further discloses “The Conversation-Goal Qualifier 140 may use deep learning neural network algorithms designed to extract contextual patterns from the messages in the conversation, users participating in the conversation, and external events from unclassified conversation” in C6L49-53; “In at least one implementation, parsing module 250 communicates the created pattern to predictive module 260. Predictive module 260 can use a predictive model to match the created pattern to one or more patterns associated with known conversation types. Predictive module 260 can then identify within a certain percentage probability that there are one or more potential conversation types” in C7L42-48; “Additionally, FIG. 5 shows that the method may include an act 520 of calculating probabilities of conversation type. Act 520 can comprise, based upon the user information and one or more words communicated in the conversation” in C9L14-17. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tsou, Jiang, Bendale and Shin with the teaching of Scarasso so as to utilize a combination of natural language processing and predictive models to estimate the relative likelihood of the possibilities of a conversation belonging to a conversation type (Scarasso, C5L60-67).
As to Claim 3, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, wherein the one or more processors are configured to generate the visual representation of the avatar based on motion data (Bendale discloses “In particular embodiments, the video output may comprise a rendering of a sequence of actions to be performed by the digital avatar” in [0033].)
As to Claim 4, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 3, further comprising one or more motion sensors configured to capture the motion data (Tsou discloses a location sensor may include a GPS sensor and a motion sensor in [0056].)
As to Claim 5, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 4, wherein the motion data is associated with movement of a manned or unmanned vehicle (Tsou discloses a location sensor may include a GPS sensor and a motion sensor in [0056]. It is well-known that a vehicle may include motion sensor (i.e. accelerometer, gyroscopes and GPS).)
As to Claim 6, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 3, wherein the motion data enables the generation of the adjusted face data (Tsou, [0040, 0044]. Bendale, [0032-0033]. Shin, [0210-0212].)
As to Claim 7, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 6, wherein the avatar facial expression is one of: surprise, fear, joy, startled, relaxed, or excitement (Tsou discloses “potential emotional states (e.g., angry or calm, sad or happy, excited or bored, etc.)” in [0036], see also [0039]. Shin, Fig 7.)
As to Claim 8, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, wherein a neural network is used to generate the adjusted face data (Tsou discloses “In some embodiments, analysis to determine an emotional state may be based on statistical research or machine learning models and can be performed by building a database with algorithms to determine the emotional state” in [0035]. Bendale also discloses “For example, the deep learning algorithms 1418 may include ANNs, such as a multilayer perceptron (MLP), an autoencoder (AE), a convolution neural network (CNN), a recurrent neural network (RNN)…” in [0073]. Shin, [0178, 0203].)
As to Claim 10, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, wherein the processing further includes using the face data in conjunction with the audio data to disambiguate the user speech from the background noise, and the one or more processors are configured to predict facial expressions of the user based on the semantical context and the face data (Jiang discloses “However, simple recording will record background noise, environmental noise, echo, etc. at the same time, and it is inevitable that non-real voice will also be recorded.” in [0004]; “Through the effective speech recognition device provided by the embodiment of the present invention, while the recording device records the speech data of the sound source object, the facial image data of the sound source object is obtained through the camera device. Combined with the open mouth image, the effective speech in the ASR recognition result of the speech data is identified, and the background noise, environmental noise, and speech content of non-sound source objects in the ASR recognition result can be accurately filtered out, effectively improving the application value of the ASR recognition result.” in [0064]. Bendale further discloses “In particular embodiments, analysis may be performed on text input, audio input, expression input, and video inputs to identify characteristics of a particular semantic context… In particular embodiments, each of the plurality of semantic contexts may be indicative of an expression” in [0030].)
As to Claim 11, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, wherein the one or more processors are configured to generate the adjusted face data based on motion data (Shin discloses “Alternatively, the robot 100 may first generate the animation character based on face information of the user. Such an animation character may also be generated by reflecting the detected facial expression landmark points of the user. For example, in the example of a user having a large nose, animation character having a large nose may be created. Additionally, the robot 100 may change the facial expression landmark points of the generated animation character to correspond to the recognized emotion information, thereby generating an avatar character expressing the specific emotion of the user.” in [0211].)
As to Claim 12, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1 further comprising a camera configured to track body movements of the user that provide context for the adjusted face data (Tsou discloses “the recipient 132 may grant the system 100 permission to capture or cause the capture of the real time response of the recipient 132 to an electronic message” in [0043], see also [0034, 0058].)
As to Claim 13, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, wherein the one or more processors are configured to select, based on the predicted type of the conversation, a set of parameters that constrains a range of avatar facial expressions used to generate the visual representation of the avatar (Tsou discloses “the visual feedback system may… determine a potential mood or emotional response of the recipient of the electronic message. The visual feedback system may provide the sender with a visual cue as to the likely emotional response of the recipient, such as a visual image of the recipient experiencing the emotional response. Using such a system, the sender of the electronic message may be more cognizant of the effect of the message being sent as the sender is able to visually observe the potential emotional response that may be caused by the electronic message they are sending” in [0021]; “In some embodiments, the system 100 may operate to determine a potential emotional state of the recipient 132 of the message and/or the target of a cyberbullying message. For example, based on the cyberbullying risk score, the software 112 may determine the potential emotional state of the recipient 132. In these and other embodiments, based on the emotional state, a corresponding visual image may be presented to the sender” in [0025]; “providing visual images for a set of known emotional states” in [0027]; “the context in which the user is generating the outgoing message may affect the cyberbullying risk score” in [0033]; “In some embodiments, analysis to determine an emotional state may be based on … machine learning models and can be performed by building a database with algorithms to determine the emotional state” in [0035]. Scarasso further discloses “Predictive module 260 can use a predictive model to match the created pattern to one or more patterns associated with known conversation types” in C7L44-46.)
As to Claim 14, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, further comprising one or more microphones configured to generate the audio data (Bendale, [0067].)
As to Claim 15, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, further comprising one or more cameras configured to generate the image data (Tsou, [0043].)
As to Claim 16, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, further comprising one or more speakers configured to play out an audio output (Bendale, [0067].)
As to Claim 17, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, further comprising a display device configured to display the visual representation of the avatar (Tsou discloses display 270 in Fig 2.)
As to Claim 18, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, further comprising a modem, wherein the image data is received from a second device via the modem (Tsou discloses “the communication component 240 may include a modem” in [0055].)
As to Claim 19, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, wherein the one or more processors are configured to send the visual representation of the avatar, an audio output, or both, to a second device (Tsou discloses “At block 1050, the captured real time response may be transmitted by the visual feedback system to the second user” in [0150].)
As to Claim 20, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 1, wherein the one or more processors are integrated in an extended reality device (Tsou, Fig 2.)
Claim 21 recites similar limitations as claim 1 but in a method form. Therefore, the same rationale used for claim 1 is applied.
Claim 23 is rejected based upon similar rationale as Claim 3.
Claim 24 is rejected based upon similar rationale as Claim 5.
Claim 25 is rejected based upon similar rationale as Claims 6.
Claim 26 is rejected based upon similar rationale as Claim 8.
Claim 27 is rejected based upon similar rationale as Claim 13.
Claim 28 is rejected based upon similar rationale as Claim 12.
Claim 29 recites similar limitations as claim 1 but in a computer readable medium form. Therefore, the same rationale used for claim 1 is applied.
Claim 30 recites similar limitations as claim 1 but in an apparatus form. Therefore, the same rationale used for claim 1 is applied.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Tsou in view of Jiang and Bendale, further in view of Shin, Scarasso and Phan (US 2022/0398795).
As to Claim 9, Tsou in view of Jiang, Bendale, Shin and Scarasso teaches The device of claim 8. The combination of Phan further teaches wherein the neural network is a variational autoencoder (Bendale discloses “For example, the deep learning algorithms 1418 may include ANNs, such as a multilayer perceptron (MLP), an autoencoder (AE), a convolution neural network (CNN), a recurrent neural network (RNN)…” in [0073]. Phan further discloses “Similarly, in some embodiments a machine learning model may be trained to reconstruct an input expression given labeled positions of portions of an expression (e.g., facial features). For example, the positions may be provided as conditions to a conditional variational autoencoder. In this way, the conditional variational autoencoder may learn to associate positions of facial features with specific expressions” in [0016].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Tsou, Jiang, Bendale, Shin and Scarasso with the teaching of Phan so that the variational autoencoder may learn to associate positions of facial features with specific expressions and modify the expression via adjusting positions of facial features (Phan, [0016]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIMING HE whose telephone number is (571)270-1221. The examiner can normally be reached on Monday-Friday, 8:30am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEIMING HE/
Primary Examiner, Art Unit 2611
Prosecution Insights