Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed 05/21/2026 has been entered. Claims 1-20 remain pending in the application.
Information Disclosure Statement
The information disclosure statement submitted on 03/18/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-13, 16-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gelfenbeyn et al. (US 20230351118 A1 hereinafter Gelfenbeyn) in view of el Kaliouby et al. (US 11073899 B2 hereinafter el Kaliouby).
As to independent claim 1, Gelfenbeyn teaches a computer-implemented method, comprising: receiving an original prompt, the original prompt comprising one or more first tokens entered by a user for input to a generative machine learning model; [input to a LLM ¶30-31 "Upon generating the input, the input may be provided to the language model. Upon receiving the input, the language model may predict, based on the input, a response to the message of the user"]
receiving sensor data from one or more sensors that sense the user while entering the original prompt; [voice, camera, sensors from user input ¶44, ¶106]
based at least on the sensor data, generating one or more second tokens [changes input (second tokens) with emotional state ¶28 "prior to sending a request to the LLM, the platform may classify and filter the user questions and messages to change words based on the personalities of AI characters, emotional states of AI characters, emotional states of users, context of a conversation, scene and environment of the conversation, and so forth."]
augmenting a prompt [changes input ¶28]
receiving a response from the generative machine learning model; and [LLM response ¶28 "adjust the response formed by the LLM by changing words and adding fillers based on the personality, role, and emotional state of the AI character. "]
outputting the response for presentation to the user. [present response ¶7 "The client-side computing device may present the response to the user."]
Gelfenbeyn does not specifically teach based at least on the sensor data, generating one or more second tokens that identify an emotional state of the user when entering the original prompt.
However, Kim teaches based at least on the sensor data, generating one or more second tokens that identify an emotional state of the user when entering the original prompt. [detect emotion while user is speaking (entering prompt) col. 3 ln. 1-7 (33) "To improve an interaction with the user, the device may detect a sentiment (e.g., emotion) of a user while speaking to the system or to another person."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the learning models disclosed by Gelfenbeyn by incorporating the based at least on the sensor data, generating one or more second tokens that identify an emotional state of the user when entering the original prompt disclosed by Kim because both techniques address the same field of machine learning and by incorporating Kim into Gelfenbeyn improves user interaction with systems and accuracy of recognition or emotion. [Kim Col. 3 ln. 1-7, ABST]
Gelfenbeyn and Kim do not specifically teach generating an augmented prompt that includes the one or more first tokens of the original prompt entered by the user and the one or more second tokens generated based on the sensor data; and inputting the augmented prompt having the one or more first tokens and the one or more second tokens to the generative machine learning model.
However, Bhardwaj teaches generating an augmented prompt that includes the one or more first tokens of the original prompt entered by the user and the one or more second tokens generated based on the sensor data; [input tokens X augmented with second tokens P to tune the prompt ¶27 " the input tokens X is prepended with a set of learnable soft tokens 103 P={p.sub.1, . . . , p.sub.n}. "]
inputting the augmented prompt having the one or more first tokens and the one or more second tokens to the generative machine learning model; [concatenated prompt as input to the PLM ¶28, ¶68 "the input text (e.g., 105 in FIG. 1) are concatenated with the final soft prompt tokens (e.g., 117 in FIG. 1) to form a model input."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the learning models disclosed by Gelfenbeyn and Kim by incorporating the generating an augmented prompt that includes the one or more first tokens of the original prompt entered by the user and the one or more second tokens generated based on the sensor data; and inputting the augmented prompt having the one or more first tokens and the one or more second tokens to the generative machine learning model disclosed by Bhardwaj because all techniques address the same field of automation and by incorporating Bhardwaj into Gelfenbeyn and Kim provides a more efficient conditioning of language models [Bhardwaj ¶4 and ¶19]
As to dependent claim 2, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 1 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the sensor data includes audio data, video data, physiological data, and cognitive data. [Gelfenbeyn audion, visuals ¶58, biometric, camera ¶106]
As to dependent claim 3, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 1 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach further comprising:
inputting the sensor data into a machine-learning classifier; [Kim estimator (classifier) Fig. 8 850 receives sensor data col. 24 ln. 18-40]
receiving an emotion category output by the machine-learning classifier in response to the sensor data; and [Kim sentiment categories Col. 6 ln. 30-40 "sentiment categories may include angry, sad, happy, surprised, and/or disgust without departing from the disclosure"]
mapping the emotion category output by the machine-learning classifier to the one or more second tokens. [Gelfenbeyn changes input (second tokens) with emotional state ¶28 ]
As to dependent claim 4, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 1 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the emotional state includes one or more emotion categories. [Kim sentiment categories Col. 6 ln. 30-40 "sentiment categories may include angry, sad, happy, surprised, and/or disgust without departing from the disclosure"]
As to dependent claim 5, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 4 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the emotional state is represented as an emotion vector that indicates magnitudes of the emotion categories. [Kim vector data and sentiment score (magnitudes) of categories Col. 36 ln. 8-28]
As to dependent claim 6, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 1 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach further comprising:
sending the sensor data to an emotion service; and [Kim estimator (classifier) Fig. 8 850 receives sensor data col. 24 ln. 18-40]
receiving the emotional state from the emotion service. [Kim generates sentiment Col. 40 ln. 14-32]
As to dependent claim 7, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 1 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the one or more second tokens include a word, a highlight, a punctuation mark, an emoji, metadata, and/or an emotion vector. [Kim text, Col. 8 ln. 2-11, vector data and sentiment score (magnitudes) Col. 36 ln. 8-28, metadata Col. 19-20 ln. 59-10]
Regarding claims 9 and 16, claims 9 and 16 are machine-readable storage medium claims and system claims respectively that correspond to the method of claim 1. Therefore, claims 9 and 16 are rejected for at least the same reasons as method of claim 1. See Gelfenbeyn memory processor and instructions ¶6] claims 10-12 sensing modalities [Gelfenbeyn voice, camera, sensors from user input ¶44, ¶106]
As to dependent claim 10, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 9 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the sensor data includes a plurality of sensing modalities. [Gelfenbeyn voice, camera, sensors from user input ¶44, ¶106]
As to dependent claim 11, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 10 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach emotional state includes a plurality of emotion categories associated with the plurality of sensing modalities. [Kim sentiment categories Col. 6 ln. 30-40 "sentiment categories may include angry, sad, happy, surprised, and/or disgust without departing from the disclosure"]
As to dependent claim 12, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 10 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach emotional state is represented as a plurality of emotion vectors associated with the plurality of sensing modalities. [Kim vector data and sentiment score (magnitudes) of categories Col. 36 ln. 8-28]
As to dependent claim 13, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 12 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach send the sensor data to an emotion service; and [Kim estimator (classifier) Fig. 8 850 receives sensor data col. 24 ln. 18-40]
receive the emotional state from the emotion service. [Kim generates sentiment Col. 40 ln. 14-32]
As to dependent claim 17, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 16 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the instructions further cause the processor to: associate the one or more second tokens that identify the emotional state with a specific first token representing a specific word from the original prompt.
As to dependent claim 19, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 16 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the one or more second tokens include at least two different numbers representing magnitudes of at least two different emotion categories. [Kim text, Col. 8 ln. 2-11, vector data and sentiment score (magnitudes) Col. 36 ln. 8-28, metadata Col. 19-20 ln. 59-10]
As to dependent claim 20, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 16 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach wherein the sensor data includes first sensor data from a contact sensor that was in contact with the user when entering the original prompt. [Gelfenbeyn fingerprint, biometric ¶106]
Claims 8 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Gelfenbeyn in view of Manfredi, as applied in the rejection of claim 1 above, and further in view of Han et al. (US 11138970 B1) hereinafter Han.
As to dependent claim 8, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 1 above that is incorporated,
Gelfenbeyn, Kim and Bhardwaj do not specifically teach selecting a particular position in which to insert the one or more second tokens in the augmented prompt based at least on timestamps associated with the sensor data.
However, Han teaches selecting a particular position in which to insert the one or more second tokens in the augmented prompt based at least on timestamps associated with the sensor data. [timestamps at the word-level at the start Col. 3 ln. 45-59 "word-level timestamps are created for the redacted words and the modified audio recording. Word-level timestamps include a start time and an end time that indicate when the audio recording commences and completes playing the word"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the learning models disclosed by Gelfenbeyn, Kim and Bhardwaj by incorporating the selecting a particular position in which to insert the one or more second tokens in the augmented prompt based at least on timestamps associated with the sensor data disclosed by Han because all techniques address the same field of automation and by incorporating Han into Gelfenbeyn, Kim and Bhardwaj protect the privacy of confidential information in machine learning with more security [Han Col. 3 ln. 4-17]
As to dependent claim 14, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 1 above that is incorporated,
Gelfenbeyn, Kim and Bhardwaj do not specifically teach wherein the instructions, when executed by the processor, further cause the system to: associate the one or more first tokens with one or more first timestamps; associate the sensor data with one or more second timestamps; and associate the emotional state with one or more third timestamps.
However, Han teaches wherein the instructions, when executed by the processor, further cause the system to: associate the one or more first tokens with one or more first timestamps; associate the sensor data with one or more second timestamps; and associate the emotional state with one or more third timestamps. [timestamps at the word-level at the start Col. 3 ln. 45-59 "word-level timestamps are created for the redacted words and the modified audio recording. Word-level timestamps include a start time and an end time that indicate when the audio recording commences and completes playing the word"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the learning models disclosed by Gelfenbeyn, Kim and Bhardwaj by incorporating the wherein the instructions, when executed by the processor, further cause the system to: associate the one or more first tokens with one or more first timestamps; associate the sensor data with one or more second timestamps; and associate the emotional state with one or more third timestamps disclosed by Han because all techniques address the same field of automation and by incorporating Han into Gelfenbeyn, Kim and Bhardwaj protect the privacy of confidential information in machine learning with more security [Han Col. 3 ln. 4-17]
As to dependent claim 15, Gelfenbeyn, Kim, Bhardwaj and Han teach the method of claim 14 above that is incorporated, Gelfenbeyn, Kim, Bhardwaj and Han further teach wherein the instructions, when executed by the processor, further cause the system to select a particular position in the augmented prompt in which to insert the one or more second tokens among the original tokens based at least on the one or more first timestamps, the one or more second timestamps, and the one or more third timestamps. [Han Fig. 3B illustrates timestamps to start from selectable to insert tokens Col. 4-5 ln. 54-36]
Claims 18 is rejected under 35 U.S.C. 103 as being unpatentable over Gelfenbeyn in view of Manfredi, as applied in the rejection of claim 16 above, and further in view of Woloshyn (US 9118773 B2)
As to dependent claim 18, Gelfenbeyn, Kim and Bhardwaj teach the method of claim 16 above that is incorporated, Gelfenbeyn, Kim and Bhardwaj further teach a sequence of augmented prompts. [Gelfenbeyn sequential dialog and actions input to AI (prompts are AI input) ¶74, ¶68 "generative models configured to follow sequential instructions for dialog and actions that are driven by a specific purpose or intent for AI-driven characters. FIG. 7A shows possible user inputs 702 and input impact for goals model 704"]
Gelfenbeyn, Kim and Bhardwaj do not specifically teach wherein the instructions further cause the processor to send a sequence of augmented prompts to the generative machine learning model at regular intervals.
However, Woloshyn teaches wherein the instructions further cause the processor to send a sequence of augmented prompts to the generative machine learning model at regular intervals. [auto initiates automation system in regular intervals Col. 31 ln. 39-47 "Automated Notation System procedures may be initiated and/or implemented manually, automatically, statically, dynamically, concurrently, and/or combinations thereof. Additionally, different instances and/or embodiments of the Automated Notation System procedures may be initiated at one or more different time intervals (e.g., during a specific time interval, at regular periodic intervals, at irregular periodic intervals, upon demand, etc.)."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the learning models disclosed by Gelfenbeyn, Kim and Bhardwaj by incorporating the wherein the instructions further cause the processor to send a sequence of augmented prompts to the generative machine learning model at regular intervals disclosed by Woloshyn because all techniques address the same field of automation and by incorporating Woloshyn into Gelfenbeyn, Kim and Bhardwaj provides easier wats to record and personalize notes on conversations [Woloshyn Col. 5 ln. 50-63]
Response to Arguments
Applicant's arguments filed 05/21/2026, with respect to 101, these rejections have been withdrawn.
Applicant's arguments filed 05/21/2026. In the remark, applicant argues that:
(1) Gelfenbeyn and el Kaliouby fail to teach "generating an augmented prompt that includes the one or more first tokens of the original prompt entered by the user and the one or more second tokens generated based on the sensor data;" as recited by amended claim 1.
As to point (1), Applicant’s arguments with respect to claim 1 have been considered but are moot in view of a new ground of rejection as set forth above of Gelfenbeyn in view of Kim and Bhardwaj.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
Hinkle et al. (US 10680995 B1) teaches emotive information and metadata (see Col. 3-4 ln. 59-25)
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEAU SPRATT whose telephone number is (571)272-9919. The examiner can normally be reached M-F 8:30-5 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on 5712127212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEAU D SPRATT/ Primary Examiner, Art Unit 2143
Prosecution Insights