Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 8 and 14 first recite “receiving, by the conversational artificial intelligence engine, an output of the large language model, wherein the output comprises text and gestures for the embodied interactive agent”, and the claims later recite “generating, by the conversational artificial intelligence engine, animations for the embodied interactive agent from the gestures and speech for the embodied interactive agent based on the text”. It is unclear whether the gestures are already included in the large language model output or alternatively are generated from the text at a later step. Additionally, the relationship between the recited text, speech, gestures, and generated animations is ambiguous because the claim does not clearly specify the source of the gestures or how the gestures are derived.
Claims 4, 11 and 17 recite “generates a text prompt for the large language model based on the user utterance and an image prompt for a visual language model, and the large language model and the visual language model return outputs” is unclear because the relationship between the recited prompt and models cannot be reasonably determined. It is ambiguous whether the claim requires generating a text prompt for the large language model and separately generating an image promote for the visual language model, or alternatively whether the text prompt itself is generated based on both the user utterance and the image prompt. Additionally, the claim fails to specify which model receives which prompt, whether the models operate independently or sequentially, and what output is returned by each model.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 7-10, 12, 14-16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Khorshid (US 2024/0045704) in view of Seo et al. (End-to-end Generative Pretraining for Multimodal Video Captioning, Computer Vision and Pattern Recognition, 2022) in view of Ramesh et al. (Walk-the-Talk: LLM driven pedestrian motion generation, IEEE, June 2024) in view of Windle et al. (Llanimation: Llama Driven Gesture Animation, Human-Computer Interaction, May, 2024).
Regarding claim 1, Khorshid discloses a method (Khorshid, [0154], “FIG. 11 illustrates an example method 1100 for morphing a virtual assistant avatar”), comprising:
receiving, by a conversational artificial intelligence engine (Khorshid, [0007], “provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0096], “the dialog manager 216 may implement reinforcement learning frameworks to improve the dialog optimization”) and from an augmented reality headset worn by a user (Khorshid, [0034], “augmented-reality (AR) smart glasses, virtual-reality (VR) headset”), a query that is made to an embodied interactive agent that is displayed in a display of the augmented reality headset (Khorshid, [0141], “the AR/VR system 130 may render the XR assistant avatar 920 in the user's home so that the user 910 may request assistance naturally from the XR assistant avatar 920…The user 910 may ask, via a voice command, “assistant, do you think the chair I just checked would fit my living room? 925” As the AR/VR system 130 detects a change of context”), wherein the query comprises audio of a user utterance (Khorshid, [0141], “the first or second form of the XR assistant avatar may be based on one or more of voice, speech”) and images or video captured by a camera of the augmented reality headset of what the user is seeing (Khorshid, [0131], “The external-facing cameras can capture images and videos of the real-world environment”);
generating, an animation for the embodied interactive agent from speech for the embodied interactive agent based on the audio (Khorshid, [0007], “provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0141], “The user 910 may ask, via a voice command, “assistant, do you think the chair I just checked would fit my living room? 925” As the AR/VR system 130 detects a change of context… Firstly, the XR assistant avatar 920 now is rendered as solid (not semi-opaque anymore). Secondly, the XR assistant avatar 920 is rendered as sitting in the user's 910 couch in the living room so that the user 910 may see how the chair looks like compared to a real person if the AR/VR system 130 later renders the chair near the couch”);
receiving, output comprises text and gestures for the embodied interactive agent (Khorshid, [0007], “the assistant system may provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0141], “the first or second form of the XR assistant avatar may be based on one or more of voice, speech, emotion, tone, pitch, appearance, size, shape, clothing, orientation, position, depth, movement, gesture, facial expression, color, shading, outline, brightness, luminescence, or transparency”);
outputting, by the conversational artificial intelligence engine, the animations and the speech to the augmented reality headset (Khorshid, [0007], “provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0154], “the AR/VR system 130 may determine a second form and a second pose based on a personalization model and the one or more user sentiments, wherein the personalization model is a machine-learning model trained on a plurality of prior reactions associated with the first user with respect to a plurality of morphing of the XR assistant avatar”. Fig. 9B);
Khorshid fails to teach “generating a prompt for a large language model based on the user utterance and the images or video”;
Seo et al. (hereinafter Seo) discloses generating a prompt based on utterance and images or video (Seo, Fig 2 illustrates given input frames and a future utterance, predict the current utterances. Both losses are applied to a triplet consisting of video frames, present utterances and a future utterance. Fig. 3 illustrates generated sentences).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the assistant system of Khorshid to incorporate the concept of the sentence generation function based on input frames, as taught by Seo. The motivation for doing so would have been enabling the virtual assistant to generate scene-aware conversational responses.
Khorshid as modified Seo does not expressly disclose “the prompt for a large language model”;
Ramesh et al. (hereinafter Ramesh) discloses a prompt for a large language model (Ramesh, III Data and Methodology, [0001], “Training a large language model for a specific task such as motion generation, motion captioning, etc. using prompt templates”. In addition, in section B. Pedestrain Motion Generation, [0003], “utilize a T5 language model [22] and pre-train it initially with our motion and language tokens generated by our tokenizer”);
providing, the prompt to the large language model (Ramesh, III Data and Methodology, [0001], “Training a large language model for a specific task such as motion generation, motion captioning, etc. using prompt templates”);
receiving, an output of the large language model for a virtual object (Ramesh, Fig. 2);
generating, animations for the virtual object based on the text (Ramesh, Fig. 2).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant avatar of Khorshid to incorporate the motion generator utilizing Large Language Models as taught by Ramesh. The motivation for doing so would have been improving realism of avatar behavior.
Though Khorshid teaches gestures and speech for the embodied interactive agent (Khorshid, [0141], “the first or second form of the XR assistant avatar may be based on one or more of voice, speech, emotion, tone, pitch, appearance, size, shape, clothing, orientation, position, depth, movement, gesture, facial expression, color, shading, outline, brightness, luminescence, or transparency”); Khorshid as modified by Seo and Ramesh does not expressly disclose “the gestures for a virtual object based on text”;
Windle et al. (hereinafter Windle) discloses gestures for a virtual object based on text (Windle, Fig. 1 illustrates generating gestures from speech using Llama2 features derived from text as the primary input, producing temporally aligned and contextually accurate gesture animation).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant animation of Khorshid as modified by Seo and Ramesh to incorporate the gesture animation generation from text, as taught Windle. The motivation for doing so would have been enhancing multimodal communication and realism of user interaction.
Regarding claim 2, Khorshid discloses receiving, by the conversational artificial intelligence engine, user location information from the augmented reality headset (Khorshid, [0005], “The assistant system may enable the user to interact with the assistant system via user inputs of various modalities (e.g., audio, voice, text, image, video, gesture, motion, location, orientation)”), action based on the user location information (Khorshid, [0116], “The location updates may be consumed by the dialog manager 216 to support various proactive/reactive scenarios”).
Khorshid as modified by Seo with the same motivation from claim 1 discloses the prompt (Seo, Fig 2 illustrates given input frames and a future utterance, predict the current utterances.).
Regarding claim 3, Khorshid discloses inferring, by the conversational artificial intelligence engine, a task goal associated with the query (Khorshid, [0109], “The content determination component may determine the communication content based on the knowledge source, communicative goal, and the user's expectations”. In addition, in paragraph [0113], “processing a user input…An assistant task may be a central concept that is shared across the whole assistant stack to understand user intention, interact with the user and the world to complete the right task for the user”), wherein the inference is based on a user interaction history (Khorshid, [0096], “the dialog intent resolution 356 may resolve the user intent associated with the current dialog session based on dialog history between the user and the assistant system 140”), environment object labels (Khorshid, [0080], “recognize interesting objects in the world through a combination of existing machine-learning models”), and user location information (Khorshid, [0082], “the capability of signals intelligence may enable the assistant system 140 to, for example, determine user location”).
Regarding claim 5, Khorshid as modified by Seo and Ramesh with the same motivation from claim 1 discloses the large language model comprises a multi-modal large language model (Ramesh, B. Pedestrian Motion Generation, [0003], “Being a unified, multi-modal architecture, our model now should be able to treat both motion and text tokens as language”).
Regarding claim 7, Khorshid discloses the display in the augmented reality headset displays the animations for the embodied interactive agent (Khorshid, Figs. 9A-9B), and a speaker in the augmented reality headset outputs the speech for the embodied interactive agent (Khorshid, [0007], “the assistant system may provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”).
Regarding claim 8, Khorshid discloses a system (Khorshid, [0005], “the assistant system may assist a user to obtain information or services”), comprising:
an augmented reality headset (Khorshid, [0034], “smart glasses, augmented-reality (AR) smart glasses, virtual-reality (VR) headset”) comprising a camera (Khorshid, [0034], “camera”), a microphone (Khorshid, [0037], “a microphone of the AR/VR system”), a display (Khorshid, [0038], “an AR/VR system 130 may include an AR/VR display device”), and a speaker (Khorshid, [0199], “speaker”), wherein the augmented reality headset is configured to be worn by a user (Khorshid, Fig. 5); and
a multi-modal conversational platform (Khorshid, [0005], “the assistant system may support mono-modal inputs (e.g., only voice inputs), multi-modal inputs (e.g., voice inputs and text inputs), hybrid/multi-modal inputs”).
The remaining limitations recite in claim 8 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claims 9-10, claims 9-10 recite functions that are similar in scope to the method steps recited in claims 2-3 and therefore are rejected under the same rationale.
Regarding claim 12, claim 12 recites function that is similar in scope to the method step recited in claim 5 and therefore is rejected under the same rationale.
Regarding claim 14, Khorshid discloses a non-transitory computer readable storage medium, including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps (Khorshid, [0196], “processor 1502 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 1502 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1504, or storage 1506”. In addition, in paragraph [0202], “a computer-readable non-transitory storage medium or media may include one or more semiconductor-based”).
The limitations recite in claim 14 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claims 15-16, claims 15-16 recite instructions that are similar in scope to the method steps recited in claims 2-3 and therefore are rejected under the same rationale.
Regarding claim 18, claim 18 recites instruction that is similar in scope to the method step recited in claim 5 and therefore is rejected under the same rationale.
Regarding claim 20, claim 20 recites instructions that are similar in scope to the method steps recited in claim 7 and therefore are rejected under the same rationale.
Claims 4, 11 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Khorshid (US 2024/0045704) in view of Seo et al. in view of Ramesh et al. in view of Windle et al., as applied to claims 1, 8 and 14, in further view of Alayrac et al. (Flamingo: a Visual Language Model for Few-Shot Learning, Computer Vision and Pattern Recognition, 2022).
Regarding claim 4, Khorshid discloses generates a text prompt based on the user utterance (Khorshid, [0055], “The ASR module 208a may allow a user to dictate and have speech transcribed as written text”);
Khorshid as modified by Seo and Ramesh with the same motivation from claim 1 discloses the text prompt for the large language model (Ramesh, III Data and Methodology, [0001], “Training a large language model for a specific task such as motion generation, motion captioning, etc. using prompt templates”), and the large language model returns output (Ramesh, Fig. 2);
Khorshid as modified by Seo, Ramesh and Windle does not expressly disclose “an image prompt for a visual language model”;
Alayrac et al. (hereinafter Alayrac) discloses an image prompt for a visual language model (Alayrac, 1 Introduction, [0001], “a Visual Language Model (VLM) that sets a new state of the art in few-shot learning on a wide range of open-ended vision and language tasks, simply by being prompted with a few input/output examples, as illustrated in Figure 1”. Fig. 1 illustrates image prompt);
the visual language model returns output (Alayrac, Fig. 1).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant system based on large language model of Khorshid as modified by Seo and Ramesh to incorporate image prompts for a visual language model as taught by Alayrac. The motivation for doing so would have been enhancing multimodal understanding.
Regarding claim 11, claim 11 recites functions that are similar in scope to the method steps recited in claim 4 and therefore are rejected under the same rationale.
Regarding claim 17, claim 17 recites instructions that are similar in scope to the method steps recited in claim 4 and therefore are rejected under the same rationale.
Claims 6, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Khorshid (US 2024/0045704) in view of Seo et al. in view of Ramesh et al. in view of Windle et al., as applied to claims 1, 8 and 14, in further view of Hinski (US 2016/0049010).
Regarding claim 6, Khorshid teaches provide to the augmented reality headset (Khorshid, Fig. 9A); Khorshid as modified by Seo, Ramesh and Windle does not expressly disclose “outputs an identification of a document”;
Hinski discloses outputs an identification of a document (Hinski, [0061], “an identification module 111, such as a collection of processor instructions, to automatically capture a document image and automatically identify the image content data…The document support data is automatically sent to the display 103 to be viewed by the user along with the document using the display module 113”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant system of Khorshid as modified by Seo, Ramesh to incorporate the document information retrieval for augmented reality display as taught by Hinsky. The motivation for doing so would have been enabling real-time retrieval of relevant documents and presentation of the retrieved documents on the augmented reality display.
Regarding claim 13, claim 13 recites functions that are similar in scope to the method steps recited in claim 6 and therefore are rejected under the same rationale.
Regarding claim 19, claim 19 recites instructions that are similar in scope to the method steps recited in claim 6 and therefore are rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/Primary Examiner, Art Unit 2611
Prosecution Insights