Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 08/12/26 have been fully considered but they are not persuasive.
Applicant’s arguments with respect to claim(s) 1, 8 and 14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 4, 11 and 17 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 4, 11, and 17 disclose wherein the prompt to the large language model comprises the response to the text prompt and the response to the image prompt; The Examiner has attempted to discern the corresponding support in applicant’s specification, and finds the support to be limited to paragraph [0054]; however, this support does not rise to the level of supporting the claim language. MPEP 2163 II A (b) states “To comply with the written description requirement of 35 U.S.C. 112, para. 1, or to be entitled to an earlier priority date or filing date under 35 U.S.C. 119, 120, or 365(c), each claim limitation must be expressly, implicitly, or inherently supported in the originally filed disclosure. When an explicit limitation in a claim "is not present in the written description whose benefit is sought it must be shown that a person of ordinary skill would have understood, at the time the patent application was filed, that the description requires that limitation.” As it is readily apparent that the recited limitations are far more explicitly detailed and limited in scope than the language of the written description, it is clear that the written description cannot support these limitations. In short, the examiner can find no basis whatsoever for concluding that the Applicant was in possession of the claimed invention including the above mentioned specific limitations at the time of filing.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4, 11 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 4, 11 and 17 recite limitation discussed above, which lacks support in the specification. In particular, the specification does not describe feeding the responses back to the large language model. Because the Examiner is unable to identify any support for this limitation in the specification, the Examiner cannot derive sufficient clarity from the specification regarding the meaning of the limitation. Therefore, the limitation renders claims 4, 11 and 17 indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 7-10, 12, 14-16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Khorshid (US 2024/0045704) in view of Seo et al. (End-to-end Generative Pretraining for Multimodal Video Captioning, Computer Vision and Pattern Recognition, 2022) in view of Ramesh et al. (Walk-the-Talk: LLM driven pedestrian motion generation, IEEE, June 2024) in view of Windle et al. (Llanimation: Llama Driven Gesture Animation, Human-Computer Interaction, May, 2024) in view of Nunally (US 2024/0339211).
Regarding claim 1, Khorshid discloses a method (Khorshid, [0154], “FIG. 11 illustrates an example method 1100 for morphing a virtual assistant avatar”), comprising:
receiving, by a conversational artificial intelligence engine (Khorshid, [0007], “provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0096], “the dialog manager 216 may implement reinforcement learning frameworks to improve the dialog optimization”) and from an augmented reality headset worn by a user (Khorshid, [0034], “augmented-reality (AR) smart glasses, virtual-reality (VR) headset”), a query that is made to an embodied interactive agent that is displayed in a display of the augmented reality headset (Khorshid, [0141], “the AR/VR system 130 may render the XR assistant avatar 920 in the user's home so that the user 910 may request assistance naturally from the XR assistant avatar 920…The user 910 may ask, via a voice command, “assistant, do you think the chair I just checked would fit my living room? 925” As the AR/VR system 130 detects a change of context”), wherein the query comprises audio of a user utterance (Khorshid, [0141], “the first or second form of the XR assistant avatar may be based on one or more of voice, speech”) and images or video captured by a camera of the augmented reality headset of what the user is seeing (Khorshid, [0131], “The external-facing cameras can capture images and videos of the real-world environment”);
generating, an animation for the embodied interactive agent from speech for the embodied interactive agent based on the audio (Khorshid, [0007], “provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0141], “The user 910 may ask, via a voice command, “assistant, do you think the chair I just checked would fit my living room? 925” As the AR/VR system 130 detects a change of context… Firstly, the XR assistant avatar 920 now is rendered as solid (not semi-opaque anymore). Secondly, the XR assistant avatar 920 is rendered as sitting in the user's 910 couch in the living room so that the user 910 may see how the chair looks like compared to a real person if the AR/VR system 130 later renders the chair near the couch”);
receiving, output comprises text and gestures for the embodied interactive agent (Khorshid, [0007], “the assistant system may provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0141], “the first or second form of the XR assistant avatar may be based on one or more of voice, speech, emotion, tone, pitch, appearance, size, shape, clothing, orientation, position, depth, movement, gesture, facial expression, color, shading, outline, brightness, luminescence, or transparency”);
animating, by the conversational artificial intelligence engine, the embodied interactive agent with the animations (Khorshid, [0134], “the XR assistant avatar may have motion and audio cues that are more animated and/or realistic to convey responses and emotions better to the user”. In addition, in paragraph [0141], “the XR assistant avatar 920 now is rendered as solid (not semi-opaque anymore). Secondly, the XR assistant avatar 920 is rendered as sitting in the user's 910 couch in the living room so that the user 910 may see how the chair looks like compared to a real person if the AR/VR system 130 later renders the chair near the couch”); and
outputting, by the conversational artificial intelligence engine, audio via the embodied interactive agent (Khorshid, [0007], “provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. In addition, in paragraph [0134], “As an example and not by way of limitation, the customization may include changing the look and feel, voice, emotions, or other attributes…the XR assistant avatar may have motion and audio cues that are more animated and/or realistic to convey responses and emotions better to the user…the assistant system 140 may provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”. Furthermore, in paragraph [0154], “the AR/VR system 130 may determine a second form and a second pose based on a personalization model and the one or more user sentiments, wherein the personalization model is a machine-learning model trained on a plurality of prior reactions associated with the first user with respect to a plurality of morphing of the XR assistant avatar”. Fig. 9B);
Khorshid fails to teach “generating a prompt for a large language model based on the user utterance and the images or video”;
Seo et al. (hereinafter Seo) discloses generating a prompt based on utterance and images or video (Seo, Fig 2 illustrates given input frames and a future utterance, predict the current utterances. Both losses are applied to a triplet consisting of video frames, present utterances and a future utterance. Fig. 3 illustrates generated sentences).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the assistant system of Khorshid to incorporate the concept of the sentence generation function based on input frames, as taught by Seo. The motivation for doing so would have been enabling the virtual assistant to generate scene-aware conversational responses.
Khorshid as modified Seo does not expressly disclose “the prompt for a large language model”;
Ramesh et al. (hereinafter Ramesh) discloses a prompt for a large language model (Ramesh, III Data and Methodology, [0001], “Training a large language model for a specific task such as motion generation, motion captioning, etc. using prompt templates”. In addition, in section B. Pedestrain Motion Generation, [0003], “utilize a T5 language model [22] and pre-train it initially with our motion and language tokens generated by our tokenizer”);
providing, the prompt to the large language model (Ramesh, III Data and Methodology, [0001], “Training a large language model for a specific task such as motion generation, motion captioning, etc. using prompt templates”);
receiving, an output of the large language model for a virtual object (Ramesh, Fig. 2);
generating, animations for the virtual object based on the text (Ramesh, Fig. 2).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant avatar of Khorshid to incorporate the motion generator utilizing Large Language Models as taught by Ramesh. The motivation for doing so would have been improving realism of avatar behavior.
Though Khorshid teaches gestures and speech for the embodied interactive agent (Khorshid, [0141], “the first or second form of the XR assistant avatar may be based on one or more of voice, speech, emotion, tone, pitch, appearance, size, shape, clothing, orientation, position, depth, movement, gesture, facial expression, color, shading, outline, brightness, luminescence, or transparency”); Khorshid as modified by Seo and Ramesh does not expressly disclose “the gestures for a virtual object based on text”;
Windle et al. (hereinafter Windle) discloses gestures for a virtual object based on text (Windle, Fig. 1 illustrates generating gestures from speech using Llama2 features derived from text as the primary input, producing temporally aligned and contextually accurate gesture animation).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant animation of Khorshid as modified by Seo and Ramesh to incorporate the gesture animation generation from text, as taught Windle. The motivation for doing so would have been enhancing multimodal communication and realism of user interaction.
Khorshid teaches generating audio for the embodied interactive agent; Khorshid as modified by Seo, Ramesh and Windle is silent with respect to “generating, audio based on the text”;
Nunally discloses generating audio for a virtual avatar based on text (Nunally, [0071], “At step 703, the system renders a virtual avatar model based on the first response…and voice may be generated via a text-to-speech model…rendering the virtual avatar model based on the first response includes generating auditory speech of the first response in a first style”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to generate the virtual assistant avatar of Khorshid using the text-to-speech model, as taught by Nunally. The motivation for doing so would have been providing more realistic virtual assistant interaction.
Regarding claim 2, Khorshid discloses receiving, by the conversational artificial intelligence engine, user location information from the augmented reality headset (Khorshid, [0005], “The assistant system may enable the user to interact with the assistant system via user inputs of various modalities (e.g., audio, voice, text, image, video, gesture, motion, location, orientation)”), action based on the user location information (Khorshid, [0116], “The location updates may be consumed by the dialog manager 216 to support various proactive/reactive scenarios”).
Khorshid as modified by Seo with the same motivation from claim 1 discloses the prompt (Seo, Fig 2 illustrates given input frames and a future utterance, predict the current utterances.).
Regarding claim 3, Khorshid discloses inferring, by the conversational artificial intelligence engine, a task goal associated with the query (Khorshid, [0109], “The content determination component may determine the communication content based on the knowledge source, communicative goal, and the user's expectations”. In addition, in paragraph [0113], “processing a user input…An assistant task may be a central concept that is shared across the whole assistant stack to understand user intention, interact with the user and the world to complete the right task for the user”), wherein the inference is based on a user interaction history (Khorshid, [0096], “the dialog intent resolution 356 may resolve the user intent associated with the current dialog session based on dialog history between the user and the assistant system 140”), environment object labels (Khorshid, [0080], “recognize interesting objects in the world through a combination of existing machine-learning models”), and user location information (Khorshid, [0082], “the capability of signals intelligence may enable the assistant system 140 to, for example, determine user location”).
Regarding claim 5, Khorshid as modified by Seo and Ramesh with the same motivation from claim 1 discloses the large language model comprises a multi-modal large language model (Ramesh, B. Pedestrian Motion Generation, [0003], “Being a unified, multi-modal architecture, our model now should be able to treat both motion and text tokens as language”).
Regarding claim 7, Khorshid discloses the display in the augmented reality headset displays the animations for the embodied interactive agent (Khorshid, Figs. 9A-9B), and a speaker in the augmented reality headset outputs the audiofl for the embodied interactive agent (Khorshid, [0007], “the assistant system may provide a way for the user to interact via the XR assistant avatar in a conversation manner (i.e., natural-language dialog), where the natural-language responses provided by the XR assistant avatar may customize to the specific application the user is engaging with”).
Regarding claim 8, Khorshid discloses a system (Khorshid, [0005], “the assistant system may assist a user to obtain information or services”), comprising:
an augmented reality headset (Khorshid, [0034], “smart glasses, augmented-reality (AR) smart glasses, virtual-reality (VR) headset”) comprising a camera (Khorshid, [0034], “camera”), a microphone (Khorshid, [0037], “a microphone of the AR/VR system”), a display (Khorshid, [0038], “an AR/VR system 130 may include an AR/VR display device”), and a speaker (Khorshid, [0199], “speaker”), wherein the augmented reality headset is configured to be worn by a user (Khorshid, Fig. 5); and
a multi-modal conversational platform (Khorshid, [0005], “the assistant system may support mono-modal inputs (e.g., only voice inputs), multi-modal inputs (e.g., voice inputs and text inputs), hybrid/multi-modal inputs”).
The remaining limitations recite in claim 8 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claims 9-10, claims 9-10 recite functions that are similar in scope to the method steps recited in claims 2-3 and therefore are rejected under the same rationale.
Regarding claim 12, claim 12 recites function that is similar in scope to the method step recited in claim 5 and therefore is rejected under the same rationale.
Regarding claim 14, Khorshid discloses a non-transitory computer readable storage medium, including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps (Khorshid, [0196], “processor 1502 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 1502 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1504, or storage 1506”. In addition, in paragraph [0202], “a computer-readable non-transitory storage medium or media may include one or more semiconductor-based”).
The limitations recite in claim 14 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claims 15-16, claims 15-16 recite instructions that are similar in scope to the method steps recited in claims 2-3 and therefore are rejected under the same rationale.
Regarding claim 18, claim 18 recites instruction that is similar in scope to the method step recited in claim 5 and therefore is rejected under the same rationale.
Regarding claim 20, claim 20 recites instructions that are similar in scope to the method steps recited in claim 7 and therefore are rejected under the same rationale.
Claims 4, 11 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Khorshid (US 2024/0045704) in view of Seo et al. in view of Ramesh et al. in view of Windle et al. in view of Nunally (US 2024/0339211), as applied to claims 1, 8 and 14, in further view of Alayrac et al. (Flamingo: a Visual Language Model for Few-Shot Learning, Computer Vision and Pattern Recognition, 2022) in view of Panagopoulou et al. (US 2024/0370718).
Regarding claim 4, Khorshid discloses generating a text prompt based on the user utterance (Khorshid, [0055], “The ASR module 208a may allow a user to dictate and have speech transcribed as written text”);
Khorshid as modified by Seo and Ramesh with the same motivation from claim 1 discloses the text prompt for the large language model (Ramesh, III Data and Methodology, [0001], “Training a large language model for a specific task such as motion generation, motion captioning, etc. using prompt templates”), and the large language model returns output (Ramesh, Fig. 2);
Khorshid as modified by Seo, Ramesh, Windle and Nunally with the same motivation from claim 1 discloses receiving, a response to the text prompt from the large language model (Nunally, [0034], “AI avatar module 330 may receive input 340 such as user input, sensor data, text, etc. and generate an output 350 such as a response provided via an avatar. For example, AI avatar module 330 may be configured to generate a response using language generation (e.g., via a large language model)”);
Khorshid as modified by Seo, Ramesh, Windle and Nunally does not expressly disclose “an image prompt for a visual language model”;
Alayrac et al. (hereinafter Alayrac) discloses an image prompt for a visual language model (Alayrac, 1 Introduction, [0001], “a Visual Language Model (VLM) that sets a new state of the art in few-shot learning on a wide range of open-ended vision and language tasks, simply by being prompted with a few input/output examples, as illustrated in Figure 1”. Fig. 1 illustrates image prompt);
response to the image prompt from the visual language model returns output (Alayrac, Fig. 1);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant system based on large language model of Khorshid as modified by Seo and Ramesh to incorporate image prompts for a visual language model as taught by Alayrac. The motivation for doing so would have been enhancing multimodal understanding.
Khorshid as modified by Seo, Ramesh, Windle, Nunally and Alayrac does not expressly disclose “prompt to the large language model comprises text prompt and image prompt”;
Panagopoulou et al. (hereinafter Panagopoulou) discloses prompt to a language model comprises text prompt and image prompt (Panagopoulou, [0025], “Multi-modal instruction model 130 comprises a language model 122 and a multimodal encoder 108 which aids in the generation of an input representation 116 for language model 122…Multi-modal instruction model 130 takes an input 102, and an instruction 112, and based on those inputs generates an output text 124. For example, an input 102 may be an image of various vegetables and other ingredients on a table. The instruction 112 may be “Can you tell me about this image in detail?””).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the large language model of Khorshid as modified by Seo, Ramesh, Windle and Nunally to receive both text input and image input, as taught by Panagopoulou. The motivation for doing so would have been allowing the model to consider both textual and visual information when generating responses.
Regarding claim 11, claim 11 recites functions that are similar in scope to the method steps recited in claim 4 and therefore are rejected under the same rationale.
Regarding claim 17, claim 17 recites instructions that are similar in scope to the method steps recited in claim 4 and therefore are rejected under the same rationale.
Claims 6, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Khorshid (US 2024/0045704) in view of Seo et al. in view of Ramesh et al. in view of Windle et al. in view of Nunally (US 2024/0339211), as applied to claims 1, 8 and 14, in further view of Hinski (US 2016/0049010).
Regarding claim 6, Khorshid teaches provide to the augmented reality headset (Khorshid, Fig. 9A); Khorshid as modified by Seo, Ramesh, Windle and Nunally does not expressly disclose “outputs an identification of a document”;
Hinski discloses outputs an identification of a document (Hinski, [0061], “an identification module 111, such as a collection of processor instructions, to automatically capture a document image and automatically identify the image content data…The document support data is automatically sent to the display 103 to be viewed by the user along with the document using the display module 113”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the virtual assistant system of Khorshid as modified by Seo, Ramesh to incorporate the document information retrieval for augmented reality display as taught by Hinsky. The motivation for doing so would have been enabling real-time retrieval of relevant documents and presentation of the retrieved documents on the augmented reality display.
Regarding claim 13, claim 13 recites functions that are similar in scope to the method steps recited in claim 6 and therefore are rejected under the same rationale.
Regarding claim 19, claim 19 recites instructions that are similar in scope to the method steps recited in claim 6 and therefore are rejected under the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/Primary Examiner, Art Unit 2611
Prosecution Insights