DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 11, and 20 are objected to because of the following informalities: “a chat function of metaverse place” is missing and article in front of “metaverse”. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7, 9-17, 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is a mental process without significantly more.
Independent claims 1, 11, and 20 recite a method, non-transitory computer readable medium, and system respectively, which under their broadest reasonable interpretation cover translating a voice communication from one user to another user according to language preference and context. The claimed steps can be performed as a series of mental steps by a human translator/interpreter.
For example, the independent claims recite: receiving a request to translate audio associated with a chat function of metaverse place of the virtual metaverse, the audio received from a first user of a plurality of users, wherein the plurality of users are associated with the metaverse place (a human interpreter can be asked to translate a speaker’s spoken statement in a group conversation such as virtual conversation); retrieving translation data associated with a second user of the plurality of users, wherein the translation data includes at least a language preference associated with the second user, and wherein the second user is associated with a user device (A human interpreter can ask or know the listener’s preferred language and know which listener is to receive the translation); converting audio received from the first user into text, wherein the audio includes input speech in a first language spoken by the first user (A human interpreter can listen to the speaker and write down or mentally note the words spoken by the speaker); translating the text into a second language, wherein the second language is defined by the language preference and wherein the translated text includes context data from the audio (A human interpreter can translate the speaker’s words into the listener’s preferred language when taking into account context, tone, emphases, meaning etc.); converting the translated text into output speech including the context data; and providing the output speech to the user device (A human interpreter can communicate the translate speech to the listener vocally).
As described above these limitations can be carried out as a series of mental steps involving listening to human speech, determining a listener’s language preference, translating the speech into listener’s language, considering context or emotion, and communicating the translated speech to the listener.
The judicial exception is not integrated into a practical application because the additional elements merely use generic computer and communication components as tools to implement the mental process. The recited metaverse place, chat function, user device, processing device, memory, computer—readable medium, audio-to-text conversion, text translation, text-to-speech conversion, and providing output speech are recited at a high level of generality and merely limit the abstract idea to a particular technological environment, namely virtual/metaverse voice chat. The claims do not recite an improvement to the operation of a computer, an improvement to speech recognition technology, machine translation technology, text-to-speech technology, or a specific technical architecture for improving voice chat systems. The claims use generic computer implementation to automate the human mental process of interpreting and translating speech between users.
The claims also do not include additional elements that amount to significantly more than the judicial exception. The additional elements, considered individually and as an ordered combination, amount to no more than generic computer implementation of receiving audio, identifying language preference, converting speech to text, translating text, converting text to speech, applying context, and transmitting output speech. These are conventional computer and communication functions used as tools to perform an abstract idea.
The remaining dependent claims fail to add patent eligible subject matter to independent claim 1:
Claims 2 and 12 simply add identifying the speaker/listening, and originating device, which a human can do mentally by identifying in their head the speaker and listener in a conversation and recognizing that the speaker initiated the request/conversation.
Claims 3 and 13 simply add applying voice output preferences that override other translation settings which a human can do mentally by following/storing the speaker’s requested vocal preferences in their head and keeping it mind when providing the translated speech.
Claims 4 and 14 simply add text moderating which a human can do by mentally reviewing speech and removing profanity/prohibited words before translating the modified statement.
Claims 5 and 15 simply add using a trained machine learning model for translation which merely automates linguistic translation using a generic component as a tool (ChatGPT) and a human translator can already linguistically translate mentally or via dictionaries.
Claims 6 and 16 simply add identifying emotion from audio which humans can do mentally when listening to someone speak and analyzing tone, pitch speech, emphasis, pauses, etc.
Claims 7 and 17 simply add pre-processing the audio to extract emotion which a human can do by first listening for vocal cues in conversation (such as volume, speech pitch, pauses etc.) before determining the speaker’s emotional tone mentally.
Claims 9 simply adds translating into multiple languages and generating multiple translated speech outputs, which a human interpreter can do by translating the same statement into multiple languages and speaking each translated version.
Claim 10 simply adds sending the different translated speech outputs to different user devices which a human can do by providing the translation to the appropriate listener (ex. Spanish translation to Spanish listener, French translation to French listener, etc.).
Claim 19 simply adds translating into multiple languages and generating multiple translated speech outputs, which a human interpreter can do by translating the same statement into multiple languages and speaking each translated version. Claim 19 also simply adds sending the different translated speech outputs to different user devices which a human can do by providing the translation to the appropriate listener (ex. Spanish translation to Spanish listener, French translation to French listener, etc.).
Claims 8 and 18 recite using a speech waveform modulator to create a modulated speech waveform that at least partially includes the context data. Unlike the translation/context identification steps discussed above, creating a modulated speech waveform is not a step that can practically be performed in the human mind or with a pen and paper. Rather it is more reasonably characterizes as audio signal processing performed by a computer-based speech synthesis/modulation component (Specification, P[0125]: “At block 610, the translated text is converted into output speech including the context data and/or emotion data. For example, a per-user or user-specific text to speech model may be provided the text, context data, and/or emotion data. The text to speech model may also be referred to as a speech synthesizer or speech synthesis model. Using the context data and/or emotion data together with speech data, the speech synthesizer may generate a speech waveform based on the speech, in the second language, and including at least one or more of the context data and/or emotion data.”). Accordingly claims 8 and 18 are considered to integrate the claimed translation process into a practical application and include additional technical limitations beyond the mental process. As a result, claims 8 and 18 are not included in the 35 U.S.C 101 rejection above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Svenchenko et al. (hereinafter Svenchenko) (US 11258734 B1) (see attached copy for paragraph numbers) in view of Ahrens (US 20230009957 A1).
Regarding claim 1, Svenchenko discloses:
in a virtual metaverse, comprising (Shevchenko, P(99): "implemented with a VR platform to augment conversations between people as avatars in virtual reality"):
receiving a request to translate audio associated with a chat function of metaverse place of the virtual metaverse, the audio received from a first user of a plurality of users (Shevchenko, P(99): "the AR/VR communication facility may be implemented with a smart speaker to intercept voice communications, analyze them, and either translate them into better-formed text and then read back or display them in a target application for confirmation or iterations." (teaches receiving/intercepting voice/audio communications from a user in an AR/VR communication facility and translating the received voice communication for output/confirmation/alteration)), wherein the plurality of users are associated with the metaverse place (Shevchenko, P(99): "augment real-time conversations being carried out through augmented reality (AR) or virtual reality (VR) platforms through an AIA AR/VR communication facility, which provides communication assistance (augmented communication) to users (senders as well as receivers)" (teaches plurality of users/senders/receivers participating in real time AR/VR conversations associated with the virtual/metaverse environment));
Svenchenko does not explicitly disclose:
A computer-implemented method of voice chat translation
retrieving translation data associated with a second user of the plurality of users, wherein the translation data includes at least a language preference associated with the second user, and wherein the second user is associated with a user device;
converting audio received from the first user into text, wherein the audio includes input speech in a first language spoken by the first user;
translating the text into a second language, wherein the second language is defined by the language preference and wherein the translated text includes context data from the audio;
converting the translated text into output speech including the context data; and
providing the output speech to the user device
However, Ahrens discloses:
A computer-implemented method of voice chat translation (Ahrens, P[0029]: "process of translating a transmission between communication devices")
retrieving translation data associated with a second user of the plurality of users, wherein the translation data includes at least a language preference associated with the second user, and wherein the second user is associated with a user device (Ahrens, P[0042]: "a first user on a communication device 104 calls a user on a second communication device 106 with the first user initiating a video call. Each user selects their native language and the audio gathering unit 110 translates the audio in the preferred language of each user. " (teaches second user/device and translation data including the second user's language preference));
converting audio received from the first user into text, wherein the audio includes input speech in a first language spoken by the first user (Ahrens, P[0043]: "an audio stream is captured by the audio gathering unit 110. In one embodiment, the audio stream is the voice of a user of a communication device 104/106. In step 404, the audio stream is converted into a digital format by the audio gathering unit 110 using any known digital conversion method. In step 406, the audio stream is converted into text" (teaches converting first user spoken audio into text));
translating the text into a second language, wherein the second language is defined by the language preference and wherein the translated text includes context data from the audio (Ahrens, P[0040]: "generates a new audio stream using the translated text", "applies the speech patterns to the newly generated audio stream", P[0042]: "Each user selects their native language", P[0043]: "correlates the sections of text with related audio portions", "The characteristics includes, but is not limited to, speed of speech, tone, pronunciation of letters and words, and any other voice characteristic" (teaches translating into a user selected language while retaining/associating audio derived context/voice characteristics with translated text/output));
converting the translated text into output speech including the context data (Ahrens, P[0040]: "generates a new audio stream using the translated text. In step 322, the audio gathering unit 110 uses digital audio processing to identify specific speech patterns of the speaker in the original audio stream and applies the speech patterns to the newly generated audio stream" (teaches associating translated/text portions with audio-derived context/voice characteristics)); and
providing the output speech to the user device (Ahrens, P[0042]: "a first user on a communication device 104 calls a user on a second communication device 106 with the first user initiating a video call. Each user selects their native language and the audio gathering unit 110 translates the audio in the preferred language of each user." (teaches providing translated output speech/audio to the respective user device)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Svenchenko in view of Ahrens. Doing so would provide the voice translation system that captures user audio, converts it to text, translates the text into each user’s preferred language and generates a translated output audio of Ahrens (Ahrens, Abstract, P[0040]-P[0042]) with the AR/VR communication facility for augmenting real-time conversations in AR/VR platforms including intercepting voice communications, analyzing them, translating speech, and providing communication assistance of senders and receivers in a virtual environment (Svenchenko, Abstract, P(99)). This combination would result in a predictable use of known speech translation and audio generation techniques in an AR/VR communication system to provide translated voice chat output to users in their preferred languages.
Regarding claim 2, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 1.
Ahrens further discloses:
wherein the request identifies
the first user and the second user, and the request originates from a computing device associated with the first user (Ahrens, P[0042]: "a first user on a communication device 104 calls a user on a second communication device 106 with the first user initiating a video call. " (teaches first user originated communication/request identifying first and second users/devices)).
Regarding claim 3, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 1.
Svenchenko further discloses:
further comprising retrieving
voice output preferences associated with the first user, wherein the voice output preferences at least partially override the translation data associated with the second user (Shevchenko, Fig. 7, P(61): "The communication transformation module 418 may provide interactive feedback 706, such as to prompt the user to specify a user/audience and the context that AIA cannot get or infer from the environment, prior user input, user settings, available communication draft and history, and the like. The AIA may prompt the user to confirm or override the receiver/audience and the context the AIA has identified. The user may update the content, context, and receivers at any time during the session." (teaches prompting the first user to specify, confirm, update, or override receiver/audience/context information used for communication transformation, which corresponds to first user voice/output preferences at least partially overriding translating/receiver data associated with second user)).
Regarding claim 4, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 1.
Svenchenko further discloses:
further comprising, prior to
translating the text into the second language, moderating the text to remove words based on a text moderation filter (Shevchenko, P(52): "intercepting a electronic communication", P(53): "removing or replacing language from the electronic communication", "The removed or replaced language may be offensive or abusive language" (teaches filtering/moderating communication content by removing words/language before output/further processing)).
Regarding claim 5, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 1.
Svenchenko further discloses:
wherein the translating comprises providing, as input to a trained machine learning model, the text and receiving, as an output from the trained machine learning model, the translated text (Shevchenko, P(68): "The personalization and targeting facility may use a style transfer module implemented with a machine, deep, or other statistical learning model or models, such as a sequence to sequence model or models that translate generated language variations from generic style into the user's authentic communication style. " (teaches providing text/language variations to a trained machine/deep/statistical learning sequence-to-sequence model and receiving translated/transformed text output in the user's communication style)).
Regarding claim 6, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 1.
Ahrens further discloses:
wherein the context data comprises emotion data extracted from the audio (Ahrens, P[0041]: "may determine the emotions conveyed by the speaker and may adjust the speaker's image to convey to the speaker's emotional state. In one embodiment, the audio gathering unit 110 may determine the emotional state of the speaker by analyzing the volume and speed of the audio as well as the facial coordinates of the speaker while speaking.").
Regarding claim 7, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 6.
Ahrens further discloses:
further comprising pre- processing the audio to extract the emotion data (Ahrens, P[0041]: "may determine the emotional state of the speaker by analyzing the volume and speed of the audio" (teaches pre-processing audio to extract emotion data)).
Regarding claim 8, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 1.
Ahrens further discloses:
wherein converting the translated text into output speech comprises using a speech waveform modulator to create a modulated speech waveform that at least partially includes the context data (Ahrens, P[0040]: "generates a new audio stream using the translated text", "uses digital audio processing to identify specific speech patterns of the speaker in the original audio stream and applies the speech patterns", P[0044]: "adjusts the voice characteristics of each audio segment to generate a modified audio output", "combines all the segments into a single audio segment." (teaches digital audio processing to generate a modified translated speech waveform including audio-derived speech-pattern context)).
Regarding claim 9, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 1.
Ahrens further discloses:
further comprising translating the text into a plurality of different languages to create a plurality of different translated texts, and converting the plurality of different translated texts into a plurality of different output speech (Ahrens, P[0042]: "Each user selects their native language and the audio gathering unit 110 translates the audio in the preferred language of each user" (teaches translating into different preferred languages for different users and generating corresponding translated audio/output speech)).
Regarding claim 10, the combination of Svenchenko and Ahrens discloses the computer-implemented method of claim 9.
Ahrens further discloses:
further comprising providing the plurality of different output speech to a plurality of user devices, wherein each output speech comprises utterances in a language associated with a respective user device of the plurality of user devices (Ahrens, P[0042]: "a first user on a communication device 104 calls a user on a second communication device 106 with the first user initiating a video call. Each user selects their native language and the audio gathering unit 110 translates the audio in the preferred language of each user." (teaches providing translated output speech to respective user devices in respective user’s selected/preferred language)).
Regarding claim 11, claim 11 recites the non-transitory computer-readable medium corresponding to the method presented in claim 1 and is rejected under the same grounds stated above.
Svenchenko further discloses:
A non-transitory computer-readable medium with instructions stored thereon that, responsive to execution by a processing device, causes the processing device to perform operations comprising (Svenchenko, P(14)):
Regarding claim 12, claim 12 recites the non-transitory computer-readable medium corresponding to the method presented in claim 2 and is rejected under the same grounds stated above.
Regarding claim 13, claim 13 recites the non-transitory computer-readable medium corresponding to the method presented in claim 3 and is rejected under the same grounds stated above.
Regarding claim 14, claim 14 recites the non-transitory computer-readable medium corresponding to the method presented in claim 4 and is rejected under the same grounds stated above.
Regarding claim 15, claim 15 recites the non-transitory computer-readable medium corresponding to the method presented in claim 5 and is rejected under the same grounds stated above.
Regarding claim 16, claim 16 recites the non-transitory computer-readable medium corresponding to the method presented in claim 6 and is rejected under the same grounds stated above.
Regarding claim 17, claim 17 recites the non-transitory computer-readable medium corresponding to the method presented in claim 7 and is rejected under the same grounds stated above.
Regarding claim 18, claim 18 recites the non-transitory computer-readable medium corresponding to the method presented in claim 8 and is rejected under the same grounds stated above.
Regarding claim 19, claim 19 recites the non-transitory computer-readable medium corresponding to the methods presented in claims 9 and 10 and is rejected under the same grounds stated above.
Regarding claim 20, claim 20 recites the system corresponding to the method presented in claim 1 and is rejected under the same grounds stated above.
a memory with instructions stored thereon (Svenchenko, P(134)); and
a processing device, coupled to the memory and operable to access the memory, wherein the instructions when executed by the processing device, cause the processing device to perform operations including (Svenchenko, P(134)).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/ Examiner, Art Unit 2655
/ANDREW C FLANDERS/ Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights