Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Introduction
This office action is in response to Applicant’s submission filed on 11/25/2024. As such, claims 1-20 have been examined.
Subject Matter Eligibility
Examiner Comment Regarding Patent Subject Matter Eligibility under 35 U.S.C. 101
Independent claims 1, 10 and 19 involve a method/device/computer-readable storage media focus on technical solution for effective communication between people that speak different language (translation in using speaker’s own unique voice) is a process that could not practically be performed as an abstract idea such as a mental process under the broadest reasonable interpretation (BRI). Accordingly, the independent claims and their dependents by virtue of their dependency, are directed towards patent eligible subject matter under step 2A prong 1 and/or 2A prong 2 because even if one can make an argument that the steps can be done as a mental process, but, the claim describes specific technological improvement in how machine processes speech, instead of using generic AI voice, it translate language in the speakers native voice and also protective privacy of speaker by keeping the speaker model in their own device, therefore the claims are patent eligible.
This is examiner perspective toward the computer readable storage media claims with regards to U.SC. 101 subject matter eligibility. In para [0118] of the instant application, it is noted in the specification that “Computer-readable storage media” may refer to media and/or devices that enable persistent and/or non-transitory storage of information in contrast to mere signal transmission, carrier waves, or signals per se. Thus, computer-readable storage media refers to non-signal bearing media. The computer-readable storage media includes hardware such as volatile and non-volatile, removable and non-removable media and/or storage devices implemented in a method or technology suitable for storage of information such as computer readable instructions, data structures, program modules, logic elements/circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage device, tangible media, or article of manufacture suitable to store the desired information and which may be accessed by a computer.” In light of the applicant’s definition or boundary toward the computer readable storage media, it is determined that the current claims relating to the computer readable storage media is within the scope of patent eligible statutory category.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-8, 10-12, 16-17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Applicant supplied reference, Trehan (US 20210312143), in view of Liu (US 20220199086), and further in view of Schlakman (US 20240386877).
Regarding Claim 1, Trehan discloses: 1. A computer-implemented method comprising: receiving, by a first computing device, speech from a speaker, the speech being in a speaker’s language pre-defined as corresponding to the speaker; ([0018] Aspects/embodiments of the present invention provides translation of a call through an application interface includes establishing a call with a first device associated with a source user to a second device associated with a target user, where the source user is speaking a source language and the target user understands and is speaking a target language.)
responsive to receiving the speech, translating the speech from the speaker’s language into translated data in a listener’s language corresponding to a second computing device; ([0018] Aspects/embodiments of the present invention provides translation of a call through an application interface includes establishing a call with a first device associated with a source user to a second device associated with a target user, where the source user is speaking a source language and the target user understands and is speaking a target language. … that the translated audio has been provided and heard.)
and communicating the pronounced audio data to the second computing device as the speech is being received by the first computing device. ([0053] The application provides a communication interface 20 that facilitates communication and real-time call translation configured with a translation program. The application includes the communication interface 20 executed by a program on a local processor on the communication device 16 which allows the first user 12 to establish a call (audio calls or video calls) to a communication device 18 associated with a second user 14 (also referred as target user) over a network which is a packet-based network in this embodiment but which may not be packet-based in other embodiments.)
Trehan does not appear to disclose the following feature which appears to describe translation using the speaker’s own unique voice feature.
However, Liu in the related art discloses: superimposing, by the first computing device, pronunciation of the speaker as modeled by a speaker pronunciation model onto the translated data to generate pronounced audio data in the listener’s language modeled so as to sound as if pronounced by the speaker, the speaker pronunciation model being trained on a voice of the speaker speaking the speaker’s language, ([0004] The embodiment also includes associating, by the processor, a custom phoneme with a standard phoneme of the standard phoneme sequence, wherein the custom phoneme includes the specific person's pronunciation of a sound in the translated word. The embodiment also includes synthesizing, by the processor, the translated text string to a translated audio signal that includes the translated word pronounced using the custom phoneme.) Also see fig. 8 and the related description 0125-0126. [The text teaches using custom phoneme from a specific person’s pronunciation reads on how the speaker pronunciation model is trained.]
Trehan and Liu are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Trehan to combine the teaching of Liu, because generating translation using personalized voices may enhance and improve end user experience (Liu, [0024]).
Trehan and Liu does not appear to teach storing the trained speaker pronunciation model in the user’s local device.
Schlakman in a related field discloses: and the speaker pronunciation model being stored at the first computing device; ([0235] In some embodiments, system 600 obtains the plurality of audio samples (e.g., via user interface 698), generates the personalized voice model, and stores the personalized voice model locally (e.g., on a user device such as 602, 670, and/or 684) in order to preserve the privacy and security of the user's personalized voice model.)
Trehan/Liu/Schlakman are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Trehan and Liu to combine the teaching of Schlakman, because storing personalized voice model locally may preserve privacy and security of the user’s personalized voice model (Schlakman, [0235]).
Regarding claim 2, Trehan/Liu/Schlakman disclose all of claim 1,
Liu further discloses: wherein: the superimposing includes overlaying the pronunciation of the speaker onto words in the translated data, ([0004] The embodiment also includes associating, by the processor, a custom phoneme with a standard phoneme of the standard phoneme sequence, wherein the custom phoneme includes the specific person's pronunciation of a sound in the translated word. The embodiment also includes synthesizing, by the processor, the translated text string to a translated audio signal that includes the translated word pronounced using the custom phoneme.)
as indicated by a superimposition model of the listener’s language that models generic pronunciation of the words in the listener’s language, ([0004] The embodiment also includes associating, by the processor, a custom phoneme with a standard phoneme of the standard phoneme sequence, wherein the custom phoneme includes the specific person's pronunciation of a sound in the translated word. The embodiment also includes synthesizing, by the processor, the translated text string to a translated audio signal that includes the translated word pronounced using the custom phoneme.)
to generate the pronounced audio data; ([0004] The embodiment also includes associating, by the processor, a custom phoneme with a standard phoneme of the standard phoneme sequence, wherein the custom phoneme includes the specific person's pronunciation of a sound in the translated word. The embodiment also includes synthesizing, by the processor, the translated text string to a translated audio signal that includes the translated word pronounced using the custom phoneme.)
and the speaker pronunciation model and the superimposition model map one or more of timbre, ([0004] The embodiment also includes associating, by the processor, a custom phoneme with a standard phoneme of the standard phoneme sequence, wherein the custom phoneme includes the specific person's pronunciation of a sound in the translated word. The embodiment also includes synthesizing, by the processor, the translated text string to a translated audio signal that includes the translated word pronounced using the custom phoneme.) [synthesize audio signal using custom phoneme reads acoustic characteristics (timbre, articulation) of that specific person to the translated words.]
Where the rationale for the combination would be similar to the one previously provided.
Regarding claim 3, Trehan/Liu/Schlakman disclose all of claim 1,
Trehan further discloses: wherein: the first computing device and the second computing device are configured for communicating with each other during a communication session via a communication server; ([0053] the present invention provides a real-time call translation system and method. Now referring to figures the present invention provides a call translation system 10 as illustrated in the FIG. 1a, FIG. 1b and FIG. 1c. In one embodiment as illustrated in FIG. 1a and FIG. 1b, the system 10 operates on a communication device 16 by a first user 12 (also referred as source user); the communication device 16 is running an application. The application provides a communication interface 20 that facilitates communication and real-time call translation configured with a translation program. The application includes the communication interface 20 executed by a program on a local processor on the communication device 16 which allows the first user 12 to establish a call (audio calls or video calls) to a communication device 18 associated with a second user 14 (also referred as target user) over a network which is a packet-based network in this embodiment but which may not be packet-based in other embodiments.)
the superimposing is performed via a communication session client executing on the first computing device; ([0053] the present invention provides a real-time call translation system and method. Now referring to figures the present invention provides a call translation system 10 as illustrated in the FIG. 1a, FIG. 1b and FIG. 1c. In one embodiment as illustrated in FIG. 1a and FIG. 1b, the system 10 operates on a communication device 16 by a first user 12 (also referred as source user); the communication device 16 is running an application. The application provides a communication interface 20 that facilitates communication and real-time call translation configured with a translation program. The application includes the communication interface 20 executed by a program on a local processor on the communication device 16 which allows the first user 12 to establish a call (audio calls or video calls) to a communication device 18 associated with a second user 14 (also referred as target user) over a network which is a packet-based network in this embodiment but which may not be packet-based in other embodiments.)
and the translating is performed at least one of in the communication session client executing on the first computing device or at least partly at the communication server. ([0053] the present invention provides a real-time call translation system and method. Now referring to figures the present invention provides a call translation system 10 as illustrated in the FIG. 1a, FIG. 1b and FIG. 1c. In one embodiment as illustrated in FIG. 1a and FIG. 1b, the system 10 operates on a communication device 16 by a first user 12 (also referred as source user); the communication device 16 is running an application. The application provides a communication interface 20 that facilitates communication and real-time call translation configured with a translation program. The application includes the communication interface 20 executed by a program on a local processor on the communication device 16 which allows the first user 12 to establish a call (audio calls or video calls) to a communication device 18 associated with a second user 14 (also referred as target user) over a network which is a packet-based network in this embodiment but which may not be packet-based in other embodiments.)
Regarding claim 4, Trehan/Liu/Schlakman disclose all of claim 1,
Schlakman further discloses: wherein the speaker pronunciation model is not communicated off the first computing device. ([0235] In some embodiments, system 600 obtains the plurality of audio samples (e.g., via user interface 698), generates the personalized voice model, and stores the personalized voice model locally (e.g., on a user device such as 602, 670, and/or 684) in order to preserve the privacy and security of the user's personalized voice model.)
Where the rationale for the combination would be similar to the one provided in claim 1.
Regarding claim 7, Trehan/Liu/Schlakman disclose all of claim 1,
Trehan further discloses: further comprising storing, at the first computing device, at least one translation model or conversion model which models speech-to-text conversion in the speaker’s language, models text translation from the speaker’s language to the listener’s language, and models text-to-audio conversion in the listener’s language, the translating being performed using the at least one translation model or conversion model. ([0066] As shown in FIG. 5, the source user initiates the call and can turn on the translation through a voice command or pressing a key button or screen touch or visual gesture to automate the translation. As discussed above, the interface 20 is configured with the translation engine 42. When the translation command is received from the user, the system starts collecting the speech of a source user through a voice collection unit 52; respectively importing the collected voice into the speech recognition unit 54 through the processor 31 to obtain confidence degrees of the voice corresponding to different alternative languages, and determining a source language used by the source user according to the confidence degrees and a preset determination rule, and converting the voice from the source language into a target language through the processor 31, then transferring the translated language to target user and playing back to the source user via the sound playing device.) Also see fig. 5 which is reproduced below for convenience, as the claim is basically describing a system where SST – translation system – TTS.
PNG
media_image1.png
509
547
media_image1.png
Greyscale
Ttrehan does not explicitly mention where the translation engine is located.
Schlakman further discloses: storing, at the first computing device, at least one translation model or conversion model ([0235] In some embodiments, system 600 obtains the plurality of audio samples (e.g., via user interface 698), generates the personalized voice model, and stores the personalized voice model locally (e.g., on a user device such as 602, 670, and/or 684) in order to preserve the privacy and security of the user's personalized voice model.)
Where the rationale for the combination would be similar to the one already provided in claim 1.
Regarding claim 8, Trehan/Liu/Schlakman disclose all of claim 1,
Trehan further discloses: further comprising: participating, by the first computing device, in a communication session via a communication session server with the second computing device; ([0066] As shown in FIG. 5, the source user initiates the call and can turn on the translation through a voice command or pressing a key button or screen touch or visual gesture to automate the translation. As discussed above, the interface 20 is configured with the translation engine 42. When the translation command is received from the user, the system starts collecting the speech of a source user through a voice collection unit 52; respectively importing the collected voice into the speech recognition unit 54 through the processor 31 to obtain confidence degrees of the voice corresponding to different alternative languages, and determining a source language used by the source user according to the confidence degrees and a preset determination rule, and converting the voice from the source language into a target language through the processor 31, then transferring the translated language to target user and playing back to the source user via the sound playing device.) [0089] The system implementations of the described technology, in which the application interface 20 is capable of executing a program to execute the translation, the interface 20 is connected with a network 36, control server 37 and a computer system capable of executing a computer program to execute the translation.
and providing, by the first computing device, the pronounced audio data to a communication session server for further routing to the second computing device. ([0066] As shown in FIG. 5, the source user initiates the call and can turn on the translation through a voice command or pressing a key button or screen touch or visual gesture to automate the translation. As discussed above, the interface 20 is configured with the translation engine 42. When the translation command is received from the user, the system starts collecting the speech of a source user through a voice collection unit 52; respectively importing the collected voice into the speech recognition unit 54 through the processor 31 to obtain confidence degrees of the voice corresponding to different alternative languages, and determining a source language used by the source user according to the confidence degrees and a preset determination rule, and converting the voice from the source language into a target language through the processor 31, then transferring the translated language to target user and playing back to the source user via the sound playing device.) [0089] The system implementations of the described technology, in which the application interface 20 is capable of executing a program to execute the translation, the interface 20 is connected with a network 36, control server 37 and a computer system capable of executing a computer program to execute the translation.
Regarding Claim 10, Trehan discloses: 10. A computing device comprising: local computer-readable storage media; ([0060] As shown in FIG. 3, the communication device comprises at least one processor 31, further the processor is connected with a memory 32 for storing data and performing translation with the communication interface 20.) an audio input device operable to receive speech; ([0060] Further an input audio device 34 (e.g. one or more microphones) and output audio device 35 (e.g. one or more speakers) are connected to the processor 31.) and at least one processor operable with the audio input device, ([0060] Further an input audio device 34 (e.g. one or more microphones) and output audio device 35 (e.g. one or more speakers) are connected to the processor 31.) and configured to:
As for the rest of the claim, they recite elements similar to claim 1, therefore the rationale applied in rejection of claim 1 is equally applicable.
Claims 11-12 and 16-17 recites limitations similar to the limitations of Claim 2-3 and 7-8 respectively, and are rejected under similar rationale.
Regarding Claim 19, Trehan discloses: 19. One or more computer-readable storage media storing computer-executable instructions that, responsive to execution by one or more processors, perform operations comprising: ([0060] As shown in FIG. 3, the communication device comprises at least one processor 31, further the processor is connected with a memory 32 for storing data and performing translation with the communication interface 20.)
As for the rest of the claim, they recite elements similar to claim 1, therefore the rationale applied in rejection of claim 1 is equally applicable.
Claims 5, 9, 14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Applicant supplied reference, Trehan (US 20210312143), in view of Liu (US 20220199086), further in view of Schlakman (US 20240386877), and furthermore in view of Agiomyrgiannakis (US 20150127349)
Regarding claim 5, Trehan/Liu/Schlakman disclose all of claim 1,
However, the combination of references does not teach or suggest using a GPU for superimposing or translation.
Agiomyrgiannakis in the related art discloses: further comprising using a graphical processing unit (GPU) of the first computing device to perform at least one of the translating or the superimposing. ([0083] Processor 406 may include one or more general purpose processors (e.g., microprocessors) and/or one or more special purpose processors (e.g., digital signal processors (DSPs), graphical processing units (GPUs), floating point processing units (FPUs), network processors, or application specific integrated circuits (ASICs)). Processor 406 may be configured to execute computer-readable program instructions 410 that are contained in data storage 408, and/or other instructions, to carry out various functions described herein.)
Trehan/Liu/Schlakman/Agiomyrgiannakis are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of teachings to combine with the teaching of Agiomyrgiannakis, because GPU can process multiple task simultaneously in real time without lag (Agiomyrgiannakis, [0083]).
Regarding claim 9, Trehan/Liu/Schlakman disclose all of claim 1,
However, the combination of references does not teach or suggest applying encryption in communicated audio data.
Agiomyrgiannakis in the related art discloses: further comprising encrypting, by the first computing device and before the communicating, the pronounced audio data, wherein the pronounced audio data communicated to the second computing device is encrypted. ([0082] In some embodiments, communication interface 404 may be configured to provide reliable, secured, and/or authenticated communications. For each communication described herein, information for ensuring reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of a message header and/or footer (e.g., packet/message sequencing information, encapsulation header(s) and/or footer(s), size/time information, and transmission verification information such as cyclic redundancy check (CRC) and/or parity check values). Communications can be made secure (e.g., be encoded or encrypted) and/or decrypted/decoded using one or more cryptographic protocols and/or algorithms, such as, but not limited to, the data encryption standard (DES), the advanced encryption standard (AES), the Rivest, Shamir, and Adleman (RSA) algorithm, the Diffie-Hellman algorithm, and/or the Digital Signature Algorithm (DSA).)
Trehan/Liu/Schlakman/Agiomyrgiannakis are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of teachings to combine with the teaching of Agiomyrgiannakis, because encryption protection will ensure communicated data is protected (Agiomyrgiannakis, [0082]).
Claims 14 and 18 recites limitations similar to the limitations of Claim 5 and 9 respectively, and are rejected under similar rationale.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Applicant supplied reference, Trehan (US 20210312143), in view of Liu (US 20220199086), further in view of Schlakman (US 20240386877), and furthermore in view of Fayan (US 20250292799)
Regarding claim 13, Trehan/Liu/Schlakman disclose all of claim 10,
However, the combination of references does not teach or suggest synchronization of audio and video to additional device.
Fayan in the related art discloses: wherein the at least one processor is further configured to synchronize the pronounced audio data and video data for synchronous output at the additional computing device. ([0025] The translated language speech, whether in a standard voice supplied by a TTS system or in the cloned voice of the original audio is used to create the language proxy 312. When the audio comprises dialog for a video composition, the translated audio is combined with the original video, which may be achieved by relinking the master clip to different, translated and optionally voice-cloned audio. Synchronization between the video and translated audio is performed using the timing information from the MT text output. The language proxy may be stored as proxy audio file 314 locally to the editor and the editing application being used or remotely, on a remote server or in the cloud.) [The text mentions that the file can be stored remotely, on a remote server or in the cloud. This means other device can access and synchronously output the aligned video and translated audio.]
Trehan/Liu/Schlakman/Fayan are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of teachings to combine with the teaching of Fayan, because audio/video synchronization would match audio to video, ensuring visual lip movement is aligned correctly to the audio (Fayan, [0025]).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Applicant supplied reference, Trehan (US 20210312143), in view of Liu (US 20220199086), further in view of Schlakman (US 20240386877), and furthermore in view of Thomson (US 20210375288)
Regarding claim 20, Trehan/Liu/Schlakman disclose all of claim 19,
However, the combination of references does not teach or suggest the first and second device being the same device.
Thomson in the related art discloses: wherein the first computing device and the second computing device are a same computing device. ([0106] At block 312, a transcription of a second communication session may be directed to a second device for presentation to the user. In these and other embodiments, the transcription may be generated by the second transcription generation technique in response to the indication from the first device. Alternately or additionally, the first device and the second device may be the same device.)
Trehan/Liu/Schlakman/Thomson are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of teachings to combine with the teaching of Thomson, because a speaker can speak and hear their own translated words using their own device (Thomson, [0106]).
Allowable Subject Matter
Claims 6 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 6, the claim discloses a specific technique of training an AI to sound exactly like a specific speaker, using specific feature extraction and superimposing these unique traits onto the generated audio.
The closest prior arts are: Pan US 20230081659 – which teaches cross speaker style transfer speech synthesis, it teaches training to implement cross style transfer. However, it does not disclose training of a pronunciation model, and only broadly mention extraction of acoustic features and does not teach the superimposing step. Fayan US 20250292799 – which discuss voice cloning and translation, as it teaches analyze acoustic features of the reference voice and learn the nuances of speech patterns, although it teaches general voice cloning concept, but it does not teach the specific steps of receiving speech, extracting the specific features and incorporating them into the model as the claim requires. Further, none of them do so in the manner as specifically claimed.
Claim 15, although in different statutory category, nevertheless they recited similar language as claim 6, therefore is subject to the condition mentioned from above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Arslan (US 20250225341) – discloses method/system for real time online voice/telephone translation. See Abstract, 0014-0015, 0027, 0033, 00046, 0048, 0054, 0062 and figs 1-4 for additional details.
Pavlovsky (US 20240194177) – teaches a method for securely synthesizing voice of a speaker during a voice call translation. See Abstract and figs 1-2, and 4 for additional details.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Philip H Lam whose telephone number is (571)272-1721. The examiner can normally be reached 9 AM-3 PM Pacific time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached on 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHILIP H LAM/ Examiner, Art Unit 2656
Prosecution Insights