DETAILED ACTION
This communication is in response to the Amendments and Arguments filed on 07/28/2026. Claims 1-20 are pending and have been examined.
Any previous objection/rejection not mentioned in this Office Action has been withdrawn by the Examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Page 10, lines 5-16, filed 07/28/2026, with respect to the Drawing and Specification have been fully considered and are persuasive. The objection of 05/19/2026 has been withdrawn.
Applicant’s arguments, see Page 10, lines 17-21, filed 07/28/2026, with respect to the Claim Interpretation under 35 U.S.C § 112(f) have been fully considered and are persuasive. The interpretation of 05/19/2026 has been withdrawn.
Applicant’s arguments with respect to claim(s) 1, 13, and 14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claim(s) 12 objected to because of the following informalities:
The system of claim 1, wherein the method further comprises: reproducing the generated output audio.
Claim 12 is not consistent with the claim language provided by the amended claims in order to overcome the “means-plus-function” interpretation. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 9, 11-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1).
Regarding claim 1, Joly teaches:
One or more processors (see Column 53, lines 31-36, where each of these devices (110/120/525) may include one or more controllers/processors (1004/1104), which may each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (1006/1106) for storing data and instructions of the respective device); and
one or more computer-readable media having stored thereon instructions that, when executed with the one or more processors, perform a method (see Column 53, lines 36-49, where the memories (1006/1106) may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magneto resistive memory (MRAM), and/or other types of memory. Each device (110/120/525) may also include a data storage component (1008/1108) for storing data and controller/processor-executable instructions. Each data storage component (1008/1108) may individually include one or more non-volatile storage types such as magnetic storage, optical storage, solid-state storage, etc. Each device (110/120/525) may also be connected to removable or external non-volatile memory and/or storage (such as a removable memory card, memory key drive, networked storage, etc.) through respective input/output device interfaces (1002/1102). Also see Column 53, lines 50-59, where computer instructions for operating each device (110/120/525) and its various components may be executed by the respective device's controller(s)/processor(s) (1004/1104), using the memory (1006/1106) as temporary “working” storage at runtime. A device's computer instructions may be stored in a non-transitory manner in non-volatile memory (1006/1106), storage (1008/1108), or an external device(s). Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or instead of software);
receiving a first input defining the semantic content of the output audio (see Column 13, lines 30-32, where method 450 may include receiving (455) input data representing natural language content for creation of synthesized speech),
identifying, from one or more latent spaces each associated with one or more possible characteristics of the output audio, one or more parameters for use in generating the output audio based at least upon the second input (see column 3, lines 19-25, where a variational autoencoder can include an encoder trained in a manner that organizes its latent space to facilitate generation of useful output from points within the latent space. Thus, the emphasis model may be trained to receive inputs (e.g., audio data and, in some cases, emphasis data) and generate outputs (e.g., acoustic embeddings). Also see Column 7, lines 6-10, where the decoder component 160 may be, for example, an acoustic model that may receive the phoneme embedding data 145 as well as acoustic embedding data 155 and output audio data 161 representing the synthesized speech); and
generating the output audio based at least upon the first input and the identified one or more parameters (see Column 3, lines 45-48, where a decoder may receive the acoustic embeddings and phoneme embeddings (e.g., summed or combined), and process them to generate audio data representing synthesized speech).
Joly fails to teach receiving a second input defining one or more desired characteristics of the output audio, wherein the second input defining the one or more desired characteristics is dynamically determined in response to an in-game event or an in-game environmental condition.
However, Pinto does teach a second input defining one or more desired characteristics determined in response to an in-game event or environmental condition (see [0004], where some aspects feature a method for automatically adjusting speech of video game characters comprising: receiving a speech script including a sequence of words to be spoken in a video game; generating one or more context tags based on a game state of the video game in which the sequence of words will be spoken. Also see [0007], where dynamically generating the one or more context tags in a video game based on a video game state that indicates a context in which the speech script is configured to be read. The context tags include at least two of: a video game title or series; a speaker attribute; a video game mode; a video game level; a location in the video game; and an event that occurred in the video game. Also see [0058], where the video game engine 163 can generate context tags describing the setting or game state in which the script will be spoken. For example, the video game engine 163 can provide context tags such as the name of the character who will speak the script, the gender of the character who will speak the script, the place of origin of the character who will speak the script, the age of the character who will speak the script, the species of the character who will speak the script, and other character attributes. The video game engine 163 can also generate context tags describing the video game title, the level, the game mode, recent events that occurred in the video game, and the like).
Joly and Pinto are both considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques, speech synthesis, and speech conversion. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly to incorporate the teachings of Pinto to dynamically determine the desired characteristics of the audio based on an in-game event or environmental condition in order to improve the character’s audio by synthesizing their voice to sound more natural and realistic according to the event currently happening in the video game (see [0002], where computer systems can read text aloud through speakers, but the result is often a monotonic, robotic, and/or unrealistic sounding voice that does not sound as though spoken by a natural person. Also, computer systems may be programmed to read a text line aloud by synthesizing sounds in a particular way to mimic human emotion. However, text spoken by a computer will sound unnatural and lack speaking adjustments that a human would make. Also see [0003], where computer generated voices are presently unrealistic in a wide variety of situations. Instead, to have realistic and appropriately sounding voices, voice actors record lines for playback in movies and in video games. The voice actors can adjust their reading of lines to sound natural and realistic in different contexts that may occur in the movies or video games, even if the voice actors repeat the same words).
Regarding claim 2, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Joly teaches wherein the first input comprises text and/or audio comprising speech (see Column 2, lines 25-27, where a speech-processing system that a user can interact with by providing natural language inputs (e.g., human speech and/or text)).
Regarding claim 3, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Joly teaches wherein the desired characteristics include one or more of:
an emotion of a speaker associated with the output audio (see Column 46, lines 22-24, where the emotion features may include data corresponding to an emotion associated with the text data 111, such as surprise, anger, or fear);
an age of a speaker associated with the output audio (see Column 46, lines 24-26, where the speaker features may include data corresponding to a type of speaker, such as sex, age, or profession);
a gender of a speaker associated with the output audio (see Column 46, lines 24-26, where the speaker features may include data corresponding to a type of speaker, such as sex, age, or profession);
a nationality of a speaker associated with the output audio (see Column 46, lines 26-28, where the accent features may include data corresponding to an accent associated with the speaker, such as Southern, Boston, English, French, or other such accent); and/or
an accent of a speaker associated with the output audio (see Column 46, lines 26-28, where the accent features may include data corresponding to an accent associated with the speaker, such as Southern, Boston, English, French, or other such accent).
Regarding claim 5, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Joly teaches wherein the method further comprises:
generating new audio to obtain the output audio (see Column 15, lines 8-10, where method 450 has a decoder that may process the combined phoneme and acoustic embedding data to generate audio data), which fully corresponds to the first alternative of “generating new audio to obtain the output audio and/or configured to modify the first input, in the case that the first input is audio comprising speech, to obtain the output audio" limitation in claim 5. As such, Joly fully meet the entirety of claim 5.
Regarding claim 9, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Joly teaches wherein the method further comprises:
selecting one or more latent spaces from which to identify parameters based at least upon the first input and/or the second input (see Column 3, lines 19-25, where a variational autoencoder can include an encoder trained in a manner that organizes its latent space to facilitate generation of useful output from points within the latent space. Thus, the emphasis model may be trained to receive inputs (e.g., audio data and, in some cases, emphasis data) and generate outputs (e.g., acoustic embeddings). Also see column 7, lines 6-10, where the decoder component 160 may be, for example, an acoustic model that may receive the phoneme embedding data 145 as well as acoustic embedding data 155 and output audio data 161 representing the synthesized speech).
Regarding claim 11, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Joly teaches wherein the method further comprises:
identifying a plurality of sets of parameters (see Column 3, lines 19-25, where a variational autoencoder can include an encoder trained in a manner that organizes its latent space to facilitate generation of useful output from points within the latent space. Thus, the emphasis model may be trained to receive inputs (e.g., audio data and, in some cases, emphasis data), and
generating the output audio in a multi-stage process, in which each stage corresponds to the use of a different one of the plurality of sets of parameters (see Column 7, lines 6-10, where the decoder component 160 may be, for example, an acoustic model that may receive the phoneme embedding data 145 as well as acoustic embedding data 155 and output audio data 161 representing the synthesized speech).
Regarding claim 12, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Joly teaches wherein the method further comprises:
reproducing the generated output audio (see Column 15, lines 14-16, where the system 100 may output the synthesized speech; for example, from the loudspeaker 112 as output audio 14).
Regarding claim 13, which recites a method, this claim is rejected as unpatentable over the same prior art and reasoning applied against claim 1. As detailed in the rejection of claim 1, the disclosed system teaches each step of the method recited in claim 13. Accordingly, claim 13 is rejected for the same reasons set forth in the rejection of claim 1.
Regarding claim 14, which recites a non-transitory machine-readable storage medium, this claim is rejected as unpatentable over the same prior art and reasoning applied against claim 1. As detailed in the rejection of claim 1, the disclosed system teaches each step of the non-transitory machine-readable storage medium recited in claim 14. Accordingly, claim 14 is rejected for the same reasons set forth in the rejection of claim 1.
Regarding claim 15, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Pinto teaches wherein the in-game event or the in-game environmental condition is associated with an action taken by a user or character in an interactive environment (see [0032], where events in video games can include talking to certain characters, completing mission objectives, fighting, equipping items, using items, obtaining items, reaching checkpoints, using skills, making storyline decisions, trading with other players, making in-game transactions, and the like).
Claim(s) 4, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1, and further in view of Lumbantobing (US 12456450 B1).
Regarding claim 4, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1 but fails to teach applying a normalization process to the first input.
However, Lumbantobing does teach applying a normalization process to the first input (see Column 6, lines 3-15, where the frame-level representation 146 can be considered a representation of the speech waveform 102 without parts or all of the speaker characteristics 118. For example, the frame-level representation 146 does not have any or most of the information regarding the speaker conditioning of the speech waveform 102. In some implementations, the frame-level representation can have durational components related to phonemes. The frame-level representation 146 may have information regarding the style and noise conditioning of the speech waveform 102 that is speaker normalized, to increase the modeling capability related to different expressivity and environmental settings. Also see Column 3, lines 40-48, where the speaker characteristics 118 can include speaker conditioning, noise conditioning, and style conditioning as described herein. Speaker conditioning can include the frequencies at which the speaker speaks. Noise conditioning can include ambient noise and background noise to the audio of the source speech waveform. Style conditioning can include changes to the voice, speaking speed, cadence, and the like due to emotion or tone of the speaker).
Joly, Pinto, and Lumbantobing are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques, speech synthesis, and speech conversion. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Lumbantobing to apply a normalization process to the first input in order to prevent specific linguistic and audio artifacts (e.g., slang or volume) in the speech input from degrading the quality and fidelity of the synthesized output, thus enabling better control of the target characteristics during the synthesis process (see Column 1, lines 5-15, where recording voices has become very common. Many types of media involve the recording of voices. The recording of spoken words and non-verbal components can be referred to as a speech waveform. Effects can be applied to speech waveforms to create certain distortions, however the distorted voices are just altered voices and can sound unnatural, and may intentionally sound unnatural. Similarly, text-to-speech can be used to generate speech waveforms. However, the speech waveforms are usually distinctly unnatural as computer-generated. These speech waveforms can have dissonant tone, style, and emotion).
Regarding claim 18, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1. Furthermore, Joly teaches:
the first input comprises input voice content (see Column 13, lines 59-67, where method 450 may include receiving (460) speaker embedding data representing desired voice characteristics for the creation of synthesized speech. The system 100 may receive the speaker embedding data from another source (e.g., one associated with the input data), retrieve the speaker embedding data from a storage (e.g., a profile storage 570), and/or receive the speaker embedding data via user input (e.g., where a user selects from available/modifiable voice characteristics)),
but fails to teach applying an initial filter to the input voice content to generate neutralized content having reduced characteristics specific to a source user providing the input voice content prior to generating the output audio.
However, Lumbantobing does teach applying an initial filter to the input voice content to generate neutralized content having reduced characteristics specific to a source user providing the input voice content prior to generating the output audio (see Column 9, lines 61-67, and Column 10, lines 1-27, where the voice conversion system can also include an invertible normalizing flow module 220. The invertible normalizing flow module 220 can be trained by the training system (for example, the invertible normalizing flow module 220 can be the invertible normalizing flow module 120 of FIG. 1 that was trained). The invertible normalizing flow module 220 can be a machine learning system that can both extract the speaker characteristics out of a representation of a speech waveform (for example, the sampled encoded audio representation 210) as well as add in new speaker characteristics (for example, the target speaker characteristics 240) into a speech waveform or a representation of the speech waveform. The invertible normalizing flow module 220 can receive the sampled encoded audio representation 210 and the source speaker characteristics 218 to generate a flow-normalized representation 222. The flow normalized representation 222 can be a normalized version of the sampled encoded audio representation 210 such that the source speaker characteristics 218 are extracted out. This process of generating a flow-normalized representation 222 can be referred to as forward flow. The invertible normalizing flow module 220 can receive the flow-normalized representation 222 and the target speaker characteristics 240 to generate a target speech representation. The target speech representation includes the target speaker characteristics 240 entangled into the flow-normalized representation 222. This entanglement of the target speaker characteristics 240 into the flow-normalized representation 222 can be referred to as reverse flow. By using the invertible normalizing flow module 220 to entangle the target speaker characteristics 240 into the flow-normalized representation 222, the expressivity and the randomness of the speaking manner of the target speaker can be preserved in the target speech waveform 214).
Joly, Pinto, and Lumbantobing are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques, speech synthesis, and speech conversion. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Lumbantobing to apply an initial filter to the first input to neutralize and reduce the characteristics of the input voice in order to prevent specific linguistic and audio artifacts (e.g., slang or volume) in the speech input from degrading the quality and fidelity of the synthesized output, thus enabling better control of the target characteristics during the synthesis process (see Column 1, lines 5-15, where recording voices has become very common. Many types of media involve the recording of voices. The recording of spoken words and non-verbal components can be referred to as a speech waveform. Effects can be applied to speech waveforms to create certain distortions, however the distorted voices are just altered voices and can sound unnatural, and may intentionally sound unnatural. Similarly, text-to-speech can be used to generate speech waveforms. However, the speech waveforms are usually distinctly unnatural as computer-generated. These speech waveforms can have dissonant tone, style, and emotion).
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1, and further in view of Wexler (US 20230045237 A1).
Regarding claim 6, which depends on claim 5, Joly in view of Pinto teaches all the limitations of claim 5 but fails to teach the parameters comprise one or more filters to be applied to the first input, and wherein the method further comprises: applying the one or more filters to the first input.
However, Wexler does teach wherein:
the parameters comprising one or more filters to be applied to the first input (see [0314], where the speech rendering process may include changing a pitch of a person speaking (e.g., such rendering may be beneficial if user 100 has difficulty in discerning particular frequencies), a cadence of the person's speech, the loudness of the person's speech, or any other characteristics of the person's speech (e.g., a filter may be applied to the person's speech to change the voice of a person from a male to a female voice), and
wherein the method further comprises: applying the one or more filters to the first input (see [0596], where a processor 210 may filter components, or add other components, to the first audio signal to enhance or reduce certain features and make the first audio signal more closely match the characteristics of another voice. For example, processor 210 may alter the voice of the individual by altering the pitch of the individual's voice. Thus, if the individual is a man, processor 210 may process the first audio signal to increase its pitch so as to more clearly match a woman's voice or may add overtones characteristic of a particular person's voice).
Joly, Pinto, and Wexler are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques or speech synthesis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Wexler to include parameters comprising one or more filters to be applied to the first input, and configured to apply the one or more filters to the first input in order to apply the desired characteristics of the second input to the semantic content of the first input, and improve the representation and clearness of the generated audio output (see [0280], where the speech of the person may be modified such that the speech clarity is improved).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1, and further in view of Deleeuw (WO 2015047248 A1).
Regarding claim 7, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1 but fails to teach modifying one or more aspects of the first input so as to substitute one or more words or phrases represented by the first input with alternatives.
However, Deleeuw does teach modifying one or more aspects of the first input so as to substitute one or more words or phrases represented by the first input with alternatives (see Page 12, lines 16-18, where the computing device 100 may modify the word choice of the natural language response, for example by using a dictionary associated with a particular emotion).
Joly, Pinto, and Deleeuw are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques or speech synthesis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Deleeuw and modify one or more aspects of the first input so as to substitute one or more words or phrases represented by the first input with alternatives in order to allow for better representation and clearness of the characteristics desired by the user, and better flexibility of expression given a specific word or phrase (see Page 1, lines 6-8, where cues can also convey social information among humans, including indicating the speaker's dominant or non-dominant status in a conversation. Also see Page 1, lines 13-14, where communicating those emotional and social cues may allow humans to increase clarity of communication).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1, and further in view of Nesta (US 20200349965 A1).
Regarding claim 8, which depends in claim 1, Joly in view of Pinto teaches all the limitations of claim 1 but fails to teach the one or more latent spaces which are configured in a hierarchical manner, such that a latent space lower in the hierarchy represents a subset of the characteristics of a latent space higher in the hierarchy.
However, Nesta does teach the one or more latent spaces are configured in a hierarchical manner, such that a latent space lower in the hierarchy represents a subset of the characteristics of a latent space higher in the hierarchy (see [0024], where a user interface is provided allowing the user to modify certain variables of the embedding which are directly linked to some high-level speech characteristics. At the same time, some prior embeddings for typical noise context can also be retrieved by inputting an audio example and by exploring the embedding space map in hierarchal manner. Also see [0032], where a hierarchical map (such as map interface 512) of noise types may allow the user to search for a sound that is similar to the one in the mixed recording. The hierarchical map can be built in a data-driven way, for example, by learning discriminant embeddings of the sound examples and by comparing them to build a semantic map of sound examples. A user could, for example, start with a category like "animal", then find "dog", and select lower branches of dog breeds by listening to sample recordings at each level).
Joly, Pinto, and Nesta are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques or speech synthesis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Nesta and configure one or more latent spaces in a hierarchical manner, such that a latent space lower in the hierarchy represents a subset of the characteristics of a latent space higher in the hierarchy in order to allow for better attribute control and a more understandable way of navigating specific speech characteristics (see [0032], where semantic spaces (e.g., noise types) can be sometimes difficult to handle from a user's perspective due to the lack of an understandable interface).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1, and further in view of Wang (US 20240274122 A1).
Regarding claim 10, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1 but fails to teach the one or more latent spaces which are generated using spectrograms of voice samples.
However, Wang does teach the one or more latent spaces which are generated using spectrograms of voice samples (see [0050], where the spectrogram encoder 210 may be trained to encode the speech represented in the spectrogram data 125 into a latent space from which the acoustic embedding decoder 220 can reproduce speech).
Joly, Pinto and Wang are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques or speech synthesis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Wang to generate one or more latent spaces using spectrograms of voice samples. Doing so would improve the cluster classification of the voice characteristics desired by the user and allow for a better representation of the natural voice (see [0050], where the spectrogram encoder 210 and the acoustic embedding decoder 220 may be trained to encode the spectrogram data 125 into acoustic embedding data Z 215 and decode it into an output audio data 185 that includes a faithful reproduction of speech represented in the spectrogram data 125).
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1, and further in view of Harazi et al. (US 20230252972 A1).
Regarding claim 16, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1 but fails to teach wherein identifying the one or more parameters comprises sampling a midpoint between a first location and a second location in the one or more latent spaces, the first location and the second location corresponding to contrasting emotion characteristics, to obtain parameters representing a neutral or blended emotion.
However, Harazi does teach wherein:
identifying the one or more parameters comprises sampling a midpoint between a first location and a second location in the one or more latent spaces, the first location and the second location corresponding to contrasting emotion characteristics, to obtain parameters representing a neutral or blended emotion (see [0112], where the emotion input module 630 can implement a machine learning technique that automatically estimates coefficients of style components or scaling factors of style components to provide the mix of different emotions or levels or intensities of a given emotion. The coefficients or scaling factors can be used by the style combination module 640 to linearly interpolate between different styles to generate the style component associated with the indicated level or intensity or mix of different styles. For example, the style combination module 640 can generate a new style component by combining multiple style components of different emotions. For example, the style combination module 640 can scale or apply a first coefficient corresponding to a first level to a first style component corresponding to a first emotion. The style combination module 640 can scale or apply a second coefficient corresponding to a second level to a second style component corresponding to a second emotion. The style combination module 640 can aggregate or interpolate between the scaled first and second style components (adjusted vector values of the different style components) to generate a new vector corresponding to the new style component representing the intensity or level of emotion specified by the emotion input module 630).
For purposes of this rejection, the term “midpoint” is being interpreted under broadest reasonable interpretation as a position within a latent space that is located between two reference positions, representing an intermediate of blended characteristics between the two reference positions. While the claim language uses the term “midpoint,” Harazi does not use the same terminology when referring to a specific equal-distance point between two emotion embeddings, and instead uses the word “interpolate.” Interpolation is a broader mathematical concept that encompasses any position along a path between two reference points, including but not limited to the exact midpoint. For example, interpolation may sample at 25%, 50%, or 75% along the path between two emotion embeddings or vectors. The mere difference in terminology does not create a substantive distinction and serve the same function of combining two reference positions or emotion embeddings or vectors to obtain an intermediate or blended characteristic.
Joly, Pinto and Harazi are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques or speech synthesis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Harazi to interpolate between two distinct latent spaces representing emotions in order to further improve the realism of the output speech at synthesis time (see [0016], where the disclosed techniques can determine that there does not exist an embedding for the first speaker and the second emotion. In response, the disclosed techniques generate, based on the first embedding and the second embedding, a third embedding for the first speaker associated with the second emotion and apply the third embedding and the text string to an acoustic generator and vocoder to generate an audio stream that includes the one or more words being spoken by the first speaker with the second emotion. Also see [0017], where this significantly improves the overall ability for the social network system to perform automated speech generation and enhances the realism associated with such generated speech. In this way, the disclosed techniques provide a much more intuitive, robust, and engaging experience and avoid having users record individual audio files for any given text string. This reduces the amount of resources needed to perform routine social networking tasks and operations).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1), and further in view of Asano et al. (US 7065490 B1).
Regarding claim 19, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1 but fails to teach wherein the one or more parameters comprise volume, pitch, and speed parameters, and wherein generating the output audio comprises using the volume, the pitch, and the speed parameters in conjunction with a dictionary library to generate the output audio.
However, does teach:
wherein the one or more parameters comprise volume, pitch, and speed parameters (see Column 1, lines 42-46, where the control means may control phonemics information and pitch information of synthesized sound output by the voice synthesizing means, and the control means may also control the speech speed or volume of synthesized sound output by the voice synthesizing means), and
wherein generating the output audio comprises using the volume, the pitch, and the speed parameters in conjunction with a dictionary library to generate the output audio (see Column 15, lines 30-46, where the dictionary storing unit 34 has stored therein word dictionaries describing part of speech information for each word, reading, accentuation, and other information thereof. Also, the analyzing grammar storing unit 35 stores analyzing syntaxes relating to restrictions of word concatenation and the like, regarding the words described in the word dictionaries in the dictionary storing unit 34. Then, the text generating unit 31 performs morpheme analysis and grammatical structure analysis of the input text based on the word dictionaries and analyzing syntaxes, and extracts information necessary to the rule voice synthesizing performed by the latter rules synthesizing unit 32. Here, examples of information necessary for rule voice synthesizing include pause positions, pitch information such as information for controlling accents and intonation, phonemics information such as the pronunciation and the like of each word, and so forth).
Joly, Pinto and Asano are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques or speech synthesis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Asano to determine the synthesized output audio parameters based on volume, pitch, speed, and a pronunciation dictionary in order to better reflect the emotional state of the output speech in a more realistic manner (see Abstract, where in the event that the emotion state of the robot represents "not angry", synthesized sound of "What is it?" is generated at the voice synthesizing unit. On the other hand, in the event that the emotion state of the robot represents "angry", synthesized sound of "Yeah, what?" is generated at the voice synthesizing unit, to express the anger. Thus, a robot with a high entertainment nature is provided).
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Joly et al. (US 12243511 B1) in view of Pinto (US 20200251089 A1), and further in view of Karlapati et al. (US 12488782 B1).
Regarding claim 20, which depends on claim 1, Joly in view of Pinto teaches all the limitations of claim 1 but fails to teach wherein the one or more latent spaces are configured in a hierarchical manner such that a first latent space lower in the hierarchy represents a specialized subset of emotion characteristics of a generic second latent space higher in the hierarchy.
However, Karlapati does teach wherein the one or more latent spaces are configured in a hierarchical manner such that a first latent space lower in the hierarchy represents a specialized subset of emotion characteristics of a generic second latent space higher in the hierarchy (see Column 3, lines 42-51, where a first prosody prediction component may process the input data (e.g., text input data) to determine first prosody embedding data for a first hierarchy level (e.g., sentence-level hierarchy, paragraph-level hierarchy, or chapter-level hierarchy) that represents the prosody of the input data at that level of hierarchy. If the first level of hierarchy is sentence-level hierarchy, for example, the first prosody embedding data may represent the emotion, speech rate, or mood of the sentence).
Joly, Pinto, and Karlapati are considered to be analogous to the claimed invention because they are in the same field of speech analysis techniques or speech synthesis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Joly and Pinto to incorporate the teachings of Karlapati and configure one or more latent spaces in a hierarchical manner, such that a latent space lower in the hierarchy represents a specialized subset of emotion characteristics of a latent space higher in the hierarchy in order to allow for better attribute control and generate speech that is closer to the intended emotion to convey (see Column 2, lines 3-9, where “prosody” refers to the manner in which a given word, sentence, paragraph, or other unit of speech is spoken. Aspects of prosody may include the rate of the speech, the loudness of the speech, how syllables, words, or sentences in the speech are emphasized, when and where pauses in the speech may be inserted, or what emotion (e.g., happy, sad, or anxious) is expressed in the speech. Also see Column 2, lines 10-22, where aspects of the present disclosure thus relate to predicting prosody to be applied to input data, such as text or other meaning representation data, and then using the predicted prosody to generate audio data representing speech that reflects the predicted prosody. The speech-synthesis component may first be trained using training data that includes text data and corresponding speech data exhibiting corresponding prosody. The training data may be, for example, audio data representing a voice actor reading written content, such as a book, and corresponding text data representing the text of the book or metadata describing the text, such as locations of word boundaries, sentence boundaries, or chapter boundaries).
Allowable Subject Matter
Claim 17 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN HONG FANG-WU whose telephone number is (571)270-0607. The examiner can normally be reached Monday - Friday, 9AM to 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras Shah can be reached at (571)-270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN HONG FANG-WU/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
10/07/2026
Prosecution Insights