Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-20 are pending. Claims 1, 18, and 20 are independent.
This Application was published as U.S. 20250028742.
Apparent priority: 20 June 2024.
An internet communications authorization form SB/439 is on file for this Application.
Note that while the Specification includes: “[0095] … In no case is the computer readable storage media a propagated signal,” Claims 18-19 are directed to “A memory” and not to “A computer readable storage media.” The more prevalent form is to claim “a non-transitory computer readable medium” or “a non-transitory computer readable storage medium” which circumvents having to refer to the Specification or a potential 35 U.S.C. 101 rejection. The term “memory,” however, is considered to have a defined structure in the context of the instant art and is not considered non-statutory.
Abstract:
Systems, methods, software, and devices are disclosed herein process context data to encode one or more semantic elements of a desired audio composition in a semantic token sequence, process the semantic token sequence to encode one or more structural elements of the desired audio composition in a structural token sequence disentangled from the semantic token sequence, and process the structural token sequence to encode one or more audio signal elements of the desired audio composition in an audio signal token sequence disentangled from the structural token sequence. The semantic token sequence, the structural token sequence, and the audio signal token sequence may then be processed to generate at least a portion of the desired audio composition.
PNG
media_image1.png
502
392
media_image1.png
Greyscale
[0043] In operation, the computing device receives input comprised of context data (step 201). The context data may be, for example, a text string that indicates a genre, mood, or other such semantic feature of the desired generative audio. Alternatively, or in addition, the context may be derived or otherwise inferred from some other type of input such as a representative clip of sound, music, speech, or the like.
[0044] … The sequence of semantic tokens function to influence the audio that is ultimately produced to be semantically similar to or representative of the desired context.
[0024] … Each level of codes represents different aspects of the audio: the top-level codes capture high-level concepts like genre and mood, the mid-level codes capture structural features such as rhythm patterns and phoneme segmentation, and the low-level codes capture basic signal properties.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Agostinelli (U.S. 11915689) in view of Yang (Yang et al., “UNIAUDIO: AN AUDIO FOUNDATION MODEL TOWARD UNIVERSAL AUDIO GENERATION, 2023, https://arxiv.org/pdf/2310.00704v5) and further in view of Santhosh Kumar (U.S. 20240184534).
Regarding Claim 1, Agostinelli teaches:
1. An audio generation method, wherein the method uses a processor coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor, carry out steps of the method, [Agostinelli, “Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them….” 30:36-32:56 teach the various hardware and software components used for implementing the method of the reference including: “Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both….” 31:40-45.]
comprising:
executing a first code generator [Agostinelli: Figure 1, 120 and 130 teach the “first code generator” or “encoder” which together generate the “semantic token sequence” of the Claim: “In some implementations, the system 100 can also include an embedding neural network 120. In some examples where the context includes an input, the system 100 can process the input to map the input to one or more embedding tokens 122, also referred to as audio embedding tokens.” 7:59-63. “… Furthermore, embeddings that are close together in the joint embedding space signify that the embeddings share semantics within and across modalities. For example, two embeddings that are close to each other can represent two text sequences that are semantically similar, two audio samples that are semantically similar, or an audio sample and a text sequence with semantically similar features….” 8:7-13.] to generate, based on context data, a semantic token sequence having one or more semantic elements of a desired audio composition encoded therein; [Agostinelli: Figure 1, “To generate audio, the system 100 receives the request 102. In some implementations, the request 102 can specify a context for the audio signal 104. In these implementations, the audio signal 104 is conditioned on the context.” The “request 102” of Figure 1 teaches the “context data” of the Claim. Figure 4 shows the generation of “semantic representation 106” from the “request 102.” See also Figure 5 where the input/ “context data” is “Hip Hop song with violin solo 502” and the output is “Generated audio 504.”]
executing a second code generator to generate, based on the semantic token sequence, a structural token sequence disentangled from the semantic token sequence and having one or more structural elements of the desired audio composition encoded therein;
executing a third code generator [Agostinelli: Figure 1, “one or more generative neural networks 108.”] to generate, based on the structural token sequence, an audio signal token sequence disentangled from the structural token sequence and having one or more audio signal elements of the desired audio composition encoded therein; and [Agostinelli: Figure 1, “one or more generative neural networks 108” which generates the “acoustic representation 110” from the “semantic representation 106” but leaves out the intermediate “structural token sequence” of the Claim. “The system 100 then generates, using one or more generative neural networks 108 and conditioned on at least the semantic representation 106, an acoustic representation 110 of the audio signal 104.” 8:37-40.]
executing a decoder [Agostinelli: Figure 1, “Decoder neural network 112.”] to obtain, based on the semantic token sequence, the structural token sequence, and the audio signal token sequence, at least a portion of the desired audio composition. [Agostinelli: Figure 1, “prediction of audio signal 104” as the output. “The audio generation system 100 generates a prediction of an audio signal 104 given a request 102 to generate an audio signal….” 7:24-26. The output of the audio signal 104 is based on the “semantic representation 106” and the “acoustic representation 110.” See also Figure 5 where the input/ “context data” is “Hip Hop song with violin solo 502” and the output is “Generated audio 504.”]
Agostinelli does not teach the intermediate “second code generator” of this Claim.
Yang teaches:
executing a first code generator [The first, second, and third encoders or code generators are taught by “… The residual vector quantization (Zeghidour et al., 2021) based on neural codecs is used in the tokenization process, resulting in overly long token sequences (one frame corresponding to multiple tokens) that cannot be processed efficiently by LLM….” Section 1, p. 2.] to generate, based on context data, a semantic token sequence having one or more semantic elements of a desired audio composition encoded therein; [Yang, “Toward universal audio generation, this work presents UniAudio, which adopts LLM techniques and is able to generate multiple types of audio (speech, sounds, music, and singing) conditioned on various input modalities, such as phoneme sequences, textual descriptions, and audio itself. The proposed UniAudio is mainly featured as follows: First, all types of audio, along with all other input modalities, are tokenized as discrete sequences….” Section 1, p. 2. “Textual descriptions” teaches the “context data.”]
executing a second code generator to generate, based on the semantic token sequence, a structural token sequence disentangled from the semantic token sequence and having one or more structural elements of the desired audio composition encoded therein; [Yang, “Toward universal audio generation, this work presents UniAudio, which adopts LLM techniques and is able to generate multiple types of audio (speech, sounds, music, and singing) conditioned on various input modalities, such as phoneme sequences, textual descriptions, and audio itself. The proposed UniAudio is mainly featured as follows: First, all types of audio, along with all other input modalities, are tokenized as discrete sequences. Specifically, a universal neural codec model is built to effectively tokenize audio regardless of the audio type, and other tokenizers are used to tokenize other different modalites. Then, UniAudio concatenates the source-target pair as a single sequence. Lastly, UniAudio performs next-token prediction using LLM.” Section 1, p. 2. The phoneme sequence of Yang teaches the “structural token sequence” of the Claim. All the modalities are treated separately and are “disentangled” from one another. “Phoneme: Phonemes are the basic units of speech pronunciation in linguistics. Phoneme sequences have multiple sources: (1) when only text is available, phoneme sequence without duration information can be obtained by text-to-phoneme mapping using a pronunciation dictionary; (2) when only speech is available, phoneme sequence with duration information is obtained by beam search of the DNN-HMM system (Hinton et al., 2012); (3) when both text and speech are available, phoneme sequence with duration information is obtained by forced alignment of the DNN-HMM system 2.” Section 2.1.2.]
executing a third code generator to generate, based on the structural token sequence, an audio signal token sequence disentangled from the structural token sequence and having one or more audio signal elements of the desired audio composition encoded therein; and [Yang, “Toward universal audio generation, this work presents UniAudio, which adopts LLM techniques and is able to generate multiple types of audio (speech, sounds, music, and singing) conditioned on various input modalities, such as phoneme sequences, textual descriptions, and audio itself. The proposed UniAudio is mainly featured as follows: First, all types of audio, along with all other input modalities, are tokenized as discrete sequences. …” Section 1, p. 2.]
executing a decoder to obtain, based on the semantic token sequence, the structural token sequence, and the audio signal token sequence, at least a portion of the desired audio composition. [Yang, Figure 1 shown below, the different streams of tokens at the bottom layer are all input to the higher levels of the decoding to generate the output of the “predicted audio sequence.”]
PNG
media_image2.png
336
816
media_image2.png
Greyscale
Agostinelli and Yang pertain to autoregressive training and generation of audio from an input context (“In addition, autoregressively generating semantic and acoustic tokens, and then mapping the acoustic tokens to an audio sample saves computational resources compared to alternatives such as processing audio samples directly.” Agostinelli, 6:14-18. “To be more detailed, as the acquisition of each token zk t is auto-regressively dependent on its prior tokens {zj t|j < k}, a local Transformer is adopted to predict the patch sequence zt in auto-regressive style.” Yang, Section 2.3.) and it would have been obvious to modify Agostinelli to include the disentangled phoneme sequence generation (structural token sequence of the Claim) in addition to the conversion of the input context to semantic tokens and generation of audio output from a plurality of separate/disentangled input token sequences as shown in Figure 1 of Yang for a complete UniAudio like system that is discussed in the background of the instant Application for the benefits of a UniAudio system as stated in the Abstract of Yang. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Agostinelli and Yang teach encoders and not “code generators” per se which according to the dependent Claims of the instant Application are trained by encoders and effectively do as indicated by the encoders that are taught by Agostinelli and Yang.
Agostinelli in Figures 4 and 8 shows the training processes for “Audio Representation Neural Network 410” which generates the “semantic tokens S.” This is the training of the “encoder” and not training a “code generator” using a trained “encoder.”
Yang teaches building an improved “neural codec model” “Specifically, a universal neural codec model is built to effectively tokenize audio regardless of the audio type …” Section 1. This is an encoder/decoder system and not a code generator per se.
Kumar teaches the “code generators” of the Claim in Figure 1 at 140:
PNG
media_image3.png
652
562
media_image3.png
Greyscale
Regarding Claim 2, Agostinelli and Yang do not teach training code generators by encoders.
Santhosh Kumar teaches:
2. The audio generation method of claim 1 further comprising:
training the first code generator using a first encoder trained to generate semantic tokens based on audio data; [Santhosh Kumar teaches the Encoders 110, 120, 130 uses to train a “source code generation decoder 140.” ‘Mechanisms are provided for generating electronic data interchange mapping source code. A source code generator, comprising a machine learning trained neural network computer model, receives input data comprising a source document data structure in a first format and a destination document data structure in a second format, different from the first format. The source code generator processes the input data to generate a source code sequence output vector. A source code is generated that maps contents of source documents in the first format to contents of destination documents in the second format based on the source code sequence output vector. The source code is output to a computing device for execution on other source documents that utilize the first format to thereby automatically generate corresponding destination documents in the second format.” Abstract.]
training the second code generator using a second encoder trained to generate structural tokens disentangled from the semantic tokens; and [Santhosh Kumar teaches the Encoders 110, 120, 130 uses to train a “source code generation decoder 140.” Any number of code generators can be trained by the method of Kumar for performance of various tasks. “[0029] In some illustrative embodiments, different instances of the trained encoder-decoder model may be provided that are specific to the particular combination of source document format, destination document format, and MRS rules/logic. …”]
training the third code generator using a third encoder trained to generate audio signal tokens disentangled from the semantic tokens and the structural tokens. [Santhosh Kumar teaches the Encoders 110, 120, 130 uses to train a “source code generation decoder 140.” Any number of code generators can be trained by the method of Kumar.]
Agostinelli/Yang and Santhosh Kumar pertain to use of encoder/decoder models and it would have been obvious to combine the code generation of Santosh Kumar with the system of combination as a precursor to the system or in place of the encoders of Agostinelli or Yang. This combination falls under combining prior art elements according to known methods to yield predictable results or simple substitution of one known element for another to obtain predictable results. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 3, Agostinelli teaches:
3. The audio generation method of claim 2 further comprising training the first encoder by at least: [Agostinelli: Figures 4 and 8 show training processes for the encoders used in the steps of audio generation.]
processing the audio data to generate semantic embeddings; [Agostinelli: in both figures the input is “target audio 450, 850.”]
executing the first encoder to map the semantic embeddings to semantic tokens; and [Agostinelli: Figure 4, “audio representation neural network 410” and Figure 8, “audio representation neural network 410.” Both generating “semantic tokens S.”]
computing first losses based on the semantic embeddings and ground-truth semantic embeddings known for an audio segment. [Agostinelli: loss calculation and minimization is part of training: “The audio representation neural network 410 can have been trained to generate representations of input audio signals. In some implementations, the audio representation neural network 410 can be pre-trained and frozen prior to the training of the audio generation system. The audio representation neural network 410 can be trained to minimize masked language model (MLM) loss and contrastive loss….” 18:1-10.]
(See also Santhosh Kumar: “[0027] The encoders and decoder are trained through machine learning processes to generate the source code based on source documents, destination documents, and MRS rules/logic. For example, the training data samples of source documents, destination documents, and MRS rules/logic are input to the respective encoders, and the corresponding source code embeddings are generated. The generated embeddings are then passed to the decoder and a final output is generated. This final output compared to ground truth to determine a loss. The loss is then used by machine learning training logic, e.g., back-propagation, to update operational parameters with adjustments to the encoders and decoder to determine modified operational parameters that attempt to reduce the loss until a convergence criterion is achieved, e.g., a loss equal to or below a threshold loss or a predetermined number of training iterations/epochs are performed. In this way, the encoders and decoder learn a correlation between input patterns of text in the respective ones of the source document, destination document, and MRS, and corresponding source code that are output as the embedding vector outputs of the decoder.”)
Regarding Claim 4, Agostinelli teaches:
4. The audio generation method of claim 3 further comprising:
updating parameters of the first code generator based on the first losses; and [Agostinelli: Figures 4 and 8 which show the training imply update of the parameters of the encoder. Optimizing the model (encoder) is the goal of any training.]
generating first residual embeddings based on differences between the semantic embeddings and the semantic tokens. [Agostinelli: “The outputs of the one or more layers of the audio representation neural network 410 can include embeddings of input audio….” 18:31-33. “The target acoustic representations for the training of the one or more generative neural networks 108 can have been generated by applying quantization to each of the respective embeddings outputted by the encoder neural network of the neural audio codec 420 at each of multiple second time steps. The quantization can be a residual vector quantization (RVQ) as described above with reference to FIG. 2.” 19:27-33.]
(See also [0027] of Santhosh Kumar above which expressly sets forth the training process in terms of loss/ground truth/parameter update.)
Regarding Claim 5, Agostinelli teaches:
5. The audio generation method of claim 4 wherein training the first code generator using the first encoder comprises:
executing the first encoder to obtain first training data, wherein the first training data comprises a first sequence of semantic tokens; [Agostinelli: Figure 4, the “first encoder” is the “one or more generative neural networks 108” which is obtaining the “semantic tokens S.”]
executing the first code generator to obtain first predicted data, wherein the first predicted data comprises a first sequence of predicted semantic tokens; [Agostinelli: Figure 4, the “first code generator” is the encoder inside the “one or more generative neural networks 108” which is obtaining the “semantic tokens S.”]
computing a first loss based on the first sequence of semantic tokens and the first sequence of predicted semantic tokens; and [Agostinelli: training of Figures 4 and 8 implies calculating loss so that it can be minimized. This is implied by training.]
updating parameters of the first code generator based on the first loss. [Agostinelli: training of Figures 4 and 8 implies calculating loss so that it can be minimized and thus obtain optimized parameters for the model/code generator. This is implied by training.]
Regarding Claim 6, Agostinelli, in Figures 4 and 8 shows a “one or more generative neural networks 108” which could include the generations of structural tokens from the input semantic tokens S but this is not taught by the reference.
Yang teaches:
6. The audio generation method of claim 4 further comprising training the second encoder by at least:
processing the first residual embeddings to generate structural tokens; and [Yang, Phoneme Tokens of Figure 1 teach the “structural tokens” of the Claim and are obtained from the residual embeddings in Yang: “… Also, a multi-scale Transformer model is proposed to handle the overly long sequences caused by the residual vector quantization-based neural codec in tokenization. …” Abstract.]
computing second losses based on the structural tokens and ground-truth structural tokens known for the audio segment. [Yang teaches the training of the UniAudio model and the training involves computing loss bases on ground truth: “Firstly, the proposed UniAudio is trained on multiple audio generation tasks jointly, which allows the model to obtain sufficient prior knowledge not only of the intrinsic properties of audio but also of the interrelationship between audio and other input modalities. Secondly, through fine-tuning, the trained model can seamlessly support more unseen audio generation tasks.” Section 1.]
(See Santhosh Kumar for calculating a loss from ground truth in training: “[0027] … The generated embeddings are then passed to the decoder and a final output is generated. This final output compared to ground truth to determine a loss. The loss is then used by machine learning training logic, e.g., back-propagation, to update operational parameters with adjustments to the encoders and decoder to determine modified operational parameters that attempt to reduce the loss until a convergence criterion is achieved, e.g., a loss equal to or below a threshold loss or a predetermined number of training iterations/epochs are performed….”)
Regarding Claim 7, Agostinelli teaches:
7. The audio generation method of claim 6 further comprising updating parameters of the second code generator based on the second losses. [Agostinelli: training of Figures 4 and 8 implies calculating loss so that it can be minimized and thus obtain optimized parameters for the model/code generator. This is implied by training. This is definitional for a training process: minimize loss to optimize model parameters.]
Regarding Claim 8, Agostinelli teaches this Claim which goes through a generic process of training only when applied to “structural tokens” as input training data. See mapping of Claim 5.
8. The audio generation method of claim 7 wherein training the second code generator using the second encoder comprises:
executing the second encoder to obtain second training data, wherein the second training data comprises a sequence of structural tokens;
executing the second code generator to obtain second predicted data, wherein the second predicted data comprises a sequence of predicted structural tokens;
computing a second loss based on the sequence of structural tokens and the sequence of predicted structural tokens; and
updating parameters of the second code generator based on the second loss.
Regarding Claim 9, Agostinelli teaches this Claim which goes through a generic process of training only when applied to “structural tokens” used to generate the “audio signal tokens.” See mapping of Claim 6.
9. The audio generation method of claim 7 further comprising training the third encoder by at least:
processing second residual embeddings to generate audio signal tokens; and
generating audio signal data based at least on the audio signal tokens.
Regarding Claim 10, Agostinelli teaches this Claim which goes through a generic process of training by minimizing loss and optimizing the parameters only when applied to third encoder of Claim 1. See mapping of Claim 7.
10. The audio generation method of claim 9 further comprising:
computing third losses based on the audio signal data and known audio signal data for the audio segment; and
updating parameters of the third code generator based on the third losses.
Regarding Claim 11, Agostinelli teaches this Claim which goes through a generic process of training only when applied to “audio signal tokens” as input training data. See mapping of Claim 5.
11. The audio generation method of claim 10 wherein training the third code generator using the third encoder comprises:
executing the third encoder to obtain third training data, wherein the third training data comprises a sequence of audio signal tokens;
executing the third code generator to obtain third predicted data, wherein the third predicted data comprises a sequence of predicted audio signal tokens;
computing a third loss based on the sequence of audio signal tokens and the sequence of predicted audio signal tokens; and
updating parameters of the third code generator based on the third loss.
Regarding Claim 12, Agostinelli teaches:
12. The audio generation method of claim 1 wherein the one or more semantic elements of the desired audio composition comprise one or more of genre, instrument, key, mood, meaning, a sound event category, and an acoustic scene. [Agostinelli: “… Examples of semantic content can also include genre, melody, harmony, and rhythmic properties for music.” 8:25-27.]
Regarding Claim 13, Agostinelli teaches:
13. The audio generation method of claim 1 wherein the one or more structural elements of the desired audio composition comprise one or more of sound texture, beat, tempo, rhythm pattern, pitch contour, scale, chord progression, and song structure. [Agostinelli: “… Examples of semantic content can also include genre, melody, harmony, and rhythmic properties for music.” 8:25-27.]
While Agostinelly teaches that rhythm that this Claim requires is included in its input, it does not categorize it under “structural elements.” Yang was cited for teaching Phonemes that are the structural elements of the Claim and Yang additionally teaches the use of “MIDI: MIDI (Zhang et al., 2022) is widely used for singing voice synthesis tasks. F0 and duration information are included in the MIDI. We use the duration information to flatten the F0 sequence, so that the frame-level F0 sequence is obtained.” Section 2.1.2. F0 sequence is pitch contour.
Regarding Claim 14, Agostinelli teaches:
14. The audio generation method of claim 1 wherein the one or more structural elements of the desired audio composition comprise grammar, syntax, speaker identity, intonation, stress, prosody, emphasis, speech rate, pauses, silences, word segmentation, phoneme segmentation, and articulatory features. [Agostinelli: “Each semantic token is selected from a vocabulary of semantic tokens and represents semantic content of the audio signal 104 at the corresponding first time step. Examples of semantic content that can be represented by the semantic tokens include linguistic content, phonetics, language syntax, and prosodic features for speech. Examples of semantic content can also include genre, melody, harmony, and rhythmic properties for music.” 8:20-27. “… Acoustic properties can include, for example, speaker identity….” 8:48. “… For example, the system can use the acoustic representations to capture speaker identity and recording conditions of an audio input of a given context.” 5:43-45. Note that grammar and syntax are taught by syntax of the reference. Speaker identity is expressly taught. “Intonation, stress, prosody, emphasis, speech rate, pauses, silences,” and “articulatory features” all fall within the “prosodic features.” “Word segmentation, phoneme segmentation,” are taught by “linguistic features.”]
While Agostinelly teaches that the parameters that this Claim requires are included in its input, it does not categorize them under “structural elements.” Yang was cited for teaching Phonemes that are the structural elements of the Claim and teaches phoneme duration in Section 2.1.2 as one of the types of information obtained from a phoneme sequence.
Regarding Claim 15, Agostinelli does not teach the use of structural elements.
Yang teaches:
15. The audio generation method of claim 1 wherein the one or more structural elements of the desired audio composition comprise event duration, event onset and offset, event patterns, and spatial features. [Yang: “Phoneme: Phonemes are the basic units of speech pronunciation in linguistics. Phoneme sequences have multiple sources: (1) when only text is available, phoneme sequence without duration information can be obtained by text-to-phoneme mapping using a pronunciation dictionary; (2) when only speech is available, phoneme sequence with duration information is obtained by beam search of the DNN-HMM system (Hinton et al., 2012); (3) when both text and speech are available, phoneme sequence with duration information is obtained by forced alignment of the DNN-HMM system.” Section 2.1.2. The phoneme information with duration can yield the list of parameters of this Clai.]
Rationale for combination as provided for Claim 1.
Regarding Claim 15, Agostinelli teaches:
16. The audio generation method of claim 1 wherein the one or more audio signal elements of the desired audio composition comprise amplitude characteristics, spectral characteristics, and temporal characteristics. [Agostinelli, “The audio sample at a given output time step can be an amplitude value of the audio wave or an amplitude value that has been compressed, companded, or both. For example, the audio sample can be a raw amplitude value or a mu-law companded representation of the amplitude value.” 1:40-45. “…In some implementations, the melody embedding neural network can be a vision transformer (ViT) that receives temporal frames of the mel spectrogram of the input audio signal and generates the melody embedding vectors for the input audio signal.” 25:21-25. “… For example, the system can use semantic representations to capture local dependencies such as phonetics and long-term semantic information such as linguistic content and temporal structure.” 5:30-34.]
Regarding Claim 17, Agostinelli is not express on this point but Yang is.
Yang teaches:
17. The audio generation method of claim 1 wherein each token sequence, of the semantic token sequence, the structural token sequence, and the audio signal token sequence, comprises a disentangled token sequence with respect to each other token sequence of the semantic token sequence, the structural token sequence, and the audio signal token sequence. [Yang, Figure 1, provided above shows that the different modalities of input including Phonemes (“structural token sequence”), MIDI (also corresponding to “structural token sequence”), Text (“semantic token sequence”), Semantic Tokens (“semantic token sequence”), and Audio (“context”) are provided as separate, namely disentangled, tokenized sequences to the transformer/encoder of Yang.]
Rationale for combination as provided for Claim 1.
Claim 18 is a system claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Mapping to structural elements of Agostinelli was provided for Claim 1.
Claim 19 is a system claim with limitations corresponding to the limitations of Claim 2 and is rejected under similar rationale.
Claim 20 is a system claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Mapping to structural elements of Agostinelli was provided for Claim 1.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Note the plurality of related X references cited by the international search authority including the following US documents:
US 11915689.
US 20240078412.
US 20240233713.
US 20240371366.
US 20250266035.
Riedmiller (U.S. 20250196347): “[0050] In implementations generating the set of tokens 120 comprises generating one or more (different) respective tokens 120 to represent the one or more aspects of each (different) respective entity 122 in the environment 108. That is, different tokens 120 can represent different respective entities 122 in the environment 108. Each token 120 can encode a disentangled aspect of an entity 122 in the environment 108, i.e., a variable that defines how the entity 122 appears in the observation of the environment 108, such as the location or pose of an object in the environment 108. Such a representation is particularly appropriate for a real-world environment 108 in which how an entity 122 appears in an observation 112 of the environment 108 tends to decompose into separate factors of variation, such that factors relevant to a particular sub-task can be selected. Information about a scene within which the entities 122, e.g., objects, are located may be represented explicitly by one or more tokens 120, or can be represented implicitly by the set of tokens 120.”
He (U.S. 20230153532): “[0012] FIG. 5C shows aspects of example pretraining of a transformer-based language-model encoder using gradient-disentangled embedded sharing (GDES).”
He (U.S. 20240062020): “Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.”
Wu (U.S. 20220383858): “[0038] In some implementations, neural network component 330 may be implemented with a transformer neural network that includes a disentangled attention neural network layer. With disentangled attention, the words (or tokens) of the input may be represented with two vectors instead of a single vector. The two vectors may represent the content of the word (what the word is) and the relative position of the word in the input. A cross attention weight for any two words in the input may then be computed using content vectors for the two words and the relative position vectors for the two words and disentangled matrices. The cross attention weight may be computed, for example, as a combination of four attention scores, where each of the four attention scores to combinations of content and relative position.”
N. Takahashi, M. K. Singh and Y. Mitsufuji, "Hierarchical disentangled representation learning for singing voice conversion," 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, 2021, pp. 1-7, doi: 10.1109/IJCNN52387.2021.9533583.
PNG
media_image4.png
398
560
media_image4.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FARIBA SIRJANI whose telephone number is (571)270-1499. The examiner can normally be reached 9 to 5, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Fariba Sirjani/
Primary Examiner, Art Unit 2659
Prosecution Insights