Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on March 29, 2025 is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 6 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 6, the claim states ”on all” is ambiguous. The phrase is unclear whether the extraction operates collectively on all encodings or separately or sequentially on every encoding.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim [ 1, 2, 3, 4, 5, 9, 18, 19, 20 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai).
Regarding claim 1,
Jiang teaches
An audio processing system configured to isolate and extract a varying number of sound sources from an audio mixture, the audio processing system comprising a processor coupled with stored instructions that, when executed by the processor, run modules of the audio processing system, the modules comprising:
[3.1. Sources and Actions "
PNG
media_image1.png
206
498
media_image1.png
Greyscale
"];
[4.2 Sound Editor
PNG
media_image2.png
582
521
media_image2.png
Greyscale
Where hx corresponds to the claimed features of audio mixture];
an audio input interface configured to produce features of an audio mixture formed by multiple sound sources;
[3.1 Sources and Actions “
PNG
media_image3.png
206
498
media_image3.png
Greyscale
[3.1 Sources and Actions “Therefore, every sound mixture is a combination of N different sources of different descriptions {˜s1, ˜s2, ..., ˜ sN}, with ˜si ̸ = ˜sj,∀i ̸ = j, and a description ˜si can identify an unique source si in the mixture.” where hx is the claimed audio mixture and s1--sn are the multiple sources. ]
a prompt input interface configured to produce a
[3.3 Text Prompts " Here, p is an open-vocabulary text prompt written in natural language by
H, which can be either a human or a large language model (LLM). Given the impracticality of hiring humans to write hundreds of thousands of prompts, we leveraged a LLM to generate text prompts p from π."];
[3.3 Text prompts "Although π = {(a1,˜s1),...,(aN, ˜sN)} itself can accurately specify a sound mixture-to-mixture editing instruction, writing an instruction in this format can be inconvenient and
difficult for humans, because we prefer to use natural language rather than numbers and symbols."];
[4.1. Prompt Reader " The Prompt Reader is a language model that behaves as the inverse of H: It generates a D-dimensional text embedding z ∈RD from a text prompt p, which encodes the information about the sources and actions specified in π. We call z a semantic filter because it is generated from text to guide the Sound Editor to execute proper filtering of sounds. "];
[Abstract “..multimodal sound mixture editor that modifies each sound source in a mixture based on user provided text instructions. “];
[Abstract “We developed a 160-hour dataset with over 100k mixtures, including speech and various audio sources, along with text prompts for diverse editing tasks like extraction, removal, and volume control.”];
an information exchanger neural network trained to
[4.2 Sound Editor “
PNG
media_image4.png
488
490
media_image4.png
Greyscale
" where z is represents the prompt and mixture encoder output hi(with mark above) is the modified audio-mixture feature and hi is the audio mixture. FiLM is mapped to the Information exchanger. Semantic filter z is mapped to the input digital encodings];
[3.Sound Mixture-to-Mixture Editing “We formulate sound mixture-to-mixture editing in this section. An example editing is shown in Figure 2.
3.1. Sources and Actions Consider a sound mixture x, " where x is the sound mixture];
an extraction neural network trained to extract a varying number of the target sound sources by
[4.2. Sound Editor “ Instead, the model functions similar to a sound extraction model to directly output a single sound that is the mix of all the edited sources”]
[4.2 Sound Editor “
PNG
media_image5.png
580
510
media_image5.png
Greyscale
PNG
media_image6.png
216
489
media_image6.png
Greyscale
PNG
media_image7.png
328
494
media_image7.png
Greyscale
" where the mask network process prompt encoding z and audio mixture representation hx and FiLM guided editing blocks process modified audio features hi(with mark above) as shown in the Equations]
an output interface configured to output the extracted target sound sources.
[4.2 Sound Editor"
PNG
media_image8.png
241
493
media_image8.png
Greyscale
" where Learnable linear decoder is the output interface, extracted latent representation hy is the extracted sound sources, and convert hy into output audio waveform y is the output the extracted sound sources].
But Jiang does not teach set of input digital encodings representing input sound prompts of at least some of the sound sources forming the audio mixture in a space of the features of the audio mixture, wherein the set of input digital encodings includes a set of target digital encodings representing target sound prompts for extracting target sound sources from the audio mixture;
But Elizalde teaches
set of input digital encodings representing input sound prompts of at least some of the sound sources forming the audio mixture in a space of the features of the audio mixture,
[2.2. Zero-Shot Classification “First, we compute audio embeddings and text embeddings for N
audios and C classes using the pretrained encoders and their projection layers.” where the multiple text embeddings for each C sound classes is the claimed set of input digital encodings];
[3.2. Evaluation setups for CLAP “Instead of using the class label, we constructed a natural language prompting, ‘this is a sound of [class label].The prompt was kept the same for all the domains
except three. For Emotion Recognition we used‘this person is feeling [class label]’, for Keyword Spotting we only use the keyword, and for Speaker Counting we used ‘[number between 0 - 10] persons speaking’.” where each class based text prompt identifies a type of sound source. When class corresponds to a sound present in the mixture, its text embedding represents an input sound prompt for one source forming that mixture];
[2.2. Zero-Shot Classification “ Second, because both the embeddings are in a common space,…”];
wherein the set of input digital encodings includes a set of
[2.2 Zero-Shot Classification “First, we compute audio embeddings and text embeddings for N
audios and C classes using the pretrained encoders and their projection layers.” where text embeddings computed for the C sound classes is a mapped to a set of digital encodings,
[3.2. Evaluation setups for CLAP Zero-shot (ZS) Evaluation studies the generalization of
CLAP to unseen classes and audios. The setup is explained in Section 2.2. Instead of using the class label, we constructed a natural language prompting, ‘this is a sound of [class label]’.” where each text embedding encodes a natural language prompt constructed from its sound class label, and multiple text embeddings corresponds to multiple sound class prompts];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde such that the input sound prompts and audio mixture features are represented in a common multimodal feature space. This modification would improve the correspondence between each sound prompt and its associated sound source, thereby enabling more accurate identification and extraction of the user selected target sounds from the mixture.
Jiang in view of Elizalde do not teach modify each of the set of target digital encodings and processing the modified target digital encodings.
However, Tsai teaches modify each of the set of target digital encodings and processing the modified target digital encodings.
[Page 6562, Section Cross modal Transformers. "In this process, each modality keeps updating its sequence via low-level external information from
the multi-head cross modal attention module. " where each modality is mapped to the target encodings and audio mixture features, keeps updating its sequence maps to modifies both set of features, and external information maps to each modality is modified using information received from the other modality].
[Figure 2 shows A->L for modifying the target/language encodings based on audio features; and L-> A for modifying audio mixture features based on target/language encodings.
PNG
media_image9.png
752
738
media_image9.png
Greyscale
" where XL maps to input/target digital encodings. XA maps to features of the audio mixtures. A-> L produces updated language features. Therefore, updated language XL depends on both. ]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai so that the target prompt encodings and audio mixture features exchange information and modify one another. This modification would improve alignment between the selected sound prompts and corresponding audio components, thereby improving identification and extraction of target sounds from the mixture.
Regarding claim 2, rejection of claim 1 is incorporated.
Jiang teaches the audio processing system of claim 1, wherein each of the input sound prompts includes one or a combination of a
[3.1. Sources and Actions " In other words, no two speech sources share the same speaking style, and no two audio sources share the same class label."];
[3.3. Text Prompts Although π = {(a1, ˜s1), ..., (aN, ˜sN)} itself accurately specifies a remixing instruction,..]"
However, Jiang does not teach learned embedding vector indicative of a reference sound sample, a recording of a reference sound sample,
But Elizalde teaches learned embedding vector indicative of a reference sound sample, a recording of a reference sound sample,
[2.1. Contrastive Language-Audio Pretraining"
PNG
media_image10.png
447
486
media_image10.png
Greyscale
" where the audio recording Xa is recording of the a reference sound sample, the resulting audio embedding Ea is the learned embedding indicative of that sample, and the class label encoded as a text embedding is the mapped to sound event class];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde so that a target sound could be specified using either a sound event class, a reference audio recording, or its learned audio embedding. This modification would provide these alternatives prompt types to increase input flexibility and allow target identification when a sound is difficult to describe with text alone.
Regarding claim 3, rejection of claim 1 is incorporated.
Jiang does not teach the audio processing system of claim 1, wherein the information
exchanger neural network comprises an attention mechanism such that the information exchanger neural network
is trained to place each of the set of input digital encodings and the features of the audio mixture in attention to each of the set of input digital encodings and the features of the audio mixture
to modify all of the target digital encodings and the features of the audio mixture.
However, Elizalde teaches the audio processing system of claim 1,
[2. Method "The input is audio and text pairs passed to an audio encoder and a text encoder. Both representations are connected in joint multimodal space with linear projections." where the text encoder outputs corresponds to the input target digital encodings representing the sound prompts and the audio encoder outputs correspond to the features representing the audio input mixture.];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde so that the sound prompt encodings and audio mixture features are represented in a common multimodal space. It would improve correspondence between the sound prompts and the mixture components, thereby enabling more accurate information exchange and identification of the target sounds.
Jiang in view of Elizalde does not teach the audio processing system of claim 1, wherein the information exchanger neural network comprises an attention mechanism such that the information exchanger neural network is trained to place each of the set of
But Tsai teaches the audio processing system of claim 1, wherein the information exchanger neural network comprises an attention mechanism such that the information exchanger neural network
is trained to place each of the set of
to modify all of the
[Section 2, Transformer Network "..built up from multiple stacks of pairwise and bidirectional cross modal attention blocks that directly attend to low-level features (while removing the self-attention)." where cross modal attention blocks correspond to the claimed attention mechanism.
[3. Proposed Method "A MulT architecture hence models all pairs of modalities with such cross modal transformers, followed by sequence models (e.g., self-attention transformer) that predicts using the fused features."]
[3.1 Cross modal Attention" …whose (i,j)-th entry measures the attention given by the i-th time step of modality α to the j-th time step of modality β." ];
[3.2 Cross modal Transformer "In this process, each modality keeps updating its sequence via low-level external information from the multi-head cross modal attention module." where the combination Tsai's first modality features correspond to the Clap's input/output digital encodings and the second modality feature correspond to Clap's audio mixture features].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai to support flexible audio or language prompts and refine the prompt and mixture features through information exchange, thereby improving the identification and extraction of requested target sounds.
Regarding claim 4, rejection of claim 1 is incorporated.
Jiang teaches the audio processing system of claim 1, wherein the extraction neural network is configured to process the
[4.2. Sound Editor "We adopted the architecture of source separation models for the SoundEditor due to their abilities to isolate individual sources in a sound mixture. However, it is essential to clarify that SoundEditor does not explicitly (in any stage) separate all the sources in the mixture, individually editing them, and sum them up. Instead, the model functions similar to a sound extraction model to directly output a single sound that is the mix of all the edited sources."
[4.2 Sound Editor "
PNG
media_image11.png
494
493
media_image11.png
Greyscale
"];
However, Jiang does not teach target digital encodings
But Elizalde teaches target digital encodings
[2. Method "The input is audio and text pairs passed to an audio encoder and a text encoder. Both representations are connected in joint multimodal space with linear projections." where audio text embeddings correspond to the target digital encodings.];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde to represent audio or language sound prompts and audio features in a common feature space, thereby providing more flexible identification of target sounds for extraction.
Jiang in view of Elizalde do not teach provide the modified features of the audio mixture and modified target digital encodings
Tsai teaches modified features of the audio mixture and modified target digital encodings
[Section 2. Transformer Network "…pairwise and bidirectional cross modal attention blocks that directly attend to low-level features (while removing the self-attention)."
[Section 3.2 Cross modal Transformer "In this process, each modality keeps updating its sequence via low-level external information from the multi-head cross modal attention module. "].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai to exchange and refine information between the target sound encodings and audio mixture features before separation, thereby improving the accuracy of target sound extraction.
Regarding claim 5, rejection of claim 4 is incorporated.
Jiang teaches the audio processing system of claim 4, wherein the conditional TSE module is executed multiple times for different
and output multiple sound sources.
[B. Sound Remixer "The Sound Remixer is an acoustic model responsible for remixing the sound mixture x into ˆy based on the semantic filter z. We adopted a similar architecture of source separation
models for the Sound Remixer due to their ability to isolate individual sources in a sound mixture. However, it is essential to clarify that Sound Remixer does not explicitly (at any stage)
separate all the sources in the mixture to remix them. Instead, the model functions similarly to a sound extraction model, directly producing a single sound that is a remix of all sources"
[4.2 Sound Editor "
PNG
media_image11.png
494
493
media_image11.png
Greyscale
"];
However, Jiang does not teach target digital encodings
Elizalde teaches target digital encodings
[2. 2 "First, we compute audio embeddings and text embeddings for N audios and C classes using the pretrained encoders and their projection layers];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde to execute Jiang's reusable TSE module several times, once for each respective target encoding, to extract and output multiple requested sounds from the audio mixture.
Jiang in view of Elizalde do not teach the modified target digital encodings
Tsai teaches the modified target digital encodings
[Section 2. Transformer Network "…pairwise and bidirectional cross modal attention blocks that directly attend to low-level features (while removing the self-attention)."
[Section 3.2 Cross modal Transformer "In this process, each modality keeps updating its sequence via low-level external information from the multi-head cross modal attention module. "].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai to exchange and refine information between the target sound encodings and audio mixture features before separation, thereby improving the accuracy of target sound extraction.
Regarding claim 9, rejection of claim 1 is incorporated.
Jiang in view of Elizalde do not teach the audio processing system of claim 1, wherein the
information exchanger neural network comprises a self-attention module.
But Tsai teach wherein the information exchanger neural network comprises a self-attention
module.
[3.2 Self-Attention Transformers and Prediction " As a final step, we concatenate the outputs from the cross modal transformers that share the same target modality to yield Z{L,V,A} ∈ RT{L,V,A}× 2d.
For example, ZL = [Z[D] V→L;Z[D] A→L]. Each of them is then passed through a sequence model to collect temporal information to make predictions. We choose the self-attention transformer (Vaswani
et al., 2017)."];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai to further process the exchanged prompt and audio mixture features, thereby capturing dependences and improving identification of the target sounds.
Regarding claim 18,
Jiang teaches a method for processing an audio mixture, comprising: producing features of the audio mixture formed by multiple sound sources;
[4.2 Sound Editor "
PNG
media_image12.png
590
511
media_image12.png
Greyscale
" where hx is the claimed audio mixture]
[4.3 Training Objective “…input mixture x, a target mixture y, and a text prompt p. The sources in the mixture {s1,s2,...,sN} and the underlying symbolic instruction π used in the creation of y and p are hidden from the system” where s1--sn are the multiple sources ]
[4.2 Sound Editor “
PNG
media_image13.png
254
515
media_image13.png
Greyscale
Where hx corresponds to the claimed features of audio mixture];
[3.3. Text Prompts " Here, p is an open-vocabulary text prompt written in natural language by
H, which can be either a human or a large language model (LLM). Given the impracticality of hiring humans to write hundreds of thousands of prompts, we leveraged a LLM to generate text prompts p from π."];
[3.3. Text prompts "Although π = {(a1,˜s1),...,(aN, ˜sN)} itself can accurately specify a sound mixture-to-mixture editing instruction, writing an instruction in this format can be inconvenient and
difficult for humans, because we prefer to use natural language rather than numbers and symbols."];
[4.1 Prompt Reader " The Prompt Reader is a language model that behaves as the inverse of H: It generates a D-dimensional text embedding z ∈RD from a text prompt p, which encodes the information about the sources and actions specified in π. We call z a semantic filter because it is generated from text to guide the Sound Editor to execute proper filtering of sounds. "];
[Abstract “..multimodal sound mixture editor that modifies each sound source in a mixture based on user provided text instructions. “];
[Abstract “We developed a 160-hour dataset with over 100k mixtures, including speech and various audio sources, along with text prompts for diverse editing tasks like extraction, removal, and volume control.”];
PNG
media_image14.png
389
499
media_image14.png
Greyscale
" where z is represents the prompt and mixture encoder output hi(with mark above) is the modified audio-mixture feature and hi is the audio mixture. FiLM is mapped to the Information exchanger. Semantic filter z is mapped to the input digital encodings];
[4.2 Sound Editor “
[
PNG
media_image15.png
217
507
media_image15.png
Greyscale
."
where x is the sound mixture];
extracting a varying number of the target sound sources by
B. Sound Remixer " Instead, the model functions similar to a sound extraction model to directly producing a single sound that is the remix of all sources."
[ 4.2 Sound Editor “
PNG
media_image16.png
218
492
media_image16.png
Greyscale
”]
PNG
media_image17.png
385
494
media_image17.png
Greyscale
" where the mask network process prompt encoding z and audio mixture representation hx and FiLM guided editing blocks process modified audio features hi(with mark above) as shown in the Equations]
And outputting the extracted target sound sources.
[4.2 Sound Editor "
PNG
media_image18.png
245
491
media_image18.png
Greyscale
" where Learnable linear decoder is the output interface, extracted latent representation hy is the extracted sound sources, and convert hy into output audio waveform y is the output the extracted sound sources].
But Jiang does not teach set of input digital encodings representing input sound prompts of at least some of the sound sources forming the audio mixture in a space of the features of the audio mixture, wherein the set of input digital encodings includes a set of target digital encodings representing
But Elizalde teaches
set of input digital encodings representing input sound prompts of at least some of the sound sources forming the audio mixture in a space of the features of the audio mixture,
[2.2. Zero-Shot Classification “First, we compute audio embeddings and text embeddings for N
audios and C classes using the pretrained encoders and their projection layers.” where the multiple text embeddings for each C sound classes is the claimed set of input digital encodings];
[3.2. Evaluation setups for CLAP “Instead of using the class label, we constructed a natural language prompting, ‘this is a sound of [class label].The prompt was kept the same for all the domains
except three. For Emotion Recognition we used‘this person is feeling [class label]’, for Keyword Spotting we only use the keyword, and for Speaker Counting we used ‘[number between 0 - 10] persons speaking’.” where each class based text prompt identifies a type of sound source. When class corresponds to a sound present in the mixture, its text embedding represents an input sound prompt for one source forming that mixture];
[2.2. Zero-Shot Classification “ Second, because both the embeddings are in a common space,…”];
[3.2. Evaluation setups for CLAP Zero-shot (ZS) Evaluation studies the generalization of
CLAP to unseen classes and audios. The setup is explained in Section 2.2. Instead of using the class label, we constructed a natural language prompting, ‘this is a sound of [class label]’.” where each text embedding encodes a natural language prompt constructed from its sound class label, and multiple text embeddings corresponds to multiple sound class prompts];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde such that the input sound prompts and audio mixture features are represented in a common multimodal feature space. This modification would improve the correspondence between each sound prompt and its associated sound source, thereby enabling more accurate identification and extraction of the user selected target sounds from the mixture.
Jiang in view of Elizalde does not teach modifying the target digital encodings and processing the modified target digital encodings.
However, Tsai teaches modifying the set of target digital encodings and processing the modified target digital encodings.
[Page 6562, Section Cross modal Transformers. "In this process, each modality keeps updating its sequence via low-level external information from
the multi-head cross modal attention module. " where each modality is mapped to the target encodings and audio mixture features, keeps updating its sequence maps to modifies both set of features, and external information maps to each modality is modified using information received from the other modality].
[Figure 2 shows A->L for modifying the target/language encodings based on audio features; and L-> A for modifying audio mixture features based on target/language encodings.
PNG
media_image9.png
752
738
media_image9.png
Greyscale
" where XL maps to input/target digital encodings. XA maps to features of the audio mixtures. A-> L produces updated language features. Therefore, updated language XL depends on both. ]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai so that the target prompt encodings and audio mixture features exchange information and modify one another. This modification would improve alignment between the selected sound prompts and corresponding audio components, thereby improving identification and extraction of target sounds from the mixture.
Regarding claim 19
Jiang does not teach executing an operation, the operation comprising
placing each of the set of input digital encodings and the features of the audio mixture in attention to each of the set of input digital encodings and the features derived of the audio mixture and,
modifying each of the set of target digital encodings and the features of the audio mixture based on the execution of the operation, to obtain
However, Elizalde teaches executing an operation, the operation comprising
[2. Method "The input is audio and text pairs passed to an audio encoder and a text encoder. Both representations are connected in joint multimodal space with linear projections." where the text encoder outputs corresponds to the input target digital encodings representing the sound prompts and the audio encoder outputs correspond to the features representing the audio input mixture.];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde so that the sound prompt encodings and audio mixture features are represented in a common multimodal space. It would improve correspondence between the sound prompts and the mixture components, thereby enabling more accurate information exchange and identification of the target sounds.
Jiang in view of Elizalde does not teach executing an operation, the operation comprising placing each of the set of
And modifying each of the
But Tsai teaches executing an operation, the operation comprising placing each of the set of
And modifying each of the
[Section 2, Transformer Network "..built up from multiple stacks of pairwise and bidirectional cross modal attention blocks that directly attend to low-level features (while removing the self-attention)." where cross modal attention blocks correspond to the claimed attention mechanism.
[3. Proposed Method "A MulT architecture hence models all pairs of modalities with such cross modal transformers, followed by sequence models (e.g., self-attention transformer) that predicts using the fused features."]
[3.1 Cross modal Attention" …whose (i,j)-th entry measures the attention given by the i-th time step of modality α to the j-th time step of modality β." ];
[3.2 Cross modal Transformer "In this process, each modality keeps updating its sequence via low-level external information from the multi-head cross modal attention module." where the combination Tsai's first modality features correspond to the Clap's input/output digital encodings and the second modality feature correspond to Clap's audio mixture features].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai to support flexible audio or language prompts and refine the prompt and mixture features through information exchange, thereby improving the identification and extraction of requested target sounds.
Regarding claim 20, rejection of claim 19 is incorporated.
Jiang in view of Elizalde do not teach the method of claim 19, wherein the modification of each
of the set of target digital encodings and the features of the audio mixture comprises executing a self- attention operation.
But Tsai teach comprises executing a self- attention operation.
[3.2 Self-Attention Transformers and Prediction " As a final step, we concatenate the outputs from the cross modal transformers that share the same target modality to yield Z{L,V,A} ∈ RT{L,V,A}× 2d.
For example, ZL = [Z[D] V→L;Z[D] A→L]. Each of them is then passed through a sequence model to collect temporal information to make predictions. We choose the self-attention transformer (Vaswani
et al., 2017)."];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai to further process the exchanged prompt and audio mixture features, thereby capturing dependences and improving identification of the target sounds.
Claim [ 6, 8, 10, 15 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view of Delcroix (Delcroix, Marc, et al. "SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning." IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2022): 121-136., hereinafter Delcroix).
Regarding claim 6, rejection of claim 1 is incorporated.
Jiang teaches the audio processing system of claim 1, wherein the extraction neural network is configured to process the
[4.2 Sound Editor "The Sound editor is an acoustic model responsible for editing the sound mixture x based on the semantic filter z. [see equation 5] We adopted the architecture of source separation
models for the Sound Editor due to their abilities to isolate individual sources in a sound mixture. However, it is essential to clarify that Sound Editor does not explicitly (at any stage)
separate all the sources in the mixture, individually editing them, and sum them up. Instead, the model functions similarly to a sound extraction model to directly output a signal sound that is the mix of all the edited sources.
[4.2 Sound Editor "
PNG
media_image19.png
313
484
media_image19.png
Greyscale
"];
Jiang does not teach modified features or conditioned on all of the modified target digital encodings
However, Elizalde teaches target digital encodings
[Abstract "We call our approach Contrastive Language-Audio Pretraining (CLAP), which connects language and audio by using two encoders and a contrastive learning objective, bringing audio and text
descriptions into a joint multimodal space."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde to permit target sounds to be specified flexibly using audio or language prompts and to support zero shot extraction of sound classes.
Jiang in view of Elizalde do not teach modified features and conditioned on all of the modified target digital encodings
But Tsai teaches modified features and modified target digital encodings
[Abstract "At the heart of our model is the directional pairwise cross modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. "];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde with Tsai so that each representation could incorporate relevant information from the other, thereby improving alignment between the requested target sounds and corresponding portions of the audio mixture.
Jiang in view of Elizalde in view of Tsai do not teach conditioned on all of the modified target digital encodings
But Delcroix teaches conditioned on all of the modified target digital encodings
[Page 128. C. Simultaneous Multi-Target Extraction "We propose instead learning to simultaneous extract a mixture of target SE classes [2]."];
[Page 128. C. Simultaneous Multi-Target Extraction "For simultaneous multi-target extraction, the embedding vector should characterize all target SE classes."]
[Page 128, C. Simultaneous Multi-Target Extraction "We learn simultaneous extraction by including multitarget training samples, performing extraction with the above multi-target embeddings"].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Delcroix to extract multiple requested sound classes in a single operation, thereby reducing the computational complexity associated with separately executing them.
Regarding claim 8, rejection of claim 1 is incorporated.
Jiang in view of Elizalde in view of Tsai do not teach the audio processing system of claim 1,
wherein the processor is configured to combine the set of input digital encodings, and the features of the audio mixture to generate a tensor.
But Delcroix teaches the processor is configured to combine the set of input digital encodings, and the features of the audio mixture to generate a tensor.
[Page 125, A. Sound Extraction NN "The sound extraction NN estimates the target signal, xs, from the mixture, y, and a D-dimensional target embedding vector,
es ∈ RD,as,
ˆ
xs =f(y,es),
where f(·) is the sound extraction NN." where es is the initial digital encoding];
[A. Sound Extraction NN, Column 2 "The time domain input signal, y, is processed by a trainable encoder layer as,
Y=Encoder(y),
where Y ∈ RD×T′ is the encoded mixture representation, T′ is the number of time frames" where encoded mixture representation Z/Y is the encoded mixture representation. The target conditioned feature representation processed by the extraction network is the generated tensor].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Delcroix’s
target embedding conditioning operation, thereby generating a target conditioned feature tensor that more effectively identifies and extracts the requested sounds from the audio fixture.
Regarding claim 10, rejection of claim 1 is incorporated.
Jiang in view of Elizalde in view of Tsai do not teach the audio processing system of claim 1,
wherein the input sound prompts include multiple sound prompts, and each sound prompt is indicated as one of the target sound prompts.
But Delcroix teaches the audio processing system of claim 1, wherein the input sound prompts
include multiple sound prompts, and each sound prompt is indicated as one of the target sound prompts.
[Page 124, B. Multi-Target Extraction "
PNG
media_image20.png
87
480
media_image20.png
Greyscale
"
[Page 125, B. Multi-Target Extraction " When using enrollment audio samples as clues, we consider here that we have separate enrollment samples for each of the target SE classes."];
[Page 125, B. Multi-Target Extraction "
PNG
media_image21.png
192
510
media_image21.png
Greyscale
" where separate enrollment audio samples (as1..) maps to multiple sound prompts, and each enrollment to target class corresponds to identified target prompt];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Delcroix’s
so that each requested target sound class is represented by its own prompt, thereby enabling the system to identify and extract multiple user selected target sounds from the audio mixture.
Regarding claim 15, rejection of claim 1 is incorporated.
Jiang in view of Elizalde in view of Tsai do not teach the audio processing system of claim 1,
wherein the set of input digital encodings is stored in a memory.
But Delcroix teaches
[Page 126, B. Clue Encoder “
PNG
media_image22.png
182
487
media_image22.png
Greyscale
where W constitutes the set of class based input digital encodings. Because W is a learned model parameter used during execution, it must be retained in computer memory]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Delcroix’s enabling lossless manipulation, precise timing control, and advanced DSP capabilities.
Claim [ 7 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view of Saijo (Saijo, Kohei, et al. "TF-Locoformer: Transformer with local modeling by convolution for speech separation and enhancement." 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC). IEEE, 2024., hereinafter, Saijo).
Regarding claim 7, rejection of claim 1 is incorporated.
Jiang in view of Elizalde in view of Tsai do not teach The audio processing system of claim 1,
wherein each of the information exchanger neural network and the extraction neural network
includes a neural network having a TF-Locoformer architecture.
However, Saijo teaches The audio processing system of claim 1, wherein each of the information exchanger neural network and the extraction neural network
includes a neural network having a TF-Locoformer architecture.
[Abstract "This work presents TF-Locoformer, a Transformer-based model with Local-modeling by Convolution. "]
[Abstract "The model uses feed forward networks (FFNs) with convolution layers, instead of linear layers, to capture local information, letting the self-attention focus
on capturing global patterns."
[Section 2.1 "The model is based on TF-domain dual-path modeling[6,7],where frequency and temporal modeling are done alternately."]
[Section 2.1 "It separates each source by complex spectral mapping, where their real and imaginary(RI)components are directly estimated."
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Saijo to improve multi-task and universal source separation. This model can jointly process and cross reference completely different sources, prompts modalities simultaneously.
Claim [ 11 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view of Ma ( Ma, Hao, et al. "Clapsep: Leveraging contrastive pre-trained model for multi-modal query-conditioned target sound extraction." arXiv preprint arXiv:2402.17455v1, 27 Feb 2024 - arxiv.org, hereinafter Ma).
Regarding claim 11, rejection of claim 1 is incorporated.
Jiang in view of Elizalde in view of Tsai do not teach The audio processing system of claim 1,
wherein the input sound prompts include multiple sound prompts, and a strict subset of the input sound prompts is indicated as the target sound prompts.
But Ma teaches the audio processing system of claim 1, wherein the input sound prompts include multiple sound prompts, and a strict subset of the input sound prompts is indicated as the target sound prompts.
[Page 3, B. Proposed CLAPSep Model " The proposed CLAPSep model extracts the target sound from a sound mixture conditioned on a series of user-specified queries.These queries are multi-modal, allowing for both text and audio modalities, and can include positive and/or negative samples
to guide the extraction process."];
[Page 4, Proposed CLAPSep Model. 1) Query Network "Eq (3) is applied to the user’s positive and negative queries separately, producing embeddings eP and eN. Finally, the two embeddings
are concatenated as the output, namely the conditional embedding c = [eP , eN] ∈ R2D" where positive prompts is included is the subset of input sound prompts.];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Ma because it would be constraint the target space to known, pre-validated reference characteristics, thereby significantly improving model stability and reduce data misalignment.
Claim [ 12 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view Pan (Pan, Zexu, et al. "Scenario-aware audio-visual TF-Gridnet for target speech extraction." 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023., hereinafter Pan).
Regarding claim 12, rejection of claim 1 is incorporated.
Jiang does not teach the audio processing system of claim 1, wherein the features of the audio mixture are in a time-frequency domain, wherein the features at each time frame
and each frequency bin comprise a vector of a same dimension as each one of the input digital encodings, such that the information exchanger neural network processes
the features of the audio mixture.
However, Elizalde teaches the audio processing system of claim 1, wherein the features of the audio mixture are in a time-frequency domain
PNG
media_image23.png
105
496
media_image23.png
Greyscale
Where F represents spectral components and T represents time frames. This implies time frequency domain input features];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde to improve correspondence between user's prompt and audio components.
Jiang in view of Elizalde in view of Tsai do not teach wherein the features at each time frame and each frequency bin comprise a vector of a same dimension as each one of the input digital encodings, such that the information exchanger neural network processes
the features of the audio mixture.
However, Pan teaches wherein the features at each time frame
and each frequency bin comprise a vector of a same dimension as each one of the input digital encodings, such that the information exchanger neural network processes
the features of the audio mixture.
[2. Methodology "It encodes a speech signal into TF representations of dimension T ×F ×D with a short-time Fourier transform (STFT) followed by a 2-dimensional convolution (Conv2D) and a layer normalization (LN) operation."]
[2. Methodology "1) An intra-frame spectral module that views the TF representation as T separate sequences of D-dimensional embeddings, each sequence having length F, and applies a BLSTM
layer with H units and a 1-dimensional deconvolution layer with kernel size I and stride J; 2) A sub-band temporal module that views the TF representation as F separate sequences of D-dimensional embeddings, each sequence having length T, and performs a similar procedure as in the intra-frame
spectral module;" where each time frequency bin contains a D dimensional vector, projecting the conditioning and audio features to the audio feature dimension means they have same vector, and fusing the embeddings with audio embeddings at every Grid Net means the information exchanges processes the audio mixture features.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Pan so that each time frequency feature vector could directly interact with the prompt encodings, thereby improving target sound identification and extraction.
Claim [ 13 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view Wang (Wang, Zhong-Qiu, et al. "TF-GridNet: Integrating full-and sub-band modeling for speech separation." IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023): 3221-3236., hereinafter Wang).
Regarding claim 13, rejection of claim 1 is incorporated.
Jiang teaches the audio processing system of claim 1, wherein the audio processing system is trained based on a loss associated with ground truth audio signals
[4.3. Training objective "Each training instance is a tuple (x, y, p) including an input mixture x, a target mixture y, and a text prompt p. The sources in the mixture {s1, s2, ..., sN} and the underlying
symbolic instruction π used in the creation of y and p are hidden from the system. LCR estimates a mixture ˆy given x and p. All model components are jointly optimized by maximizing the signal-to-noise ratio (SNR) between the target and the estimated mixture." where y is target mixture and ˆy is estimated mixture, therefore training is based on comparing ground truth and extracted/estimated audio].
However, Jiang does not teach source category in a set of source categories of the target sound prompts,
But Elizalde teaches source category in a set of source categories of the target sound prompts,
[2.2. Zero-Shot Linear Classification " Let’s consider a target dataset with C class labels and N test audios. First, we compute audio embeddings and text embeddings for N audios and C classes using the pretrained encoders and their
projection layers."]
[3.3. Evaluation setups for CLAP " Instead of using the class label, we constructed a natural language prompting, ‘this is a sound of [class label]’."];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang with Elizalde to identify the source categories corresponding to the target sounds, thereby enabling flexible category based selection and extraction.
Jiang in view of Elizalde in view of Tsai do not teach such that the loss is computed as a sum of losses wherein the loss for each source category is a permutation-invariant loss.
But Wang teaches such that the loss is computed as a sum of losses wherein the loss for each source category is a permutation-invariant loss.
[Page 3225 “
PNG
media_image24.png
466
442
media_image24.png
Greyscale
"where
PNG
media_image25.png
28
43
media_image25.png
Greyscale
Is the target ground truth for signal c,
PNG
media_image26.png
33
32
media_image26.png
Greyscale
Is the predicted separated signal for source c,
And
PNG
media_image27.png
46
61
media_image27.png
Greyscale
Sums the losses for all sources and utterance level PIT makes the source to output assignment permutation invariant].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Wang to correctly match separated outputs with corresponding ground truth sources regardless of output order, thereby improving training accuracy and separation performance.
Claim [ 14 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view Kong (Kong, Qiuqiang, et al. "Source separation with weakly labelled data: An approach to computational auditory scene analysis." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020., hearinafter, Kong).
Regarding claim 14, rejection of claim 1 is incorporated.
Jiang teaches the audio processing system of claim 1, further comprising:
a prompt selection model configured to select prompts from a
[5. Dataset, Editing Tasks " For every mixture, we first evenly sampled a task t ∈ T (e.g. target speech extraction),…"];
[5. Dataset, Text Prompts “We requested GPT-3.5 Turbo to write a text prompt five times from the assigned instruction π for each mixture."]
an example selector configured to
[5.. Dataset, Sound Sources" Sources Speech sources were from TextrolSpeech [38] and audio sources were from VGGSound [60] (for training and in-domain evaluation) or FSD50K [61] (for zero-shot
evaluation). Speakers and audios were split into training, validation, and testing sets."]
and a mixture creator configured to mix the
[5. Dataset, Sound Mixtures “We generated 100k, 5k, and 10k mixtures of 2 speech + 2 audio sources from TextrolSpeech and VGGSound for training, validation, and testing."];
Jiang in view of Elizalde in view of Tsai do not teach
predetermined set of allowed sound prompts
randomly sample
stored in a database
wherein the audio samples are determined based on a type of each prompt
However, Kong teaches
predetermined set of allowed sound prompts
[4. Experiments "Audio Set is a large-scale audio dataset with an ontology of 527 sound classes [20] in the released version"
[3.3 Inference "Only the sound classes predicted as present in the audio clip will be separated. For the k-th sound event, we use one-hot encoding as a conditional vector ck = {0,..,0,1,0,...,0}, where the k-th index of ck is 1 while others 0. " where sound classes (type of sound prompt) constitute a predetermined set of allowed sound prompts];
randomly sample
[3.2. Source Separation " To begin with, we randomly select two sound classes from Audio Set.... "];
stored in a database
[4. EXPERIMENTS " AudioSet is a large-scale audio dataset with an ontology of 527 sound classes [20] in the released version"];
[Introduction "On the other hand, AudioSet is a large-scale dataset contains 5000 hours of data with 527 sound classes. AudioSet is a weakly labelled dataset, that is, only the tags of an audio clip are provided, without the occurrence time of the sound classes, neither the clean sources."];
wherein the audio samples are determined based on a type of each prompt
[3.2 Sound separation "For each sound class, an audio clip containing the sound class is selected."];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Kong by selecting a sound event class prompt, retrieving audio examples corresponding to that selected class from Audio set, and mixing the randomly selected examples to create training mixtures. This combination would automatically generate correctly labeled, diverse training mixtures in which audio content corresponds to the selected sound prompt, thereby improving training.
Claim [ 16 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view Yan (Yan, Jia, Suzhi Bi, and Ying-Jun Angela Zhang. "Optimal model placement and online model splitting for device-edge co-inference." IEEE Transactions on Wireless Communications 21.10 (2022): 8354-8367., hereinafter Yan).
Regarding claim 16, rejection of claim 1 is incorporated.
Jiang in view of Clap in view of Tsai do not teach the audio processing system of claim 1, wherein the set of input digital encodings is received from a remote device.
But Yan teaches input digital encodings is received from a remote device.
[Abstract "In particular, the WD executes the first few layers of the DNN and sends the intermediate features to the edge server that processes the remaining layers of the DNN." where intermediate features are digital encodings and edge server receives those encodings from the remote wireless device. Since WD sends the intermediate features to a separate edge server, therefore the device is functionally remote from server ];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Yan thereby allowing the audio processing system to receive the input digital encodings from a remote device while distributing neural network processing between the device and server and reducing the servers computational burden.
Claim [ 17 ] are rejected under 35 U.S.C. 103 as being unpatentable over Jiang (Jiang, Xilin, et al. "Listen, chat, and edit: Text-guided soundscape modification for enhanced auditory experience." arXiv e-prints (2024): arXiv-2402., hereinafter Jiang) in view of Elizalde (Elizalde, Benjamin, et al. "Clap learning audio concepts from natural language supervision." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023., hereinafter Elizalde) and in further view of Tsai (Tsai, Yao-Hung Hubert, et al. "Multimodal transformer for unaligned multimodal language sequences." Proceedings of the 57th annual meeting of the association for computational linguistics. 2019., hereinafter Tsai) and in further view Rakshit (US 11307825 B1, hereinafter Rakshit).
Regarding claim 17, rejection of claim 1 is incorporated.
Jiang in view of Clap in view of Tsai do not teach The audio processing system of claim 1, wherein the prompt input interface comprises a user interface (UI) including a plurality of display options,
the plurality of display options comprising at least:
a first UI element associated with a selection of an audio signal;
a second UI element associated with a selection of the input set of sound prompts; and
a third UI element associated with a selection of the target set of sound prompts.
But Rakshit teaches
a first UI element associated with a selection of an audio signal;
[Column 7, lines 30-35 "Selection indicator 270 represents an indication of the user selecting an icon displayed on display area 210 of the
user's personal device. In some embodiments, the user's personal device presents the user icons representing different
sounds separated from a mixture of sounds in a sound stream and enables the user to select a sound and record the sound" where each displayed sound icon is a UI element associated with audio signal]
a second UI element associated with a selection of the input set of sound prompts; and
[Column 9, lines 32-37 "In some embodiments in which a larger number of separated sounds are identified and assigned a category icon,
sound selection program 300 may display a limited number of icons at one time on the display area of the AR glasses, with a paging selection to display the next set of icons to be
considered by the user." where category icons constitute an input set of sound prompts (because sound class is an allowable sound prompt].
a third UI element associated with a selection of the target set of sound prompts.
[Column 7, lines 31-32 " Selection indicator 270 represents an indication of the user selecting an icon displayed on display area 210 of the
user's personal device. " where selected sound icons constitute the target set of sound prompts ]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jiang in view of Elizalde in view of Tsai with Rakshit
to allow a user to conveniently select an audio signal, view available sound prompts categories, and designate one or more categories as target sounds, thereby improving user control and ease of operating.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHEZA ABDUL AZIZ whose telephone number is (571)272-9610. The examiner can normally be reached Monday-Friday 7:30am-5pm Alternate Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHEZA ABDUL AZIZ/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights