Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Introduction
This office action is in response to Applicant’s response to submission filed on 1/6/2025. Claims 1-21 are pending of which claims 1, 19 and 21 are independent. As such, claims 1-21 have been examined.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 8, 11, 13-14, 16-17, and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Cordourier (US 20240430634), in view of Sharma (US 20240185875).
Regarding Claim 1, Cordourier discloses: 1. A computer-implemented method for audio enhancement, comprising: encoding input audio in a frequency domain by a first encoder using one or more first learned parameters to produce first latent vectors; ([0039] frequency domain 2D input surface then may be input into the NN 118, and with one 2D frequency surface representing a single frame of frequency bins x spectrum values, … form the NN 118 may have a frequency encoder that receives multi-microphone frequency domain input,)
encoding the input audio in a time domain by a second encoder using one or more second learned parameters to produce second latent vectors; ([0039] The time domain input 1D vector …a time domain encoder that receives multi-microphone time domain input, and a decoder that outputs binaural audio signals.)
fusing the first latent vectors and the second latent vectors in the time domain to produce fused latent vectors; ([0039] The time domain encoder and time domain decoder form a U-net type of neural network in this example. The output of each encoder may be combined at a bottleneck of the U-net time domain encoder-decoder to form input for the decoder.)
Cordourier does not appear to disclose processing the fused latent vectors and a reference vector in the time domain according to one or more third learned parameters to generate output audio, the reference vector corresponding to audio recorded using equipment or in an environment that is different from the input audio, wherein the output audio corresponds to a version of the input audio that is recorded using equipment or in an environment having qualities consistent with the reference vector.
Sharma in the related art discloses: and processing the fused latent vectors and a reference vector in the time domain according to one or more third learned parameters to generate output audio, the reference vector corresponding to audio recorded using equipment or in an environment that is different from the input audio, wherein the output audio corresponds to a version of the input audio that is recorded using equipment or in an environment having qualities consistent with the reference vector. ([0013] As will be discussed in greater detail below, implementations of the present disclosure generate a conditioning vector as an input to neural network which allows for the augmentation of an input speech signal to have the background acoustics of a target signal. This approach has the advantage of augmenting an input speech segment based on example field recordings, by using a non-intrusive estimate of the background acoustic properties. Furthermore, neural networks of the present disclosure include neural architectures which allow for noise and reverberation augmentation in both directions (i.e., clean audio signal segments to noisy audio signal segments, or noisy audio signal segments to cleaner audio signal segments).) Also see para 0016, 0021, 0030, 0042-0043, and 0049 and fig. 9.
Cordourier and Sharma are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Cordourier to combine the teaching of Sharma, because the present disclosure allows for data augmentation of input speech signals that takes a clean speech into a target background environment that the user perfers (Sharma, [0013]).
Regarding claim 2, Cordourier/Sharma teach all the element of claim 1,
Sharma further discloses: wherein the qualities include at least one of microphone channel characteristics or recording conditions of the reference vector. ([0043] data augmentation process 10 generates 404 an augmented audio signal segment with background acoustic properties of the target audio signal segment by processing an input audio signal segment, the target audio signal segment, the noise spectrum, and the noise neural embedding using a neural network. As discussed above with an acoustic neural embedding, data augmentation process 10 generates an augmented audio signal segment with background acoustic properties of the target audio signal segment by processing input audio signal segment 214, target audio signal segment 200, noise spectrum 204, and noise neural embedding 502 using neural network 216.)
Where the rationale for the combination would be similar to the one already provided.
Regarding claim 3, Cordourier/Sharma teach all the element of claim 1,
Cordourier further discloses: wherein the input audio is encoded simultaneously by the first encoder and the second encoder. ([0052] Process 200 may include “generate binaural audio signals” 206, and specifically by having a neural network emulate binaural audio signals during a run-time (or live or in real-time). This operation 206 may include “simultaneously input a version of the individual audio signals into a neural network” 208, and specifically to “input versions of the audio signals into both a time domain encoder and a frequency domain encoder” 210.)
Regarding claim 4, Cordourier/Sharma teach all the element of claim 1,
Sharma further discloses: wherein the reference vector is associated with a different audio source compared with the input audio. ([0013] implementations of the present disclosure generate a conditioning vector as an input to neural network which allows for the augmentation of an input speech signal to have the background acoustics of a target signal. This approach has the advantage of augmenting an input speech segment based on example field recordings, by using a non-intrusive estimate of the background acoustic properties. Furthermore, neural networks of the present disclosure include neural architectures which allow for noise and reverberation augmentation in both directions (i.e., clean audio signal segments to noisy audio signal segments, or noisy audio signal segments to cleaner audio signal segments).)
Where the rationale for the combination would be similar to the one already provided.
Regarding claim 5, Cordourier/Sharma teach all the element of claim 1,
Cordourier further discloses: wherein the reference vector is associated with studio recording equipment or a studio recording environment. ([0030] the input device 105 to be used during a run time to capture audio signals may be any computing device with live microphones such as mobile devices such as a laptop, notebook, tablet, smartphone, or any smart device. Otherwise, input device 105 may be a desktop computer, extension monitor with mics, or other computing device with microphones. In some forms, the binaural audio emulation unit 114 is on the same device as the microphones mentioned above. In other examples, the binaural audio emulation unit 114 is on one or more of the devices mentioned above while the device 105 with microphones is a separate listening or input device that provides audio signals, such as peripheral devices with microphone arrays, such as headset microphone wand with multiple microphone arrays, studio and/or stand-alone (or free-standing) microphone arrays to name a few examples, and as long as such microphone arrays are wirelessly or wired to a device with the binaural audio emulation unit 114 to provide audio signals from the microphone array 104 to the binaural audio emulation unit 114.)
Regarding claim 8, Cordourier/Sharma teach all the element of claim 1,
Sharma further discloses: wherein occurrence of at least one of background noise, reverberation, or an echo is removed in the output audio. ([0033] As discussed above, data augmentation process 10 generates 116 a multiplied filter to represent the reverberation present in the target audio signal segment without any extra reverberation present in the input audio signal segment. Accordingly, the resulting multiplied filter is able to add reverberation when the input audio signal segment does not include reverberation present in the target audio signal segment and/or is able to remove or reduce reverberation when the input audio signal segment includes reverberation not present in the target audio signal segment.,)
Where the rationale for the combination would be similar to the one already provided.
Regarding claim 11, Cordourier/Sharma teach all the element of claim 1,
Cordourier further discloses: wherein the input audio is associated with at least one of speech, environmental sounds, or a musical instrument. ([0025] The audio sources may be arbitrary source inputs, such as speech, music, or other audio events. [0047]Process 200 may include “receive, by processor circuitry, multiple audio signals from multiple microphones and overlapping in a same time and associated with a same at least one audio source” 202. This may include many different types of audio environments with many different types of audio input or source (or listening) devices as long as the input device has at least two microphones or microphone array with a pattern shape and directional sensitivity as described above.)
Regarding claim 13, Cordourier/Sharma teach all the element of claim 1,
Cordourier further discloses: evaluating a loss function using the output audio and ground truth audio to compute a loss value; and updating the first parameters, the second parameters, and the third parameters based on the loss value. ([0093] Referring to FIGS. 8A-8C for the testing results, the spectrograms for the raw input signals (‘4 channel input’) 800 used in the training and testing of the binaural audio emulation neural network is shown along with the ground truth NN binaural output spectrograms 802 and the binaural HATS target network outputs 804. As shown by the strong similarity between the NN binaural output 802 from the neural network and the HATS target signals 804, the results demonstrate that the binaural audio emulation neural network is able to replicate the binaural spectrogram extremely well.) [updating weight/parameter of neural network during training is obvious and implied as that is the basic and foundation of training a neural network]
Regarding claim 14, Cordourier/Sharma teach all the element of claim 13,
Cordourier further discloses: wherein training audio associated with the ground truth audio corresponds to a variety of at least one of ([0025] The NN is trained with real audio signals generated by simultaneously capturing emitted audio with both non-binaural sets of microphones, such as a microphone array, and a binaural recording device with two microphones at ear locations of a headset or mannequin head. The binaural audio signals are used as the ground truth NN target or expected audio signals for supervised training, while the non-binaural audio signals of the set of multiple microphones are input to the NN to generate NN output or estimated audio signals. The target and output audio signals are then compared in a loss function to determine when the NN is ready for run-time use. The audio sources may be arbitrary source inputs, such as speech, music, or other audio events. After the network is trained, and during an inference or run-time, the system then can produce binaural-emulated 3D sound based on non-binaural multiple microphone inputs.)
Regarding claim 16, Cordourier/Sharma teach all the element of claim 13,
Cordourier further discloses: wherein the loss function includes at least one of ([0080] Process 400 may include “determine loss using a loss function” 420, and including “compare output and target binaural audio signals” 422. This may be performed separately for time and frequency domains so that determining the loss also includes “determine time and frequency domain losses” 424.)
Regarding claim 17, Cordourier/Sharma teach all the element of claim 1,
Cordourier further discloses: wherein the method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system for performing one or more generative AI operations; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; a system implemented at least partially using cloud computing resources; ([0119] In various implementations, content services device(s) 1130 may be hosted by any national, international and/or independent service and thus accessible to platform 1102 via the Internet, for example. Content services device(s) 1130 may be coupled to platform 1102 and/or to display 1120, speaker subsystem 1160, and microphone subsystem 1170. Platform 1102 and/or content services device(s) 1130 may be coupled to a network 1165 to communicate (e.g., send and/or receive) media information to and from network 1165. Content delivery device(s) 1140 also may be coupled to platform 1102, speaker subsystem 1160, microphone subsystem 1170, and/or to display 1120.) a system using or deploying one or more inference microservices; a system that incorporates one or more machine learning models deployed in a service or microservice along with an OS-level virtualization package (e.g., a container).
Sharma further discloses: a system for performing simulation operations; ([0016] As discussed above, current methods for data augmentation rely on simulating various aspects of the signal processing pipeline separately, each of which rely on estimates or prior knowledge of the corrupting process (i.e., known room characteristics, noise type, etc.). Implementations of the present disclosure use a neural network to apply such degradations in an automated manner. Moreover, implementations of the present disclosure perform both degradation and cleaning of an input speech signal based upon the background acoustics determined for a target speech signal. In this manner, the present disclosure allows for data augmentation of input speech signals for training speech processing systems based on an acoustic neural embedding/conditioning vector and allows speech data from TTS-based systems to be used for generating training data.)
a system implemented at least partially using cloud computing resources; ([0057] Data augmentation process 10s may be a server application and may reside on and may be executed by a computer system 1000, which may be connected to network 1002 (e.g., the Internet or a local area network). Computer system 1000 may include various components, examples of which may include but are not limited to: a personal computer, a server computer, a series of server computers, a mini computer, a mainframe computer, one or more Network Attached Storage (NAS) systems, one or more Storage Area Network (SAN) systems, one or more Platform as a Service (PaaS) systems, one or more Infrastructure as a Service (IaaS) systems, one or more Software as a Service (Saas) systems, a cloud-based computational system, and a cloud-based storage platform.
Where the rationale for the combination would be similar to the one already provided.
Regarding Claim 19, Cordourier: discloses:19. A system for audio enhancement, comprising: a memory that stores input audio; ([0018] The material disclosed herein also may be implemented as instructions stored on a machine-readable medium or memory, which may be read and executed by one or more processors.) and one or more processors coupled to the memory to perform operations including: ([0018] The material disclosed herein also may be implemented as instructions stored on a machine-readable medium or memory, which may be read and executed by one or more processors.)
As for the rest of the claim, they recite similar elements to the method of claim 1, therefore the rationale applied in rejection of claim 1 is also applicable.
Claim 20 is a system claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 21 is a system claim with limitations similar to the limitations of Claim 19, but is broader than claim 19 and is rejected under similar rationale.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Cordourier (US 20240430634), in view of Sharma (US 20240185875), and further in view of Tagliasacchi (US 20230395087).
Regarding claim 9, Cordourier/Sharma teach all the element of claim 1,
Cordourier/Sharma do not disclose input audio processed in portion of duration in under 100 ms.
Tagliasacchi in the related art discloses: wherein the input audio is processed in portions having durations of less than 100 milliseconds. ([0129] All signals may be assumed to be sampled at 16 KHz. The mel spectrogram can be calculated with a window length of 1024 samples (64 ms) and a hop length of 160 (10 ms) and with 128 bins. The optimization may be performed with a training dataset using step size 0.005 and 1000 iterations, which can be performed in less than a minute on a single GPU due to the small number of parameters. For the microphone model, the systems and methods may include fixing the STFT window length to 2048 (128 ms) and hop length to 160 in the experiments. Both the speech enhancement network and the microphone model may operate on gain normalized signals. As an illustration, FIG. 9 shows example parameters of the microphone model learned from a single audio sample in the MOBIPHONE dataset.)
Cordourier/Sharma/Tagliasacchi are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Cordourier/Sharma to combine the teaching of Tagliasacchi, because the present disclosure enable improved model robustness (e.g., to microphone variability) by generating augmented training data in which the training data can be augmented to reflect different potential microphone transformations (Tagliasacchi, [0031]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Cordourier (US 20240430634), in view of Sharma (US 20240185875), further in view of Tagliasacchi (US 20230395087), and furthermore in view of Freed (US 20220310088).
Regarding claim 10, Cordourier/Sharma/Tagliasacchi teach all the element of claim 9,
Cordourier/Sharma/Tagliasacchi do not teach or suggest wherein a length of the durations is programmable.
Freed in the related art discloses: wherein a length of the durations is programmable. ([0050] In some embodiments, the orchestration layer 110 revises the time threshold to increase the duration based on which the length of the voice input segment is determined.)
Cordourier/Sharma/Tagliasacchi/Freed are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Cordourier/Sharma/Tagliasacchi to combine the teaching of Freed, because capturing complete voice input segment leads to better speech recognition (Freed, [0050]).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Cordourier (US 20240430634), in view of Sharma (US 20240185875), further in view of Khorram (US 20240203406).
Regarding claim 15, Cordourier/Sharma teach all the element of claim 14,
Cordourier/Sharma do not disclose wherein the training audio is generated by applying random augmentations to the ground truth output audio, and the random augmentations include at least one of inserting random noise, modifying a room impulse response, or modifying frequency content of the ground truth output audio.
Khorram in the related art discloses: wherein the training audio is generated by applying random augmentations to the ground truth output audio, and the random augmentations include at least one of inserting random noise, modifying a room impulse response, or modifying frequency content of the ground truth output audio. ([0040] Applying data augmentation to the acoustic frames 306 furthers the acoustic diversity of the audio frames used to train the speech recognition model 200. The data augmentation module 360 may include a time masking component that masks portions of the acoustic frames 306. Other techniques applied by the data augmentation module 360 may include adding/injecting noise and/or adding reverberation of the labeled audio samples 305. One data augmentation technique includes using multistyle training (MTR) to inject a variety of environmental noises to the labeled audio samples 305. Another data augmentation technique that the data augmentation module 360 may apply in addition to, or in lieu of, MTR, includes using spectrum augmentation (SpecAugment) to make the acoustics of the labeled audio samples 305 closer to the adverse acoustics of other labeled audio samples 305. In combination, MTR and SpecAugment may inject noises into the labeled audio samples 305, tile random external noise sources along time and inserted before and overlapped onto the representation, and filtering the noise-injective labeled audio samples prior to training the speech recognition model 200.)
Cordourier/Sharma/Khorram are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Cordourier/Sharma to combine the teaching of Khorram, because the training technique described maximize training data diversity by augmenting and creating additional data (cost savings), prevents overfitting and lead to more robust model (Khorram, [0040]).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Cordourier (US 20240430634), in view of Sharma (US 20240185875), further in view of E. Ayanoglu et al., "Diversity Coding for Transparent Self-Healing and Fault-Tolerant Communication Networks," IEEE Transactions on Communications, vol. 41, no. 11, pp. 1677-1685, November, 1993.).
Regarding claim 18, Cordourier/Sharma teach all the element of claim 1,
Cordourier/Sharma do not disclose wherein at least one of: the encoding in the frequency domain, the encoding in the time domain, the fusing, or the processing is implemented to include advanced error correction, fault-tolerance, and self-healing capabilities.
Ayanoglu in the related art discloses: wherein at least one of: the encoding in the frequency domain, the encoding in the time domain, the fusing, or the processing is implemented to include advanced error correction, fault-tolerance, and self-healing capabilities. ([sect I. Introduction] discloses self-healing. [sect II. Diversity coding] discloses error correction for errors in the channel. [sect III. Applications and Extensions] discloses fault-tolerant parallel transmission of continuous amplitude discrete time signals,)
Cordourier/Sharma/Ayanoglu are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Cordourier/Sharma to combine the teaching of Ayanoglu, because the technique described introduce error correction in communication paths to create self-healing and fault-tolerant network channels (Ayanoglu, [Conclusion]).
Allowable Subject Matter
Claims 6-7, and 12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Notwithstanding, said aforementioned teachings of prior art cited is respectfully reconsidered and found to fail to teach or fairly suggest either individually or in a reasonable combination the presented limitations in claims 6-7, and 12, as specifically recited.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang, Y., Ju, Z., Tan, X., He, L., Wu, Z., & Bian, J. (2023). Audit: Audio editing by following instructions with latent diffusion models. Advances in Neural Information Processing Systems, 36, 71340-71357. -discloses editing audio by replacing the original source environment into a targeted audio environment that suits the user’s need. See Abstract and figs 1-3 for additional details.
Tang, Z., Bryan, N. J., Li, D., Langlois, T. R., & Manocha, D. (2020). Scene-aware audio rendering via deep acoustic analysis. IEEE transactions on visualization and computer graphics, 26(5), 1991-2001.- discloses a method/system for recording audio in various environment and use the recording to make virtual 3D models sound like from the environment. See Abstract and fig. 2 for additional details.
Khokhlov, Y., Zatvornitskiy, A., Medennikov, I., Sorokin, I., Prisyach, T., Romanenko, A., ... & Petrov, O. (2019). R-vectors: New technique for adaptation to room acoustics. In Proc. Interspeech 2019 (pp. 1243-1247). -disclose use of R-vector for room acoustic adaptation. See Abstract and sections 2-3 for additional details.
Fejzzo (US 20230186926) discloses a method/system for machine learning based key generation for key guided audio signal transformation. “A method comprise: receiving input audio and target audio having a target audio characteristic; using a first neural network, trained to generate key parameters that represent the target audio characteristic based on one or more of the target audio and the input audio, generating the key parameters; and configuring a second neural network, trained to be configured by the key parameters, with the key parameters to cause the second neural network to perform a signal transformation of the input audio, to produce output audio having an output audio characteristic corresponding to and that matches the target audio characteristic.” See Abstract, para 0017, 0019, 0021-0024 for additional details.
Ahn (US 20240265261) discloses style transfer of acoustic environment. See Abstract, and para 0019, 0041, 0046, 0048 and figs 3, 5 and 6 for additional details.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Phillip H Lam whose telephone number is (571)272-1721. The examiner can normally be reached 9 AM-2 PM, and 4-6 PM Pacific time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached on (571) 272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHILIP H LAM/ Examiner, Art Unit 2656
Prosecution Insights