Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 10 and 19 are amended. Claims 1-20 are presented for examination.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang ( SPATIALCODEC: Neural Spatial Speech Coding ) and further in view of Bataev( US 20240135920)
Regarding claim 10, Wang teaches A codec for performing neural spatial audio coding, comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code ( neural spatial audio framework, Abstract, Introduction ) including: receiving code configured to cause the at least one processor to receive an audio signal comprising a plurality of channels ( multi-channel audio, Under Introduction ) ; selecting code configured to cause the at least one processor to select a channel from the plurality of channels as a reference channel ( reference channel, Under Introduction) ; first performing code configured to cause the at least one processor to perform a short time fourier transform (STFT) on the reference channel to generate a frequency domain reference channel ( short time fourier transform on the reference channel , 3.1. Single-Channel Sub-band Codec (First Branch) ) ; first inputting code configured to cause the at least one processor to input the frequency domain reference channel into a first codec that comprises a first neural network for encoding the frequency domain reference channel and a second neural network for decoding the frequency domain reference channel ( encoder decoder network, Fig1), ; second performing code configured to cause the at least one processor to perform the STFT on the plurality of channels minus the channel selected as the reference channel to generate a spatial covariance matrix ( STFT is performed on a M-1 channel, Fig 1) ; second inputting code configured to cause the at least one processor to input the spatial covariance matrix and the frequency domain reference channel into a second codec that comprises a third neural network for encoding the spatial covariance matrix and the frequency domain reference channel (concat -
PNG
media_image1.png
414
494
media_image1.png
Greyscale
, Under 3.2 Spatial code (second branch) ) and a fourth neural network for decoding the spatial covariance matrix; ( decoding, Under 3.2) reconstructing code configured to cause the at least one processor to reconstruct the audio signal based on an output of the first codec and an output of the second codec to generate a reconstructed audio signal ( decoder for reconstruction in branch 1 and branch 2, Introduction, Fig 1) ; first training code configured to cause the at least one processor to train the first codec based on a comparison of the output of the first codec with the frequency domain reference channel ( training a neural network, Fig 1) ; and second training code configured to cause the at least one processor to train the second codec based on a comparison of the reconstructed audio signal with the plurality of channels of the audio signal minus the reference channel (
PNG
media_image2.png
450
482
media_image2.png
Greyscale
, Under 3.3 Training and Loss)
Wang does not teach the concept computing device having a virtual machines where codec reside, wherein the first codec and the second codec are included in the virtual machine
However, Bataev teaches computing device having a virtual machines where codec reside, wherein the first codec and the second codec are included in the virtual machine
(systems implemented using an edge device, systems incorporating one or more virtual machines (VMs), and the system including the neural audio encoders, Para 0025, 0029, 0031,0072- 0073, 0088)
Wang teaches the entire concept of claim 10, its well known in the art that codec can run on multiple computing/edge devices including virtual machine and it would have been obvious to include virtual machines to install codec since virtual machine can provide more flexibility compared to traditional hardware.
Regarding claim 11, Wang as above in claim 10, teaches , wherein the first neural network of the first codec is a two-dimensional convolutional neural network (2D-CNN), and the second neural network of the first codec is a transpose of the 2D-CNN ( Then the whole encoder-decoder architectures are 2D-CNNs with residual blocks treating real-imaginary as the channel dimension, Under 3.1)
Regarding claim 12, Wang as above in claim 10, teaches, wherein the third neural network of the second codec is a two-dimensional convolutional neural network (2D-CNN), and the fourth neural network of the first codec is a transpose of the 2D-CNN ( ( Then the whole encoder-decoder architectures are 2D-CNNs with residual blocks treating real-imaginary as the channel dimension, Under 3.1)
Regarding claim 13, Wang as above in claim 10, teaches, wherein the second codec further comprises a plurality complex ratio filters, wherein a total number of the complex ratio filters corresponds to the plurality of audio channels minus the reference channel ( complex ratio filter, Under 3.2)
Regarding claim 14, Wang as above in claim 10, teaches, The codec according to claim 10, wherein the reconstructing code further comprises (i) third performing code configured to cause the at least one processor to perform an inverse STFT on an output of the first codec to generate a reconstructed reference channel, (ii) third inputting code configured to cause the at least one processor to input the reconstructed reference channel and the output of the second codec into a filter, and (iii) fourth performing code configured to cause the at least one processor to perform an inverse STFT on an output of the filter to obtain the reconstructed audio signal (STFT and inverse STFT for reconstruction, Fig 1)
Regarding claim 15, Wang as above in claim 10, teaches, wherein the first training code is based on an signal-to-noise ratio (SNR) loss function applied to the frequency domain reference channel and the output of the first codec ( SNR loss, Fig 1, Under 3.3 Training and Loss)
Regarding claim 16, Wang as above in claim 10, teaches, wherein the second training code is based on an signal-to-noise ratio (SNR) loss applied to each non-reference channel and corresponding reconstructed non-reference channel ( SNR loss is applied to all the channel, Fig 1)
Regarding claim 17, Wang as above in claim 10, teaches, wherein the first codec further comprises a first quantizer that quantizes an output of the first neural network and provides the quantized output of the first neural network to the second neural network( encoder having a quantizer codec, Under 3.1) , and wherein the second code further comprises a second quantizer that quantizes an output of the third neural network and provides the quantized output of the third neural network to the second neural network ( encoder quantizer codec, Under 3.2)
Regarding claim 18, Wang as above in claim 10, teaches, wherein the audio signal is captured by a microphone comprising a plurality of arrays corresponding to the plurality of channels of the audio signal ( microphone, Under Introduction )
Regarding claim 1, arguments analogous to claim 10, are applicable.
Regarding claim 2, arguments analogous to claim 11, are applicable.
Regarding claim 3, arguments analogous to claim 12, are applicable.
Regarding claim 4, arguments analogous to claim 13, are applicable.
Regarding claim 5, arguments analogous to claim 14, are applicable.
Regarding claim 6, arguments analogous to claim 15, are applicable.
Regarding claim 7 , arguments analogous to claim 16, are applicable.
Regarding claim 8, arguments analogous to claim 17, are applicable.
Regarding claim 9, arguments analogous to claim 18, are applicable.
Regarding claim 19, arguments analogous to claim 10, are applicable.
2nd Rejection
Claims 1, 10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sharma ( US 20240005946) and further in view of Zhong ( Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement) and further in view of Bataev( US 20240135920)
Regarding claim 1, Sharma teaches a method performed by at least one processor for performing neural spatial audio coding, comprising: receiving an audio signal comprising a plurality of channels ( multi-channel audio signal, Para 0007) ; selecting a channel from the plurality of channels as a reference channel (reference channel for e.g. Ch1, Para 0055) ; performing a short time fourier transform (STFT) on the reference channel to generate a frequency domain reference channel (performing short time fourier transform to estimate RTF, Para 0080, Fig 10 ) ; inputting the frequency domain reference channel into a first codec that comprises a first neural network for encoding the frequency domain reference channel and a second neural network for decoding the frequency domain reference channel ( encoder decoder network, Fig 2 and Fig 3- encoder decoder network where the ref channel is an input – furthermore the encoder decoder are neural networks) ; performing the STFT on the plurality of channels minus the channel selected as the reference channel to generate a RTF( finding RTF, fig 10) ; inputting the spatial covariance matrix and the frequency domain reference channel into a second codec that comprises a third neural network for encoding the RTF and the frequency domain reference channel and a fourth neural network for decoding the RTF; reconstructing the audio signal based on an output of the first codec ( Fig 4c- channel 1 from the spectral decoder and channel 2 stream are input to the spatial encoder 430 which is later decoder ( second and third neural nets, Para 0080, , Fig 4) and an output of the second codec to generate a reconstructed audio signal (Fig 2 and Fig 3, decoded signal using the RTF and residual ) ; training the first codec based on a comparison of the output of the first codec with the frequency domain reference channel ( training based on the loss, Para 0071 0073) ; and training the second codec based on a comparison of the reconstructed audio signal with the plurality of channels of the audio signal minus the reference channel( training, fig 4b the CNN 420 or neural net architecture can be trained to minimize the difference between the output from this block 430 and the Ch.2 signal as one loss term, in addition to the ASR loss, Para 0080)
Although Sharma teaches the concept of generating RTF, Sharma does not explicitly teach performing the STFT on the plurality of channels minus the channel selected as the reference channel to generate a spatial covariance matrix ; inputting the spatial covariance matrix and the frequency domain reference channel into a second codec that comprises a third neural network for encoding the spatial covariance matrix and the frequency domain reference channel and a fourth neural network for decoding the spatial covariance matrix; reconstructing the audio signal based on an output of the first codec
However Zhong teaches the concept of STFT on the plurality of channels minus the channel selected as the reference channel to generate a spatial covariance matrix ( calculate the spatial covariance matrix, Fig 1, Under 3.2. Multi-frame multichannel Wiener filter) ; inputting the spatial covariance matrix and the frequency domain reference channel into a second codec that comprises a third neural network for encoding the spatial covariance matrix and the frequency domain reference channel ( Xbfi is concatenated with the covariance matrix, Fig 1; Under 3, Methods; wherein the network is a neural network – specifically Conv-TasNet, --For the second and third networks, the separated time-domain outputs of the previous network are concatenated with the time-domain mixture signal as the input features to produce separated estimates, Under 4.3 Networks ) and a fourth neural network for decoding the spatial covariance matrix ( decoding using DCN++ , Fig 1)
Sharma has a structure for the reconstruction of the audio as descried in the claimed invention. Sharma’s concept differed by the claim in the sense Shama uses relative transfer functions ( RTF) instead of covariance matrix. Covariance matrix is a known technique as described in the Zhong as they explore different ways of calculating it to concatenate with the input signal to do the sound separation and hence it would have been obvious to one of the ordinary skill in the art before effective filing date to use covariance matrix instead of RTF for obtaining a spatial information since covariance matrix is well known technique for capturing multi source spatial information ( Abstract, Zhong)
Sharma modified by Zhong does not teach the concept computing device having a virtual machines where codec reside, wherein the first codec and the second codec are included in the virtual machine
However, Bataev teaches computing device having a virtual machines where codec reside, wherein the first codec and the second codec are included in the virtual machine
(systems implemented using an edge device, systems incorporating one or more virtual machines (VMs), and the system including the neural audio encoders, Para 0025, 0029, 0031,0072- 0073, 0088)
Sharma and Zhong teaches the entire concept of claim 1, its well known in the art that codec can run on multiple computing/edge devices including virtual machine and it would have been obvious to include virtual machines to install codec since virtual machine can provide more flexibility compared to traditional hardware.
Regarding claim 10, arguments analogous to claim 1, are applicable.
Regarding claim 19, arguments analogous to claim 1, are applicable.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Richa Sonifrank whose telephone number is (571)272-5357. The examiner can normally be reached M-T 7AM - 5:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Phan Hai can be reached at (571)272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Richa Sonifrank/Primary Examiner, Art Unit 2654
Prosecution Insights