DETAILED ACTION
Introduction
Applicant's submission filed on 04/04/2024 has been entered. Claims 1-20 are pending in the application and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over R. Li, et. al., "Multi-Stream End-to-End Speech Recognition," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 646-655, 2020 in view of Kristjansson et. al., US Patent 10,937441.
PNG
media_image1.png
562
712
media_image1.png
Greyscale
Regarding claim 1, Li teaches an end-to-end automatic speech recognition (ASR) system, comprising :a first encoder configured for close-talk input captured by a close-talk input mechanism(see Li, pg. 649, sect III A : For simplicity, multi-stream model with N=2 is depicted in Fig. 1, where two encoders in parallel take different input features, X(1) with T(1) frames and X(2) with T(2) frames, respectively. Each encoder operates on different temporal resolution with subsampling factor s(1) and s(2), where subsampling could be performed in RNN or maxpooling layer in CNN; Stream 1 is interpreted as close talk and Stream 2 is interpreted as far talk); a second encoder configured for far-talk input captured by a far-talk input mechanism (see Li, pg. 649, sect III A : For simplicity, multi-stream model with N=2 is depicted in Fig. 1, where two encoders in parallel take different input features, X(1) with T(1) frames and X(2) with T(2) frames, respectively. Each encoder operates on different temporal resolution with subsampling factor s(1) and s(2), where subsampling could be performed in RNN or maxpooling layer in CNN; Stream 1 is interpreted as close talk and Stream 2 is interpreted as far talk) and the first output of the first encoder and the second output of the second encoder are weighted according to encoder-selection probabilities to produce a combined output(see Li, pg. 4 sect. III B to generate the stream attention). However, Li fails to teach an encoder selection layer configured to select the first encoder, the second encoder, or both, based on the close-talk input, the far-talk input, or both, wherein upon determining that signals from both the close-talk input mechanism and the far-talk input mechanism are present for a speech segment: the first encoder processes the first speech feature derived from the close-talk input to produce a first output, and the second encoder processes the second speech feature derived from the far-talk input to produce a second output.
However, Kristjansson teaches an encoder selection layer configured to select the first encoder, the second encoder, or both, based on a first speech feature
PNG
media_image2.png
545
426
media_image2.png
Greyscale
derived from the close-talk input, a second speech feature derived from the far-talk input, or both (see Kristjansson , col 39, lines 15-45 discusses determination of far-talk or close talk ; interpreted as encoder selection), wherein upon determining that signals from both the close-talk input mechanism and the far-talk input mechanism are present for a speech segment (see Kristjansson , col 7 lines 53-55 double talk ): the first encoder processes the first speech feature derived from the close-talk input to produce a first output, and the second encoder processes the second speech feature derived from the far-talk input to produce a second output (see Kristjansson , fig. 10, 14, col 41 line 60 – col 42, line 1 processing of near talk and far talk features; first and second encoder ); and the first output of the first encoder and the second output of the second encoder are weighted according to encoder-selection probabilities to produce a combined output (see Kristjansson, col 42, lines 46-56 and col 43 lines 38-48 discuss weighting the signals for combined output ).
Li and Kristjansson are considered to be analogous to the claimed invention because they relate to audio processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Li on a multi-stream architecture within the joint CTC/Attention framework with the target signals selection based on local speech or remote speech teachings of Li to improve audio processing based on current system conditions ( see Kristjansson, col 2 lines 11-27).
Regarding claim 2, Li in view of Kristjansson teaches the system according to claim 1. Kristjansson further teaches wherein the combined output is a weighted average of the first output and the second output (see Kristjansson, col 31 lines 25-61 discuss the reference signal based on average power values and in double talk conditions; weighted average).
Regarding claim 3, Li in view of Kristjansson teaches the system according to claim 1. Li further teaches wherein the far-talk input is a multi-channel far-talk input that is processed by a neural network, wherein the neural network learns mapping the multi-channel far-talk input to an enhanced single-channel input (see Li, pg. 652, sec V, Note that for each array, multi-channel input was synthesized into a single-channel audio using Delay-and-Sum beamforming technique with BeamformIt Toolkit).
Regarding claim 4, Li in view of Kristjansson teaches the system according to claim 1. Kristjansson further teaches wherein the far-talk input is a multi-channel far-talk input that is processed by a beamformer, wherein the beamformer turns the multi-channel far-talk input to a single-channel input using a data-adaptive beamforming solution (see Kristjansson, col 9 lines 38-54 discusses an example of beamforming based on filter and sum structure(data-adapting beamforming solution), there are other methods mentioned too ).
Regarding claim 5, Li in view of Kristjansson teaches the system according to claim 1. Kristjansson further teaches wherein the first speech feature is at least one of a short-time Fourier transform (STFT), a Mel-frequency Cepstral Coefficient (MFCC), or a filter bank derived from the close-talk input (see Kristjansson, col 32 line 65-col 33 line 15 discusses filter bank to process the signal for near-end single-talk detection ).
Regarding claim 6, Li in view of Kristjansson teaches the system according to claim 1. Kristjansson further teaches wherein the first speech feature is at least one of a short-time Fourier transform (STFT), a Mel-frequency Cepstral Coefficient (MFCC), or a filter bank derived from the far-talk input (see Kristjansson, col 35 line 50 – col 36 line 8 discusses filter bank to process the signal for far-end talk detection ).
Regarding claim 7, Li in view of Kristjansson teaches the system according to claim 1 . Kristjansson further teaches wherein the close-talk input mechanism comprises a first type of input device (see Kristjansson, Fig. 1, 112).
Regarding claim 9, Li in view of Kristjansson teaches the system according to claim 1 . Kristjansson further teaches wherein the far-talk input mechanism comprises a second type of input device (see Kristjansson, col 3 line 28-33, Fig. 1, 114).
Regarding claim 10, Li in view of Kristjansson teaches the system according to claim 1 . Kristjansson further teaches wherein the second type of input device comprises a microphone array (see Kristjansson, col 9, lines 29-31 The device 110 may operate using a microphone array 114 comprising multiple microphones, where beamforming techniques may be used to isolate desired audio including speech ).
Regarding claim 11, is directed to a method claim corresponding to the system claim presented in claim 1 and is rejected under the same grounds stated above regarding claim 1.
Regarding claim 12, is directed to a method claim corresponding to the system claim presented in claim 2 and is rejected under the same grounds stated above regarding claim 2.
Regarding claim 13, is directed to a method claim corresponding to the system claim presented in claim 3 and is rejected under the same grounds stated above regarding claim 3.
Regarding claim 14, is directed to a method claim corresponding to the system claim presented in claim 4 and is rejected under the same grounds stated above regarding claim 4.
Regarding claim 15, is directed to a method claim corresponding to the system claim presented in claim 5 and is rejected under the same grounds stated above regarding claim 5.
Regarding claim 17, is directed to a method claim corresponding to the system claim presented in claim 7 and is rejected under the same grounds stated above regarding claim 7.
Regarding claim 19, is directed to a method claim corresponding to the system claim presented in claim 9 and is rejected under the same grounds stated above regarding claim 9.
Regarding claim 20, is directed to a method claim corresponding to the system claim presented in claim 9 and is rejected under the same grounds stated above regarding claim 20.
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over R. Li, et. al., "Multi-Stream End-to-End Speech Recognition," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 646-655, 2020 in view of Kristjansson et. al., US Patent 10,937441further in view of Flanagan et. al., US Patent 5,737,485.
Regarding claim 8, Li in view of Kristjansson teaches the system according to claim 7. Li in view of Kristjansson fail to teach wherein the first type of input device comprises a headphone or an MP3 recorder.
However, Flanagan teaches wherein the first type of input device comprises a headphone or an MP3 recorder (see Flanagan, Fig. 1, close talking 8 ( microphone), Fig. 1, 2 distant talking ( microphone array) ).
Li, Kristjansson and Flanagan are considered to be analogous to the claimed invention because they relate to End-to-End (E2E) speech recognition. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Li and Kristjansson on a multi-stream architecture and the acoustic-based classification teachings with the using microphone arrays and neural network teachings of Flanagan to improve speech recognition for microphone-array distant from the speaker ( see Flanagan col 1 line 55-col2, line 9).
Regarding claim 18, is directed to a method claim corresponding to the system claim presented in claim 8 and is rejected under the same grounds stated above regarding claim 8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Le Roux, et. al, US Patent 10,811,000 teaches end-to-end automatic speech recognition (ASR) systems used with encoder-decoder architecture to recognize multiple speakers. (see Le Roux, Col. 9 lines 58-Col. 10, lines 12)
Shah, et. al., US Patent Application Publication 2017/0208391 teaches using the far end signal for echo cancellation at a near end microphone (see Shah, Fig. 3).
X Wang, et.al., “Exploring end-to-end multi-channel ASR with bias information for meeting transcription,” in Proc. of IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China, 2021, pp. 833–840, IEEE teaches Joint frontend/back-end model trained with multi-channel meeting data and bias information and Session-based decoding strategy for target-speaker ASR (see Wang, Fig. 1).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANDINI SUBRAMANI whose telephone number is (571)272-3916. The examiner can normally be reached Monday - Friday 12:00pm - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh M Mehta can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NANDINI SUBRAMANI/Examiner, Art Unit 2656
/BHAVESH M MEHTA/Supervisory Patent Examiner, Art Unit 2656