DETAILED ACTION
This action is in response to the request for continued examination filed 11/21/2025, claims 1, 4-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Request for Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/21/2025 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 and 8 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments filed 11/21/2025 regarding claim 16 have been fully considered but they are not persuasive. Specifically, applicant argues that Zhang et al (US 2020/0202867) does not disclose the limitations of “using an orientation with a maximum probability being a position of the current sound source as an effective sound source orientation”, Applicant argues “Zhang only describes the “probability of an active speaker located along the PDF.” Zhang, para. [0037]. However, the speaker receives the audio signal from the audio source, instead of the audio source itself. Thus the probability in Zhang is not related to the sound source”. In Paragraph 34, Zhang describes utilizing a neural network to determine an angular orientation of the user with respect to a microphone array, describing the determination of the sound sources orientation from the microphone that receives the signal, which as described is the speech of a speaker within a room. Paragraph 35 further describes the generation of a SSL (Sound Source Localization) distribution, which also utilizes weighting values to determine likely positions and/or orientations of the speaker. Paragraph 37 further describes this SSL distribution and the usage of a Probability Distribution Function (PDF) which indicates an active speaker along the distribution function, that is to say, the PDF displays probabilities for different positions of the speaker. FIG. 7 further displays an aspect of the machine learning system, which displays the recording of the probability of a speaker’s location corresponding to each orientation angle. Further, Paragraph 45 states “In this example, the joint speaker location/speaker identification neural network 34 may utilize the phase information features 110 of the current audio signal to locate the source of the human speech at an angular orientation of 29 degrees with a relatively high degree of probability.”, displaying the determination of an angular orientation based upon a high degree of probability. Based upon the above reasoning, Zhang et al teaches the limitation of “using an orientation with a maximum probability being a position of the current sound source as an effective sound source orientation.”, as such the rejection of claim 16 under 35 U.S.C. 102(a)(2) is maintained.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim 16 is rejected under 35 U.S.C. 102(a)(2) as being anticipated by Zhang et al, US Publication No. 2020/0202867.
Regarding Claim 16, Zhang et al teaches One or more memories storing thereon computer-readable instructions (FIG. 1, Non-Volatile Storage 18) that, when executed by one or more processors (FIG. 1, Processor 26), cause the one or more processors to perform acts comprising: acquiring a current audio signal captured by a microphone array, the microphone array including at least two microphones (FIG. 1, Microphone array 48, Paragraph 23, "In some examples, microphone array 48 may utilize 7 microphones. In other examples, any suitable number of microphones may be utilized.");
generating spatial distribution information of a current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones (Paragraph 29, "As illustrated in FIG. 4, the joint speaker location/speaker identification neural network 34 processes the magnitude features 100 and phase information features 110 through a plurality of layers to yield a plurality of speaker identification and location features described below.", FIG. 8A, Steps 804, 808, 812, 816);
and identifying whether the current audio signal is an overlapping speech based on the spatial distribution information of the current sound source and in combination with a conversion relationship between a single speech and the overlapping speech learned from historical audio signals (Paragraph 51, "Similarly, the joint speaker location/speaker identification neural network 34 also may determine voice overlap (Overlap) characteristics from a multi-channel audio signal. In some examples, where a user embedding comprises Overlap characteristics, the joint speaker location/speaker identification neural network 34 may determine if the Overlap characteristics indicate that the audio data contains two or more people speaking at the same time. Where the Overlap characteristics indicate that two or more voices are overlapping, then the joint speaker location/recognition engine 14 may refrain from utilizing this user embedding to enhance a speaker profile. In a similar manner, determining that the Overlap characteristics indicate that multiple voices are overlapping also may be utilized to manage a variety of other functionalities of different user computing devices.", See further, Paragraphs 3 and 20.);
in response to determining that the current audio signal is identified as the overlapping speech, determining at least two effective sound source orientations based on the spatial distribution information of the current sound source (Paragraph 40, "In this example, the multi-channel audio signal 600 is analyzed to determine that user Charlie 330 speaks for a period of time followed by a break, then Alice 340 speaks for a longer period of time followed by a break, and then Bob 350 speaks. As this conversation continues and each speaker remains at their respective angular locations, the joint speaker location/speaker identification neural network 34 learns to associate each speaker's identity with her or his angular location with respect to the audio-conferencing device 310.");
or in response to determining that the current audio signal is identified as a single speech, using an orientation with a maximum probability being a position of the current sound source as an effective sound source orientation (FIG. 8A, Step 812, 816, See further, Paragraphs 34, 35, and 37.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867 in view of Visser, US Publication No. 2012/0224456.
Regarding Claim 1, Zhang et al teaches A method comprising: acquiring a current audio signal captured by a microphone array, the microphone array including at least two microphones (FIG. 1, Microphone array 48, Paragraph 23, "In some examples, microphone array 48 may utilize 7 microphones. In other examples, any suitable number of microphones may be utilized.");
generating spatial distribution information of a current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones (Paragraph 29, "As illustrated in FIG. 4, the joint speaker location/speaker identification neural network 34 processes the magnitude features 100 and phase information features 110 through a plurality of layers to yield a plurality of speaker identification and location features described below.", FIG. 8A, Steps 804, 808, 812, 816);
and identifying that the current audio signal is an overlapping speech based on the spatial distribution information of the current sound source and in combination with a conversion relationship between a single speech and the overlapping speech learned from historical audio signals (Paragraph 51, "Similarly, the joint speaker location/speaker identification neural network 34 also may determine voice overlap (Overlap) characteristics from a multi-channel audio signal. In some examples, where a user embedding comprises Overlap characteristics, the joint speaker location/speaker identification neural network 34 may determine if the Overlap characteristics indicate that the audio data contains two or more people speaking at the same time. Where the Overlap characteristics indicate that two or more voices are overlapping, then the joint speaker location/recognition engine 14 may refrain from utilizing this user embedding to enhance a speaker profile. In a similar manner, determining that the Overlap characteristics indicate that multiple voices are overlapping also may be utilized to manage a variety of other functionalities of different user computing devices.", Further, See Paragraphs 3 and 20.).
Zhang does not further teach, the generating the spatial distribution information of the current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones including: calculating a wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones, wherein the wave arrival spectrogram reflects the spatial distribution information of the current sound source, the calculating the wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones comprises: accumulating the phase difference information of the current audio signal captured by respective two microphones for an orientation in a position space to obtain a probability of the orientation being a position of the current sound source and generating the wave arrival spectrogram corresponding to the current audio signal based on a probability of each orientation in the position space being a position of the current sound source.
However, Visser et al, in a similar invention in the same field of endeavor teaches, the generating the spatial distribution information of the current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones including: calculating a wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones (Paragraph 61, “FIG. 1A shows an example of a multi-microphone device D10 that includes an array of two microphones MC10 and MC20. In other examples of device D10, the microphone array may include more than two microphones (e.g., three, four, or more). In such cases, the microphones may be disposed in a linear or two- or three-dimensional pattern, and the spacing between adjacent microphones in the array may be uniform or non-uniform.”, Paragraph 79, “The DOA estimation method may be configured to calculate, for each of a plurality of the calculated phase differences, a corresponding indication of the DOA.”, Paragraph 100, “Task T100 may be configured to estimate the range from the detected ultrasound signal in the time domain or in a frequency domain, such as a subband domain or a transform domain (e.g., an FFT domain). In a subband-domain example, task T100 passes the time-domain received signal through a bank of one or more time-domain bandpass filters and measures the output energy of each subband. In a transform-domain example, task T100 calculates a spectrogram of the received signal and monitors an evolution of the energy at the peak frequency (e.g., 40 kHz) over time (see, e.g., FIG. 20).”), wherein the wave arrival spectrogram reflects the spatial distribution information of the current sound source (Paragraph 101, “Task T100 may be configured to determine the time-axis location of a received echo as the location of an energy peak in the FFT domain. For a time-domain signal, task T100 may be configured to determine the time-axis location of a received echo as the location of the peak of a region of samples whose energy (individually or, alternatively, collectively) is above an echo detection threshold value.”).
the calculating the wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones comprises: accumulating the phase difference information of the current audio signal captured by respective two microphones for an orientation in a position space (Paragraph 91, “It may be desirable in certain applications to localize the position of each of one or more directional sound sources relative to an audio sensing device. In addition to DOA estimation, for example, it may be desirable to obtain information regarding the range (i.e., distance) of each directional sound source relative to the microphone array. It may be desirable to use such information to track the position over time of a moving directional source, such as a human speaker, relative to the audio sensing device.”, Paragraph 97, “In such cases, the application may be configured to use phase differences to estimate DOA in response to an indication that the source is close to the device. Near-field noise reduction methods may also be used in such cases. In response to an indication that the source is far away from the device, the application may be configured to use an energy-difference-based DOA estimation method (and/or a different phase-difference-based method) instead.”), to obtain a probability of the orientation being a position of the current sound source (Paragraph 73, “A DOA estimate may also be obtained by directly using the BSS unmixing matrix W and the microphone spacing. Such a technique may include estimating the source DOA (e.g., for each source-microphone pair) by using back-projection of separated source signals, using an inverse (e.g., the Moore-Penrose pseudo-inverse) of the unmixing matrix W, followed by single-source DOA estimation on the back-projected data. Such a DOA estimation method is typically robust to errors in microphone gain response calibration. The BSS unmixing matrix W is applied to the m microphone signals X.sub.1 to X.sub.M, and the source signal to be back-projected Y.sub.j is selected from among the outputs of matrix W. A DOA for each source-microphone pair may be computed from the back-projected signals using a technique such as GCC-PHAT or SRP-PHAT. A maximum likelihood and/or multiple signal classification (MUSIC) algorithm may also be applied to the back-projected signals for source localization. The back-projection methods described above are illustrated in FIG. 37.”, explains the usage of multiple probability-based functions to determine a DOA and/or a position of the sound source.),
and generating the wave arrival spectrogram corresponding to the current audio signal based on a probability of each orientation in the position space being a position of the current sound source (Paragraph 100, “Task T100 may be configured to estimate the range from the detected ultrasound signal in the time domain or in a frequency domain, such as a subband domain or a transform domain (e.g., an FFT domain). In a subband-domain example, task T100 passes the time-domain received signal through a bank of one or more time-domain bandpass filters and measures the output energy of each subband. In a transform-domain example, task T100 calculates a spectrogram of the received signal and monitors an evolution of the energy at the peak frequency (e.g., 40 kHz) over time (see, e.g., FIG. 20).”, Paragraph 101, “Task T100 may be configured to determine the time-axis location of a received echo as the location of an energy peak in the FFT domain. For a time-domain signal, task T100 may be configured to determine the time-axis location of a received echo as the location of the peak of a region of samples whose energy (individually or, alternatively, collectively) is above an echo detection threshold value. The echo detection threshold value may be fixed or adaptive, and it may be desirable to limit the maximum width (in samples) of the region. Task T100 may be configured to identify the peak as the highest-energy sample of the region or, alternatively, as the center in time of the region.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of generating a wave arrival spectrogram corresponding to the current audio signal, utilizing phase difference information, being utilized to determine the probability of a location of the sound source, as taught by Visser et al, with the system as taught by Zhang et al. The motivation being that the utilization of spectrograms to determine source locations is utilized in audio based systems when attempting to locate speakers within a desired location, as well as the utilization of probability function when developing and determining these sound source positioning.
Regarding Claim 17, Zhang et al teaches all the limitations of claim 16, but does not further teach wherein the generating the spatial distribution information of the current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones comprises: calculating a wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones, wherein the wave arrival spectrogram reflects the spatial distribution information of the current sound source.
However, Visser et al, in a similar invention in the same field of endeavor teaches, the generating the spatial distribution information of the current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones including: calculating a wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones (Paragraph 61, “FIG. 1A shows an example of a multi-microphone device D10 that includes an array of two microphones MC10 and MC20. In other examples of device D10, the microphone array may include more than two microphones (e.g., three, four, or more). In such cases, the microphones may be disposed in a linear or two- or three-dimensional pattern, and the spacing between adjacent microphones in the array may be uniform or non-uniform.”, Paragraph 79, “The DOA estimation method may be configured to calculate, for each of a plurality of the calculated phase differences, a corresponding indication of the DOA.”, Paragraph 100, “Task T100 may be configured to estimate the range from the detected ultrasound signal in the time domain or in a frequency domain, such as a subband domain or a transform domain (e.g., an FFT domain). In a subband-domain example, task T100 passes the time-domain received signal through a bank of one or more time-domain bandpass filters and measures the output energy of each subband. In a transform-domain example, task T100 calculates a spectrogram of the received signal and monitors an evolution of the energy at the peak frequency (e.g., 40 kHz) over time (see, e.g., FIG. 20).”), wherein the wave arrival spectrogram reflects the spatial distribution information of the current sound source (Paragraph 101, “Task T100 may be configured to determine the time-axis location of a received echo as the location of an energy peak in the FFT domain. For a time-domain signal, task T100 may be configured to determine the time-axis location of a received echo as the location of the peak of a region of samples whose energy (individually or, alternatively, collectively) is above an echo detection threshold value.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of creating a wave arrival spectrogram corresponding to the phase difference information of the current audio signal, captured by the at least two microphones, as taught by Visser et al, with the system as taught by Zhang et al. The motivation being to create a representation of the audio signals, to enhance the performed processing and machine learning process.
Regarding Claim 18, Zhang et al in view of Visser et al teaches all the limitations of claim 17, and Visser et al further teaches, wherein the calculating the wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones comprises: accumulating the phase difference information of the current audio signal captured by respective two microphones for an orientation in a position space (Paragraph 91, “It may be desirable in certain applications to localize the position of each of one or more directional sound sources relative to an audio sensing device. In addition to DOA estimation, for example, it may be desirable to obtain information regarding the range (i.e., distance) of each directional sound source relative to the microphone array. It may be desirable to use such information to track the position over time of a moving directional source, such as a human speaker, relative to the audio sensing device.”, Paragraph 97, “In such cases, the application may be configured to use phase differences to estimate DOA in response to an indication that the source is close to the device. Near-field noise reduction methods may also be used in such cases. In response to an indication that the source is far away from the device, the application may be configured to use an energy-difference-based DOA estimation method (and/or a different phase-difference-based method) instead.”), to obtain a probability of the orientation being a position of the current sound source (Paragraph 73, “A DOA estimate may also be obtained by directly using the BSS unmixing matrix W and the microphone spacing. Such a technique may include estimating the source DOA (e.g., for each source-microphone pair) by using back-projection of separated source signals, using an inverse (e.g., the Moore-Penrose pseudo-inverse) of the unmixing matrix W, followed by single-source DOA estimation on the back-projected data. Such a DOA estimation method is typically robust to errors in microphone gain response calibration. The BSS unmixing matrix W is applied to the m microphone signals X.sub.1 to X.sub.M, and the source signal to be back-projected Y.sub.j is selected from among the outputs of matrix W. A DOA for each source-microphone pair may be computed from the back-projected signals using a technique such as GCC-PHAT or SRP-PHAT. A maximum likelihood and/or multiple signal classification (MUSIC) algorithm may also be applied to the back-projected signals for source localization. The back-projection methods described above are illustrated in FIG. 37.”, explains the usage of multiple probability-based functions to determine a DOA and/or a position of the sound source.);
and generating the wave arrival spectrogram corresponding to the current audio signal based on a probability of each orientation in the position space being a position of the current sound source (Paragraph 100, “Task T100 may be configured to estimate the range from the detected ultrasound signal in the time domain or in a frequency domain, such as a subband domain or a transform domain (e.g., an FFT domain). In a subband-domain example, task T100 passes the time-domain received signal through a bank of one or more time-domain bandpass filters and measures the output energy of each subband. In a transform-domain example, task T100 calculates a spectrogram of the received signal and monitors an evolution of the energy at the peak frequency (e.g., 40 kHz) over time (see, e.g., FIG. 20).”, Paragraph 101, “Task T100 may be configured to determine the time-axis location of a received echo as the location of an energy peak in the FFT domain. For a time-domain signal, task T100 may be configured to determine the time-axis location of a received echo as the location of the peak of a region of samples whose energy (individually or, alternatively, collectively) is above an echo detection threshold value. The echo detection threshold value may be fixed or adaptive, and it may be desirable to limit the maximum width (in samples) of the region. Task T100 may be configured to identify the peak as the highest-energy sample of the region or, alternatively, as the center in time of the region.”).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Visser, even further in view of Kahn. Zhang et al, US Publication No. 2020/0202867, in view of Visser, US Publication No. 2012/0224456, even further in view of Kahn et al, US Publication No. 2006/0149558.
Regarding Claim 4, Zhang et al in view of Visser et al teaches all the limitations of claim 1, but does not further teach wherein the identifying that the current audio signal is the overlapping speech based on the spatial distribution information of the current sound source and in combination with the conversion relationship between the single speech and the overlapping speech learned from the historical audio signals comprises: calculating peak information of the spatial distribution information of the current sound source as a current observation state of a Hidden Markov model (HMM); using the single speech and the overlapping speech as two hidden states of the HMM; inputting the current observation state into the HMM and, in conjunction with a jump relationship between the two hidden states learned by the HMM, calculating a probability of a hidden state corresponding to the current observation state by taking a historical observation state as a precondition; and identifying that the current audio signal is the overlapping speech based on the probability of the hidden state corresponding to the current observation state
However, Kahn et al in a similar invention in the same field of endeavor teaches, wherein the identifying that the current audio signal is the overlapping speech based on the spatial distribution information of the current sound source and in combination with the conversion relationship between the single speech and the overlapping speech learned from the historical audio signals comprises: calculating peak information of the spatial distribution information of the current sound source as a current observation state of a Hidden Markov model (HMM) (Paragraph 25, "During the decoding phase, the acoustic data is typically processed with Hidden Markov Models, probabilistic methods that include parameters defined for states, transition probabilities, and observation likelihoods.");
using the single speech and the overlapping speech as two hidden states of the HMM (Paragraph 297, "In sequence 1210, the process may index each audio-aligned text segment based upon speaker identity. In one approach, the locate speaker-specific segments and index steps may be performed simultaneously. The indexing may reference start/duration times of segments ("time stamps") and sequentially number each segment. During sequence 1210, the index may also be include speaker identity, speaker overlap, and other segment-related data.");
inputting the current observation state into the HMM and, in conjunction with a jump relationship between the two hidden states learned by the HMM, calculating a probability of a hidden state corresponding to the current observation state by taking a historical observation state as a precondition (Paragraph 610, “In some instances, use of common segmentation speech parameters 635 among different speakers based upon historical or group data may be sufficient. In one approach, the speech segmentation parameters are stored as part of the speech user profile 312 (FIG. 3) and loaded by a user management application that represents part of a workflow manager described in association with FIG. 1.”);
and identifying that the current audio signal is the overlapping speech based on the probability of the hidden state corresponding to the current observation state (Paragraph 372, "Speech recognition may be used for single channel and multichannel multispeaker settings. In addition to the benefits of automating transcription, speech recognition 476 may also assist in determination of sequence in cases of speech overlap. Using "ROVER" techniques, different speech user profile 312 (FIG. 3) or configuration parameters, or both, may be used to create 1 through N transcribed session file s478.", Paragraph 389, "In one approach, the preprocessing application (FIG. 4) may use "ROVER" techniques to determine a best match based upon output from two or more instances of the same or different speech recognition programs 476 using different configurable options or different speech user profiles. These techniques may determine best results based upon confidence scores ").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of using a Hidden Markov Model to help in identifying the speech signal as an overlapping speech signal or a single speaker signal, as taught by Kahn et al, with the system as taught by Zhang et al in view of Visser et al. The motivation being to allow for a better process of speech recognition, and to further implement the speech recognition features.
Claims 5-7 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867, in view of Visser, US Publication No. 2012/0224456, even further in view of Sannino et al, US Publication No. 2014/0192999.
Regarding Claim 5, Zhang et al in view of Visser et al teaches all the limitations of claim 1, and Zhang et al further teaches, in response to determining that the current audio signal is identified as the overlapping speech, determining at least two effective sound source orientations based on the spatial distribution information of the current sound source (Paragraph 58, "At 832 the method 800 may include utilizing the location characteristics of the user embedding to determine an angular orientation of the user with respect to a microphone array that captured the multi-channel audio signal of the utterance spoken by the user. At 836 the method 800 may include outputting the angular orientation along with the identity of the person matched to the user."); Zhang et al in view of Visser et al does not further teach performing speech enhancement on audio signals in the at least two effective sound source orientations, and performing speech recognition on the enhanced audio signals in the at least two effective sound source orientations respectively.
However, Sannino et al in a similar invention in the same field of endeavor teaches, performing speech enhancement on audio signals in the at least two effective sound source orientations (Paragraph 67, " An embodiment thus allows for selecting and outputting an acoustic signal from more than one source, e.g., from the presenter in a conference and a member of the audience asking a question, while screening all other ambient sounds, like, for instance, murmuring of other members of the audience, background noise from technical equipment like air conditioning, fans, or projectors, or from other sources of noise, e.g., street noise. Hence, the sound quality of the selected and possibly transmitted or recorded acoustic signal can be significantly improved.", Paragraph 68, "Additionally, certain filters like anti-echo-filters, babble-noise-filters, other noise filters, etc., may be applied to the temporally sampled acoustic signal, the Fourier transformed matrix representation, or the obtained spectrum to improve the signal quality and/or the SNR.");
and performing speech recognition on the enhanced audio signals in the at least two effective sound source orientations respectively (Paragraph 66, "Since the obtained one-dimensional spectrum ideally represents the acoustic signal from a single acoustic source in the direction of arrival, it may be used to direct the reception by the plurality of microphones at a specific acoustic source, e.g., a specific speaker in a conference room. Various characteristics of the obtained one-dimensional spectrum, such as pitch, range, central frequency, dominant frequencies, cepstral coefficients, etc., or recognition of female/male/child speaker, voice recognition results, etc., may be used to select at least one directional acoustic signal out of a plurality of directional acoustic signals corresponding to a plurality of acoustic sources.").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of performing sound enhancement in the two sound orientations, and performing speech recognition on the enhanced audio signals, as taught by Sannino et al, with the system as taught by Zhang et al in view of Visser et al. The motivation being to allow for a better speech recognition process, by ensuring the audio signals are optimized for such functions.
Regarding Claim 6, Zhang et al in view of Visser et al, further in view of Sannino et al teach all the limitations of claim 5, and Zhang et al further teaches, wherein the determining the at least two effective sound source orientations based on the spatial distribution information of the current sound source comprises: in response to determining that the spatial distribution information of the current sound source comprises a probability of a respective orientation being a position of the current sound source, taking two orientations with maximum probabilities being positions of the current sound source as effective sound source orientations (Paragraph 40, "In this example, the multi-channel audio signal 600 is analyzed to determine that user Charlie 330 speaks for a period of time followed by a break, then Alice 340 speaks for a longer period of time followed by a break, and then Bob 350 speaks. As this conversation continues and each speaker remains at their respective angular locations, the joint speaker location/speaker identification neural network 34 learns to associate each speaker's identity with her or his angular location with respect to the audio conferencing device 310.").
Regarding Claim 7, Zhang et al in view of Visser et al teaches all the limitations of claim 1, but does not further teach wherein before the identifying that the current audio signal is the overlapping speech, the method further comprises: calculating a direction of arrival (DOA) of the current audio signal based on the spatial 35distribution information of the current sound source; selecting, according to the DOA, one microphone from the at least two microphones as a target microphone; and performing voice activity detection (VAD) on the current audio signal captured by the target microphone to determine that the current audio signal is a speech signal.
However, Sannino in a similar invention in the same field of endeavor teaches, wherein before the identifying that the current audio signal is the overlapping speech, the method further comprises: calculating a direction of arrival (DOA) of the current audio signal based on the spatial distribution information of the current sound source (Paragraph 40, "Once a first peak has been determined in the spectrum, the direction of arrival (DOA) of the acoustic signal at least one of the plurality of microphones is calculated according to an embodiment based on the determined first peak. The first peak may be determined as the global maximum in the spectrum obtained based on the application of the Fourier transform. From the location of the determined first peak in the (D+1)-dimensional Fourier space, an angle or multiple angles may be determined between a line connecting the origin of the Fourier space and the determined first peak and the axis or multiple axes of the spatial frequency dimensions of the Fourier space.");
selecting, according to the DOA, one microphone from the at least two microphones as a target microphone (Paragraph 66, "Since the obtained one-dimensional spectrum ideally represents the acoustic signal from a single acoustic source in the direction of arrival, it may be used to direct the reception by the plurality of microphones at a specific acoustic source, e.g., a specific speaker in a conference room.");
and performing voice activity detection (VAD) on the current audio signal captured by the target microphone to determine that the current audio signal is a speech signal (Paragraph 66, "Various characteristics of the obtained one-dimensional spectrum, such as pitch, range, central frequency, dominant frequencies, cepstral coefficients, etc., or recognition of female/male/child speaker, voice recognition results, etc., may be used to select at least one directional acoustic signal out of a plurality of directional acoustic signals corresponding to a plurality of acoustic sources.").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of utilizing the DOA to determine target microphones, and performing VAD on the selected audio signal, as taught by Sannino, with the system as taught by Zhang et al in view of Visser et al. The motivation being to enhance the speech recognition process, and allow for a more efficient process.
Claims 8 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867, in view of Benattar, US Patent No. 9980042 B1.
Regarding Claim 8, Zhang et al teaches, a device comprising: a microphone array (FIG. 1, Microphone array 48); one or more processors (FIG. 1, Processor 26); and one or more memories storing thereon computer-readable instructions (FIG. 1, Non-Volatile Storage 18) that, when executed by the one or more processors, cause the one or more processors to perform acts comprising: acquiring a current audio signal captured by the microphone array, the microphone array including at least two microphones (FIG. 1, Microphone array 48, Paragraph 23, "In some examples, microphone array 48 may utilize 7 microphones. In other examples, any suitable number of microphones may be utilized.");
generating spatial distribution information of a current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones (Paragraph 29, "As illustrated in FIG. 4, the joint speaker location/speaker identification neural network 34 processes the magnitude features 100 and phase information features 110 through a plurality of layers to yield a plurality of speaker identification and location features described below.", FIG. 8A, Steps 804, 808, 812, 816);
and identifying that the current audio signal is an overlapping speech based on the spatial distribution information of the current sound source and in combination with a conversion relationship between a single speech and the overlapping speech learned from historical audio signals (Paragraph 51, "Similarly, the joint speaker location/speaker identification neural network 34 also may determine voice overlap (Overlap) characteristics from a multi-channel audio signal. In some examples, where a user embedding comprises Overlap characteristics, the joint speaker location/speaker identification neural network 34 may determine if the Overlap characteristics indicate that the audio data contains two or more people speaking at the same time. Where the Overlap characteristics indicate that two or more voices are overlapping, then the joint speaker location/recognition engine 14 may refrain from utilizing this user embedding to enhance a speaker profile. In a similar manner, determining that the Overlap characteristics indicate that multiple voices are overlapping also may be utilized to manage a variety of other functionalities of different user computing devices.", See further, Paragraphs 3 and 20.).
Zhang does not further teach, Determining whether the current audio signal is a speech signal; in response to determining that the current audio is not a speech signal, determining not to recognize the current audio signal or in response to determining that the current audio signal is the speech signal, recognizing the current audio signal.
However, Benattar et al, in a similar invention in the same field of endeavor teaches, Determining whether the current audio signal is a speech signal; in response to determining that the current audio is not a speech signal, determining not to recognize the current audio signal (Column 18, Lines 21-24, “If the detection condition active decision 611 is no, then the process goes to deselect beams at 613, which may be the same as deselect step 609, and start loop 614 passes back to start loop 601”, the detection condition active decision is the determination on if speech is present, if so the beam analysis is not performed, and further checks for speech before activating the analysis later, the conditions of activation are further explained in the quotation below.); or in response to determining that the current audio signal is the speech signal, recognizing the current audio signal (Column 17, Lines 50-55, US 9980042 B1, “If the determination 602 is that there is no active beam, determination 611 tests whether the detection condition is active. The detection condition is any condition that the analysis process is monitoring. Audio conditions may include voice activity detection, keyword detection, speaker detection, and direction of arrival detection.”, Displays the usage of VAD to determine whether or not to process the current audio signal, allowing for the system to be selective based upon if speech is detected. ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of not recognizing the current audio signal if it is determined that the current audio signal is not a speech signal, and to recognize the current audio signal if it is determined that the current audio signal is a speech signal, as taught by Benattar et al, with the system as taught by Zhang et al. The motivation being that selectively processing signals based upon whether speech is present allows for the system to not analyze signals that are noise or other sound sources, which in a system which prioritizes speech, allows for it to not process unnecessary signals, which saves the system time and resources.
Regarding Claim 15, Zhang et al in view of Benattar et al teaches all the limitations of claim 8, and Zhang et al further teaches, wherein the device is a conference device, a sound pickup device, a robot, a smart set-top box, a smart TV, a smart speaker, or a smart vehicle- mounted device (Paragraph 25, “In different examples the user computing device 10 may take a variety of forms. In some examples, user computing device 10 may take the form of a standalone device that may be placed in home or work environment. For example, the standalone device may comprise or be communicatively coupled with an intelligent personal assistant program, and may enable a user to interact with the assistant program via voice commands. For example, a user may control a music player application, interact with a personal assistant application, request information, and perform other actions by speaking commands to the standalone device. In other examples user computing device 10 may take the form of an audio or audio/visual conferencing device, a desktop computer, laptop computer, tablet computer, mobile computer, smartphone, set-top device, gaming console, stand-alone display, or any other type of suitable computing device. In some examples, user computing device 10 may be embedded or otherwise included with other devices, components, vehicles, buildings, or other objects to enable the collection and exchange of data among devices.”).
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867, in view of Benattar, US Patent No. 9980042 B1, even further in view of Visser, US Publication No. 2012/0224456.
Regarding Claim 9, Zhang et al in view of Benattar teaches all the limitations of claim 8, but does not further teach wherein the generating the spatial distribution information of the current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones comprises: calculating a wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones wherein the wave arrival spectrogram reflects the spatial distribution information of the current sound source.
However, Visser et al in a similar invention in the same field of endeavor teaches, wherein the generating the spatial distribution information of the current sound source corresponding to the current audio signal based on phase difference information of the current audio signal captured by the at least two microphones comprises: calculating a wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones (Paragraph 61, “FIG. 1A shows an example of a multi-microphone device D10 that includes an array of two microphones MC10 and MC20. In other examples of device D10, the microphone array may include more than two microphones (e.g., three, four, or more). In such cases, the microphones may be disposed in a linear or two- or three-dimensional pattern, and the spacing between adjacent microphones in the array may be uniform or non-uniform.”, Paragraph 79, “The DOA estimation method may be configured to calculate, for each of a plurality of the calculated phase differences, a corresponding indication of the DOA.”, Paragraph 100, “Task T100 may be configured to estimate the range from the detected ultrasound signal in the time domain or in a frequency domain, such as a subband domain or a transform domain (e.g., an FFT domain). In a subband-domain example, task T100 passes the time-domain received signal through a bank of one or more time-domain bandpass filters and measures the output energy of each subband. In a transform-domain example, task T100 calculates a spectrogram of the received signal and monitors an evolution of the energy at the peak frequency (e.g., 40 kHz) over time (see, e.g., FIG. 20).”), wherein the wave arrival spectrogram reflects the spatial distribution information of the current sound source (Paragraph 101, “Task T100 may be configured to determine the time-axis location of a received echo as the location of an energy peak in the FFT domain. For a time-domain signal, task T100 may be configured to determine the time-axis location of a received echo as the location of the peak of a region of samples whose energy (individually or, alternatively, collectively) is above an echo detection threshold value.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of creating a wave arrival spectrogram corresponding to the phase difference information of the current audio signal, captured by the at least two microphones, as taught by Visser et al, with the system as taught by Zhang et al in view of Benattar. The motivation being to create a representation of the audio signals, to enhance the performed processing and machine learning process.
Regarding Claim 10, Zhang et al in view of Benattar, even further in view of Visser et al teaches all the limitations of claim 9, and Visser et al further teaches, wherein the calculating the wave arrival spectrogram corresponding to the current audio signal based on the phase difference information of the current audio signal captured by the at least two microphones comprises: accumulating the phase difference information of the current audio signal captured by respective two microphones for an orientation in a position space (Paragraph 91, “It may be desirable in certain applications to localize the position of each of one or more directional sound sources relative to an audio sensing device. In addition to DOA estimation, for example, it may be desirable to obtain information regarding the range (i.e., distance) of each directional sound source relative to the microphone array. It may be desirable to use such information to track the position over time of a moving directional source, such as a human speaker, relative to the audio sensing device.”, Paragraph 97, “In such cases, the application may be configured to use phase differences to estimate DOA in response to an indication that the source is close to the device. Near-field noise reduction methods may also be used in such cases. In response to an indication that the source is far away from the device, the application may be configured to use an energy-difference-based DOA estimation method (and/or a different phase-difference-based method) instead.”), to obtain a probability of the orientation being a position of the current sound source (Paragraph 73, “A DOA estimate may also be obtained by directly using the BSS unmixing matrix W and the microphone spacing. Such a technique may include estimating the source DOA (e.g., for each source-microphone pair) by using back-projection of separated source signals, using an inverse (e.g., the Moore-Penrose pseudo-inverse) of the unmixing matrix W, followed by single-source DOA estimation on the back-projected data. Such a DOA estimation method is typically robust to errors in microphone gain response calibration. The BSS unmixing matrix W is applied to the m microphone signals X.sub.1 to X.sub.M, and the source signal to be back-projected Y.sub.j is selected from among the outputs of matrix W. A DOA for each source-microphone pair may be computed from the back-projected signals using a technique such as GCC-PHAT or SRP-PHAT. A maximum likelihood and/or multiple signal classification (MUSIC) algorithm may also be applied to the back-projected signals for source localization. The back-projection methods described above are illustrated in FIG. 37.”, explains the usage of multiple probability-based functions to determine a DOA and/or a position of the sound source.);
and generating the wave arrival spectrogram corresponding to the current audio signal based on a probability of each orientation in the position space being a position of the current sound source (Paragraph 100, “Task T100 may be configured to estimate the range from the detected ultrasound signal in the time domain or in a frequency domain, such as a subband domain or a transform domain (e.g., an FFT domain). In a subband-domain example, task T100 passes the time-domain received signal through a bank of one or more time-domain bandpass filters and measures the output energy of each subband. In a transform-domain example, task T100 calculates a spectrogram of the received signal and monitors an evolution of the energy at the peak frequency (e.g., 40 kHz) over time (see, e.g., FIG. 20).”, Paragraph 101, “Task T100 may be configured to determine the time-axis location of a received echo as the location of an energy peak in the FFT domain. For a time-domain signal, task T100 may be configured to determine the time-axis location of a received echo as the location of the peak of a region of samples whose energy (individually or, alternatively, collectively) is above an echo detection threshold value. The echo detection threshold value may be fixed or adaptive, and it may be desirable to limit the maximum width (in samples) of the region. Task T100 may be configured to identify the peak as the highest-energy sample of the region or, alternatively, as the center in time of the region.”).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867, in view of Benattar, US Patent No. 9980042 B1, even further in view of Kahn et al, US Publication No. 2006/0149558.
Regarding Claim 11, Zhang et al in view of Benattar teaches all the limitations of claim 8, but does not further teach wherein the identifying that the current audio signal is the overlapping speech based on the spatial distribution information of the current sound source and in combination with the conversion relationship between the single speech and the overlapping speech learned from the historical audio signals comprises: calculating peak information of the spatial distribution information of the current sound source as a current observation state of a Hidden Markov model (HMM); using the single speech and the overlapping speech as two hidden states of the HMM; inputting the current observation state into the HMM and, in conjunction with a jump relationship between the two hidden states learned by the HMM, calculating a probability of a hidden state corresponding to the current observation state by taking a historical observation state as a precondition; and identifying that the current audio signal is the overlapping speech based on the probability of the hidden state corresponding to the current observation state.
However, Kahn et al in a similar invention in the same field of endeavor teaches, wherein the identifying that the current audio signal is the overlapping speech based on the spatial distribution information of the current sound source and in combination with the conversion relationship between the single speech and the overlapping speech learned from the historical audio signals comprises: calculating peak information of the spatial distribution information of the current sound source as a current observation state of a Hidden Markov model (HMM) (Paragraph 25, "During the decoding phase, the acoustic data is typically processed with Hidden Markov Models, probabilistic methods that include parameters defined for states, transition probabilities, and observation likelihoods.");
using the single speech and the overlapping speech as two hidden states of the HMM (Paragraph 297, "In sequence 1210, the process may index each audio-aligned text segment based upon speaker identity. In one approach, the locate speaker-specific segments and index steps may be performed simultaneously. The indexing may reference start/duration times of segments ("time stamps") and sequentially number each segment. During sequence 1210, the index may also include speaker identity, speaker overlap, and other segment-related data.");
inputting the current observation state into the HMM and, in conjunction with a jump relationship between the two hidden states learned by the HMM, calculating a probability of a hidden state corresponding to the current observation state by taking a historical observation state as a precondition (Paragraph 610 “In some instances, use of common segmentation speech parameters 635 among different speakers based upon historical or group data may be sufficient. In one approach, the speech segmentation parameters are stored as part of the speech user profile 312 (FIG. 3) and loaded by a user management application that represents part of a workflow manager described in association with FIG. 1.”);
and identifying that the current audio signal is the overlapping speech based on the probability of the hidden state corresponding to the current observation state (Paragraph 372, "Speech recognition may be used for single channel and multichannel multispeaker settings. In addition to the benefits of automating transcription, speech recognition 476 may also assist in determination of sequence in cases of speech overlap. Using "ROVER" techniques, different speech user profile 312 (FIG. 3) or configuration parameters, or both, may be used to create 1 through N transcribed session file s478.", Paragraph 389, "In one approach, the preprocessing application (FIG. 4) may use "ROVER" techniques to determine a best match based upon output from two or more instances of the same or different speech recognition programs 476 using different configurable options or different speech user profiles. These techniques may determine best results based upon confidence scores ").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of using a Hidden Markov Model to help in identifying the speech signal as an overlapping speech signal or a single speaker signal, as taught by Kahn et al, with the system as taught by Zhang et al in view of Benattar. The motivation being to allow for a better process of speech recognition, and to further implement the speech recognition features.
Claims 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867, in view of Benattar, US Patent No. 9980042 B1, even further in view of Sannino et al, US Publication No. 2014/0192999.
Regarding Claim 12, Zhang et al in view of Benattar teaches all the limitations of claim 8, and Zhang et al further teaches, wherein the acts further comprise: in response to determining that the current audio signal is identified as the overlapping speech, determining at least two effective sound source orientations based on the spatial distribution information of the current sound source (Paragraph 58, "At 832 the method 800 may include utilizing the location characteristics of the user embedding to determine an angular orientation of the user with respect to a microphone array that captured the multi-channel audio signal of the utterance spoken by the user. At 836 the method 800 may include outputting the angular orientation along with the identity of the person matched to the user."), they do not further teach performing speech enhancement on audio signals in the at least two effective sound source orientations; and performing speech recognition on the enhanced audio signals in the at least two effective sound source orientations respectively.
However, Sannino et al in a similar invention in the same field of endeavor teaches, performing speech enhancement on audio signals in the at least two effective sound source orientations (Paragraph 67, " An embodiment thus allows for selecting and outputting an acoustic signal from more than one source, e.g., from the presenter in a conference and a member of the audience asking a question, while screening all other ambient sounds, like, for instance, murmuring of other members of the audience, background noise from technical equipment like air conditioning, fans, or projectors, or from other sources of noise, e.g., street noise. Hence, the sound quality of the selected and possibly transmitted or recorded acoustic signal can be significantly improved.", Paragraph 68, "Additionally, certain filters like anti-echo-filters, babble-noise-filters, other noise filters, etc., may be applied to the temporally sampled acoustic signal, the Fourier transformed matrix representation, or the obtained spectrum to improve the signal quality and/or the SNR.");
and performing speech recognition on the enhanced audio signals in the at least two effective sound source orientations respectively (Paragraph 66, "Since the obtained one-dimensional spectrum ideally represents the acoustic signal from a single acoustic source in the direction of arrival, it may be used to direct the reception by the plurality of microphones at a specific acoustic source, e.g., a specific speaker in a conference room. Various characteristics of the obtained one-dimensional spectrum, such as pitch, range, central frequency, dominant frequencies, cepstral coefficients, etc., or recognition of female/male/child speaker, voice recognition results, etc., may be used to select at least one directional acoustic signal out of a plurality of directional acoustic signals corresponding to a plurality of acoustic sources.").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of performing sound enhancement in the two sound orientations, and performing speech recognition on the enhanced audio signals, as taught by Sannino et al, with the system as taught by Zhang et al in view of Benattar. The motivation being to allow for a better speech recognition process, by ensuring the audio signals are optimized for such functions.
Regarding Claim 13, Zhang et al in view of Benattar, further in view of Sannino et al teaches all the limitations of claim 12, and Zhang et al further teaches, wherein the determining the at least two effective sound source orientations based on the spatial distribution information of the current sound source comprises: in response to determining that the spatial distribution information of the current sound source comprises a probability of a respective orientation being a position of the current sound source, taking two orientations with maximum probabilities being positions of the current sound source as effective sound source orientations (Paragraph 40, "In this example, the multi-channel audio signal 600 is analyzed to determine that user Charlie 330 speaks for a period of time followed by a break, then Alice 340 speaks for a longer period of time followed by a break, and then Bob 350 speaks. As this conversation continues and each speaker remains at their respective angular locations, the joint speaker location/speaker identification neural network 34 learns to associate each speaker's identity with her or his angular location with respect to the audio conferencing device 310.").
Regarding Claim 14, Zhang et al in view of Benattar teaches all the limitations of claim 8, but does not further teach, wherein before the identifying that the current audio signal is the overlapping speech, the acts further comprise: calculating a direction of arrival (DOA) of the current audio signal based on the spatial distribution information of the current sound source; selecting, according to the DOA, one microphone from the at least two microphones as a target microphone; and performing voice activity detection (VAD) on the current audio signal captured by the target microphone to determine that the current audio signal is a speech signal.
Sannino however, in a similar invention in the same field of endeavor teaches, wherein before the identifying that the current audio signal is the overlapping speech, the acts further comprise: calculating a direction of arrival (DOA) of the current audio signal based on the spatial distribution information of the current sound source (Paragraph 40, "Once a first peak has been determined in the spectrum, the direction of arrival (DOA) of the acoustic signal at least one of the plurality of microphones is calculated according to an embodiment based on the determined first peak. The first peak may be determined as the global maximum in the spectrum obtained based on the application of the Fourier transform. From the location of the determined first peak in the (D+1)-dimensional Fourier space, an angle or multiple angles may be determined between a line connecting the origin of the Fourier space and the determined first peak and the axis or multiple axes of the spatial frequency dimensions of the Fourier space.");
selecting, according to the DOA, one microphone from the at least two microphones as a target microphone (Paragraph 66, "Since the obtained one-dimensional spectrum ideally represents the acoustic signal from a single acoustic source in the direction of arrival, it may be used to direct the reception by the plurality of microphones at a specific acoustic source, e.g., a specific speaker in a conference room.");
and performing voice activity detection (VAD) on the current audio signal captured by the target microphone to determine that the current audio signal is a speech signal (Paragraph 66, "Various characteristics of the obtained one-dimensional spectrum, such as pitch, range, central frequency, dominant frequencies, cepstral coefficients, etc., or recognition of female/male/child speaker, voice recognition results, etc., may be used to select at least one directional acoustic signal out of a plurality of directional acoustic signals corresponding to a plurality of acoustic sources.").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of utilizing the DOA to determine target microphones, and performing VAD on the selected audio signal, as taught by Sannino, with the system as taught by Zhang et al in view of Benattar. The motivation being to enhance the speech recognition process, and allow for a more efficient process.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867, in view of Kahn et al, US Publication No. 2006/0149558.
Regarding Claim 19, Zhang et al teaches all the limitations of Claim 16, but does not further teach wherein the identifying that the current audio signal is the overlapping speech based on the spatial distribution information of the current sound source and in combination with the conversion relationship between the single speech and the overlapping speech learned from the historical audio signals comprises: calculating peak information of the spatial distribution information of the current sound source as a current observation state of a Hidden Markov model (HMM); using the single speech and the overlapping speech as two hidden states of the HMM; inputting the current observation state into the HMM and, in conjunction with a jump relationship between the two hidden states learned by the HMM, calculating a probability of a hidden state corresponding to the current observation state by taking a historical observation state as a precondition; and identifying that the current audio signal is the overlapping speech based on the probability of the hidden state corresponding to the current observation state.
However, Kahn et al in a similar invention in the same field of endeavor teaches, wherein the identifying that the current audio signal is the overlapping speech based on the spatial distribution information of the current sound source and in combination with the conversion relationship between the single speech and the overlapping speech learned from the historical audio signals comprises: calculating peak information of the spatial distribution information of the current sound source as a current observation state of a Hidden Markov model (HMM) (Paragraph 25, "During the decoding phase, the acoustic data is typically processed with Hidden Markov Models, probabilistic methods that include parameters defined for states, transition probabilities, and observation likelihoods.");
using the single speech and the overlapping speech as two hidden states of the HMM (Paragraph 297, "In sequence 1210, the process may index each audio-aligned text segment based upon speaker identity. In one approach, the locate speaker-specific segments and index steps may be performed simultaneously. The indexing may reference start/duration times of segments ("time stamps") and sequentially number each segment. During sequence 1210, the index may also include speaker identity, speaker overlap, and other segment-related data.");
inputting the current observation state into the HMM and, in conjunction with a jump relationship between the two hidden states learned by the HMM, calculating a probability of a hidden state corresponding to the current observation state by taking a historical observation state as a precondition (Paragraph 610 “In some instances, use of common segmentation speech parameters 635 among different speakers based upon historical or group data may be sufficient. In one approach, the speech segmentation parameters are stored as part of the speech user profile 312 (FIG. 3) and loaded by a user management application that represents part of a workflow manager described in association with FIG. 1.”);
and identifying that the current audio signal is the overlapping speech based on the probability of the hidden state corresponding to the current observation state (Paragraph 372, "Speech recognition may be used for single channel and multichannel multispeaker settings. In addition to the benefits of automating transcription, speech recognition 476 may also assist in determination of sequence in cases of speech overlap. Using "ROVER" techniques, different speech user profile 312 (FIG. 3) or configuration parameters, or both, may be used to create 1 through N transcribed session file s478.", Paragraph 389, "In one approach, the preprocessing application (FIG. 4) may use "ROVER" techniques to determine a best match based upon output from two or more instances of the same or different speech recognition programs 476 using different configurable options or different speech user profiles. These techniques may determine best results based upon confidence scores ").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of using a Hidden Markov Model to help in identifying the speech signal as an overlapping speech signal or a single speaker signal, as taught by Kahn et al, with the system as taught by Zhang et al. The motivation being to allow for a better process of speech recognition, and to further implement the speech recognition features.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al, US Publication No. 2020/0202867, in view of Sannino et al, US Publication No. 2014/0192999.
Regarding Claim 20, Zhang et al teaches all the limitations of claim 16, and further teaches the determining the at least two effective sound source orientations based on the spatial distribution information of the current sound source comprises: in response to determining that the spatial distribution information of the current sound source comprises a probability of a respective orientation being a position of the current sound source, taking two orientations with maximum probabilities being positions of the current sound source as effective sound source orientations (Paragraph 40, "In this example, the multi-channel audio signal 600 is analyzed to determine that user Charlie 330 speaks for a period of time followed by a break, then Alice 340 speaks for a longer period of time followed by a break, and then Bob 350 speaks. As this conversation continues and each speaker remains at their respective angular locations, the joint speaker location/speaker identification neural network 34 learns to associate each speaker's identity with her or his angular location with respect to the audio conferencing device 310."); but does not further teach in response to determining that the current audio signal is identified as the overlapping speech, performing speech enhancement on audio signals in the at least two effective sound source orientations; and performing speech recognition on the enhanced audio signals in the at least two effective sound source orientations respectively.
However, Sannino et al in a similar invention in the same field of endeavor teaches, in response to determining that the current audio signal is identified as the overlapping speech, performing speech enhancement on audio signals in the at least two effective sound source orientations (Paragraph 67, " An embodiment thus allows for selecting and outputting an acoustic signal from more than one source, e.g., from the presenter in a conference and a member of the audience asking a question, while screening all other ambient sounds, like, for instance, murmuring of other members of the audience, background noise from technical equipment like air conditioning, fans, or projectors, or from other sources of noise, e.g., street noise. Hence, the sound quality of the selected and possibly transmitted or recorded acoustic signal can be significantly improved.", Paragraph 68, "Additionally, certain filters like anti-echo-filters, babble-noise-filters, other noise filters, etc., may be applied to the temporally sampled acoustic signal, the Fourier transformed matrix representation, or the obtained spectrum to improve the signal quality and/or the SNR.");
and performing speech recognition on the enhanced audio signals in the at least two effective sound source orientations respectively (Paragraph 66, "Since the obtained one-dimensional spectrum ideally represents the acoustic signal from a single acoustic source in the direction of arrival, it may be used to direct the reception by the plurality of microphones at a specific acoustic source, e.g., a specific speaker in a conference room. Various characteristics of the obtained one-dimensional spectrum, such as pitch, range, central frequency, dominant frequencies, cepstral coefficients, etc., or recognition of female/male/child speaker, voice recognition results, etc., may be used to select at least one directional acoustic signal out of a plurality of directional acoustic signals corresponding to a plurality of acoustic sources.").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to combine the teachings of performing sound enhancement in the two sound orientations, and performing speech recognition on the enhanced audio signals, as taught by Sannino et al, with the system as taught by Zhang et al. The motivation being to allow for a better speech recognition process, by ensuring the audio signals are optimized for such functions.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M NEECE whose telephone number is (703)756-1941. The examiner can normally be reached 10am - 7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, CAROLYN EDWARDS can be reached at (571)-270-7136. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN MAGUIRE NEECE/Examiner, Art Unit 2692
/CAROLYN R EDWARDS/ Supervisory Patent Examiner, Art Unit 2692