DETAILED ACTION
This office action is in response to Applicant’s Request for Continued Examination (RCE), received on 04/16/2026. Claims 1, 2, 5-7, 10-12, 15 have been amended. Claims 1-15 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/16/2026 has been entered.
Response to Arguments
Applicant’s arguments, see pgs. 8-12, filed 04/16/2026, with respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 (Lindrup in view of Sivaraman), with respect to Applicant’s points 2.1 and 2.2, have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Xie (US-20190341041-A1) and Graham et al. (US-20250088795-A1), hereinafter Graham.
Xie discloses “The method of an electronic apparatus includes obtaining first voice data and second voice data. The method includes allowing a background speaker model that is generated based on voice signals of a plurality of speakers to be trained based on the second voice data. The method includes determining whether the first voice data is voice data corresponding to a first registered speaker associated with the electronic apparatus. The method includes allowing a first registered speaker model that is configured to recognize a voice of the first registered speaker to be trained based on the first voice data and the background speaker model, based on determining that the first voice data is voice data corresponding to the first registered speaker” (abstract). Specifically, the registered speaker model and background speaker models of Xie will be taken in view Lindrup.
Graham discloses “an invention that that mutes specific talkers using at least one beamforming microphone array 102 that is configured to generate N audio signals 108 where each audio signal is associated with a spatial pickup pattern 130, the microphone array(s) 102 are located in a room 200; a processor 104 and memory 105 operably coupled to the microphone array 102, the processor 104 configured to execute the following steps: (a) selectively mute or unmute an individual talker the room with a mute function 106 that controls whether to mute or unmute the individual talker T1-T7 that is picked up by one or more of the individual audio signals, the mute function 106 includes speech learning that learns to identify different talkers in real time to allow the mute function 106 to identify transitions from one talker to another talker in the room 200, (b) output an audio signal 110 based on the selective muting of the talkers T1-T7 in the room 200” (abstract). Graham is being incorporated for operations related to muting specific signals/speakers in view of the multi-speaker model(s)/acoustic fingerprint(s) of Lindrup in view of Xie.
Applicant’s arguments with respect to claim(s) 1 (argument 2.3, see pgs. 12-13) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Specifically, Sivaraman has been removed from the combination of art used to reject the independent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lindrup et al. (US-20250046330-A1), hereinafter Lindrup, in view of Xie (US-20190341041-A1), further in view of Graham et al. (US-20250088795-A1), hereinafter Graham.
Regarding claim 1, Lindrup discloses: a method performed in an environment of a user ([0024] “audio input signal” will be generally be construed to mean an electrical signal representing a sound from the sound environment, [0057] the audio device system is configured to enable the user to select between different methods for selecting the first sound signal, [Allowing a user to select sound signals, wherein those are gathered from an environment, indicates the user is in the environment (required for them to be able to hear the audio)]), a plurality of people is located in the environment ([0073] it can be assumed that the user of the audio device system is engaged in a conversation with the person represented by said sound source signal, [A conversation requires a plurality of people in a same environment]), the user having a mobile device used for obtaining user input ([0106] the system may additionally comprise at least one of a second audio device and an external device such as a smart phone, [0107] The audio device system 200 comprises an acoustical-electrical input transducer block (typically comprising two microphones) 201 and an analogue-digital converter (not shown for reasons of clarity), which provides an input signal), the user having at least one hearable device used for providing audio output to the user ([0025] the term “audio output signal” will generally be construed to mean an electrical signal representing a sound to be output by an electrical-acoustical output transducer of an audio device of an audio device system, [0116] The methods and systems of the present invention according to the disclosed embodiments may generally be implemented in both hearing aids and in audio devices that are not hearing aids (i.e. they do not comprise means for compensating a hearing loss), but nevertheless comprise both acoustical-electrical input transducers and electro-acoustical output transducers), the method comprising:
capturing a first noisy audio signal from the environment ([Fig. 1, 101], [0031] a plurality of sound source signals each representing a sound source of the present sound environment are provided, [0037] According to an embodiment the mixed audio signal may also comprise non-speech signals, [Wherein input of a plurality of sound source signals will clearly result in a mixed signal at the receiver, wherein non-speech signals track to noise in the context of improving speech intelligibility of Lindrup (see abstract). Further, see [0010] which discloses mixed audio typically comprising multiple speakers]);
extracting from the first noisy audio signal a first separate audio signal that represents a first voice ([0049] an own voice signal representing the voice of the audio device system user is detected and in response hereto the own voice signal is selected as the first sound signal), and a second separate audio signal that represents a second voice ([Fig. 1, 103], [0058] In a third step 103 of the method according to the present embodiment the speech content of the first sound signal is compared with the speech content of the provided sound source signals, [Comparing a first sound signal, previously defined to represent a voice, to a second, i.e. sound source signal, wherein sound source signals can each represent individual speakers derived from a mixed signal (see [0010]), indicating at least a second separate audio signal that represent a second voice]),
combining the first and second separate audio signals to obtain a first enhanced audio signal ([0099] an audio output is provided based on said output signal, wherein the contribution to the audio output from the remaining sound source signals is suppressed compared to the contribution from the output signal, [Suppressing contributions from remaining sound source signals, i.e. second signals, compared to the contribution of an output signal, i.e. that most closely corresponding to the first sound signal (see comparison definition of Step 103, [0062]-[0064]; therefore, representative of the first sound signal), indicates a combination of the two separate audio signals to obtain enhanced audio]),
whereby the first enhanced audio signal comprises the first separate audio signal and the second separate audio signal ([0099] In a fifth step 105 of the method according to the present embodiment an audio output is provided based on said output signal, wherein the contribution to the audio output from the remaining sound source signals is suppressed compared to the contribution from the output signal, [An output signal based on a first sound signal and additional remaining sound sources indicates a first enhances audio containing at least first and second separate audio signals]);
outputting to the user, via the at least one hearable device ([0116] The methods and systems of the present invention according to the disclosed embodiments may generally be implemented in both hearing aids and in audio devices that are not hearing aids (i.e. they do not comprise means for compensating a hearing loss), but nevertheless comprise both acoustical-electrical input transducers and electro-acoustical output transducers), the first enhanced audio signal ([Fig. 1, 105], [Providing an output with sound suppression performed, i.e. enhanced, on remaining, i.e. not first, sound signals, wherein that output is defined to be sent to hearing aids which process output through transducers, indicating generated sound to the user]).
Lindrup does not disclose:
said extracting is performed by first and second speech separation models based on first and second respective acoustic fingerprints, wherein the first and second fingerprints correspond to the first and second voices, respectively;
obtaining an instruction to mute an entity associated with the first voice;
capturing a second noisy audio signal from the environment subsequently to said capturing the first noisy audio signal;
extracting from the second noisy audio signal a third separate audio signal that represents the second voice, said extracting is performed by the second speech separation model based on the second acoustic fingerprint, wherein execution of the first speech separation model is terminated based on the instruction and is not performed by said extracting;
generating a second enhanced audio signal based on the third separate audio signal, wherein the first voice is not extracted from the second noisy audio signal and is not incorporated into the second enhanced audio signal; and
outputting to the user, via the at least one hearable device, the second enhanced audio signal.
Xie discloses:
said extracting is performed by first and second speech separation models based on first and second respective acoustic fingerprints ([Fig. 3, First Registered Speaker Model 307 and Background Speaker Model 306 based on First Voice Signal 301 and Second Voice Signal 302], [0087] the electronic apparatus 1000 may allow the background speaker model 306 generated based on the voice signals of the plurality of speakers to be trained using the first voice data 303 in the same manner as in the case of the second voice data 304, [0052] The voice feature extracted by the electronic apparatus 1000 may be stored in a form of, for example, voice data and may be used for generation of a speaker model, registration of the speaker model, voice recognition of a speaker, [Voice data in the context of Xie tracks to acoustic fingerprints in view of them being used to generate/register speaker models]), wherein the first and second fingerprints correspond to the first and second voices, respectively ([As previously disclosed, generating first and second voice data corresponding to first and second voice signals indicates each data, i.e. fingerprint, to correspond to its respective signal]).
Lindrup and Xie are considered analogous art within speaker separation for target audio extraction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Lindrup to incorporate the teachings of Xie, because of the novel way to perform voice cleaning of first and second voice data before training speaker models on said voice data, improving the quality of training data used for speaker-specific models which will also improve the performance of the models (Xie, [0057]).
Lindrup in view of Xie does not disclose:
obtaining an instruction to mute an entity associated with the first voice.
Graham discloses:
obtaining an instruction to mute an entity associated with the first voice ([0067] the processor 104 is configured to selectively mute a sound source in the audio signals 108 as represented by a mute function 106. A sound source in the audio signals 108 includes any source that contributes to the audio signal 108. For example, a sound source may include a talker that is within a beam of the BMA 102).
Lindrup, Xie, and Graham are considered analogous art within sound source/speech separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Lindrup in view of Xie to incorporate the teachings of Graham, because of the novel way to provide improved flexibility to voice signal processing capabilities through the use of mute/pass functions using beams of a beamformer to selectively mute/unmute one or more specific talkers or sounds in a room, facilitating improved feature extraction. It would be obvious to operate the sound source signal separator of Lindrup using the beams of Graham as Lindrup discloses an embodiment which contains a plurality of beamformers ([0038]) (Graham, [0021], [0029]).
Xie further discloses:
capturing a second noisy audio signal from the environment subsequently to said capturing the first noisy audio signal ([Fig. 1, S202 “Receive second voice signal via second channel that is different than first channel”], [Wherein the second voice signal may be containing no required voice data “non-mute features”, i.e. a signal of noise, as disclosed in [0055]]);
extracting from the second noisy audio signal a third separate audio signal that represents the second voice ([Fig. 2, S203 “Obtain…second voice data”]), said extracting is performed by the second speech separation model based on the second acoustic fingerprint ([Considering the previously disclosed background speaker model which identifies speakers based on voice data, i.e. acoustic fingerprints, in view of the voiceprints of Lindrup, indicating the voice data extraction operation to be based on a second acoustic fingerprint (used to train the background speaker model) corresponding to any background speaker])
Graham further discloses:
wherein execution of the first speech separation model is terminated based on the instruction and is not performed by said extracting ([0105] After all interfering talker's voices have been muted by the mute functions 1061 to 106-N, the combination function 118 may select the from among the muted audio signals 109-1 to 109-N to output the muted audio signal 109 with the largest power. As this is performed after the muting functions 106 have been performed, the remaining audio in the muted audio signals 109-1 to 109-N may represent desired audio, [Determining to mute audio signals, wherein each signal would be separated using the respective models of Xie, indicates termination of said model through the muting operation, effectively not extracting the muted signal from the combination 118]);
generating a second enhanced audio signal based on the third separate audio signal ([Fig. 6, Combination 118 used for selectively muting], [0106] The mute function 106 is performed on the combined audio signal 122 to generate the modified audio signal 110. That is, the mute function 106 may be performed to selectively mute a sound source in the combined audio signal 122), wherein the first voice is not extracted from the second noisy audio signal and is not incorporated into the second enhanced audio signal ([As previously disclosed, choosing to mute sound sources indicates not extracting said source from the combination, i.e. muting, before generation of the modified output 110]); and
outputting to the user, via the at least one hearable device ([In view of the previously disclosed hearable device of Lindrup]), the second enhanced audio signal ([0070] The processor 104 is configured to output the modified audio signal 110 based on the selective muting of the sound source).
Regarding claim 2, Lindrup in view of Xie, further in view of Graham discloses: the method of claim 1.
Lindrup further discloses:
wherein each of the first and second speech separation models comprise at least one of: a generative model ([The examiner would like to note that due to the disjunctive nature of the claim, this element does not require a mapping]), a discriminative model ([Defining a “sound source separation” neural network indicates that it is discriminatory in nature, i.e. between sources, wherein this could be extended to the models of Xie without a change in functionality to Xie as a Gaussian Mixture Model (GMM) which forms the models of Xie ([0023]) necessarily discriminates between speakers]), or a beamforming based model ([The examiner would like to note that due to the disjunctive construction of the claim, not all elements require a mapping. Further, Lindrup defines using a beamformer for identifying sound source signals, indication a beamforming model for audio separation, see ([0048])]).
Regarding claim 3, Lindrup in view of Xie, further in view of Graham discloses: the method of claim 1.
Lindrup further discloses:
wherein the first and second acoustic fingerprints are retained in a database of pre-generated acoustic fingerprints ([0054] comparing the generated voice print to previously obtained voice prints that each are associated to a specific speaker [Comparing newly generated voiceprints to previously obtained voiceprints indicates a required retaining of the obtained voiceprints for obtaining/comparing. The obtained voiceprints are “pre-generated” as compared to the newly generated prints. Lindrup defines the system as being able to be performed using a smart phone (see [0006], [0106]) indicating a database, i.e. the phone storage, for storing/retrieving previously generated voiceprints]).
Regarding claim 4, Lindrup in view of Xie, further in view of Graham discloses: the method of claim 3.
Lindrup further discloses:
wherein the first and second acoustic fingerprints are pre- generated based on respective first and second audio records of corresponding entities ([0054] One implementation comprises generating a “voice print” of data derived from a given audio signal and comparing the generated voice print to previously obtained voice prints that each are associated to a specific speaker, whereby the speaker of said given audio signal may be identified as the person associated with said previously obtained voice print, [Comparing previously generated voiceprints to new voiceprints indicates the previously generated are “pre-generated” as compared to the newly generated voiceprints. Further, obtaining previously generated voiceprints which correspond to specific speakers indicates those speakers represent entities corresponding to the audio]), the first and second audio records comprising at least one of:
past vocal communications with the user ([The ability to have a previously obtained voiceprint compared to a new voiceprint indicates the obtained voiceprint is from a past vocal communication with the user, i.e. an enrollment]);
a designated enrollment audio ([The examiner would like to note that due to the disjunctive construction of the claim, these element do not require mappings]); and
a social media platform ([The examiner would like to note that due to the disjunctive construction of the claim, these element do not require mappings]).
Regarding claim 5, Lindrup in view of Xie, further in view of Graham discloses: the method of claim 1.
Graham further discloses:
wherein the mobile device is configured to enable the user to control activation and deactivation of entities associated with voices ([0159] a user could speak to their mobile device: “Alexa, mute the audio of user number 1”, or “Cortana, unmute Bob's audio.” In this embodiment, a display on the mobile app would show the list of users whose voices have been recognized by the system, but speech would be used to control muting and unmuting of users).
Regarding claim 6, Lindrup discloses: a computer program product comprising a non-transitory computer readable storage medium retaining program instructions ([0106] a second audio device and an external device such as a smart phone, [A smartphone will inherently have non-transitory storage media]), which program instructions when read by a processor ([A smartphone will inherently have a processor]), cause the processor to perform a method in an environment of a user ([0024] “audio input signal” will be generally be construed to mean an electrical signal representing a sound from the sound environment, [0057] the audio device system is configured to enable the user to select between different methods for selecting the first sound signal, [Allowing a user to select sound signals, wherein those are gathered from an environment, indicates the user is in the environment (required for them to be able to hear the audio)]), a plurality of people is located in the environment ([0073] it can be assumed that the user of the audio device system is engaged in a conversation with the person represented by said sound source signal, [A conversation requires a plurality of people in a same environment]), the user having a mobile device used for obtaining user input ([0106] the system may additionally comprise at least one of a second audio device and an external device such as a smart phone, [0107] The audio device system 200 comprises an acoustical-electrical input transducer block (typically comprising two microphones) 201 and an analogue-digital converter (not shown for reasons of clarity), which provides an input signal), the user having at least one hearable device used for providing audio output to the user ([0025] the term “audio output signal” will generally be construed to mean an electrical signal representing a sound to be output by an electrical-acoustical output transducer of an audio device of an audio device system, [0116] The methods and systems of the present invention according to the disclosed embodiments may generally be implemented in both hearing aids and in audio devices that are not hearing aids (i.e. they do not comprise means for compensating a hearing loss), but nevertheless comprise both acoustical-electrical input transducers and electro-acoustical output transducers), the method comprising:
capturing a first noisy audio signal from the environment ([Fig. 1, 101], [0031] a plurality of sound source signals each representing a sound source of the present sound environment are provided, [0037] According to an embodiment the mixed audio signal may also comprise non-speech signals, [Wherein input of a plurality of sound source signals will clearly result in a mixed signal at the receiver, wherein non-speech signals track to noise in the context of improving speech intelligibility of Lindrup (see abstract). Further, see [0010] which discloses mixed audio typically comprising multiple speakers]);
extracting from the first noisy audio signal a first separate audio signal that represents a first voice ([0049] an own voice signal representing the voice of the audio device system user is detected and in response hereto the own voice signal is selected as the first sound signal), and a second separate audio signal that represents a second voice ([Fig. 1, 103], [0058] In a third step 103 of the method according to the present embodiment the speech content of the first sound signal is compared with the speech content of the provided sound source signals, [Comparing a first sound signal, previously defined to represent a voice, to a second, i.e. sound source signal, wherein sound source signals can each represent individual speakers derived from a mixed signal (see [0010]), indicating at least a second separate audio signal that represent a second voice]),
combining the first and second separate audio signals to obtain a first enhanced audio signal ([0099] an audio output is provided based on said output signal, wherein the contribution to the audio output from the remaining sound source signals is suppressed compared to the contribution from the output signal, [Suppressing contributions from remaining sound source signals, i.e. second signals, compared to the contribution of an output signal, i.e. that most closely corresponding to the first sound signal (see comparison definition of Step 103, [0062]-[0064]; therefore, representative of the first sound signal), indicates a combination of the two separate audio signals to obtain enhanced audio]),
whereby the first enhanced audio signal comprises the first separate audio signal and the second separate audio signal ([0099] In a fifth step 105 of the method according to the present embodiment an audio output is provided based on said output signal, wherein the contribution to the audio output from the remaining sound source signals is suppressed compared to the contribution from the output signal, [An output signal based on a first sound signal and additional remaining sound sources indicates a first enhances audio containing at least first and second separate audio signals]);
outputting to the user, via the at least one hearable device ([0116] The methods and systems of the present invention according to the disclosed embodiments may generally be implemented in both hearing aids and in audio devices that are not hearing aids (i.e. they do not comprise means for compensating a hearing loss), but nevertheless comprise both acoustical-electrical input transducers and electro-acoustical output transducers), the first enhanced audio signal ([Fig. 1, 105], [Providing an output with sound suppression performed, i.e. enhanced, on remaining, i.e. not first, sound signals, wherein that output is defined to be sent to hearing aids which process output through transducers, indicating generated sound to the user]).
Lindrup does not disclose:
said extracting is performed by first and second speech separation models based on first and second respective acoustic fingerprints, wherein the first and second fingerprints correspond to the first and second voices, respectively;
obtaining an instruction to mute an entity associated with the first voice;
capturing a second noisy audio signal from the environment subsequently to said capturing the first noisy audio signal;
extracting from the second noisy audio signal a third separate audio signal that represents the second voice, said extracting is performed by the second speech separation model based on the second acoustic fingerprint, wherein execution of the first speech separation model is terminated based on the instruction and is not performed by said extracting;
generating a second enhanced audio signal based on the third separate audio signal, wherein the first voice is not extracted from the second noisy audio signal and is not incorporated into the second enhanced audio signal; and
outputting to the user, via the at least one hearable device, the second enhanced audio signal.
Xie discloses:
said extracting is performed by first and second speech separation models based on first and second respective acoustic fingerprints ([Fig. 3, First Registered Speaker Model 307 and Background Speaker Model 306 based on First Voice Signal 301 and Second Voice Signal 302], [0087] the electronic apparatus 1000 may allow the background speaker model 306 generated based on the voice signals of the plurality of speakers to be trained using the first voice data 303 in the same manner as in the case of the second voice data 304, [0052] The voice feature extracted by the electronic apparatus 1000 may be stored in a form of, for example, voice data and may be used for generation of a speaker model, registration of the speaker model, voice recognition of a speaker, [Voice data in the context of Xie tracks to acoustic fingerprints in view of them being used to generate/register speaker models]), wherein the first and second fingerprints correspond to the first and second voices, respectively ([As previously disclosed, generating first and second voice data corresponding to first and second voice signals indicates each data, i.e. fingerprint, to correspond to its respective signal]).
Lindrup and Xie are considered analogous art within speaker separation for target audio extraction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Lindrup to incorporate the teachings of Xie, because of the novel way to perform voice cleaning of first and second voice data before training speaker models on said voice data, improving the quality of training data used for speaker-specific models which will also improve the performance of the models (Xie, [0057]).
Lindrup in view of Xie does not disclose:
obtaining an instruction to mute an entity associated with the first voice.
Graham discloses:
obtaining an instruction to mute an entity associated with the first voice ([0067] the processor 104 is configured to selectively mute a sound source in the audio signals 108 as represented by a mute function 106. A sound source in the audio signals 108 includes any source that contributes to the audio signal 108. For example, a sound source may include a talker that is within a beam of the BMA 102).
Lindrup, Xie, and Graham are considered analogous art within sound source/speech separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Lindrup in view of Xie to incorporate the teachings of Graham, because of the novel way to provide improved flexibility to voice signal processing capabilities through the use of mute/pass functions using beams of a beamformer to selectively mute/unmute one or more specific talkers or sounds in a room, facilitating improved feature extraction. It would be obvious to operate the sound source signal separator of Lindrup using the beams of Graham as Lindrup discloses an embodiment which contains a plurality of beamformers ([0038]) (Graham, [0021], [0029]).
Xie further discloses:
capturing a second noisy audio signal from the environment subsequently to said capturing the first noisy audio signal ([Fig. 1, S202 “Receive second voice signal via second channel that is different than first channel”], [Wherein the second voice signal may be containing no required voice data “non-mute features”, i.e. a signal of noise, as disclosed in [0055]]);
extracting from the second noisy audio signal a third separate audio signal that represents the second voice ([Fig. 2, S203 “Obtain…second voice data”]), said extracting is performed by the second speech separation model based on the second acoustic fingerprint ([Considering the previously disclosed background speaker model which identifies speakers based on voice data, i.e. acoustic fingerprints, in view of the voiceprints of Lindrup, indicating the voice data extraction operation to be based on a second acoustic fingerprint (used to train the background speaker model) corresponding to any background speaker])
Graham further discloses:
wherein execution of the first speech separation model is terminated based on the instruction and is not performed by said extracting ([0105] After all interfering talker's voices have been muted by the mute functions 1061 to 106-N, the combination function 118 may select the from among the muted audio signals 109-1 to 109-N to output the muted audio signal 109 with the largest power. As this is performed after the muting functions 106 have been performed, the remaining audio in the muted audio signals 109-1 to 109-N may represent desired audio, [Determining to mute audio signals, wherein each signal would be separated using the respective models of Xie, indicates termination of said model through the muting operation, effectively not extracting the muted signal from the combination 118]);
generating a second enhanced audio signal based on the third separate audio signal ([Fig. 6, Combination 118 used for selectively muting], [0106] The mute function 106 is performed on the combined audio signal 122 to generate the modified audio signal 110. That is, the mute function 106 may be performed to selectively mute a sound source in the combined audio signal 122), wherein the first voice is not extracted from the second noisy audio signal and is not incorporated into the second enhanced audio signal ([As previously disclosed, choosing to mute sound sources indicates not extracting said source from the combination, i.e. muting, before generation of the modified output 110]); and
outputting to the user, via the at least one hearable device ([In view of the previously disclosed hearable device of Lindrup]), the second enhanced audio signal ([0070] The processor 104 is configured to output the modified audio signal 110 based on the selective muting of the sound source).
Regarding claim 7, Lindrup in view of Xie, further in view of Graham discloses: the computer program product of claim 6.
Lindrup further discloses:
wherein each of the first and second speech separation models comprise at least one of: a generative model ([The examiner would like to note that due to the disjunctive nature of the claim, this element does not require a mapping]), a discriminative model ([Defining a “sound source separation” neural network indicates that it is discriminatory in nature, i.e. between sources, wherein this could be extended to the models of Xie without a change in functionality to Xie as a Gaussian Mixture Model (GMM) which forms the models of Xie ([0023]) necessarily discriminates between speakers]), or a beamforming based model ([The examiner would like to note that due to the disjunctive construction of the claim, not all elements require a mapping. Further, Lindrup defines using a beamformer for identifying sound source signals, indication a beamforming model for audio separation, see ([0048])]).
Regarding claim 8, Lindrup in view of Xie, further in view of Graham discloses: the computer program product of claim 6.
Lindrup further discloses:
wherein the first and second acoustic fingerprints are retained in a database of pre-generated acoustic fingerprints ([0054] comparing the generated voice print to previously obtained voice prints that each are associated to a specific speaker [Comparing newly generated voiceprints to previously obtained voiceprints indicates a required retaining of the obtained voiceprints for obtaining/comparing. The obtained voiceprints are “pre-generated” as compared to the newly generated prints. Lindrup defines the system as being able to be performed using a smart phone (see [0006], [0106]) indicating a database, i.e. the phone storage, for storing/retrieving previously generated voiceprints]).
Regarding claim 9, Lindrup in view of Xie, further in view of Graham discloses: the computer program product of claim 8.
Lindrup further discloses:
wherein the first and second acoustic fingerprints are pre- generated based on respective first and second audio records of corresponding entities ([0054] One implementation comprises generating a “voice print” of data derived from a given audio signal and comparing the generated voice print to previously obtained voice prints that each are associated to a specific speaker, whereby the speaker of said given audio signal may be identified as the person associated with said previously obtained voice print, [Comparing previously generated voiceprints to new voiceprints indicates the previously generated are “pre-generated” as compared to the newly generated voiceprints. Further, obtaining previously generated voiceprints which correspond to specific speakers indicates those speakers represent entities corresponding to the audio]), the first and second audio records comprising at least one of:
past vocal communications with the user ([The ability to have a previously obtained voiceprint compared to a new voiceprint indicates the obtained voiceprint is from a past vocal communication with the user, i.e. an enrollment]);
a designated enrollment audio ([The examiner would like to note that due to the disjunctive construction of the claim, these element do not require mappings]); and
a social media platform ([The examiner would like to note that due to the disjunctive construction of the claim, these element do not require mappings]).
Regarding claim 10, Lindrup in view of Xie, further in view of Graham discloses: the computer program product of claim 6.
Graham further discloses:
wherein the mobile device is configured to enable the user to control activation and deactivation of entities associated with voices ([0159] a user could speak to their mobile device: “Alexa, mute the audio of user number 1”, or “Cortana, unmute Bob's audio.” In this embodiment, a display on the mobile app would show the list of users whose voices have been recognized by the system, but speech would be used to control muting and unmuting of users).
Regarding claim 11, Lindrup discloses: an apparatus comprising a processor and coupled memory ([0106] a second audio device and an external device such as a smart phone, [Smartphones inherently have a processor coupled to memory]), the processor being adapted to perform in an environment of a user ([0024] “audio input signal” will be generally be construed to mean an electrical signal representing a sound from the sound environment, [0057] the audio device system is configured to enable the user to select between different methods for selecting the first sound signal, [Allowing a user to select sound signals, wherein those are gathered from an environment, indicates the user is in the environment (required for them to be able to hear the audio)]), a plurality of people is located in the environment ([0073] it can be assumed that the user of the audio device system is engaged in a conversation with the person represented by said sound source signal, [A conversation requires a plurality of people in a same environment]), the user having a mobile device used for obtaining user input ([0106] the system may additionally comprise at least one of a second audio device and an external device such as a smart phone, [0107] The audio device system 200 comprises an acoustical-electrical input transducer block (typically comprising two microphones) 201 and an analogue-digital converter (not shown for reasons of clarity), which provides an input signal), the user having at least one hearable device used for providing audio output to the user ([0025] the term “audio output signal” will generally be construed to mean an electrical signal representing a sound to be output by an electrical-acoustical output transducer of an audio device of an audio device system, [0116] The methods and systems of the present invention according to the disclosed embodiments may generally be implemented in both hearing aids and in audio devices that are not hearing aids (i.e. they do not comprise means for compensating a hearing loss), but nevertheless comprise both acoustical-electrical input transducers and electro-acoustical output transducers), the method comprising:
capturing a first noisy audio signal from the environment ([Fig. 1, 101], [0031] a plurality of sound source signals each representing a sound source of the present sound environment are provided, [0037] According to an embodiment the mixed audio signal may also comprise non-speech signals, [Wherein input of a plurality of sound source signals will clearly result in a mixed signal at the receiver, wherein non-speech signals track to noise in the context of improving speech intelligibility of Lindrup (see abstract). Further, see [0010] which discloses mixed audio typically comprising multiple speakers]);
extracting from the first noisy audio signal a first separate audio signal that represents a first voice ([0049] an own voice signal representing the voice of the audio device system user is detected and in response hereto the own voice signal is selected as the first sound signal), and a second separate audio signal that represents a second voice ([Fig. 1, 103], [0058] In a third step 103 of the method according to the present embodiment the speech content of the first sound signal is compared with the speech content of the provided sound source signals, [Comparing a first sound signal, previously defined to represent a voice, to a second, i.e. sound source signal, wherein sound source signals can each represent individual speakers derived from a mixed signal (see [0010]), indicating at least a second separate audio signal that represent a second voice]),
combining the first and second separate audio signals to obtain a first enhanced audio signal ([0099] an audio output is provided based on said output signal, wherein the contribution to the audio output from the remaining sound source signals is suppressed compared to the contribution from the output signal, [Suppressing contributions from remaining sound source signals, i.e. second signals, compared to the contribution of an output signal, i.e. that most closely corresponding to the first sound signal (see comparison definition of Step 103, [0062]-[0064]; therefore, representative of the first sound signal), indicates a combination of the two separate audio signals to obtain enhanced audio]),
whereby the first enhanced audio signal comprises the first separate audio signal and the second separate audio signal ([0099] In a fifth step 105 of the method according to the present embodiment an audio output is provided based on said output signal, wherein the contribution to the audio output from the remaining sound source signals is suppressed compared to the contribution from the output signal, [An output signal based on a first sound signal and additional remaining sound sources indicates a first enhances audio containing at least first and second separate audio signals]);
outputting to the user, via the at least one hearable device ([0116] The methods and systems of the present invention according to the disclosed embodiments may generally be implemented in both hearing aids and in audio devices that are not hearing aids (i.e. they do not comprise means for compensating a hearing loss), but nevertheless comprise both acoustical-electrical input transducers and electro-acoustical output transducers), the first enhanced audio signal ([Fig. 1, 105], [Providing an output with sound suppression performed, i.e. enhanced, on remaining, i.e. not first, sound signals, wherein that output is defined to be sent to hearing aids which process output through transducers, indicating generated sound to the user]).
Lindrup does not disclose:
said extracting is performed by first and second speech separation models based on first and second respective acoustic fingerprints, wherein the first and second fingerprints correspond to the first and second voices, respectively;
obtaining an instruction to mute an entity associated with the first voice;
capturing a second noisy audio signal from the environment subsequently to said capturing the first noisy audio signal;
extracting from the second noisy audio signal a third separate audio signal that represents the second voice, said extracting is performed by the second speech separation model based on the second acoustic fingerprint, wherein execution of the first speech separation model is terminated based on the instruction and is not performed by said extracting;
generating a second enhanced audio signal based on the third separate audio signal, wherein the first voice is not extracted from the second noisy audio signal and is not incorporated into the second enhanced audio signal; and
outputting to the user, via the at least one hearable device, the second enhanced audio signal.
Xie discloses:
said extracting is performed by first and second speech separation models based on first and second respective acoustic fingerprints ([Fig. 3, First Registered Speaker Model 307 and Background Speaker Model 306 based on First Voice Signal 301 and Second Voice Signal 302], [0087] the electronic apparatus 1000 may allow the background speaker model 306 generated based on the voice signals of the plurality of speakers to be trained using the first voice data 303 in the same manner as in the case of the second voice data 304, [0052] The voice feature extracted by the electronic apparatus 1000 may be stored in a form of, for example, voice data and may be used for generation of a speaker model, registration of the speaker model, voice recognition of a speaker, [Voice data in the context of Xie tracks to acoustic fingerprints in view of them being used to generate/register speaker models]), wherein the first and second fingerprints correspond to the first and second voices, respectively ([As previously disclosed, generating first and second voice data corresponding to first and second voice signals indicates each data, i.e. fingerprint, to correspond to its respective signal]).
Lindrup and Xie are considered analogous art within speaker separation for target audio extraction. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Lindrup to incorporate the teachings of Xie, because of the novel way to perform voice cleaning of first and second voice data before training speaker models on said voice data, improving the quality of training data used for speaker-specific models which will also improve the performance of the models (Xie, [0057]).
Lindrup in view of Xie does not disclose:
obtaining an instruction to mute an entity associated with the first voice.
Graham discloses:
obtaining an instruction to mute an entity associated with the first voice ([0067] the processor 104 is configured to selectively mute a sound source in the audio signals 108 as represented by a mute function 106. A sound source in the audio signals 108 includes any source that contributes to the audio signal 108. For example, a sound source may include a talker that is within a beam of the BMA 102).
Lindrup, Xie, and Graham are considered analogous art within sound source/speech separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Lindrup in view of Xie to incorporate the teachings of Graham, because of the novel way to provide improved flexibility to voice signal processing capabilities through the use of mute/pass functions using beams of a beamformer to selectively mute/unmute one or more specific talkers or sounds in a room, facilitating improved feature extraction. It would be obvious to operate the sound source signal separator of Lindrup using the beams of Graham as Lindrup discloses an embodiment which contains a plurality of beamformers ([0038]) (Graham, [0021], [0029]).
Xie further discloses:
capturing a second noisy audio signal from the environment subsequently to said capturing the first noisy audio signal ([Fig. 1, S202 “Receive second voice signal via second channel that is different than first channel”], [Wherein the second voice signal may be containing no required voice data “non-mute features”, i.e. a signal of noise, as disclosed in [0055]]);
extracting from the second noisy audio signal a third separate audio signal that represents the second voice ([Fig. 2, S203 “Obtain…second voice data”]), said extracting is performed by the second speech separation model based on the second acoustic fingerprint ([Considering the previously disclosed background speaker model which identifies speakers based on voice data, i.e. acoustic fingerprints, in view of the voiceprints of Lindrup, indicating the voice data extraction operation to be based on a second acoustic fingerprint (used to train the background speaker model) corresponding to any background speaker])
Graham further discloses:
wherein execution of the first speech separation model is terminated based on the instruction and is not performed by said extracting ([0105] After all interfering talker's voices have been muted by the mute functions 1061 to 106-N, the combination function 118 may select the from among the muted audio signals 109-1 to 109-N to output the muted audio signal 109 with the largest power. As this is performed after the muting functions 106 have been performed, the remaining audio in the muted audio signals 109-1 to 109-N may represent desired audio, [Determining to mute audio signals, wherein each signal would be separated using the respective models of Xie, indicates termination of said model through the muting operation, effectively not extracting the muted signal from the combination 118]);
generating a second enhanced audio signal based on the third separate audio signal ([Fig. 6, Combination 118 used for selectively muting], [0106] The mute function 106 is performed on the combined audio signal 122 to generate the modified audio signal 110. That is, the mute function 106 may be performed to selectively mute a sound source in the combined audio signal 122), wherein the first voice is not extracted from the second noisy audio signal and is not incorporated into the second enhanced audio signal ([As previously disclosed, choosing to mute sound sources indicates not extracting said source from the combination, i.e. muting, before generation of the modified output 110]); and
outputting to the user, via the at least one hearable device ([In view of the previously disclosed hearable device of Lindrup]), the second enhanced audio signal ([0070] The processor 104 is configured to output the modified audio signal 110 based on the selective muting of the sound source).
Regarding claim 12, Lindrup in view of Xie, further in view of Graham discloses: the apparatus of claim 11.
Lindrup further discloses:
wherein each of the first and second speech separation models comprise at least one of: a generative model ([The examiner would like to note that due to the disjunctive nature of the claim, this element does not require a mapping]), a discriminative model ([Defining a “sound source separation” neural network indicates that it is discriminatory in nature, i.e. between sources, wherein this could be extended to the models of Xie without a change in functionality to Xie as a Gaussian Mixture Model (GMM) which forms the models of Xie ([0023]) necessarily discriminates between speakers]), or a beamforming based model ([The examiner would like to note that due to the disjunctive construction of the claim, not all elements require a mapping. Further, Lindrup defines using a beamformer for identifying sound source signals, indication a beamforming model for audio separation, see ([0048])]).
Regarding claim 13, Lindrup in view of Xie, further in view of Graham discloses: the apparatus of claim 11.
Lindrup further discloses:
wherein the first and second acoustic fingerprints are retained in a database of pre-generated acoustic fingerprints ([0054] comparing the generated voice print to previously obtained voice prints that each are associated to a specific speaker [Comparing newly generated voiceprints to previously obtained voiceprints indicates a required retaining of the obtained voiceprints for obtaining/comparing. The obtained voiceprints are “pre-generated” as compared to the newly generated prints. Lindrup defines the system as being able to be performed using a smart phone (see [0006], [0106]) indicating a database, i.e. the phone storage, for storing/retrieving previously generated voiceprints]).
Regarding claim 14, Lindrup in view of Xie, further in view of Graham discloses: the apparatus of claim 13.
Lindrup further discloses:
wherein the first and second acoustic fingerprints are pre- generated based on respective first and second audio records of corresponding entities ([0054] One implementation comprises generating a “voice print” of data derived from a given audio signal and comparing the generated voice print to previously obtained voice prints that each are associated to a specific speaker, whereby the speaker of said given audio signal may be identified as the person associated with said previously obtained voice print, [Comparing previously generated voiceprints to new voiceprints indicates the previously generated are “pre-generated” as compared to the newly generated voiceprints. Further, obtaining previously generated voiceprints which correspond to specific speakers indicates those speakers represent entities corresponding to the audio]), the first and second audio records comprising at least one of:
past vocal communications with the user ([The ability to have a previously obtained voiceprint compared to a new voiceprint indicates the obtained voiceprint is from a past vocal communication with the user, i.e. an enrollment]);
a designated enrollment audio ([The examiner would like to note that due to the disjunctive construction of the claim, these element do not require mappings]); and
a social media platform ([The examiner would like to note that due to the disjunctive construction of the claim, these element do not require mappings]).
Regarding claim 15, Lindrup in view of Xie, further in view of Graham discloses: the apparatus of claim 11.
Graham further discloses:
wherein the mobile device is configured to enable the user to control activation and deactivation of entities associated with voices ([0159] a user could speak to their mobile device: “Alexa, mute the audio of user number 1”, or “Cortana, unmute Bob's audio.” In this embodiment, a display on the mobile app would show the list of users whose voices have been recognized by the system, but speech would be used to control muting and unmuting of users).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gu et al. (“Multi-modal Multi-channel Target Speech Separation”) discloses “a general multi-modal framework for target speech separation by utilizing all the available information of the target speaker, including his/her spatial location, voice characteristics and lip movements. Also, under this framework, we investigate on the fusion methods for multi-modal joint modeling. A factorized attention-based fusion method is proposed to aggregate the high-level semantic information of multi-modalities at embedding level. This method firstly factorizes the mixture audio into a set of acoustic subspaces, then leverages the target’s information from other modalities to enhance these subspace acoustic embeddings with a learnable attention scheme.” (abstract). See entire document.
Marti et al. (US-11284183-B2) discloses “a computer-implemented method comprising receiving an input signal representing an ambient auditory environment of a user, generating, from the input signal, a set of ambient audio signals that includes a first component signal and a second component signal, generating, based on the first component signal, a first inverse signal that is a polar inverse of the first component signal, removing the first component signal from the set of ambient audio signals, generating a first composite signal that includes at least the first inverse signal and the second component signal, and driving an audio output device to produce soundwaves based on the first composite signal.” (abstract). See entire document.
Yue et al. (US-20160086609-A1) discloses “a method, an electronic system and a non-transitory computer readable storage medium for recognizing audio commands in an electronic device. The electronic device obtains audio data based on an audio signal provided by a user and extracts characteristic audio fingerprint features from the audio data. The electronic device further determines whether the corresponding audio signal is generated by an authorized user by comparing the characteristic audio fingerprint features with an audio fingerprint model for the authorized user and with a universal background model that represents user-independent audio fingerprint features, respectively. When the corresponding audio signal is generated by the authorized user of the electronic device, an audio command is extracted from the audio data, and an operation is performed according to the audio command” (abstract). See entire document.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights