DETAILED ACTION
The Response filed on 02/05/2026 has been correspondingly accepted and considered in the office action. Claims 1-7, 9-15 and 17-22 are pending. Claims 1, 18 and 22 are independent and amended. Dependent Claims 2-7, 9, 10, 12-14 and 17 are also amended. Claims 8 and 16 are cancelled. Claims 19-22 are new.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Claims 1-7 and 9-17 stand rejected under 35 U.S.C. § 103. Applicant’s arguments with respect to Claims 1-7 and 9-17 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
In order to expedite prosecution, and as to the material from the Specifications that are not in the Claim and are argued by the Applicant, please note Xu et al., ("Spex: Multi-scale time domain speaker extraction network." IEEE/ACM transactions on audio, speech, and language processing 28 (2020): 1370-1384, hereinafter, Xu) and Li et al., ("Dual-path modeling for long recording speech separation in meetings." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, hereinafter, Li).
For at least the supra provided reasons, Applicant's arguments have been fully considered but they are not persuasive.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-14 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Li.
Regarding Claim 1,
Xu discloses an audio signal processing method (Xu, Title, Abstract, "…speaker extraction network (SpEx) that converts the mixture speech into multi-scale embedding coefficients..."), the method comprising:
acquiring, by using a voice registration module based on a first audio signal (Figs.2 and 3, Reference Speech), a first hidden state corresponding to the voice registration module, wherein the first hidden state represents a feature of audio in the first audio signal from a registration sound source (Figs.2 and 3, A. SpEx Architecture, 1) Speaker Encoder , "…The model that converts speech samples x(t) into feature representation is called speaker encoder g(·), and the resulting feature representation is called speaker g(x) embedding...");
initializing, based on the first hidden state corresponding to the voice registration module, a second hidden state of a voice extraction module, wherein the second hidden state represents the feature of audio in a second audio signal from the registration sound source (Fig.3, 1) Speaker Encoder, blstm, "…we propose a multi-task learning algorithm to incorporate the speaker encoder as part of the SpEx network..."3) Speaker Extractor , "…The speaker extractor, as shown in Fig. 3, is conditioned on the speaker embedding both during training and at run-time inference to estimate a filter mask...");
extracting, based on the second hidden state of the voice extraction module, a target audio signal from the second audio signal (Fig.3, 3) Speaker Extractor, "…The selective filter can be modelled by a mask that has been well studied in speech separation literature, such as ideal binary mask (IBM) [66], ideal ratio mask (IRM)...We obtain the modulated response Si [7] for each scale i = 1, 2, 3 of the target speaker by applying the receptive mask Mi on the embedding coefficient Ei of the mixture signal in each scale..."; Fig.3, 4) Speech Decoder, "…The decoder reconstructs the time domain speech signal from the modulated responses..." ).
Xu discloses the speaker extractor conditioned on the speaker embedding and the stacked temporal convolutional network (TCN) to capture long-range dependency of the speech signal. But Xu does not explicitly discloses the limitation, "the second hidden state of the voice extraction module is based on the first hidden state corresponding to the voice registration module and a historical second hidden state of the voice extraction module."
However, Li, in the analogous field of endeavor, discloses the dual-path modeling in the (continuous speech separation) CSS framework using RNN, transformer, and bidirectional long-short memory (BLSTM) (Li, Introduction, Fig.1).
Li discloses successively updating the second hidden state of the voice extraction module, wherein each update of the second hidden state of the voice extraction module is based on the first hidden state corresponding to the voice registration module and a historical second hidden state of the voice extraction module (Li, Fig.1, 2. CSS: TASK DEFINITION AND BASELINE, "…The mask based BLSTM (i.e., updating historical second hidden states (from continuous mixture input) using BLSTM) separation network is used as the baseline in this work, with phase sensitive mask [31] as network output...").
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified the speaker extraction network taught by Xu, which incorporates a speaker encoder to represent the target speaker with RNN-based or transformer-based dual path system of Li for continuous speech separation with a reasonable expectation of success to access information across windows that are far apart in time, while maintaining the separation performance for each local window, thus making them a promising choice for long sequence modeling (Li,1. Introduction).
Regarding Claim 2,
Xu in view of Li discloses the method according to claim 1,
The method according to wherein the voice registration module comprises a first encoding module and a hidden state analysis module, and wherein the acquiring of the first hidden state corresponding to the voice registration module comprises: extracting, by using the first encoding module, a first audio feature of the first audio signal; and performing, by using the hidden state analysis module based on the first audio feature, feature extraction to acquire the first hidden state of the hidden state analysis module during feature extraction (Xu, Figs.2 and 3, A. SpEx Architecture, 1) Speaker Encoder , "…The model that converts speech samples x(t) into feature representation is called speaker encoder g(·), and the resulting feature representation is called speaker g(x) embedding..."; Li, Fig.1: A):The segmentation stage splits the long recording into short windows with window size K and hop length P, "…Then the segmented entire meeting can be presented as a three-D tensor T...a feature extraction module is applied to form the feature for separation step...").
Regarding Claim 3,
Xu in view of Li discloses the method according to claim 2, wherein the performing, by using the hidden state analysis module based on the first audio feature, of the feature extraction to acquire the first hidden state of the hidden state analysis module during feature extraction comprises (Xu, Fig.3, 1) Speaker Encoder):
for each frame in the first audio signal, successively performing the following based on an order of each frame: performing, by using the hidden state analysis module based on a preceding first hidden state corresponding to a frame preceding a current frame and the first audio feature of the current frame, feature extraction to acquire the first hidden state of the hidden state analysis module at the current frame at a time of feature extraction; and updating the first hidden state of the hidden state analysis module based on the acquired first hidden state (Xu, Fig.3, 1) Speaker Encoder, blstm , "…The model that converts speech samples x(t) into feature representation is called speaker encoder g(·), and the resulting feature representation is called speaker g(x) embedding...").
Regarding Claim 4,
Xu in view of Li discloses the method according to claim 2, wherein the performing, based on the first audio feature of the feature extraction to acquire the first hidden state of the hidden state analysis module during feature extraction comprises: for each frame in the first audio signal, successively performing the following based on an inverse order of each frame: performing, by using the hidden state analysis module based on a subsequent first hidden state corresponding to a frame subsequent to a current frame and the first audio feature of the current frame, feature extraction to acquire the first hidden state of the hidden state analysis module at the current frame at a time of feature extraction; and updating the first hidden state of the hidden state analysis module based on the acquired first hidden state (Xu, Fig.3, 1) Speaker Encoder, blstm , "…The model that converts speech samples x(t) into feature representation is called speaker encoder g(·), and the resulting feature representation is called speaker g(x) embedding...").
Regarding Claim 5,
Xu in view of Li discloses the method according to claim 2, wherein the extracting, by using the first encoding module, of the first audio feature of the first audio signal comprises:
It is well known to a person of ordinary skill in the art that a common approach is to perform speaker extraction in frequency-domain, and reconstruct the time-domain signal from the extracted magnitude and estimated phase spectra (Xu, I. Introduction).
Therefore, it is construed that the speaker encoder would be easily realized in frequency-domain after performing time-frequency transform (Xu, II. TIME-DOMAIN SPEAKER EXTRACTION NETWORK, "…In a frequency-domain implementation, a STFT module serves as the speech encoder that transforms time-domain speech signal into spectrum, with magnitude and phase, while an inverse STFT serves as the speech decoder...").
Li discloses performing a time-frequency transform process on the first audio signal to obtain sub-band features corresponding to at least two preset frequency bands; and extracting, by using the first encoding module respectively corresponding to each preset frequency band based on the sub-band features of the preset frequency band, the first audio feature corresponding to the preset frequency band (Li, Fig.1: A):The segmentation stage splits the long recording into short windows with window size K and hop length P, "…Denote
W
∈
R
L
×
F
as the magnitude spectrum of the single channel continuous mixture input , where F is the number of frequency bins and L is the number of frames...Then the segmented entire meeting can be
presented as a three-D tensor
T
=
[
D
1
;
∙
∙
∙
;
D
B
]
∈
R
B
×
K
×
F
, on top of which, a feature extraction module is applied to form the feature for separation step, which has the shape
T
∈
R
B
×
K
×
N
with N referring to feature dimension...."; 4.2. Model Configurations and Training Details, "…In the feature extraction, the size of short-time Fourier transformation(STFT) is 512-point and the hop length is 256…").
Regarding Claim 6,
Xu in view of Li discloses the method according to claim 2, wherein the hidden state analysis module comprises at least one of: a recurrent neural network, an attention network, a transformer network, or a convolutional network (Xu, Fig.3, "...we use a bidirectional long-short term memory (BLSTM) to encode the context information of the reference speech into a speaker embedding..."; Li, 1. Introduction, "…We compare two kinds of the most popular models for the dual-path modeling, the RNN and transformer...we compare the dual-path bidirectional long-short memory (DP-BLSTM) with the baseline BSLTM on different window sizes..."; 4.2. Model Configurations and Training Details, "...The transformer baseline contains 10 transformer encode layers; the attention dimension is 256, and 4-head multi-head attention is used...").
Regarding Claim 7,
Xu in view of Li discloses the method according to claim 1, wherein the extracting, based on the second hidden state of the voice extraction module, of the target audio signal from the second audio signal comprises:
extracting, by using a second encoding module, a second audio feature of the second audio signal (Xu, Fig.3, Speech Encoder, "…The input mixture speech y(t) can be encoded to embedding coefficients using a convolutional neural network in a similar way as other end-to-end speech processing systems...this paper proposes to encode the mixture speech into multi-scale speech embeddings...");
extracting, by using the voice extraction module based on the second hidden state and the second audio feature, mask information corresponding to the target audio signal from the second audio signal (Xu, Fig.3, 3) Speaker Extractor, "…The selective filter can be modelled by a mask that has been well studied in speech separation literature, such as ideal binary mask (IBM) [66], ideal ratio mask (IRM)...We obtain the modulated response Si [7] for each scale i = 1, 2, 3 of the target speaker by applying the receptive mask Mi on the embedding coefficient Ei of the mixture signal in each scale...");and
determining, by using a decoding module based on the second audio feature of the second audio signal and the mask information, the target audio signal (Xu, Fig.3, 4) Speech Decoder, "…The decoder reconstructs the time domain speech signal from the modulated responses..." ).
Regarding Claim 9,
Xu in view of Li discloses the method according to claim 7, the method according to wherein the second audio signal comprises at least one block, and each of the at least one block comprises at least one frame (Li, Fig.1: A):The segmentation stage splits the long recording into short windows with window size K and hop length P, "…Denote
W
∈
R
L
×
F
as the magnitude spectrum of the single channel continuous mixture input , where F is the number of frequency bins and L is the number of frames...Then the segmented entire meeting can be presented as a three-D tensor
T
=
[
D
1
;
∙
∙
∙
;
D
B
]
∈
R
B
×
K
×
F
, on top of which, a feature extraction module is applied to form the feature for separation step, which has the shape
T
∈
R
B
×
K
×
N
with N referring to feature dimension...."); and
wherein the successively updating of the second hidden state of the voice extraction module comprises: predicting, based on the first hidden state corresponding to the voice registration module (Xu, Fig.3, 1) Speaker Encoder, blstm, "…we propose a multi-task learning algorithm to incorporate the speaker encoder as part of the SpEx network..."3) Speaker Extractor , "…The speaker extractor, as shown in Fig. 3, is conditioned on the speaker embedding both during training and at run-time inference to estimate a filter mask...") and
the historical second hidden state of the voice extraction module, the second hidden state of the voice extraction module when processing a current block; and updating the second hidden state of the voice extraction module based on the predicted second hidden state (Li, Fig.1, 2. CSS: TASK DEFINITION AND BASELINE, "…The mask based BLSTM (i.e., updating historical second hidden states (from continuous mixture input) using BLSTM) separation network is used as the baseline in this work, with phase sensitive mask [31] as network output...").
Regarding Claim 10,
Xu in view of Li discloses the method according to claim 9, wherein the predicting, based on the first hidden state corresponding to the voice registration module and the historical second hidden state of the voice extraction module, the second hidden state of the voice extraction module when processing the current block comprises:
predicting, by using a window attention module based on the first hidden state corresponding to the voice registration module and the at least one preceding second hidden state of the voice extraction module, the second hidden state of the voice extraction module when processing the current block (Xu, Fig.2,"The speaker encoder emulates the top-down voluntary focus of cognitive process with the target speaker as the attention task"; "…x(t) is the reference speech of the target speaker to form an attention..."; Li, 4.2. Model Configurations and Training Details, "...The transformer baseline contains 10 transformer encode layers; the attention dimension is 256, and 4-head multi-head attention is used..."; 4.5. Comparison Window Lengths and Online Processing, "…The bidirectional modeling (BLSTM or self-attention) is used for the cross-window information interaction..." ).
Regarding Claim 11,
Xu in view of Li discloses the method according to claim 9, wherein the historical second hidden state of the voice extraction module comprises: the second hidden state of the voice extraction module when the voice extraction module processes a preset frame of a preset block preceding the current block (Li, 1. Introduction, "…We compare two kinds of the most popular models for the dual-path modeling, the RNN and transformer...we compare the dual-path bidirectional long-short memory (DP-BLSTM) with the baseline BSLTM on different window sizes...").
Claim 12 is a method claim with limitations similar to the limitations of Claims 7 and 9 and is rejected under similar rationale.
Regarding Claim 13,
Xu in view of Li discloses the method according to 7, the method according to wherein the second audio feature comprises sub-band features of at least two preset frequency bands of the second audio signal (Li, Fig.1: A):The segmentation stage splits the long recording into short windows with window size K and hop length P, "…Denote
W
∈
R
L
×
F
as the magnitude spectrum of the single channel continuous mixture input , where F is the number of frequency bins and L is the number of frames...Then the segmented entire meeting can be presented as a three-D tensor
T
=
[
D
1
;
∙
∙
∙
;
D
B
]
∈
R
B
×
K
×
F
, on top of which, a feature extraction module is applied to form the feature for separation step, which has the shape
T
∈
R
B
×
K
×
N
with N referring to feature dimension...."),
wherein the mask information comprises mask information of each preset frequency band (Li, Fig.1, 2. CSS: TASK DEFINITION AND BASELINE, "…The mask based BLSTM (i.e., updating historical second hidden states (from continuous mixture input) using BLSTM) separation network is used as the baseline in this work, with phase sensitive mask [31] as network output..."), and
wherein the determining, by using the decoding module based on the second audio feature of the second audio signal and the mask information of the target audio signal comprises : determining, by using the decoding module respectively corresponding to each preset frequency band based on the sub-band features of each preset frequency band and the mask information, predicted features of each preset frequency band (Li, Fig.1: A):The segmentation stage splits the long recording into short windows with window size K and hop length P, "…Denote
W
∈
R
L
×
F
as the magnitude spectrum of the single channel continuous mixture input , where F is the number of frequency bins and L is the number of frames...; Fig.1, 2. CSS: TASK DEFINITION AND BASELINE, "…The mask based BLSTM (i.e., updating historical second hidden states (from continuous mixture input) using BLSTM) separation network is used as the baseline in this work, with phase sensitive mask [31] as network output...") and
determining the target audio signal based on the predicted features of each preset frequency band (Li, Fig.1: C): The stitching stage concatenates the separated windows into continuous outputs which only contain non-overlapped speech, 2. CSS: TASK DEFINITION AND BASELINE "…Then for each window, streams output
O
b
∈
R
K
×
F
×
C
are estimated by the separation module, where C is the number of the output channels...final result is estimated by a simple overlap-and-add step to connect the local separation result to form output
S
O
c
∈
R
K
×
F
×
C
with the same length as mixed signal...").
Claim 14,
Xu in view of Li discloses the method according to claim 7, wherein the voice extraction module comprises at least one of following: a recurrent neural network, an attention network, a transformer network, or a convolutional network (Li, Fig.1, 1. Introduction, "…We compare two kinds of the most popular models for the dual-path modeling, the RNN and transformer...we compare the dual-path bidirectional long-short memory (DP-BLSTM) with the baseline BSLTM on different window sizes..."; 4.2. Model Configurations and Training Details, "...The transformer baseline contains 10 transformer encode layers; the attention dimension is 256, and 4-head multi-head attention is used...").
Regarding Claim 17,
Xu in view of Li discloses the method according claim 1,wherein the registration sound source is a target speaker (Xu, Fig.2, "…The speaker encoder encodes the reference speech x(t) into a speaker embedding, that is the feature representation of the target speaker...").
Claims 15 and 18-22 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Li further in view of Kim et al., (US Pub No. 2021/0327418, hereinafter, Kim).
Regarding Claim 15,
Xu in view of Li discloses the method according to claim 1, but does not explicitly discloses the user interface and interaction for the voice/speech extraction. However, Kim, in the analogous field of endeavor, discloses outputting an audio signal to be processed to a user (Kim, par [114], "…Voice data or image data collected by the input unit 120 are analyzed and processed as a user's control command…");
receiving processing instructions from the user (Kim, Fig.4, paras [214-229], "…the user registration may be performed through a user registration interface..."; "…a user may easily manage user registration information for users registered through an interface..."); and
determining the first audio signal and the second audio signal based on the processing instructions and the audio signal to be processed (Kim, Fig.7, par [273], "…the voice signal of the first user is referred to as a first voice signal 702a, the voice signal of the second user is referred to as a voice signal 702b, and the combined signal of the first voice signal 702a and the second voice signal 702b is referred to as a third voice signal 702, the information indicating each user and the third voice signal are input, as input data, to the ANN for generating the voice extraction filter...").
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a speaker extraction network for continuous speech with the target speaker encoder of Xu in view Li with a voice registration module of AI apparatus for extracting a voice of each user form a voice input of Kim with a reasonable expectation of success to ensure effective and high precision voice recognition performance under the abnormal noise environment (e.g., the voice of the surrounding people, surrounding music, or TV sound (Kim, paras [002-004]).
Claim 18 is a device claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale.
Additionally, Kim discloses an electronic device comprising: a microphone configured to receive a first audio signal and a second audio signal; and a processor configured to (Kim, par [295], "…the processor 180 may receive the input voice signal through the microphone 122 of the AI apparatus 100 to perform the voice control or may receive the input voice signal...").
...
Rationale for combination is similar to that provided for Claim 15.
Regarding Claim 19,
Xu in view of Li further in view of Kim discloses the electronic device according to claim 18. Claim 19 is an electronic device claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Regarding Claim 20,
Xu in view of Li further in view of Kim discloses the electronic device according to claim 18.
Claim 20 is an electronic device claim with limitations similar to the limitations of Claim 7 and is rejected under similar rationale
Regarding Claim 21,
Xu in view of Li further in view of Kim discloses the electronic device according to claim 18.
Claim 21 is an electronic device claim with limitations similar to the limitations of Claim 17 and is rejected under similar rationale.
Claim 22 is a non-transitory computer-readable storage media claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale.
Additionally, Kim discloses one or more non-transitory computer-readable storage media storing instructions that, when executed by at least one processor of an electronic device individually or collectively, cause the electronic device to perform operations (Kim, par [360], "…the above-described method may be implemented as a processor-readable code in a medium where a program is recorded. Examples of a processor-readable medium may include hard disk drive (HDD), solid state drive (SSD)...")
Rationale for combination is similar to that provided for Claim 15.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Luo et al., ("Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, hereinafter, Luo) discloses a dual-path recurrent neural network (DPRNN), a simple yet effective method for organizing RNN layers in a deep structure to model extremely long sequences (Luo, Abstract).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/Examiner, Art Unit 2656
/BHAVESH M MEHTA/Supervisory Patent Examiner, Art Unit 2656
Prosecution Insights