Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-17 are pending. Claims 1 and 11 are independent.
This Application was published as US 20230326478.
Apparent priority is 04/06/2022.
Applicant’s amendments and arguments are considered but are either unpersuasive or moot in view of the new grounds of rejection that, if presented, were necessitated by the amendments to the Claims.
This action is Final.
Response to Arguments
35 USC 103
Applicant's arguments with respect to 35 USC 103 are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5, 7-12, 14-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nesta et al. (US 20200349965 A1) in view of Kong et al. (“Source Separation with Weakly Labelled Data: An Approach to Computational Auditory Scene Analysis”).
Regarding claim 1, Nesta discloses: A sound processing system to extract a target sound signal, ("[0018] The present disclosure provides improved systems and methods for denoising and target signal enhancement…”)
the sound processing system comprising: at least one processor; and memory having instructions stored thereon that, when executed by the at least one processor, cause the sound processing system to: ("[0036] … The audio processing device 600 includes audio input components 602, a memory 610, a processor 620, a communications interface 622 and a user interface 624...”)
collect a mixture of sound signals; ("[0025]… At test time, the noisy input signal is provided to the network together with the embeddings…”; see also: “[0030] In some embodiments, a user may have a recording of a mixed signal containing both speech and noise and may wish to remove the noise to extract a high-quality speech signal…”)
collect a query identifying the target sound source to be extracted from the mixture of sound signals, the query comprising one or more identifiers; extract from the query, each identifier of the one or more identifiers, ("[0030]… For example, as illustrated in FIG. 5 a user interface 510 may be provided allowing the user to input information on the source speaker...”- input information reads on a query. fig. 5 shows identifiers of gender, pitch, language, and speed. [0033] further describes how the input settings are processed.)
said each identifier being present in a predetermined set of one or more identifiers, (Fig. 5 shows a interface for predetermined identifiers)
each identifier defining at least one of mutually inclusive and mutually exclusive characteristics of the target sound source; (gender and language each show mutually exclusive options; all of the identifiers can be mutually inclusive with each other.)
determine any logical operators connecting the extracted one or more identifiers; (as shown in Fig. 5, the speech source identifiers are a logical AND, whereas the Noise Source are a logical NOT. See also Fig. 1C.)
transform the extracted one or more identifiers and any logical operators into a digital representation predetermined for querying the mixture of sound signals; ("[0025]… The architecture includes pre-processing networks that convert information about the target (e.g., speech) and noise characteristics provided by the user into the embeddings...”)
identify, based on the digital representation, one of a plurality of sound sources contributing sound signals to the mixture of sound signals as the target sound source; (“[0033] The data pre-processing program takes as input the speaker feature settings provided by the user and the noise recording. It outputs a fixed-sized embedding for each of these that can be used in the denoising program along with the mixed signal…” – the target embedding identifies the target source.)
execute a neural network trained to extract all the sound signals corresponding to the identified target sound source from the mixture of sound signals, by combining the digital representation with intermediate outputs of intermediate layers of the neural network processing the mixture of sound signals, wherein the neural network is trained with machine learning to extract different sound signals identified in a predetermined set of digital representations; ("[0021] Referring to FIG. 1C, a noise reduction neural network (NR-NN) 150 is trained with random speech and noise sequences to produce an enhanced signal 160. The inputs include the audio signal 152 (i.e., the noisy signal), the corresponding speech embedding 154 and the corresponding noise embedding 156 obtained through the pre-processing neural networks 120 and 100, respectively…”)
and output the extracted target sound source. (Fig. 1C shows Enhanced signal 160 is output.)
Nesta does not explicitly disclose combining the digital representation with intermediate outputs of intermediate layers of the neural network.
Kong discloses combining the digital representation with intermediate outputs of intermediate layers of the neural network. (“In addition to the anchor segment, a condition vector is used as an extra input to control what source to separate.” Pg. 3, para 1; “The condition vectors is mapped to embedding vectors by a learnable matrix. The embedding vectors are added to after each ReLU operation in all layers as a bias.” Pg. 4, para 2)
Nesta and Kong are considered analogous art to the claimed invention because they disclose methods for source separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Nesta by adding the embedding vector after each layer. Doing so would have been beneficial to control what sources to separate. (Kong pg. 4 para 2) This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding claim 2, Nesta discloses: 2. The sound processing system of claim 1, wherein sound signals in the mixture of sound signals are collected from a plurality of sound sources with facilitation of one or more microphones, (“[0037]…The audio input components 602 may include an audio sensor array comprising one or more microphones, anti-aliasing filters, analog-to-digital converter circuitry, echo cancellation circuitry, and other audio processing circuitry and components…”)
wherein each sound source of the plurality of sound sources corresponds to at least one of a speaker, a person or an individual, an industrial equipment, a vehicle, or a natural sound. (“[0030] In some embodiments, a user may have a recording of a mixed signal containing both speech and noise and may wish to remove the noise to extract a high-quality speech signal…” – speech reads on a speaker and a person or an individual. [0031] discloses a dog barking which reads on a natural sound. Fig. 3 discloses road noise which reads on a vehicle, and airport which reads on industrial equipment.)
Regarding claim 3, Nesta discloses: 3. The sound processing system of claim 1, wherein the predetermined set of one or more identifiers is associated with a plurality of sound sources, (Fig. 5 shows the identifiers are associated with both speech and noise sources)
wherein the each of the one or more identifiers in the predetermined set of one or more identifiers comprises at least one of: a loudest sound source identifier, quietest sound source identifier, a farthest sound source identifier, a nearest sound source identifier, a female speaker identifier, a male speaker identifier, and a language specific sound source identifier. (Fig. 5 shows identifiers associated with female, male, and language identifiers.)
Regarding claim 4, Nesta discloses: The sound processing system of claim 1, wherein the one or more identifiers are combined using any of the determined logical operators to extract the target sound source having mutually inclusive and exclusive characteristics, wherein any of the determined logical operators comprise at least one of: NOT operator, AND operator, and OR operator, wherein NOT operator is used with any single identifier of the one or more identifiers. (as shown in Fig. 5, the speech source identifiers are a logical AND, whereas the Noise Source are a logical NOT.)
Regarding claim 5, Nesta discloses: The sound processing system of claim 1, wherein the neural network is trained using the predetermined set of digital representations of a plurality of combinations of identifiers in the predetermined set of one or more identifiers. ([0021] discloses the neural networks is trained on the speech and noise embeddings. [0030] and Fig. 5 show that the embeddings are based on predetermined sets of identifiers.)
Regarding claim 7, Nesta does not disclose the additional limitations.
Kong discloses: The sound processing system of claim 1, wherein the digital representation is represented by at least one of: a one hot conditional vector, a multi-hot conditional vector, and text description. ("If an anchor segment sj only contains clean source for the k-th sound class, then cj should be one hot encoding of the k-th sound class. On the other hand, if an anchor segment sj contains multiple sound sources then cj should have positive values for all occurred sound sources and zero for other sound classes that do not occur in sj.” Pg. 3, para 3 – this is a multi-hot conditional vector).
Nesta and Kong are considered analogous art to the claimed invention because they disclose methods for source separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Nesta by using a multi-hot conditioning vector. Doing so would have been beneficial so that clean sources of training data are not required. (Kong pg. 3 para 3)
Regarding claim 8, Nesta discloses: The sound processing system of claim 1, wherein the intermediate layers of the neural network comprise one or more intertwined blocks, wherein each of the one or more intertwined blocks comprise at least one of: a feature encoder, a conditioning network, a separation network, and a feature decoder, wherein the conditioning network comprises a feature-invariant linear modulation (FiLM) layer that takes as an input the mixture of sound signals and the digital representation and modulates the input into the conditioning input, wherein the FiLM layer processes the conditioning input and sends the processed conditioning input to the separation network. ("[0034]…The denoising network could be based on a denoising autoencoder structure with a dense or convolutional network…” – this reads on at least a feature encoder)
Regarding claim 9, Nesta discloses: The sound processing system of claim 8, wherein the separation network comprises a convolution block layer that utilizes the conditioning input to separate the target sound source from the mixture of sound signals, wherein the separation network is configured to produce a latent representation of the target sound source. ("[0034] In some embodiments, the denoising network is trained on examples of mixtures of signals generated by summing individual noise and speech signals with varying signal-to-noise ratios. The latent variable representations of the ground truth signals and input these to the network along with the mixed signals may also be generated. The denoising network could be based on a denoising autoencoder structure with a dense or convolutional network. The embeddings can be inputted to the denoising network by concatenating them with the audio input in the visible layer or by inserting them in a deeper layer. The latter would induce the network to correlate the embeddings with a higher level latent representation of the sound and would allow the structure of the neural network in the first layers to be more meaningful for the nature of the audio input signals.”)
Regarding claim 10, Nesta discloses: The sound processing signal of claim 8, wherein the feature decoder converts a latent representation of the target sound source produced by the separation network into an audio waveform. (“[0021] Referring to FIG. 1C, a noise reduction neural network (NR-NN) 150 is trained with random speech and noise sequences to produce an enhanced signal 160…”; [0034] discloses a latent representation.)
Regarding claim 11, arguments analogous to claim 1 are applicable.
Regarding claim 12, arguments analogous to claim 2 are applicable.
Regarding claim 14, arguments analogous to claim 4 are applicable.
Regarding claim 15, arguments analogous to claim 5 are applicable.
Regarding claim 16, Nesta discloses: The computer-implemented method of claim 11, further comprising: generating one or more queries associated with the mutually inclusive and exclusive characteristics of the target sound source during training of the neural network. ("[0034] In some embodiments, the denoising network is trained on examples of mixtures of signals generated by summing individual noise and speech signals with varying signal-to-noise ratios. The latent variable representations of the ground truth signals and input these to the network along with the mixed signals may also be generated....”)
Regarding claim 17, arguments analogous to claim 8 are applicable.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nesta in view of Kong as applied in claim 1 above, further in view of Shanahan et al. (US 20070136336 A1).
Regarding claim 6, Nesta and Kong do not disclose the additional limitations.
Shanahan discloses: 6. The sound processing system of claim 1, wherein the neural network is trained using a positive example selector and a negative example selector to extract the target sound source. ("Assembling positive and negative examples for a training set is well known to those of ordinary skill in the art" [0039])
Nesta, Kong, and Shanahan are considered analogous art to the claimed invention because they discuss methods of training machine learning models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination with the teaching of Shanahan to use positive and negative examples in the training data. This would have been a known method with predictable results.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nesta in view of Kong as applied in claim 11 above, in further view of Chang et al. (MIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition), and Jun (US 20230119203 A1).
Regarding claim 13, Nesta discloses: The sound processing system of claim 11, wherein the predetermined set of one or more identifiers are associated with a plurality of sound sources, (Fig. 5 shows the identifiers are associated with both speech and noise sources)
wherein each of the one or more identifiers in the predetermined set of one or more identifiers comprises at least one loudest sound source identifier, quietest sound source identifier, farthest sound source identifier, nearest sound source identifier, (not explicitly disclosed)
female speaker identifier, male speaker identifier, and language specific sound source identifier. (Fig. 5 shows identifiers associated with female, male, and language identifiers.)
Nesta does not disclose: wherein each of the one or more identifiers in the predetermined set of one or more identifiers comprises at least one loudest sound source identifier, quietest sound source identifier, farthest sound source identifier, and nearest sound source identifier. Neither does Kong.
Chang discloses: loudest sound source identifier, quietest sound source identifier, ("we sort the multi-speaker data in ascending order of SNR between the loudest and quietest speaker" Pg. 3, Section 2.2)
Nesta, Kong, Chang and Jun are considered analogous art to the claimed invention because they discuss methods of identifying target sounds. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination with the teaching of Chang to use a loudest and quietest source identifier. This would have been beneficial because recognition accuracy is different at different energy levels (Chang Pg. 3, Section 2.2]).
Chang does not disclose: farthest sound source identifier, nearest sound source identifier.
Jun discloses: farthest sound source identifier, nearest sound source identifier, ("display the position information of other persons who reproduce the sound sources on a screen of the user terminal in the order of their distance from the closest to the farthest" [0023])
Nesta, Kong, Chang, and Jun are considered analogous art to the claimed invention because they discuss methods of identifying target sounds. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination with the teaching of Jun to use a nearest and farthest source identifier. This would have been beneficial in order to make use of position information (Jun [0023]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Swaminathan et al. (US 20230298561 A1). Swaminathan discloses a target source separation method using combined sound source representations. Ref fig. 1. [0041] discloses a multi-hot vector.
Liu et al. (“Separate What You Describe: Language-Queried Audio Source Separation”). Liu discloses a language queried audio source separation system. Liu does not explicitly disclose predetermined identifiers.
Hao et al. (“Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction”). Hao is not prior art but is cited for relevance. Hao discloses a method of text-guided target speaker extraction.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JON C MEIS whose telephone number is (703)756-1566. The examiner can normally be reached Monday - Thursday, 8:30 am - 5:30 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached on 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JON CHRISTOPHER MEIS/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
