Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 26 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention. The claim comprises drafting errors including the system configured to “receiving receive input audio data,” which appears redundant and will be construed as “configured to receive input audio data,” for the purposes of the art rejection infra; and “produce space embeddings,” will be construed as “produce latent space embeddings,” for the purposes of the art rejection infra. Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 26 rejected under 35 U.S.C. 102a1 as being anticipated by Kegler: Deep Speech Inpainting of Time-frequency Masks (provided by Examiner, copyright 2020 and hereinafter Ke).
Regarding claim 1
Ke teaches:
A method, comprising:
receiving, by a control system configured to implement at least one neural network, input audio data and feature weightings (Ke: Abstract; § 2.1-2.3; Fig 1: system receives time-frequency representations of speech recordings segmented into a matrix of feature weightings and selectively supplying additional mask data in which the underlying signal is degraded by inclusion of mask data, masked features, etc.);
producing, by the control system and based at least in part on the input audio data and the feature weightings, latent space embeddings (Ke: § 2.1-2.3; Table 1: system encodes the input, features, mask data, etc. upon encoding blocks of a Unet which produce hidden, latent space, etc. embeddings upon a plurality of 2D convolution layers which utilize the input data, corrupted data, and mask data),
wherein the input audio data corresponds to an input mathematical space (Ke: § 2.1: such as the input 128X128 segmented signal) and
wherein the latent space embeddings comprise mathematical representations of the input audio data indicated by the feature weightings in a latent space that is a different mathematical space from the input mathematical space (Ke: § 2.1-2.3; Table 1: the internal convolution layers processed on the encoding layers operate upon dimensional data different from and distinct to the actual input time-frequency segment as shown in the table) and
wherein the latent space embeddings correspond with unmasked portions of the input audio data (Ke: § 2.1-2.3, 5; Fig 2; Table 1: the mask corrupting the input comprises portions of the provided feature weightings and is known; as such only the non-masked portions are processed and the various latent space dimensional representation of the signal and used to by the decoder blocks of the Unet to generate a recovered degraded signal for comparison to the input audio data segment); and
constructing, by the control system, a modified audio signal based on the latent space embeddings (Ke: § 2.1-2.3; Fig 2; Table 1: encoder/decoder blocks of the Unet generate recovered degraded signal to for comparison to the input audio segment data).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-14, 22-26 rejected under 35 U.S.C. 103 as being unpatentable over Kegler: Deep Speech Inpainting of Time-frequency Masks (provided by Examiner, copyright 2020 and hereinafter Ke) further in view of Nesta: 20200349965 hereinafter Ne.
Regarding claim 1
Ke teaches:
A method, comprising:
receiving, by a control system configured to implement at least one neural network, input audio data and feature weightings (Ke: Abstract; § 2.1-2.3; Fig 1: system receives time-frequency representations of speech recordings segmented into a matrix of feature weightings and selectively supplying additional mask data in which the underlying signal is degraded by inclusion of mask data, masked features, etc.);
producing, by the control system and based at least in part on the input audio data and the feature weightings, latent space embeddings (Ke: § 2.1-2.3; Table 1: system encodes the input, features, mask data, etc. upon encoding blocks of a Unet which produce hidden, latent space, etc. embeddings upon a plurality of 2D convolution layers which utilize the input data, corrupted data, and mask data),
wherein the input audio data corresponds to an input mathematical space (Ke: § 2.1: such as the input 128X128 segmented signal) and
wherein the latent space embeddings comprise mathematical representations of the input audio data indicated by the feature weightings in a latent space that is a different mathematical space from the input mathematical space (Ke: § 2.1-2.3; Table 1: the internal convolution layers processed on the encoding layers operate upon dimensional data different from and distinct to the actual input time-frequency segment as shown in the table) and
wherein the latent space embeddings correspond with unmasked portions of the input audio data (Ke: § 2.1-2.3, 5; Fig 2; Table 1: the mask corrupting the input comprises portions of the provided feature weightings and is known; as such only the non-masked portions are processed and the various latent space dimensional representation of the signal and used to by the decoder blocks of the Unet to generate a recovered degraded signal for comparison to the input audio data segment); and
constructing, by the control system, a modified audio signal based on the latent space embeddings (Ke: § 2.1-2.3; Fig 2; Table 1: encoder/decoder blocks of the Unet generate recovered degraded signal to for comparison to the input audio segment data).
The recited mask data of Ke may not teach the broadest reasonable interpretation of the recited feature weightings which result in the adaption of the input signal with respect to actual noise within the signal which drives the construction of the signal based on latent space embeddings of the noise derived features.
In a related field of endeavor Ne teaches:
A system, method, etc. comprising: receiving, by a control system configured to implement at least one neural network, input audio data and feature weightings; (Ne: Abstract; ¶ 19-26, 36-40; Fig 1C, 6: audio processing device comprising a trained neural network operable to receive an audio signal and generate an enhanced signal; such as providing an embedding vector to the neural network, the vector comprising an audio signal; and speech and noise embeddings, features, characteristics, etc.; weights, coefficients, etc. thereof such as for training a noise reduction neural network ); and
producing, by the control system and based at least in part on the input audio data and the feature weightings, latent space embeddings (Ne: ¶ 19-26, 34; Fig 1B: DNN trained to classify audio receives audio data, samples thereof, and feature weightings thereof, providing processed data thereof to a plurality of hidden layers such as comprising an autoencoder neural network),
wherein the input audio data corresponds to an input mathematical space (Ne: ¶ 19-26: DNN trained to classify audio receives plurality of samples provide same to noise reduction neural network, hidden layers, latent spaces thereof) and
wherein the latent space embeddings comprise mathematical representations of the input audio data indicated by the feature weightings in a latent space that is a different mathematical space from the input mathematical space (Ne: ¶ 19-26; Fig 1A,-1C: a hidden layer outputs a noise classification; an inner most speech embedding layer corresponds to an autoencoder and generates a learned speech embedding vector dimensionally reduced from the input) and
wherein the latent space embeddings correspond with unmasked portions of the input audio data (Ne: ¶ 19-26; Fig 2: speech embeddings computed from, corresponding to, etc. an unmasked, speech isolated potions of the input audio based on directions from the classifier) such that based on a masking noise/speech designation provided by the classifier a clearer, less noisy speech signal, generates dimensionally distinct embeddings based thereon which are provided to the noise reduction neural network); and
constructing, by the control system, a modified audio signal based on the latent space embeddings (Ne: ¶ 19-26, 36-40; Fig 1C, 6: an enhanced signal is generated by the noise reduction neural network using the input audio and feature weightings such as speech, noise embeddings based thereon).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to utilize the feature weightings, pipeline components therefor as taught or suggested by Ne to improve or augment the input mask of Ke for at least the purpose of adapting the system to real world noise reduction by providing a route to address generating of a clean voice signal in the presence of real world ecological noises; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 2
Ke in view of Ne teaches or suggests:
The method of claim 1, wherein the feature weightings comprise mask data (Ke: § 2.1-2.3: features comprise mask data used to reconstruct an input signal segment). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 3
Ke in view of Ne teaches or suggests:
The method of claim 1, wherein the mask data is derived from estimations of signal and noise in the input audio data (Ke: § 2.1-2.3; Fig 1, 2; Table 1: features comprise mask data used to reconstruct an input signal segment); (Ne: ¶ 19-26, 36-40; Fig 1C, 6: such as by improving the mask using speech/noise classifications, embedding vectors, etc.). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 4
Ke in view of Ne teaches or suggests:
The method of claim 1, wherein the control system is configured to implement a convolutional neural network configured to perform weighted convolution and wherein the weighted convolution is based, at least in part, on the feature weightings (Ke: § 2.1-2.3; Fig 1, 2; Table 1: such as by conducting partial convolution across plural encoder layers based on weighting the input by the mask data). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 5
Ke in view of Ne teaches or suggests:
The method of claim 1, wherein producing the latent space embeddings involves applying, by the control system, a contextual encoding process (Ke: Abstract; § 1, 2.1-2.3, etc. Fig 1, 2; Table 1: processing by plural encoder/decoder blocks effects context based retrieval of masked features, etc. by speech inpainting). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 6
Ke in view of Ne teaches or suggests:
The method of claim 5, wherein the at least one neural network has been trained to implement the contextual encoding process (Ke: § 2.4, etc.: system trained to reconstruct based on masking); (Ne: Abstract, etc. system trained to generate an enhanced target signal). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 7
Ke in view of Ne teaches or suggests:
The method of claim 1, further comprising applying, to the latent space embeddings and by the control system, a hidden representation process, to produce a representation of the input audio data in the latent space (Ke: Abstract; § 1, 2.1-2.3, etc. Fig 1, 2; Table 1: system produces a sequence of hidden representations of the input audio across encoder block latent space). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 8
Ke in view of Ne teaches or suggests:
The method of claim 7, further comprising applying, by the control system, a contextual decoding process to the representation of the input audio data in the latent space, to produce the modified audio signal (Ke: Abstract; § 1, 2.1-2.3, etc. Fig 1, 2; Table 1: each decoding block up-samples the latent space representation of the encoders to generate a restored degraded speech segment). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 9
Ke in view of Ne teaches or suggests:
The method of claim 8, further comprising producing a residual signal based, at least in part, on the modified audio signal and a version of the input audio data. Examiner takes official notice that the production of a residual signal between an enhanced signal and an input signal was well-known in the art before the effective filing date of the instant invention and would have comprised an obvious inclusion for at least the purpose of providing speech enhancement, an improved model for speech enhancement, etc.; one of ordinary skill in the art would have expected only predictable results therefrom. The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 10
Ke in view of Ne teaches or suggests:
The method of claim 9, wherein the version of the input audio data comprises frequency binned audio data (Ke: Abstract; § 2.1-2.3; Fig 1: system receives time-frequency representations of speech recordings as input); (Ne: Figs 1A, 1B, 2, etc.: such as by input of time frequency represented audio samples). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 11
Ke in view of Ne teaches or suggests:
The method of claim 9, wherein the modified audio signal is in a frequency domain and wherein producing the residual signal involves transforming a frequency domain version of the residual signal into a time domain (Ke: § 2.2, 2.3; Fig 2: recovered degraded audio signal reconstructed into the time domain, such as as a waveform). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 12
Ke in view of Ne teaches or suggests:
The method of claim 1,wherein the input audio data and the feature weightings correspond to frequency bands (Ke: Abstract; § 2.1-2.3; Fig 1: system receives time-frequency representations of speech recordings and the mask corresponds to frequency bands thereof); (Ne: ¶ 19-26, 36-40; Fig 1A-1C, (Ne: Figs 1A, 1B, 2, etc.: such as by input of time frequency represented audio samples operations thereon to represent same). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 13
Ke in view of Ne teaches or suggests:
The method of claim 1,wherein the input audio data has been pre-conditioned according to one or more audio data processing methods (Ne: ¶ 37: such as by performance of filtering, cancellation, conversion, etc. thereon). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 14
Ke in view of Ne teaches or suggests:
The method of claim 13, wherein the input audio data has been pre- conditioned according to at least one of an echo cancellation process, an echo suppression process, a noise suppression process or a beamforming process (Ne: ¶ 37: such as by performance of filtering, cancellation, conversion, etc. thereon). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 22
Ke in view of Ne teaches or suggests:
The method of any claim 1, wherein the control system is configured for speech representation learning (SRL) (Ke: § 1, 2.1-2.4, etc.; Fig 2: system learns representations of speech based on exploitation of context information therein such that an input noisy speech signal is decomposed into stage-scaled representative signals and then reconstructed therefrom; such as by a sequence of convolutional encoding and decoding, including partial convolution thereby which generate a prediction in the form of the recovered degraded signal which is reified against the actual input to improve the model).
Regarding claim 23
Ke in view of Ne teaches or suggests:
The method of claim 22, wherein the at least one neural network includes an SRL encoder (Ke: § 2.2, 2.3; Fig 2: such as the Unet encoders of the figure which are used to regenerate the signal from representations thereof); (Ne: ¶ 19, 20: Figures 1A-C, 2: the DNN and autoencoders of the figures perform encoding of speech representations). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 24
Ke in view of Ne teaches or suggests:
The method of claim 23, wherein the SRL encoder comprises a convolutional encoder (Ke: § 2.1-2.4; Fig 2; Table 1: encoder comprises plural convolution layers); (Ne: § 34: system comprises convolutional network with autoencoder layer). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 25
Ke in view of Ne teaches or suggests:
The method of claim 23, wherein the convolutional encoder includes partial convolution layers (Ke: § 2.1-2.3; Fig 1, 2; Table 1: such as by conducting partial convolution across plural encoder layers based on weighting the input by the mask data). The claim is considered obvious over Ke as modified by Ne as addressed in the base claim as it would have been obvious to apply the further teaching of Ke and/or Ne to the modified device of Ke and Ne; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 26—the claim is considered to recite substantially similar subject matter to claim 1 and is similarly rejected.
Claims 15-18 rejected under 35 U.S.C. 103 as being unpatentable over Kegler: Deep Speech Inpainting of Time-frequency Masks (provided by Examiner, copyright 2020 and hereinafter Ke) further in view of Nesta: 20200349965 hereinafter Ne as applied to claims 1-14, 22-26 supra, and further in view of Li: U-shaped Transformer with Frequency-Band Aware Attention for Speech Enhancement (provided by Examiner, copyright 2021).
Regarding claim 15
Ke in view of Ne teaches or suggests the method of claim 1, wherein the at least one neural network has been trained to using a masking process for producing embeddings but does not explicitly discuss the at least one neural network also been trained to implement an attention-based masking process for producing embeddings.
In a related field of endeavor Li teaches a system, method for speech enhancement using frequency aware attention such as by adding transformer self-attention to the encoder and decoder layers of a Unet type architecture (Li: Abstract; § 3)
wherein the at least one neural network has also been trained to implement an attention-based masking process for producing embeddings (Li: § 3; Fig 1: such as by using the transformer enhanced Unet type architecture to generate masked attention maps conditioned on an ideal ratio mask of target speech to noisy speech and noise speech).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to improve the Ke in view of Ne speech enhancement pipeline by the inclusion of self-attention as taught or suggested by Li to the encoder and decoder blocks of Ke in view of Ne such as for better directing the masking, improving estimation accuracy, providing more accurate target speech recovery, etc.; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 16
Ke in view of Ne in view of Li teaches or suggests:
The method of claim 15, wherein at least one of the attention-based masking process or a contextual encoding process has been trained to recognize and to compensate for one or more errors in the masking process (Ke: § 3.1-3.3: contextual encoding process to minimize reconstruction error, such as by improving gains of masking); (Ne: ¶ 34: system sums signal, noise over context such as by adjusting, varying, etc. ratios thereof); (Li: § 3: system trained with differential weightings to compensate for errors across frequency bands). The claim is considered obvious over Ke as modified by Ne, and Li as addressed in the base claim as it would have been obvious to apply the further teaching of Ke, Ne, and/or Li to the modified device of Ke, Ne and Li; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 17
Ke in view of Ne in view of Li teaches or suggests:
The method of claim 15, wherein the at least one neural network has been trained according to mask data and according to contaminated audio signals output from an audio augmentation process (Ke: § 3.1-3.3: Unet trained on mask data and contaminated audio signals produced by mask data); (Ne: ¶ 34: system sums signal, noise, over context such as by adjusting, varying, etc. ratios thereof); (Li: § 3, 4: system trained with differential weightings based on signal and noise estimations of the input signal and a contaminated noise speech output based thereon to thereby compensate for errors introduced across frequency bands). The claim is considered obvious over Ke as modified by Ne, and Li as addressed in the base claim as it would have been obvious to apply the further teaching of Ke, Ne, and/or Li to the modified device of Ke, Ne and Li; one of ordinary skill in the art would have expected only predictable results therefrom.
Regarding claim 18
Ke in view of Ne in view of Li teaches or suggests:
The method of claim 17, wherein the audio augmentation process involves adding noise, adding reverberations, adding audio signals corresponding to speech or other interfering audio sources, or combinations thereof (Ke: § 3.2: system adds noise); (Ne: ¶ 4: system trains based on noise, interfering speech); (Li: § 4: system adds noise, interfering, undesired, etc. speech). The claim is considered obvious over Ke as modified by Ne, and Li as addressed in the base claim as it would have been obvious to apply the further teaching of Ke, Ne, and/or Li to the modified device of Ke, Ne and Li; one of ordinary skill in the art would have expected only predictable results therefrom.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL C MCCORD whose telephone number is (571)270-3701. The examiner can normally be reached 730-630 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, CAROLYN EDWARDS can be reached at (571) 270-7136. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL C MCCORD/Primary Examiner, Art Unit 2692
Prosecution Insights