DETAILED ACTION
This office action is in response to Applicant’s Request for Continued Examination (RCE), received on 06/09/2026. Claims 1, 2, 4, 6, 15, 16, and 18-20 have been amended. Claims 8 and 17 have been cancelled. Claims 1-7, 9-16, 18-20 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/09/2026 has been entered.
Response to Arguments
Applicant's arguments filed 06/09/2026, see pgs. 9-10, with respect to “Rejections under 35 U.S.C. 103” have been fully considered but they are not persuasive.
Applicant’s representative asserts, “Applicant respectfully submits that Jungmaier, Gao, and Kitazawa fail to teach or render obvious ‘applying a phase shift to the time-frequency representation of the speech signals during the inference operation’ as recited in amended claim 1. In its rejection of previously pending claim 8, the Office admits that Jungmaier, Gao, and Kitazawa fail to disclose ‘redefining the target area . . . is performed via a phase shift to the at least one audio input device.’ (Office action, p. 25). Instead, the Office relies on U.S. Patent Application Publication No. 2016/0192073 (Poornachandran et al., hereinafter Poornachandran). (Id.).
The Office cites Par. [0015] of Poornachandran to teach ‘wherein the redefining ... is performed via a phase shift.’ (Id.). Par. [0015] of Poornachandran describes the functions of an ‘audio processing device 108’ of an ‘electronic device 100,’ such as a smart headset. More specifically, the cited paragraph describes how ‘the directionality of the microphone array 110 can be adjusted by shifting the phase of the received audio signals.’ However, this refers to the processing performed within an electronic device (e.g., a headset). For example, the ‘audio processing device 108’ can be configured to adjust the directionality of the microphone array during a processing operation of the headset. However, Poornachandran fails to disclose performing a phase shift during the inference operation of a machine learning model. As recited in the amendments of claim 1 presented herein, ‘a phase shift’ is applied ‘to the time-frequency representation of the speech signals during the inference operation.’ Applicant submits that Poornachandran fails to disclose any aspect related to a machine learning model and especially any aspect related to performing a phase shift during an inference operation of such a model.
For at least the reasons set forth above, Applicant respectfully submits that Jungmaier, Gao, Kitazawa, and Poornachandran fail to teach each and every element of amended claim 1. Independent claims 15 and 19 are amended similarly as amended claim 1.”
In response, the examiner would like to refer to the combination of art used in conjunction for the rejection. Specifically, the examiner will be referring to Kitazawa, previously cited in the independent claims for “updating…phase parameters applied to the speech signals…” (pg. 6 of Final action, mailed 03/09/2026). Kitazawa discloses a phase regulator 1060, wherein the regulation is performed based on a determined change in relative position of sound sources ([Col 7, Lines 20-60]), wherein “a spectrogram [is] obtained by performing time-frequency conversion on the input signal for each channel” ([Col. 4, Lines 10-15]). It can be seen in Fig. 1 that the step of phase regulation 1060 is performed after FFT unit 1040, indicating a time-frequency, i.e. spectrogram, signal arriving at the phase regulator. This is taken in view of Gao. Gao discloses performing “source separating, source localization, beamforming operations…to reduce the computational resources associated with applying machine learning models to such tasks” (Gao, [0030]) and “multi-modal audio source channelization system to train and apply sophisticated machine learning models to both more efficiently and more effectively isolate targeted audio” ([0028], wherein “apply[ing]” after training indicates an inference), further wherein “The audio source feature classification model 604 may be trained using isolate source audio features as input, where inferred output is the desired classifications, and where ground truth data is the actual classifications. In various examples, the model is trained to minimize error between the ground truth data and the inferred output” ([0146]). The examiner asserts that inference output necessarily is produced via “an inference operation” as currently claimed; therefore, the examiner asserts that Gao discloses a machine learning model with an inference operation. An inference operation for one aspect of the invention of Kitazawa indicates the entire system with which the aspect is contained to have an inference operation; therefore, it is reasonable to extend inference to the separation. It would be obvious to apply the phase regulation of Kitazawa to the beamforming of Gao as Kitazawa discloses their solution to provide “a technique capable of stably performing sound source separation even when the relative positions of a sound source and sound pickup device change” ([Col. 2, Lines 55-60]), which would improve the source separation system of Gao through allowing beamforming which can follow moving targets.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 19-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claims are directed to “A storage device for storing computer-readable instructions”. [0114] of the instant app discloses “computing device 1700 may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape”, this does not reference instructions, nor is the storage device non-transitory. Referring to the ordinary meaning of a “storage device”, a definition from Dictionary.com reads “a thing made for a particular purpose; an invention or contrivance, especially a mechanical or electrical one”. This does not exclude transitory signals, i.e. electrical ‘things’, indicating the storage device for storing instructions to be software per se.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 6-7, 9-16, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jungmaier et al. (US-20190265345-A1), hereinafter Jungmaier, in view of Gao et al. (US-20250259639-A1), hereinafter Gao, further in view of Kitazawa (US-9712937-B2).
Regarding claim 1, Jungmaier discloses: a method of extracting sound from a coverage area having a system with at least one audio input device ([Fig. 1, Coverage Area 80, Microphone system 40], [0105] extract a signal from a specific direction, [0020] a microphone system 40, such as a microphone array), the method comprising:
receiving speech signals from one or more audio sources in a plurality of audio sources within the coverage area ([Fig. 1, Facility 20], [0019] detect commands issued by persons 50 located in the facility 20 [In view of Fig. 1 wherein the persons 50 would clearly be producing speech signals, i.e. commands, and the facility is within the coverage area 80]); and,
generating a time-frequency representation of the speech signals from the at least one audio input device ([0106] In frequency domain beamforming the microphone signal is e.g. separated into narrowband frequency bins using a short time Fourier transform (STFT) and the data in each bin is processed separately, [A Fourier transform representing frequency components over intervals of time indicates the transform to be a time-frequency representation]).
Jungmaier does not disclose:
defining, by a machine learning model during an inference operation and based on the time-frequency representation, a target area that is a portion of the coverage area, the target area being definable by a trained criterion.
Gao discloses:
defining, by a machine learning model during an inference operation ([0140] The term “audio source feature classification model” refers to a data construct that describes defined operations of a machine learning model that is configured to generate one or more isolate source audio categories based on isolate source audio features, [0146] The audio source feature classification model 604 may be trained using isolate source audio features as input, where inferred output is the desired classifications, and where ground truth data is the actual classifications. In various examples, the model is trained to minimize error between the ground truth data and the inferred output, [Inferred output indicates an inference operation to generate said output, wherein this can be extended to the speech separation as feature classification is related to sound source separation in order to know what to separate]) and based on the time-frequency representation ([In view of the previously disclosed time-frequency representation generated in Gao]), a target area that is a portion of the coverage area ([0030] Improved multi-modal audio source channelization systems are configured to improve processing speed of denoising, echo removal, source separating, source localizing, beamforming operations, and/or to reduce the computational resources associated with applying machine learning models to such tasks, [Wherein the beamforming operation of Gao is dependent upon a target area of interest ([0065]), indicating the target area, i.e. beam(s), determination to be performed using machine learning models as part of the beamforming and/or denoising operations of Gao, wherein the examiner asserts that audio classification (using the audio source feature classification model) is relevant for determining where to direct a beam]), the target area being definable by a trained criterion ([0065] Each identified target audio source (e.g., a target speaker) or area of interest within a video signal sample may be associated with one or more speaker source features, which are aspects or characteristics of the target audio source or area of interest. Like isolate source audio features discussed above in connection with the audio feature extraction model, speaker source features are used as input for a multi-modal audio source channelization model to generate a source separated channel audio sample, [0082] Multi-modal audio source channelization models discussed herein may be generative adversarial networks (GANs). An example GAN may receive multiple inputs during training including features (e.g., isolate source audio features, speaker source features, isolate source spatial input data), the original audio (e.g., multi-source audio signal sample) and video (e.g., video signal sample) that was used to create those features, and other real or actual audio (e.g., target audio source) as ground truth data for a discriminator to use to estimate the accuracy of generated audio, [Identifying target audio sources with associated areas of interest (as disclosed in [0065]) using speaker source features, wherein the model is later trained using speaker source features (as disclosed in [0082]) indicates a training criterion to learn how to determine target areas]).
Jungmaier and Gao are considered analogous art within speech source separation/cancellation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier to incorporate the teachings of Gao, because of the novel way to improve multi-modal audio source channelization system through the reduction of computational resources required for these tasks by using machine learning models to perform said tasks (Gao, [0030]).
Jungmaier further discloses:
redefining, during the inference operation ([In view of the inference operation of Gao]), the target area into a redefined target area ([0059] as a function at least of the direction of the persons determined based on the radar system 30, the microphone system 40 steers one (or more) audio beam 100 in the corresponding direction so as to enhance the reception of the audio inputs generated by the person, [Steering a beam in a new direction (in the situation of a non-constant person direction function) indicates that new direction to be a new, redefined target area]).
Jungmaier in view of Gao does not disclose:
the redefining including
applying a phase shift to the time-frequency representation of the speech signals during the inference operation; and
updating, in a time-frequency domain, phase parameters applied to the speech signals for the at least one audio input device based on updated sound-source information for the coverage area, the redefining reusing the time-frequency representation so as to avoid an additional inference event by the machine learning model such that negligible additional computational cost is incurred.
Kitazawa discloses:
the redefining including
applying a phase shift to the time-frequency representation of the speech signals during the inference operation ([Fig. 1, applying Phase Regulator 1060 to an output signal from FFT Unit 1040], [Col. 4, Lines 10-15] a spectrogram obtained by performing time-frequency conversion on the input signal for each channel is output to the phase regulator 1060, [Col. 4, Lines 65-67] when phase regulation is performed for each frame, the phase difference between the channels does not change with time any longer, [Col. 5, Lines 1-10] When a plurality of sound sources exist, phase regulation is performed on each sound source… when sound sources A and B exist, a signal obtained by correcting the relative position change of sound source A and a signal obtained by correcting the relative position change of sound source B are generated, [Phase regulation which maintains a constant phase difference between sources through time indicates a shift to phases to keep them constant with relation to each other as determined through the relative positions. “Correcting the relative position change of sound source[s]” is a phase shift]); and
updating, in a time-frequency domain, phase parameters applied to the speech signals for the at least one audio input device based on updated sound-source information for the coverage area ([Col. 4, Lines 34-40] The detected relative positional relationship between the sound pickup unit 1010 and sound source is output to the phase regulator 1060. The relative positional relationship herein mentioned is, for example, the direction (angle) of a sound source with respect to the sound pickup unit 1010. The phase regulator 1060 performs phase regulation on the input frequency spectrum), the redefining reusing the time-frequency representation so as to avoid an additional inference event by the machine learning model such that negligible additional computational cost is incurred ([As disclosed in the cited section above, regulating the phase of a frequency spectrum (indicating the spectrum has an innate phase prior to regulation) tracks to the regulation being a form of updating phase parameters applied to speech signals, wherein the relative positional relationship of a pickup and output tracks to sound-source information. Further, as there is only one spectrum being analyzed, this indicated “reusing” the time-frequency representation for regulation of the phase of the spectrum]).
Jungmaier, Gao, and Kitazawa are considered analogous art within sound source separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier in view of Gao to incorporate the teachings of Kitazawa, because of the novel way to perform sound source separation even when the relative positions of a sound source and sound pickup device change through use of a phase regulator which regulated phases of received sound signals in accordance with determined position changes of transmitters/receivers, improving upon traditional sound source separation methods which cannot function in variable environments (Kitazawa, [Col. 2, Lines 40-67], [Col. 3, Lines 1-10]).
Jungmaier further discloses:
extracting the speech signals from the target area using the time-frequency representation and the redefined target area ([0105] a microphone array is used to form a spatial filter which can extract a signal from a specific direction and reduce the contamination of the signals from other directions, [Wherein the beam in the specific direction is the redefined target area in view of the previously disclosed steering, i.e. redefining, beams of Jungmaier]); and
transmitting the speech signals to a receiver ([0020] The systems 30, 40 and the processing module 70 are operatively coupled together, [0040] each subsystem 90 includes an integrated circuit 92, one or more transmit antennas 94, one or more receive antennas 96, and a substrate 98. In some embodiments, the one or more transmit antennas 94 are arranged in a transmit antenna array, and/or the one or more receive antennas 96 are arranged in a receive antenna array, [Defining the speech subsystems to be containing receivers and transmitters, wherein those speech subsystems are connected to the processing module 70 (see Fig. 2), indicating a transmission of the speech signals gathered by the subsystems to the processing module, i.e. a receiver]).
Regarding claim 2, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 1.
Gao further discloses:
wherein the defining the target area that is the portion of the coverage area is performed by the machine learning model ([0030] Improved multi-modal audio source channelization systems are configured to improve processing speed of denoising, echo removal, source separating, source localizing, beamforming operations, and/or to reduce the computational resources associated with applying machine learning models to such tasks, [Wherein the beamforming operation of Gao is dependent upon a target area of interest ([0065]), indicating the target area determination to be performed using machine learning models as part of the beamforming and/or denoising operations of Gao]).
Regarding claim 3, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 1.
Jungmaier further discloses: wherein the trained criterion includes dimensions of the target area ([Fig. 1, Radius R of coverage area 80], [Defining a radius of a coverage area, wherein a target area can represent the entirety of the coverage area, see [0055] of the instant app, indicates the radius to be a dimension of the target area, wherein a circular target area also requires an angular dimension of 360 degrees]).
Regarding claim 6, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 1.
Jungmaier further discloses:
wherein the redefining the target area toward the speech signals during the inference operation includes aligning a target area centerline of the target area to an audio source center point of the one or more audio sources that are producing the speech signals ([Fig. 1, Coverage Area 80/Target Area 20], [Fig. 9A, Audio Beam 100, Person 50], [In view of the plurality of people 50 in the target area 20 of Fig. 1, determining to target a beam towards a specific person, as shown in Fig. 9a, indicates a redefining a target area, i.e. beam 100, through aligning a target area centerline to an audio source center point, i.e. person 50, as the beam and person are centered about the same angle]).
Regarding claim 7, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 1.
Jungmaier further discloses:
wherein the receiving the speech signals from the one or more audio sources in the plurality of audio sources within the coverage area is performed via the at least one audio input device ([0057] the microphone system 40 is configured to generate one or more audio beam 100 (FIG. 9A) via each of which the microphone system is configured to receive audio inputs. These audio inputs are adapted to include the vocal commands mentioned above).
Regarding claim 9, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 7.
Jungmaier further discloses:
wherein the at least one audio input device includes a microphone array having a plurality of audio input devices ([0020] a microphone system 40, such as a microphone array of plural microphones).
Gao further discloses:
wherein the microphone array is provided by a laptop computer ([0044] Multi-modal audio source channelization systems as discussed herein may be implemented as part of a digital signal processing apparatus, and/or as software, or a software plugin, that is configured for execution on a laptop, [0050] The multi-source audio signal sample may be generated by a single microphone, by an array microphone [Multi-model audio source channelization requiring a multi-source audio for channelization indicating the microphone array is provided by the laptop implementing the channelization]).
Regarding claim 10, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 1.
Jungmaier further discloses:
mixing the extracted speech signals ([0105] a microphone array is used to form a spatial filter which can extract a signal from a specific direction and reduce the contamination of the signals from other directions, i.e. a source of interest may be selected while minimizing undesired interfering signals, [Reducing contamination from other signals indicates that they are also separated/extracted in order to be appropriately reduced, reduction reasonably tracks to a form of mixing]).
Regarding claim 11, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 1.
Jungmaier further discloses:
wherein the extracting the speech signals from the target area forms extracted speech signals ([0106] filtering the microphone signals and combining the outputs to extract (by constructive combining) the desired signal and reject (by destructive combining) interfering signals according to their spatial location, i.e. beamforming may separate sources with overlapping frequency content that originate at different spatial locations, [In view of the previously disclosed target area 20 and predefined threshold distance THD which are used for determining whether sound sources are within a predefined threshold distance (spatial location) to be steered towards, i.e. extracted, indicating this threshold distance radius, i.e. 20 and/or THD of Fig. 1, to be representative of a target area]), the method further comprising mixing the extracted speech signals ([0105] a microphone array is used to form a spatial filter which can extract a signal from a specific direction and reduce the contamination of the signals from other directions, i.e. a source of interest may be selected while minimizing undesired interfering signals, [Reducing contamination from other signals indicates that they are also separated/extracted in order to be appropriately reduced, reduction reasonably tracks to a form of mixing]).
Regarding claim 12, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 11.
Jungmaier further discloses:
wherein the mixing includes at least one of masking and muting the extracted speech signals ([0105] a microphone array is used to form a spatial filter which can extract a signal from a specific direction and reduce the contamination of the signals from other directions, i.e. a source of interest may be selected while minimizing undesired interfering signals which may also be called beamforming, [Reducing contamination from other signals indicates that they are also separated/extracted in order to be appropriately reduced . Reducing the contribution/contamination of a signal is equivalent to muting that signal. Further, the examiner would like to note that due to the disjunctive nature of the claim, a masking operation does not requires a mapping]).
Regarding claim 13, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 1.
Jungmaier further discloses:
wherein the extracting the speech signals from the target area includes suppressing all speech signals from outside the target area ([0069] the microphone system 40 is configured to generate the audio beam for a person 50 as a function of at least the direction of the person relative to the apparatus only if the detected distance of the person 50 relative to the apparatus 10 is inferior or equal to the predetermine threshold distance TH.sub.D., [Determining not to generate a beam for audio beyond the threshold distance indicates that speech signals outside the target area, i.e. the circle formed by the threshold radius, are suppressed. Determining not to collect noise from signals effectively suppresses those signals in view of the microphone array receiving audio]).
Regarding claim 14, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 13.
Jungmaier further discloses: wherein the extracting the speech signals from the target area further includes suppressing at least one of interfering speech signals and background noise from the speech signals within the target area ([0062] a noise cancellation process of the audio inputs received via the audio beam 100 may be performed based on the audio inputs received by the audio beam and audio inputs received by another beam not defined as a function of the direction of the person, and which will therefore pickup background noise relative to the considered person, [0105] The generation of the audio beams per se may be implemented through any known process. In generating the audio beam, a microphone array is used to form a spatial filter which can extract a signal from a specific direction and reduce the contamination of the signals from other directions, [Reducing the effect of sounds from different beams on the main beam, wherein those other beams are representative of background noise, indicates suppression of interfering speech signals and/or background noise as in the context of a multi-person speaking environment an interfering noise signal is reasonably understood to also be background noise. Further, the examiner would like to note that due to the disjunctive nature of the claim, both elements do not require a mapping]).
Regarding claim 15, Jungmaier discloses: a system for extracting speech signals from a coverage area ([Fig. 1, Coverage Area 80, People 50], [0105] extract a signal from a specific direction [Extracting signals corresponding to the locations of people indicates those signals to be speech]), the system comprising:
one or more computing systems ([0146] general components and functionality that may be used to implement portions of the embodiment radar and audio system and/or an external computer), each of the computing system having:
a processor ([Fig. 14, CPU 1502]);
a storage in communication with the processor ([Fig. 14, Memory 1504 connected to processor 1502 through bus 1508]); and,
a plurality of audio input devices ([Fig. 1, Microphone system 40], [Wherein that microphone system is defined to be an array, i.e. plurality of devices, (see [0020])]),
the one or more computing systems being configured to:
receive speech signals from one or more audio sources in a plurality of audio sources within the coverage area ([Fig. 1, Facility 20], [0019] detect commands issued by persons 50 located in the facility 20 [In view of Fig. 1 wherein the persons 50 would clearly be producing speech signals, i.e. commands, and the facility is within the coverage area 80]); and,
generate a time-frequency representation of the speech signals from the at least one audio input device ([0106] In frequency domain beamforming the microphone signal is e.g. separated into narrowband frequency bins using a short time Fourier transform (STFT) and the data in each bin is processed separately, [A fourier transform representing frequency components over intervals of time indicates the transform to be a time-frequency representation]).
Jungmaier does not disclose:
define, by a machine learning model during an inference operation and based on the time-frequency representation, a target area that is a portion of the coverage area, the target area being definable by a trained criterion.
Gao discloses:
define, by a machine learning model during an inference operation ([0140] The term “audio source feature classification model” refers to a data construct that describes defined operations of a machine learning model that is configured to generate one or more isolate source audio categories based on isolate source audio features, [0146] The audio source feature classification model 604 may be trained using isolate source audio features as input, where inferred output is the desired classifications, and where ground truth data is the actual classifications. In various examples, the model is trained to minimize error between the ground truth data and the inferred output, [Inferred output indicates an inference operation to generate said output, wherein this can be extended to the speech separation as feature classification is related to sound source separation in order to know what to separate]) and based on the time-frequency representation ([In view of the previously disclosed time-frequency representation generated in Gao]), a target area that is a portion of the coverage area ([0030] Improved multi-modal audio source channelization systems are configured to improve processing speed of denoising, echo removal, source separating, source localizing, beamforming operations, and/or to reduce the computational resources associated with applying machine learning models to such tasks, [Wherein the beamforming operation of Gao is dependent upon a target area of interest ([0065]), indicating the target area, i.e. beam(s), determination to be performed using machine learning models as part of the beamforming and/or denoising operations of Gao, wherein the examiner asserts that audio classification (using the audio source feature classification model) is relevant for determining where to direct a beam]), the target area being definable by a trained criterion ([0065] Each identified target audio source (e.g., a target speaker) or area of interest within a video signal sample may be associated with one or more speaker source features, which are aspects or characteristics of the target audio source or area of interest. Like isolate source audio features discussed above in connection with the audio feature extraction model, speaker source features are used as input for a multi-modal audio source channelization model to generate a source separated channel audio sample, [0082] Multi-modal audio source channelization models discussed herein may be generative adversarial networks (GANs). An example GAN may receive multiple inputs during training including features (e.g., isolate source audio features, speaker source features, isolate source spatial input data), the original audio (e.g., multi-source audio signal sample) and video (e.g., video signal sample) that was used to create those features, and other real or actual audio (e.g., target audio source) as ground truth data for a discriminator to use to estimate the accuracy of generated audio, [Identifying target audio sources with associated areas of interest (as disclosed in [0065]) using speaker source features, wherein the model is later trained using speaker source features (as disclosed in [0082]) indicates a training criterion to learn how to determine target areas]).
Jungmaier and Gao are considered analogous art within speech source separation/cancellation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier to incorporate the teachings of Gao, because of the novel way to improve multi-modal audio source channelization system through the reduction of computational resources required for these tasks by using machine learning models to perform said tasks (Gao, [0030]).
Jungmaier further discloses:
redefine, during the inference operation ([In view of the previously disclosed inference operation of Gao]), the target area into a redefined target area ([0059] as a function at least of the direction of the persons determined based on the radar system 30, the microphone system 40 steers one (or more) audio beam 100 in the corresponding direction so as to enhance the reception of the audio inputs generated by the person, [Steering a beam in a new direction (in the situation of a non-constant person direction function) indicates that new direction to be a new, redefined target area]).
Jungmaier in view of Gao does not disclose:
the redefining including
applying a phase shift to the time-frequency representation of the speech signals during the inference operation; and
updating, in a time-frequency domain, phase parameters applied to the speech signals for the at least one audio input device based on updated sound-source information for the coverage area, the redefining reusing the time-frequency representation so as to avoid an additional inference event by the machine learning model such that negligible additional computational cost is incurred.
Kitazawa discloses:
the redefining including
applying a phase shift to the time-frequency representation of the speech signals during the inference operation ([Fig. 1, applying Phase Regulator 1060 to an output signal from FFT Unit 1040], [Col. 4, Lines 10-15] a spectrogram obtained by performing time-frequency conversion on the input signal for each channel is output to the phase regulator 1060, [Col. 4, Lines 65-67] when phase regulation is performed for each frame, the phase difference between the channels does not change with time any longer, [Col. 5, Lines 1-10] When a plurality of sound sources exist, phase regulation is performed on each sound source… when sound sources A and B exist, a signal obtained by correcting the relative position change of sound source A and a signal obtained by correcting the relative position change of sound source B are generated, [Phase regulation which maintains a constant phase difference between sources through time indicates a shift to phases to keep them constant with relation to each other as determined through the relative positions. “Correcting the relative position change of sound source[s]” is a phase shift]); and
updating, in a time-frequency domain, phase parameters applied to the speech signals for the at least one audio input device based on updated sound-source information for the coverage area ([Col. 4, Lines 34-40] The detected relative positional relationship between the sound pickup unit 1010 and sound source is output to the phase regulator 1060. The relative positional relationship herein mentioned is, for example, the direction (angle) of a sound source with respect to the sound pickup unit 1010. The phase regulator 1060 performs phase regulation on the input frequency spectrum), the redefining reusing the time-frequency representation so as to avoid an additional inference event by the machine learning model such that negligible additional computational cost is incurred ([As disclosed in the cited section above, regulating the phase of a frequency spectrum (indicating the spectrum has an innate phase prior to regulation) tracks to the regulation being a form of updating phase parameters applied to speech signals, wherein the relative positional relationship of a pickup and output tracks to sound-source information. Further, as there is only one spectrum being analyzed, this indicated “reusing” the time-frequency representation for regulation of the phase of the spectrum]).
Jungmaier, Gao, and Kitazawa are considered analogous art within sound source separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier in view of Gao to incorporate the teachings of Kitazawa, because of the novel way to perform sound source separation even when the relative positions of a sound source and sound pickup device change through use of a phase regulator which regulated phases of received sound signals in accordance with determined position changes of transmitters/receivers, improving upon traditional sound source separation methods which cannot function in variable environments (Kitazawa, [Col. 2, Lines 40-67], [Col. 3, Lines 1-10]).
Jungmaier further discloses:
extract the speech signals from the target area using the time-frequency representation and the redefined target area ([0105] a microphone array is used to form a spatial filter which can extract a signal from a specific direction and reduce the contamination of the signals from other directions, [Wherein the beam in the specific direction is the redefined target area in view of the previously disclosed steering, i.e. redefining, beams of Jungmaier]); and
transmit the speech signals to a receiver ([0020] The systems 30, 40 and the processing module 70 are operatively coupled together, [0040] each subsystem 90 includes an integrated circuit 92, one or more transmit antennas 94, one or more receive antennas 96, and a substrate 98. In some embodiments, the one or more transmit antennas 94 are arranged in a transmit antenna array, and/or the one or more receive antennas 96 are arranged in a receive antenna array, [Defining the speech subsystems to be containing receivers and transmitters, wherein those speech subsystems are connected to the processing module 70 (see Fig. 2), indicating a transmission of the speech signals gathered by the subsystems to the processing module, i.e. a receiver]).
Regarding claim 16, Jungmaier in view of Gao, further in view of Kitazawa discloses: the system of claim 15.
Gao further discloses:
wherein the target area is defined via the machine learning model ([0030] Improved multi-modal audio source channelization systems are configured to improve processing speed of denoising, echo removal, source separating, source localizing, beamforming operations, and/or to reduce the computational resources associated with applying machine learning models to such tasks, [Wherein the beamforming operation of Gao is dependent upon a target area of interest ([0065]), indicating the target area determination to be performed using machine learning models as part of the beamforming and/or denoising operations of Gao]).
Regarding claim 19, Jungmaier discloses: a storage device for storing computer-readable instructions that, when executed by at least on processor ([0146] CPU 1502 executes instructions in an executable program stored, for example in a non-transitory computer readable storage medium), cause the at least on processor to:
receive speech signals from one or more audio sources in a plurality of audio sources within the coverage area ([Fig. 1, Facility 20], [0019] detect commands issued by persons 50 located in the facility 20 [In view of Fig. 1 wherein the persons 50 would clearly be producing speech signals, i.e. commands, and the facility is within the coverage area 80]); and,
generate a time-frequency representation of the speech signals from the at least one audio input device ([0106] In frequency domain beamforming the microphone signal is e.g. separated into narrowband frequency bins using a short time Fourier transform (STFT) and the data in each bin is processed separately, [A fourier transform representing frequency components over intervals of time indicates the transform to be a time-frequency representation]).
Jungmaier does not disclose:
define, by a machine learning model during an inference operation and based on the time-frequency representation, a target area that is a portion of the coverage area, the target area being definable by a trained criterion.
Gao discloses:
define, by a machine learning model during an inference operation ([0140] The term “audio source feature classification model” refers to a data construct that describes defined operations of a machine learning model that is configured to generate one or more isolate source audio categories based on isolate source audio features, [0146] The audio source feature classification model 604 may be trained using isolate source audio features as input, where inferred output is the desired classifications, and where ground truth data is the actual classifications. In various examples, the model is trained to minimize error between the ground truth data and the inferred output, [Inferred output indicates an inference operation to generate said output, wherein this can be extended to the speech separation as feature classification is related to sound source separation in order to know what to separate]) and based on the time-frequency representation ([In view of the previously disclosed time-frequency representation generated in Gao]), a target area that is a portion of the coverage area ([0030] Improved multi-modal audio source channelization systems are configured to improve processing speed of denoising, echo removal, source separating, source localizing, beamforming operations, and/or to reduce the computational resources associated with applying machine learning models to such tasks, [Wherein the beamforming operation of Gao is dependent upon a target area of interest ([0065]), indicating the target area, i.e. beam(s), determination to be performed using machine learning models as part of the beamforming and/or denoising operations of Gao, wherein the examiner asserts that audio classification (using the audio source feature classification model) is relevant for determining where to direct a beam]), the target area being definable by a trained criterion ([0065] Each identified target audio source (e.g., a target speaker) or area of interest within a video signal sample may be associated with one or more speaker source features, which are aspects or characteristics of the target audio source or area of interest. Like isolate source audio features discussed above in connection with the audio feature extraction model, speaker source features are used as input for a multi-modal audio source channelization model to generate a source separated channel audio sample, [0082] Multi-modal audio source channelization models discussed herein may be generative adversarial networks (GANs). An example GAN may receive multiple inputs during training including features (e.g., isolate source audio features, speaker source features, isolate source spatial input data), the original audio (e.g., multi-source audio signal sample) and video (e.g., video signal sample) that was used to create those features, and other real or actual audio (e.g., target audio source) as ground truth data for a discriminator to use to estimate the accuracy of generated audio, [Identifying target audio sources with associated areas of interest (as disclosed in [0065]) using speaker source features, wherein the model is later trained using speaker source features (as disclosed in [0082]) indicates a training criterion to learn how to determine target areas]).
Jungmaier and Gao are considered analogous art within speech source separation/cancellation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier to incorporate the teachings of Gao, because of the novel way to improve multi-modal audio source channelization system through the reduction of computational resources required for these tasks by using machine learning models to perform said tasks (Gao, [0030]).
Jungmaier further discloses:
redefine, during the inference operation ([In view of the previously disclosed inference operation of Gao]), the target area into a redefined target area ([0059] as a function at least of the direction of the persons determined based on the radar system 30, the microphone system 40 steers one (or more) audio beam 100 in the corresponding direction so as to enhance the reception of the audio inputs generated by the person, [Steering a beam in a new direction (in the situation of a non-constant person direction function) indicates that new direction to be a new, redefined target area]).
Jungmaier in view of Gao does not disclose:
the redefining including
applying a phase shift to the time-frequency representation of the speech signals during the inference operation; and
updating, in a time-frequency domain, phase parameters applied to the speech signals for the at least one audio input device based on updated sound-source information for the coverage area, the redefining reusing the time-frequency representation so as to avoid an additional inference event by the machine learning model such that negligible additional computational cost is incurred.
Kitazawa discloses:
the redefining including
applying a phase shift to the time-frequency representation of the speech signals during the inference operation ([Fig. 1, applying Phase Regulator 1060 to an output signal from FFT Unit 1040], [Col. 4, Lines 10-15] a spectrogram obtained by performing time-frequency conversion on the input signal for each channel is output to the phase regulator 1060, [Col. 4, Lines 65-67] when phase regulation is performed for each frame, the phase difference between the channels does not change with time any longer, [Col. 5, Lines 1-10] When a plurality of sound sources exist, phase regulation is performed on each sound source… when sound sources A and B exist, a signal obtained by correcting the relative position change of sound source A and a signal obtained by correcting the relative position change of sound source B are generated, [Phase regulation which maintains a constant phase difference between sources through time indicates a shift to phases to keep them constant with relation to each other as determined through the relative positions. “Correcting the relative position change of sound source[s]” is a phase shift]); and
updating, in a time-frequency domain, phase parameters applied to the speech signals for the at least one audio input device based on updated sound-source information for the coverage area ([Col. 4, Lines 34-40] The detected relative positional relationship between the sound pickup unit 1010 and sound source is output to the phase regulator 1060. The relative positional relationship herein mentioned is, for example, the direction (angle) of a sound source with respect to the sound pickup unit 1010. The phase regulator 1060 performs phase regulation on the input frequency spectrum), the redefining reusing the time-frequency representation so as to avoid an additional inference event by the machine learning model such that negligible additional computational cost is incurred ([As disclosed in the cited section above, regulating the phase of a frequency spectrum (indicating the spectrum has an innate phase prior to regulation) tracks to the regulation being a form of updating phase parameters applied to speech signals, wherein the relative positional relationship of a pickup and output tracks to sound-source information. Further, as there is only one spectrum being analyzed, this indicated “reusing” the time-frequency representation for regulation of the phase of the spectrum]).
Jungmaier, Gao, and Kitazawa are considered analogous art within sound source separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier in view of Gao to incorporate the teachings of Kitazawa, because of the novel way to perform sound source separation even when the relative positions of a sound source and sound pickup device change through use of a phase regulator which regulated phases of received sound signals in accordance with determined position changes of transmitters/receivers, improving upon traditional sound source separation methods which cannot function in variable environments (Kitazawa, [Col. 2, Lines 40-67], [Col. 3, Lines 1-10]).
Jungmaier further discloses:
extract the speech signals from the target area using the time-frequency representation and the redefined target area ([0105] a microphone array is used to form a spatial filter which can extract a signal from a specific direction and reduce the contamination of the signals from other directions, [Wherein the beam in the specific direction is the redefined target area in view of the previously disclosed steering, i.e. redefining, beams of Jungmaier]); and
transmit the speech signals to a receiver ([0020] The systems 30, 40 and the processing module 70 are operatively coupled together, [0040] each subsystem 90 includes an integrated circuit 92, one or more transmit antennas 94, one or more receive antennas 96, and a substrate 98. In some embodiments, the one or more transmit antennas 94 are arranged in a transmit antenna array, and/or the one or more receive antennas 96 are arranged in a receive antenna array, [Defining the speech subsystems to be containing receivers and transmitters, wherein those speech subsystems are connected to the processing module 70 (see Fig. 2), indicating a transmission of the speech signals gathered by the subsystems to the processing module, i.e. a receiver]).
Claim(s) 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jungmaier in view of Gao, further in view of Kitazawa, further in view of Ying et al. (US-20230333205-A1), hereinafter Ying.
Regarding claim 4, Jungmaier in view of Gao, further in view of Kitazawa discloses: the method of claim 3.
Jungmaier in view of Gao, further in view of Kitazawa does not disclose:
wherein the dimensions are defined in polar coordinates such that the target area is in a form of a circular sector.
Ying discloses:
wherein the dimensions are defined in polar coordinates such that the target area is in a form of a circular sector ([Fig. 3], [Fig. 5], [0116] In the H-plane, a location of the radar is an origin, an x-axis is a polar axis, and coordinates of the object in the plane may be indicated as (r.sub.1, α), where a indicates the azimuth. In the E-plane, a location of the radar may be an origin, a z-axis is a polar axis, and coordinates of the object may be indicated as (r.sub.2, β), where β indicates the pitch angle, [In view of the target area clearly being a circular sector as shown in Fig. 5]).
Jungmaier, Gao, Kitazawa, and Ying are considered analogous art within sound source localization. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier in view of Gao, further in view of Kitazawa to incorporate the teachings of Ying, because of the novel way to fuse a first angle with an incident angle to determine the location of a sound source relative to a microphone array regardless of whether or not the sound source is static or moving, improving the accuracy of voice data extraction (Ying, [0006]).
Regarding claim 5, Jungmaier in view of Gao, further in view of Kitazawa, further in view of Ying discloses: the method of claim 4.
Jungmaier further discloses:
wherein the receiving the speech signals from the one or more audio sources in the plurality of audio sources within the coverage area is performed via the at least one audio input device ([0057] the microphone system 40 is configured to generate one or more audio beam 100 (FIG. 9A) via each of which the microphone system is configured to receive audio inputs. These audio inputs are adapted to include the vocal commands mentioned above).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jungmaier in view of Gao, further in view of Kitazawa, further in view of Poornachandran et al. (US-20160192073-A1), hereinafter Poornachandran.
Regarding claim 18, Jungmaier in view of Gao, further in view of Kitazawa discloses: the system of claim 15.
Jungmaier further discloses:
wherein the target area is defined as a circular sector centered about the plurality of audio input devices ([Fig. 1, Facility 20, Microphone System 40]).
Jungmaier in view of Gao, further in view of Kitazawa does not disclose:
wherein the phase shift is based on a phase difference between each audio input devices in the plurality of audio input devices.
Poornachandran discloses:
wherein the phase shift is based on a phase difference between each audio input devices in the plurality of audio input devices ([0015] the directionality of the microphone array 110 can be adjusted by shifting the phase of the received audio signals and then adding the audio signals together, [Determining phase shifts in order for the signals to be added indicates each phase shift is based on a phase difference between audio devices in order to align the phases for the adding operation]), the phase difference being based on both an incident angle of each of the audio input devices in the plurality of audio input devices and a distance therebetween ([0015] processing the audio signals in such a way as to amplify certain components of the audio signal based on the relative position of the corresponding sound source… Processing the audio signals in this way creates a directional audio pattern such sounds received from some angles are more amplified compared to sounds received from other angles… Control over the directionality of the microphone array 110 will be determined, at least in part, by the number of microphones and their spatial arrangement on the electronic device 100, [Control of the microphone directionality, performed through phase shifting as previously disclosed, being dependent upon the spatial arrangement of microphones indicates an incident angle and distance are relevant parameters for phase shifting as they are both disclosed as factors for determining contribution/amplification to be applied to signals]).
Jungmaier, Gao, Kitazawa, and Poornachandran are considered analogous art within speech source localization. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier in view of Gao, further in view of Kitazawa to incorporate the teachings of Poornachandran, because of the novel way to continuously monitor background noise, filtering irrelevant noises to improve the quality of monitored background noise for notifications or other “noise” sounds that the wearer will want to hear (Poornachandran, [0002]).
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jungmaier in view of Gao, further in view of Kitazawa, further in view of Markovich-Golan et al. (US-10096328-B1), hereinafter Markovich-Golan.
Regarding claim 20, Jungmaier in view of Gao, further in view of Kitazawa discloses: the storage device of claim 19.
Jungmaier further discloses:
wherein the speech signals are acquired by a plurality of audio input devices ([0057] the microphone system 40 is configured to generate one or more audio beam 100 (FIG. 9A) via each of which the microphone system is configured to receive audio inputs. These audio inputs are adapted to include the vocal commands mentioned above).
Jungmaier in view of Gao, further in view of Kitazawa does not disclose:
wherein the target area is adjusted by performing the phase shift on the speech signals using a short-time Fourier transform representation of the plurality of audio input devices.
Markovich-Golan discloses:
wherein the target area is adjusted by performing the phase shift on the speech signals using a short-time Fourier transform representation of the plurality of audio input devices ([Col. 3, Lines 15-25] These beamforming filters scale and phase shift the signals from each of the microphones. Beamformer circuit 108 is configured to apply those weights to the signals received from each of the microphones, to generate a signal y(k) which is an estimate of the speech signal s(k) through the steered beam 120. The application of beamforming weights has the effect of focusing the array 106 on the current position of the speech source 102 and reducing the impacts of the noise sources 104. The signal estimate y(k) is transformed back to the time-domain using an inverse short time Fourier transform (ISTFT), [Col. 3, Lines 35-40] audio signals received from the microphones are transformed to the short time Fourier transform (STFT), [Transforming “back” into the time-domain using an ISTFT indicates a required STFT transformation before performing the operations above. Estimating a signal through a steered beam indicates the steering is a target area adjustment]).
Jungmaier, Gao, Kitazawa, and Markovich-Golan are considered analogous art within sound source localization. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Jungmaier in view of Gao, further in view of Kitazawa to incorporate the teachings of Markovich-Golan, because of the novel way to apply beamforming weights to microphone arrays, enabling a beam to be steered to follow a moving speech source, resulting in improved quality of received speech in the presence of noise (Markovich-Golan, [Col. 1, Lines 55-67]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Manuselis et al. (US-20140188455-A1) discloses “this disclosure provides systems and methods to facilitate real-time language translation between two speakers. A system may include an audio beamforming microphone configured to detect speech in a first language from a first speaker and to detect speech in a second language from a second speaker; a language translation module configured to translate the speech in the first language to text in the second language and to translate the speech in the second language to text in the first language; and a display element configured with a first display element side and a second display element side and further configured to display the text in the first language on the first display element side and to display the text in the second language on the second display element side.” (abstract), Consider the phase shifting described in [0019].
Wang et al. (US-20220182756-A1) discloses “Audio content is controllably directed to or from an aircraft occupant, such as the pilot or a passenger using a microphone array of at least two transducers, positioned onboard an aircraft to pick up utterances of the aircraft occupant. An electronically steerable transducer array coupled to an onboard an aircraft audio system, such as an avionics communication system or in-flight entertainment system supplies audio content to the aircraft occupant. A signal processor coupled to the microphone array processes utterances sensed by the transducers using a time of arrival algorithm to determine an utterance originating direction. A controller circuit then supplies a pointing direction control signal to the steerable beamforming speaker array or microphone array based on the utterance originating direction to cause the steerable transducer array to direct the audio content in the direction of the sensed utterance and thereby direct the audio content to or from the aircraft occupant” (abstract). See discussion phase shifting in the context of beam steering of [0037].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
