CTNF 18/825,677 CTNF 96407 DETAILED ACTION Claims 1-20 of the instant application are pending and have been examined. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement 06-52 The information disclosure statement (IDS) submitted on 09/05/2024, 02/26/2025, and 11/18/2025 were filed. The submission(s) is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Specification 06-31 AIA The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claim(s) 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. More specifically directed to the abstract idea grouping of: mental processes and mathematical concepts. The independent claim(s) recite(s): 1. A method of processing a first audio using one or more artificial neural networks, the method comprising: extracting a plurality of pieces of first feature data from target information related to a second audio; generating second feature data based on the plurality of pieces of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information; and separating the second audio from the first audio based on the second feature data. 11. An electronic device for processing a first audio using one or more artificial neural networks comprising: one or more processors comprising processing circuitry; and memory storing one or more instructions that, when executed by the one or more processors individually or collectively, cause the electronic device using the one or more artificial neural networks to perform operations comprising: [the limitations as in claim 1, above.] 20. A computer-readable recording medium having stored therein a program that, when executed by a computer, causes the computer to perform a method comprising: [the limitations as in claim 1, above.] This reads on a human (e.g., mentally and/or using pen and paper): observing / evaluating by listening to audio and identifying characteristics (pitch, tempo, rhythm, etc.); performing mathematical calculations and manipulations; and performing mental analysis and judgment by distinguishing between different audio components through human auditory perception and cognition This judicial exception is not integrated into a practical application because for example: claims 1 and 11 recite “one or more artificial neural networks”, claim 11 further recites “an electronic device for processing…,” “one or more processors…,” “processing circuitry…,” “memory…,” while claim 20 recites “a computer-readable recording medium,” and “a computer” . As an example, in ¶ [0131] of the as filed specification, it is disclosed: “…The one or more processors included in the processor 1010 may be general-purpose processors such as CPUs, microprocessor units (MPUs), APs, or DSPs, dedicated graphics processors such as GPUs or VPUs, dedicated artificial intelligence processors such as NPUs, or dedicated communication processors such as communication processors (CPs)…” . Therefore, a general-purpose computer or computing device is described and mainly used as an application thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical idea because it does not impose any meaningful limits on practicing the abstract idea. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using a computer is listed as a general computing device as noted. The claim is not patent eligible. With respect to claims 2 and 12, the claim(s) recite: 2 and 12. […] wherein the plurality of types of the target information comprise at least one of a visual type, an auditory type, or a text type. This reads on a human (e.g., mentally and/or using pen and paper): wherein the identification of characteristics is performed on either visual, auditory or text information. No additional limitations are present. With respect to claims 3 and 13, the claim(s) recite: 3 and 13. […] wherein the generating of the second feature data comprises: separating each of the plurality of pieces of first feature data in a feature space by applying the plurality of feature separation vectors to the plurality of pieces of first feature data. This reads on a human (e.g., mentally and/or using pen and paper): separating characteristics/information into a predetermined/predefined space (e.g., table) No additional limitations are present. With respect to claims 4 and 14, the claim(s) recite: 4 and 14. […] wherein the plurality of feature separation vectors are vectors trained respectively according to a corresponding one of the plurality of types of the target information. This reads on a human (e.g., mentally and/or using pen and paper): where the predetermined/predefined space (e.g., table, rows, etc.) are predefined with specifics with respect to the characteristics/information extracted No additional limitations are present. With respect to claims 5 and 15, the claim(s) recite: 5 and 15. […] wherein the generating of the second feature data comprises: adding, to each of the plurality of pieces of first feature data, a corresponding feature separation vector from among the plurality of feature separation vectors. This reads on a human (e.g., mentally and/or using pen and paper): wherein the performing mathematical manipulations comprise adding to the characteristics/information extracted, the predetermined/predefined space (e.g., table, rows, etc.) No additional limitations are present. With respect to claims 6 and 16, the claim(s) recite: 6 and 16. […] wherein the separating of the second audio from the first audio based on the second feature data comprises: converting the first audio into a spectrogram by applying a short-time Fourier transform to the first audio; extracting first domain feature data of the first audio from the spectrogram; performing a first operation based on the first domain feature data of the first audio, and the second feature data; performing a first self-attention operation on a result of the first operation; extracting second domain feature data of the first audio from the spectrogram; performing a second operation based on the second domain feature data of the first audio, and the second feature data; performing a second self-attention operation on a result of the second operation; and performing one or more cross-attention operations on the result of the first self-attention operation and the result of the second self-attention operation. This reads on a human (e.g., mentally and/or using pen and paper): wherein the performing mental analysis and judgment by distinguishing between different audio components through human auditory perception and cognition comprise: using a predetermined set of (mathematical) steps or rules (e.g., Short-time Fourier transform) to manipulate/transform data to generate a spectrogram (time-frequency analysis); extracting or identifying data from the spectrogram performing another predetermined set of (mathematical) steps or rules; predetermined another set of (mathematical) steps or rules on the result from the previous step; extracting or identifying second data from the spectrogram performing another predetermined set of (mathematical) steps or rules; predetermined another set of (mathematical) steps or rules on the result from the previous step; predetermined another set of (mathematical) steps or rules on the result from the previous step with the results associated with the first set of data and the second set of data. No additional limitations are present. With respect to claims 7 and 17, the claim(s) recite: 7 and 17. […] further comprising: determining whether a first type of the target information is present; based on determining that the first type of the target information is not present, generating a plurality of pieces of third feature data from a second type of the target information corresponding to a previously separated part of the second audio; generating fourth feature data based on the plurality of feature separation vectors and the plurality of pieces of third feature data; and separating the second audio from the first audio based on the fourth feature data. This reads on a human (e.g., mentally and/or using pen and paper): determining if there is a specific type of information present; if the information is not present, using previously available data; generating data using the data in the predetermined/predefined space (e.g., table, rows, etc.); separating information data (from audio spoken by different humans, e.g., different frequencies), based on generated or predefined data above. No additional limitations are present. With respect to claims 8 and 18, the claim(s) recite: 8 and 18. […] wherein the target information comprises a plurality of pieces of target information, and the extracting of the plurality of pieces of first feature data from the target information related to the second audio comprises: determining reliability of each of the plurality of pieces of target information; selecting the target information with a highest reliability from among the plurality of pieces of target information; and extracting the plurality of pieces of first feature data from the target information with the highest reliability. This reads on a human (e.g., mentally and/or using pen and paper): wherein the observing / evaluating by listening to audio and identifying characteristics (pitch, tempo, rhythm, etc.) comprises: determining how reliable data is (i.e., predetermined rules / definition); selecting data with highest reliability (i.e., mathematical concept); extracting data from the information with the highest reliability. No additional limitations are present. With respect to claims 9 and 19, the claim(s) recite: 9 and 19. […] wherein the target information comprises a plurality of pieces of target information, and the extracting of the plurality of pieces of first feature data from the target information related to the second audio further comprises: receiving a user input for selecting the target information; and extracting the plurality of pieces of first feature data from the target information corresponding to the user input from among the plurality of pieces of target information. This reads on a human (e.g., mentally and/or using pen and paper): wherein the observing / evaluating by listening to audio and identifying characteristics (pitch, tempo, rhythm, etc.) comprises: receiving a request from another human and extracting data from the information corresponding to the received request from the other human. No additional limitations are present. With respect to claim 10, the claim(s) recite: 10. The method of claim 1, wherein the one or more artificial neural networks are trained by using a type of target information randomly selected from among the plurality of types of target information. This reads on a human (e.g., mentally and/or using pen and paper): defining the predetermined set of steps or rules used to perform the separation by randomly selecting data. No additional limitations are present. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-02-aia AIA This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. 07-21-aia AIA Claim s 1-7, 10-17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al ("Multi-Modal Multi-Channel Target Speech Separation," in IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 530-541, March 2020, doi: 10.1109/JSTSP.2020.2980956.) and further in view of Rahami et al. ("Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10483-10492, doi: 10.1109/CVPR52688.2022.01024.) . As to independent claim 1, Gu et al. teaches: 1. A method of processing a first audio using one or more artificial neural networks ( see ¶ 1 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “In this work, we address the task of separating the target speaker from a multi-channel speech mixture , by making use of target information from the target speaker’s direction, lip movements and speaker embedding...” and ¶ 2 of C. Video Stream under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “In this work, we try to separate the speech of Mandarin speakers . And due to the concern that there are a few lipreading datasets for Mandarin to train our lip reading network, we investigate the effects of joint training of both video and audio stream from scratch, only using the speech separation objective function (see Section III-F). As shown in Fig. 2 (the middle stream), we follow the work in [10], [21] and take gray frames as the input to the lip reading network. The structure of our lip reading network is similar to the one proposed by[10],which consists of a spatio-temporal convolution layer and a 18-layer ResNet [39], to capture the spatio-temporal dynamics of the lip movements.”) , the method comprising: extracting a plurality of pieces of first feature data from target information related to a second audio ( see Fig. 1 ( Extract target information : directional feature extractor, lip embedding network, speaker embedding network and after the Extract target information : directional feature, lip embedding, speaker embedding ) and ¶ 2 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “Fig. 1, the proposed system is a multi-stream architecture which takes four inputs: (i) noisy multi-channel mixture waveforms, (ii) target speaker’s direction calculated by face detection, (iii) video frames of cropped lip regions, (iv) enrollment audio(s) of the target speaker . The system directly outputs estimated monaural target speech, while all other interfering signals are suppressed.”) ; generating second feature data based on the plurality of pieces of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information ( see Fig. 1 ( after the Extract target information : directional feature, lip embedding, speaker embedding) and Fig. 2 (acoustic embeddings, lip embeddings, speaker embeddings AND Multimodality fusion (which includes all three: acoustic embeddings, lip embeddings, and speaker embeddings)) and ¶ Fig. 2 caption : “Fig. 2. Our proposed multi-modal target speech separation framework with three streams: audio, video and speaker embedding stream. The video steam extracts frame-level lip embeddings from the lip video. The speaker embedding stream processes the enrollment audio(s) of the target speaker and generates an utterance-level speaker embedding . The audio stream takes the noisy multi-channel speech mixture and the target speaker’s direction as input, extracting acoustic embeddings . Then, the multi-modality fusion module will combine these embeddings and feed into proceeding fusion blocks, which outputs a T-F mask for the target speaker. The output of our framework is the estimated target speech waveform.”) ; and separating the second audio from the first audio based on the second feature data ( see Fig. 1 (based on embeddings from after the Extract target information : directional feature, lip embedding, speaker embedding + mixture speech [Wingdings font/0xE0] Multimodal Separation Network [Wingdings font/0xE0] target speech ) and ¶ 2 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION citation as in limitation(s) above, more specifically : “…outputs estimated monaural target speech, while all other interfering signals are suppressed” and ¶ Fig. 2 caption “… multi-modality fusion module will combine these embeddings and feed into proceeding fusion blocks, which outputs a T-F mask for the target speake r. The output of our framework is the estimated target speech waveform .””) . However, Gu et al. does not explicitly teach, but Rahimi et al. does teach: a method of processing a first audio using one or more artificial neural networks ( see Fig. 2 ( Noisy Audio [Wingdings font/0xE0] 1D CNN [Wingdings font/0xE0] Transformer Encoder [Wingdings font/0xE0] 1D CNN [Wingdings font/0xE0] Clean Audio ) and ¶ 1 of 3.1. Architecture : “Audio, Visual, and Text representations. The model has three input streams: one ingesting an audio waveform a ∈ RTa, one the corresponding video input v ∈ R3×Tv×H×W, and one a textual representation s = (s1,s2··· ,sns ) of the sentence being uttered. Similar to [13], we extract a representation of the noisy audio, A ∈ Rta×c, directly from the input waveform a using the encoder part of a U-Net, which consists of 1D convolution layers. We also use a spatio termporal CNN [1] to obtain visual representations, V ∈ Rtv×c…”) Gu et al. and Rahimi et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in speech separation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gu et al. to incorporate the teachings of Rahimi et al. of a method of processing a first audio using one or more artificial neural networks which provides the benefit of achieving improved performance (¶ 5.2. Societal Impact under 5. Discussion of Rahimi et al.). As to independent claim 11, Gu et al. further teaches: 11. An electronic device for processing a first audio using one or more artificial neural networks ( see footnote on page 532 : “Separated samples are presented at https://moplast.github.io/” and ¶ A. Dataset Under IV. EXPERIMENTS PROCEDURES : “The audio-visual corpus used for experiments is collected from Youtube, in which Mandarin accounts for the vast majority. To select relatively high quality videos, a signal-to-noise (SNR) estimator is used to filter out videos with low SNR speech, and a face detection model is used to further remove the videos without the speaker face. After selection, there are about 1,000 speakers and 53,000 clean utterances in total. A mouth region detection program is run on the target speaker’s video to capture the the lip movements. The sampling rate for audio and video are 16 kHz and 25 fps respectively . The multi-talker multi-channel mixtures are simulated with steps in Algorithm 1 . The simulated dataset contains 160,000, 15,000 and 1,200 multi- channel noisy and reverberant mixtures for training, validation and testing …”) comprising: one or more processors comprising processing circuitry ( see ¶ 2 of B. Impact of Different Modalities under V. Results and Analysis : “Also, the real-time factor (RTF) is also reported for computation measurement. The real time factor is defined as the GPU processing time (s) divided by the audio time (s). The RTF result is evaluated on the whole test set and it indicates that whether the model is fast enough for real-time processing .”) ; and memory storing one or more instructions that, when executed by the one or more processors individually or collectively, cause the electronic device using the one or more artificial neural networks to perform operations ( see footnote on page 532 : “Separated samples are presented at https://moplast.github.io/” and ¶ A. Dataset Under IV. EXPERIMENTS PROCEDURES and see ¶ 2 of B. Impact of Different Modalities under V. Results and Analysis citations as in limitations above and further Algorithm 1 : “Data Simulation Process of Audio-Visual Mandarin Dataset.”) comprising: [the limitations as in claim 1, above, taught by Gu et al. in combination with Rahimi et al.]. Rahimi et al. further teaches: An electronic device for processing a first audio using one or more artificial neural networks comprising: … instructions that, … cause the electronic device using the one or more artificial neural networks to perform operations ( see Fig. 2 (Noisy Audio [Wingdings font/0xE0] 1D CNN [Wingdings font/0xE0] Transformer Encoder [Wingdings font/0xE0] 1D CNN [Wingdings font/0xE0] Clean Audio) and ¶ 1 of 3.1. Architecture : “Audio, Visual, and Text representations. The model has three input streams: one ingesting an audio waveform a ∈ RTa, one the corresponding video input v ∈ R3×Tv×H×W, and one a textual representation s = (s1,s2··· ,sns ) of the sentence being uttered. Similar to [13], we extract a representation of the noisy audio, A ∈ Rta×c, directly from the input waveform a using the encoder part of a U-Net, which consists of 1D convolution layers. We also use a spatio termporal CNN [1] to obtain visual representations, V ∈ Rtv×c…” and ¶ 1 of 3. Method : “…We refer the reader to the project webpage for full architectural details.” (https://www.robots.ox.ac.uk/~vgg/research/voiceformer/: “The code for this project will be released soon”)) Gu et al. and Rahimi et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in speech separation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gu et al. to incorporate the teachings of Rahimi et al. of an electronic device for processing a first audio using one or more artificial neural networks comprising: … instructions that, … cause the electronic device using the one or more artificial neural networks to perform operations which provides the benefit of achieving improved performance (¶ 5.2. Societal Impact under 5. Discussion of Rahimi et al.). As to independent claim 20, Gu et al. further teaches: 20. A computer-readable recording medium having stored therein a program that, when executed by a computer ( see footnote on page 532 : “Separated samples are presented at https://moplast.github.io/” and ¶ A. Dataset Under IV. EXPERIMENTS PROCEDURES and see ¶ 2 of B. Impact of Different Modalities under V. Results and Analysis and Algorithm 1 citations as in claim 11, above. ) , causes the computer to perform a method ( see Figs. 1-2 and ¶ 1 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION and ¶ 2 of C. Video Stream under III. MULTI-MODAL MULTI-CHANNEL SEPARATION citations as in claim 1, above. ) comprising: [the limitations as in claim 1, above, taught by Gu et al. in combination with Rahimi et al.]. Regarding claims 2 and 12, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 2 and 12. The method/electronic device of claims 1 and 11, wherein the plurality of types of the target information comprise at least one of a visual type , an auditory type , or a text type ( see ¶ 3 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “In this work, we integrate all the target information into one framework, in order to achieve more superior and robust separation performance under challenging scenarios. As illustrated in Fig. 1, the proposed system is a multi-stream architecture which takes four inputs: (i) noisy multi- channel mixture waveforms, (ii ) target speaker’s direction calculated by face detection , (iii) video frames of cropped lip regions , (iv) enrollment audio(s) of the target speaker.”) . Regarding claims 3 and 13, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 3 and 13. The method/electronic device of claims 1 and 11, wherein the generating of the second feature data ( see Fig. 1 and ¶ Fig. 2 captions as presented in claims 1 and 11, above. ) comprises: separating each of the plurality of pieces of first feature data in a feature space by applying the plurality of feature separation vectors to the plurality of pieces of first feature data ( see ¶ 4-5 of E. Multi-Modality Fusion under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “…we propose to factorize the acoustic embeddings into a set of acoustic subspaces (e.g., phone subspaces, speaker subspaces) and utilize information from other modalities to aggregate them with selective attention. […] Specifically, we take the audio-visual fusion as an example, illustrated in Fig. 5. Firstly, the acoustic embeddings A are factorized into different acoustic subspaces with parallel linear transformations W1 a,W2 a , . . .,WH a , where H is the number of subspaces and the acoustic representation in h-th subspace at the t-th time step is denoted as aht = AtWh a ∈ R1×P , where P is dimension of each subspace. Then, the lip embeddings V are mapped from the D-dimensional space to a H-dimensional space, where each dimension h is expected to contain bias information that corresponds to the h-th acoustic subspace . Next, these mapped lip embeddings are passed to a softmax layer and then produce the estimated posterior for each subspace at each time step, calculated as v = softmax(VWv) = [v1, v2, . . ., vH] ∈ RT×H. Finally, the fused audio-visual embedding (AVE) is obtained by summing up the weighted contribution of different acoustic subspaces:”) . Regarding claims 4 and 14, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 4 and 14. The method/electronic device of claims 1 and 11, wherein the plurality of feature separation vectors are vectors trained respectively according to a corresponding one of the plurality of types of the target information ( see III. MULTI-MODAL MULTI-CHANNEL SEPARATION : ¶ 4 : “ B. Audio Stream : …How to obtain the target speaker’s direction θ. During training, the direction of target speaker θ is known, because the multi-channel audios for training are generated by simulation (see Algorithm 1).” […] ¶ 10-13 : “ C. Video Stream : For the video stream, the majority of previous audio-visual speech separation approaches [9]–[11], [21] adopt the pretraining strategy. Before jointly training with the audio stream, they firstly set a lip reading objective to train the video stream , called lip reading network. The input of the lip reading network can either be a sequence of images of cropped lip regions [10] or the face embedding of the target speaker [9]. The network is trained to estimate the word-level or phone-level posteriors [12], [13]. The supervision information is formed with the speech transcription. In this work, we try to separate the speech of Mandarin speakers. And due to the concern that there are a few lipreading datasets for Mandarin to train our lip reading network, we investigate the effects of joint training of both video and audio stream from scratch , only using the speech separation objective function (see Section III-F). […] To intuitively observe the l earned patterns of lip embeddings through joint training of the audio and video stream , Fig. 3 visualizes the lip embeddings obtained from a sample of the Mandarin-mix dataset.” ¶ 15 : “ D. Speaker Embedding : As discussed in Section II-A, speaker embedding is a kind of bias signal that informs the separation network of the target information and enables target speaker separation. Here, we introduce a pre-trained speaker model and utilize its produced embedding to characterize the target speaker . The speaker model was pre-trained on speaker verification task [41],with 4 convolution layers followed by a fully connected layer.”) . Regarding claims 5 and 15, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 5 and 15. The method/electronic device of claims 1 and 11, wherein the generating of the second feature data ( see Fig. 1 and ¶ Fig. 2 captions as presented in claims 1 and 11, above. ) comprises: adding, to each of the plurality of pieces of first feature data, a corresponding feature separation vector from among the plurality of feature separation vectors ( see ¶ 4-5 of E. Multi-Modality Fusion under III. MULTI-MODAL MULTI-CHANNEL SEPARATION citations as in claims 3 and 13, and further ¶ 5-6 : “…Finally, the fused audio-visual embedding (AVE) is obtained by summing up the weighted contribution of different acoustic subspaces : (Equation) (4), where σ is the sigmoid activation function. As for using factorized attention for acoustic and speaker embedding fusion, the audio-speaker embedding can be calculated by ASEt = σ( H h=1 softmax(stWs)haht ), where Ws is the weight matrix that converts the speaker embedding st from the speaker space to acoustic subspaces. Compared to direct concatenation, the factorized attention sums over all possible speakers or acoustic context guided by cross-modal information . The interaction of embeddings of different modalities in various subspaces enables the deep semantic information capturing and selection .”) . Regarding claims 6 and 16, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 6 and 16. The method/electronic device of claims 1 and 11, wherein the separating of the second audio from the first audio based on the second feature data ( see Fig. 1 and 2 and ¶ 2 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION citations as in claims 1 and 11, above. ) comprises: converting the first audio into a spectrogram by applying a short-time Fourier transform to the first audio ( see Fig. 2 ( STFT conv1d ) and ¶ 1-4 of B. Audio Stream under E. Multi-Modality Fusion under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : ¶ 1 : “ B. Audio Stream : The detailed paradigm of audio stream processing is illustrated as the top stream in Fig. 2. A STFT convolution 1 d layer is used to map the multi-channel mixture waveforms to complex spectrograms. Based on the complex spectrograms, the single-channel spectral feature and multi-channel spatial feature are extracted. Apart from the target speaker independent spectral and spatial features, a directional feature is extracted according to the spatial direction of target speaker.”) ; extracting first domain feature data of the first audio from the spectrogram ( see Fig. 2 ( audio [Wingdings font/0xE0] STFT conv1d [Wingdings font/0xE0] spectrograms (directional feature extractor) [Wingdings font/0xE0] IPD, LPS [Wingdings font/0xE0] concatenation with directional feature (extracted from spectrograms and target speaker direction information) [Wingdings font/0xE0] Audio blocks [Wingdings font/0xE0] acoustic embeddings ) and ¶ 1-4 of B. Audio Stream under E. Multi-Modality Fusion under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : ¶ 2 : “ B. Audio Stream : U-channel raw mixture waveform y, a standard STFT module is used for spectrum analysis . STFT transforms the signal to a complex domain that can be decomposed into magnitude and phase components . Given a window function w with length N, the multi-channel complex spectrogram Y calculated by standard STFT is written as:”) ; performing a first operation based on the first domain feature data of the first audio, and the second feature data ( see Fig. 2 (acoustic embeddings, lip embeddings, and multi-modality fusion) and Fig. 5 ( factorized attention when fusing audio embeddings with video embeddings) and ¶ 1 of B. Audio Stream under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “…All of the features are then concatenated and fed into the audio blocks, which consist of stacked dilated convolutional layers with exponentially growing dilation factors, following [22]. This design supports a long reception field to capture more sufficient contextual information. The output of the audio blocks are the acoustic embeddings A ∈ RT×E, where E is the output convolution channels of the conv1d layers.”) ; performing a first self-attention operation on a result of the first operation ( see Fig. 2 ( audio [Wingdings font/0xE0] STFT conv1d [Wingdings font/0xE0] spectrograms (directional feature extractor) [Wingdings font/0xE0] IPD, LPS [Wingdings font/0xE0] concatenation with directional feature (extracted from spectrograms and target speaker direction information) [Wingdings font/0xE0] Audio blocks [Wingdings font/0xE0] acoustic embeddings ) and Fig. 5 ( factorized attention when fusing audio embeddings A with video embeddings V to obtain the audio-visual embedding AVE) and ¶ 3-4 of E. Multi-modality Fusion under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “2) Factorized Attention: In recent speech recognition work, a factorized layer is proposed [42] for fast adaptation to the acoustic context. In speech recognition literature, a factor characterizes a set of speakers or a specific acoustic environment [43]. The factorized layer uses a set of parameters to process each acoustic class and these parameters are dependent on external factors that represent the acoustic conditions. Inspired by this, we propose to factorize the acoustic embeddings into a set of acoustic subspaces (e.g., phone subspaces, speaker subspaces) and utilize information from other modalities to aggregate them with selective attention …”) ; extracting second domain feature data of the first audio from the spectrogram ( see Fig. 2 ( multi-modality fusion [Wingdings font/0xE0] fusion blocks [Wingdings font/0xE0] target speech mask ) and ¶ 1-2 of F. End-to-End Training under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “The fusion blocks are followed by a 1 × 1−conv layer and a nonlinear activation function (rectified linear Unit (ReLU) in this work), which produces the estimated magnitude mask ∈ RT×F for target speech. Then, the estimated target speech complex spectrogram can be obtained by multiplying the reference channel of mixture complex spectrogramY1 by the estimated mask . Finally, the iSTFT operation is used to convert the estimated target speech spectrogram back to the waveform. To optimize the network from end to end, instead of using a time-domain mean squared error (MSE) loss, the speech separation metric scale-invariant signal-to-distortion (SI-SDR) is used to directly optimize the separation performance, since it has been proven better for speech separation [45].”) ; performing a second operation based on the second domain feature data of the first audio, and the second feature data ( see ¶ 1-2 of B. Impact of Different Modalities under V. Results and Analysis : “After the investigations of modality fusion approaches, in this subsection, we further analyze the impact of different modalities. The aim is to verify each modality, along with the reasonable multi-modality fusion, is effective in multi-channel target speech separation . Table IV reports the performances of target separation models with different modalities input. All the models included LPS and IPDs in input and trained on the whole training dataset. The fusion method for models with more than one modality is chosen according to the best result achieved in Section V-A. Specifically, factorized attention for audio-speaker, factorized attention for audio-visual, and factorized attention + concatenation for multi-modal . Also, the real-time factor (RTF) is also reported for computation measurement. The real time factor is defined as the GPU processing time (s) divided by the audio time (s). The RTF result is evaluated on the whole test set and it indicates that whether the model is fast enough for real-time processing.”) ; performing a second self-attention operation on a result of the second operation ( see ¶ 1 of D. Speaker Embedding under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “…To achieve more discriminative speaker embeddings, self-attention is adopted as the frame-level feature aggregation strategy . The input to the speaker model is an enrollment utterance of the target speaker. The speaker model outputs the utterance-level speaker embedding s ∈ R1×G, where G is the speaker embedding dimension . To match the time steps of the audio stream, the speaker embedding is tiled in time as S = [. . ., st, . . .] ∈ RT×G, where st = s.” and ¶ 7 of E. Multi-Modality Fusion under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “…Following our previous work [27], we compute the attention using the priori knowledge of angle difference between speakers. Specifically, when the angle difference ad between speakers is small , the weight score that applied to spatial and directional features is relatively low, calculated as: att(ad) = 2 ∗ max (σ(ad) − 0.5, 0) (5) where σ(ad) = 1/(1 + exp(−w(ad − b))) is the sigmoid score denotes how much emphasis should be put on spatial features and directional feature, w and b are trainable parameters. Note that the rules can take other factors into consideration, such as the whether the face is sufficiently frontal-facing, etc.”) ; and Rahimi et al. further teaches: performing one or more cross-attention operations on the result of the first self-attention operation and the result of the second self-attention operation ( see ¶ 4 of 3.1. Architecture under 3. Method : “The Transformer bottleneck fuses the three inputs together, allowing for full cross-attention between all modality combinations ..” and ¶ 3 of 4.4 Results under 4. Experiments: “Cross-modal attention. To assess the effectiveness of our design for cross-modal attention through a concatenation of the modalities , we examine the attention in the transformer bottleneck. The attention maps in Figure 3 reveal the correspondence between the audio tokens and the other modalities. attend to the features in the corresponding modalities. This indicates that the model is able to implicitly learn the audio-visual and audio-text alignments and confirms our intuition that it does not require manual alignment or common temporal rates.”) . Gu et al. and Rahimi et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in speech separation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gu et al. to incorporate the teachings of Rahimi et al. of performing one or more cross-attention operations on the result of the first self-attention operation and the result of the second self-attention operation which provides the benefit of achieving improved performance (¶ 5.2. Societal Impact under 5. Discussion of Rahimi et al.). Regarding claims 7 and 17, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 7 and 17. The method/electronic device of claims 1 and 11, further comprising: determining whether a first type of the target information is present ( see ¶ 1-3 of C. Modality Robustness under V. Results and Analysis : “There may be many cases in practical that one of the modalities is unavailable or unreliable. In order to demonstrate the robustness of our multi-modal model in real-world scenarios, we tested it under two particular cases: temporarily missing lip information and estimation error of the target speaker’s direction. 1) Impact of Missing Lip Information: In practical, the lip information may be invalid in many cases. For example, the transmission of high-resolution video may be not stable enough, thus the frames may drop randomly. Moreover, the target speaker may temporarily turn his face away from the camera, or his lip may be obstructed by the microphone. We regard these scenarios as the missing of lip information . When one frame is missing, this absent frame will be filled up with the latest previous frame in our experiment. We compare the performance of multi-modal model, to lip-only and audio-visual model when randomly dropping out 0%, 10%, 20% and 50% frames. Results are presented in Table V. For lip-only model, the dropping of frames have an obvious negative effect on the overall performance. While for models integrated with other complementary modalities, the negative influence is alleviated. Especially for the multi-modal model, the performance decrease is less than 2% when existing 50% frame drops. This confirms the robustness of our multi-modal model to the missing of visual information.”) ; based on determining that the first type of the target information is not present, generating a plurality of pieces of third feature data from a second type of the target information corresponding to a previously separated part of the second audio ( see ¶ 1-3 of C. Modality Robustness under V. Results and Analysis citation as in limitation above, more specifically : “… When one frame is missing, this absent frame will be filled up with the latest previous frame in our experiment…”) ; generating fourth feature data based on the plurality of feature separation vectors and the plurality of pieces of third feature data ( see ¶ 1-3 of C. Modality Robustness under V. Results and Analysis citation as in limitation above, more specifically : “…For lip-only model, the dropping of frames have an obvious negative effect on the overall performance . While for models integrated with other complementary modalities, the negative influence is alleviated. Especially for the multi-modal model, the performance decrease is less than 2% when existing 50% frame drops . This confirms the robustness of our multi-modal model to the missing of visual information.”) ; and separating the second audio from the first audio based on the fourth feature data ( see ¶ 1-3 of C. Modality Robustness under V. Results and Analysis citation as in limitation above, more specifically : “…This confirms the robustness of our multi-modal model to the missing of visual information” and further Table V : SA-SDR (dB) Performances of Target Separation Models with Different Dropout Rates on Lip Video Frames) . Regarding claim 10, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Rahimi et al. further teaches: 10. The method of claim 1, wherein the one or more artificial neural networks are trained by using a type of target information randomly selected from among the plurality of types of target information ( see ¶ 3 of 4.2. Datasets, training & evaluation protocol under 4. Experiments : “ Training sequences . Approximately 23,000 samples from the LRS2 dataset and over 100,000 samples from the LRS3 dataset were used at training time. The samples were mixed together randomly at each training pass leading to the generation of numerous new and unseen examples . The speech signals are mixed together in sequences of 4 seconds for the version of the model trained on audio and video input. The starting point of each sequence was randomly chosen as a data augmentation method .”) . Gu et al. and Rahimi et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in speech separation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gu et al. to incorporate the teachings of Rahimi et al. of wherein the one or more artificial neural networks are trained by using a type of target information randomly selected from among the plurality of types of target information which provides the benefit of achieving improved performance (¶ 5.2. Societal Impact under 5. Discussion of Rahimi et al.) . 07-21-aia AIA Claim s 8 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al ("Multi-Modal Multi-Channel Target Speech Separation," in IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 530-541, March 2020, doi: 10.1109/JSTSP.2020.2980956.) and further in view of Rahami et al. ("Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10483-10492, doi: 10.1109/CVPR52688.2022.01024.) as applied to claims 1 and 11, and further in view of Yamada et al. (US 20120035927 A1) . Regarding claims 8 and 18, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 8 and 18. The method/electronic device of claims 1 and 11, wherein the target information comprises a plurality of pieces of target information ( see Fig. 1 ( Extract target information : directional feature extractor, lip embedding network, speaker embedding network and after the Extract target information : directional feature, lip embedding, speaker embedding ) and ¶ 2 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “Fig. 1, the proposed system is a multi-stream architecture which takes four inputs: (i) noisy multi-channel mixture waveforms, (ii) target speaker’s direction calculated by face detection, (iii) video frames of cropped lip regions, (iv) enrollment audio(s) of the target speaker . The system directly outputs estimated monaural target speech, while all other interfering signals are suppressed.”) , and the extracting of the plurality of pieces of first feature data from the target information related to the second audio ( see Fig. 1 and ¶ 2 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION citations as in limitation above. ) comprises: However, Gu et al. in combination with Rahimi et al. does not explicitly teach, but Yamada et al. does teach: determining reliability of each of the plurality of pieces of target information ( see ¶ [0059] : “In addition, as a main process of the information processing apparatus 100 according to the present disclosure, a user position and a user as a specification process of the user are identified based on input information from a plurality of information input units (camera 21, and microphones 31 to 34). Usages of the identified result are not particularly limited. Various uncertain information is included in the image information and the voice information input from the camera 21 and the plurality of microphones 31 to 34 . In the information processing apparatus 100 according to the present disclosure, a stochastic process is performed with respect to the uncertain information included in the input information , and the information being subjected to the stochastic process is integrated to information estimated to be highly accurate . By this estimation process, robustness is improved to thereby perform analysis with high accuracy.”) ; selecting the target information with a highest reliability from among the plurality of pieces of target information ( see ¶ [0059] citation as in limitation above : “…In the information processing apparatus 100 according to the present disclosure, a stochastic process is performed with respect to the uncertain information included in the input information , and the information being subjected to the stochastic process is integrated to information estimated to be highly accurate .”) ; and extracting the plurality of pieces of first feature data from the target information with the highest reliability ( see ¶ [0059] citation as in limitation above : “…In the information processing apparatus 100 according to the present disclosure, a stochastic process is performed with respect to the uncertain information included in the input information, and the information being subjected to the stochastic process is integrated to information estimated to be highly accurate .”) . Gu et al., Rahimi et al. , and Yamada et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in speech detection/separation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gu et al. in combination with Rahimi et al. to incorporate the teachings of Yamada et al. of determining reliability of each of the plurality of pieces of target information; selecting the target information with a highest reliability from among the plurality of pieces of target information; and extracting the plurality of pieces of first feature data from the target information with the highest reliability which provides the benefit of robustness being improved to thereby perform analysis with high accuracy ([0059] of Yamada et al.) . 07-21-aia AIA Claim s 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gu et al ("Multi-Modal Multi-Channel Target Speech Separation," in IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 530-541, March 2020, doi: 10.1109/JSTSP.2020.2980956.) and further in view of Rahami et al. ("Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 10483-10492, doi: 10.1109/CVPR52688.2022.01024.) as applied to claims 1 and 11, and further in view of Ko et al. (US 20140059169 A1) Regarding claims 9 and 19, Gu et al. in combination with Rahimi et al. teach the limitations as in claims 1 and 11, above. Gu et al. further teaches: 9 and 19. The method/electronic device of claims 1 and 11, wherein the target information comprises a plurality of pieces of target information ( see Fig. 1 ( Extract target information : directional feature extractor, lip embedding network, speaker embedding network and after the Extract target information : directional feature, lip embedding, speaker embedding ) and ¶ 2 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION : “Fig. 1, the proposed system is a multi-stream architecture which takes four inputs: (i) noisy multi-channel mixture waveforms, (ii) target speaker’s direction calculated by face detection, (iii) video frames of cropped lip regions, (iv) enrollment audio(s) of the target speaker . The system directly outputs estimated monaural target speech, while all other interfering signals are suppressed.”) , and the extracting of the plurality of pieces of first feature data from the target information related to the second audio ( see Fig. 1 and ¶ 2 of A. Overview under III. MULTI-MODAL MULTI-CHANNEL SEPARATION citations as in limitation above. ) comprises: However, Gu et al. in combination with Rahimi et al. do not explicitly teach, but Ko et al. does teach: receiving a user input for selecting the target information ( see ¶ [0044] : “The detecting the at least one piece of target information may include selecting at least one display area according to user input information based on the screen displayed on the device.”) ; and extracting the plurality of pieces of first feature data from the target information corresponding to the user input from among the plurality of pieces of target information ( see ¶ [0044] citation as in limitation above and further [0043] : “According to an aspect of another exemplary embodiment, there is provided an information transmission method performed by a device, the method including: detecting at least one piece of target information from a screen displayed on the device ; receiving a first information transmission request signal regarding the detected at least one piece of target information from the outside of the device; and transmitting at least one of the detected at least one piece of target information and metadata of the detected at least one piece of target information to an external input device in response to the received first information transmission request signal.”) . Gu et al., Rahimi et al. , and Ko et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in speech detection/separation. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gu et al. in combination with Rahimi et al. to incorporate the teachings of Ko et al. of receiving a user input for selecting the target information; and extracting the plurality of pieces of first feature data from the target information corresponding to the user input from among the plurality of pieces of target information which provides the benefit of extracting and transmitting target information corresponding to selection by user input ([0013-0014] of Ko et al.) . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Regarding audio feature data extraction or generation (pertinent to claims 1, 11, and 20): Agaskar et al. (US 12136428 B1, ¶ at col. 18, line 59) Bafna et al. (US 11769491 B1, ¶ at col. 5, line 53) Regarding cross-attention associated to audio separation (pertinent to claims 6 and 16): Wang (CN 112071330 A, last paragraph starting on page 8). Regarding user input associated with target information extraction (pertinent to claims 9 and 19): Kong (CN 113885743 A, abstract). Any inquiry concerning this communication or earlier communications from the examiner should be directed to Keisha Y Castillo-Torres whose telephone number is (571)272-3975. The examiner can normally be reached Monday - Friday, 9:00 am - 4:00 pm (EST). Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. Keisha Y. Castillo-Torres Examiner Art Unit 2659 /Keisha Y. Castillo-Torres/Examiner, Art Unit 2659 Application/Control Number: 18/825,677 Page 2 Art Unit: 2659 Application/Control Number: 18/825,677 Page 4 Art Unit: 2659 Application/Control Number: 18/825,677 Page 5 Art Unit: 2659 Application/Control Number: 18/825,677 Page 6 Art Unit: 2659 Application/Control Number: 18/825,677 Page 7 Art Unit: 2659 Application/Control Number: 18/825,677 Page 8 Art Unit: 2659 Application/Control Number: 18/825,677 Page 9 Art Unit: 2659 Application/Control Number: 18/825,677 Page 10 Art Unit: 2659 Application/Control Number: 18/825,677 Page 11 Art Unit: 2659 Application/Control Number: 18/825,677 Page 12 Art Unit: 2659 Application/Control Number: 18/825,677 Page 13 Art Unit: 2659 Application/Control Number: 18/825,677 Page 14 Art Unit: 2659 Application/Control Number: 18/825,677 Page 15 Art Unit: 2659 Application/Control Number: 18/825,677 Page 16 Art Unit: 2659 Application/Control Number: 18/825,677 Page 17 Art Unit: 2659 Application/Control Number: 18/825,677 Page 18 Art Unit: 2659 Application/Control Number: 18/825,677 Page 19 Art Unit: 2659 Application/Control Number: 18/825,677 Page 20 Art Unit: 2659 Application/Control Number: 18/825,677 Page 21 Art Unit: 2659 Application/Control Number: 18/825,677 Page 22 Art Unit: 2659 Application/Control Number: 18/825,677 Page 23 Art Unit: 2659 Application/Control Number: 18/825,677 Page 24 Art Unit: 2659 Application/Control Number: 18/825,677 Page 25 Art Unit: 2659 Application/Control Number: 18/825,677 Page 26 Art Unit: 2659 Application/Control Number: 18/825,677 Page 27 Art Unit: 2659 Application/Control Number: 18/825,677 Page 28 Art Unit: 2659 Application/Control Number: 18/825,677 Page 29 Art Unit: 2659 Application/Control Number: 18/825,677 Page 30 Art Unit: 2659 Application/Control Number: 18/825,677 Page 31 Art Unit: 2659 Application/Control Number: 18/825,677 Page 32 Art Unit: 2659 Application/Control Number: 18/825,677 Page 33 Art Unit: 2659 Application/Control Number: 18/825,677 Page 34 Art Unit: 2659
