DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
As per the instant Application having Application number 18/469,909 the examiner acknowledges the applicant's submission of the amendment dated 06/26/2026. At this point, claims 1, 6, 9, 14, 17, 21, 24, and 29 have been amended. Claims 1-30 are pending.
Response to Arguments
Applicant’s arguments, see Applicant Arguments/Remarks Made in an Amendment, filed 06/26/2026, with respect to the 35 U.S.C. 112(b) rejection of claims 6, 14, 22, and 29 have been fully considered and are persuasive. The 35 U.S.C. 112(b) rejection of claims 6, 14, 22, and 29 has been withdrawn.
Applicant’s arguments, see Applicant Arguments/Remarks Made in an Amendment, filed 06/26/2026, with respect to the 35 U.S.C. 103 rejection of claims 1-30 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al., (US 2022/0277728 A1, hereinafter Zhang) in view of YUAN K. et al., ("Incorporating Convolution Designs into Visual Transformers", 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, 10 October 2021, pp. 559-568., part of the applicant submitted prior art and hereinafter Yuan) and Mihal et al., (US 11,620,495 B1, hereinafter Mihal).
Regarding claims 1, 9, 17, and 24, taking claim 9 as exemplary:
Zhang shows:
“An apparatus, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to: receive, by an artificial neural network (ANN) model, a set of tokens corresponding to an input;” (Paragraph [0076]: “The exemplary neural TTS systems according to the embodiments of the disclosure are described above with reference to FIGS. 2, 11, and 12, and the exemplary methods for generating speech through neural TTS synthesis according to the embodiments of the present disclosure are described accordingly. The above systems and methods intend to generate speech corresponding to a text input based on both a phone feature and context features of the text input.” In paragraph [0079]: “At step 1320, a phone sequence may be identified from the text input, such as a current sentence, by various techniques such as LTS.” In paragraph [0080]: “At step 1330, the phone sequence may be updated by adding a begin token and/or an end token to the beginning and/or the end of the phone sequence, respectively. In an implementation, a mute phone may be used as the begin token and the end token.” In paragraph [0081]: “At step 1340, a phone feature may be generated based on the updated phone sequence. For example, the updated phone sequence may be converted into a phone embedding vector sequence through a phone embedding model. The phone embedding vector sequence may then be provided to an encoder, such as the encoder 110 in FIG. 1, to generate the phone feature corresponding to the updated phone sequence.” In paragraph [0133]: “The apparatus 1800 may comprise at least one processor 1810. The apparatus 1800 may further comprise a memory 1820 connected with the processor 1810. The memory 1820 may store computer-executable instructions that, when executed, cause the processor 1810 to perform any operations of the methods for generating speech through neural TTS synthesis according to the embodiments of the present disclosure as mentioned above.” And in paragraph [0134]: The embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may comprise instructions that, when executed, cause one or more processors to perform any operations of the methods for generating speech through neural TTS synthesis according to the embodiments of the present disclosure as mentioned above.” – the above processor and memory addresses the processor implemented method of claim 1 (dependent claims 2-8), the apparatus including a memory and processor of claim 9 (dependent claims 10-16), the non-transitory computer-readable medium having program code recorded thereon of claim 17 (dependent claims 18-23), and the apparatus means for that includes a processor (according the 112f analysis above) of claim 24 (dependent claims 25-30).)
“process, by a token interaction block of the ANN model, the set of tokens according to each channel of the input to generate a spatial mixture of a set of features for each channel of the input;” (Paragraph [0084]: “as shown in FIG. 11, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to an attention unit to obtain first attended mixed features. At the same time, global features in the context features may be cascaded with the first attended mixed features after passing its own attention unit, thereby obtaining second mixed features. The second mixed features may be provided to a decoder to generate acoustic features corresponding to the second mixed features. The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” In paragraph [0085]: “In yet another implementation, as shown in FIG. 12, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to the attention unit to obtain first attended mixed features. Meanwhile, averaging pooling may be performed on global features in the context features to obtain average global features. The average global features may be cascaded with the first attended mixed features to obtain second mixed features. The second mixed features may be provided to a decoder to generate acoustic features corresponding to the second mixed features. The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.”)
“generate, by a feed forward network (FFN) block of the ANN model, a mixture of channel features based on the spatial mixture of the set of features for each channel of the input;” (Paragraph [0036]: “The decoder 140 may include a pre-net 142 consisted of feed-forward layers, Long Short Term Memories (LSTMs) 144, a linear projection 146, a post-net 148 consisted of convolution layers, etc. The LSTMs 144 may receive an input from the pre-net 142 and provide its output to the linear projection 146 while the processing by the LSTMs 144 is affected by the attention unit 130. The linear projection 146 may provide its output to the pre-net 142 and the post-net 148, respectively. Finally, the output of the post-net 148 is combined with the output of the linear projection 146 to produce the acoustic features 150. In an implementation, the linear projection 146 may also be used to generate stop tokens.” And in paragraph [0041]: “The generated phone feature 210 and context features 218 may be combined into mixed features through a cascading unit 220. An attention unit 222 may apply an attention mechanism on the mixed features, such as a location sensitive attention mechanism. The attended mixed features may be provided to a decoder 224. The decoder 224 may correspond to the decoder 140 in FIG. 1. The decoder 224 may generate, based on the attended mixed features, acoustic features corresponding to the mixed features. The acoustic features may then be provided to a vocoder 226. The vocoder 226 may correspond to the vocoder 160 in FIG. 1. A speech waveform 170 corresponding to the text input 202 may be generated by the vocoder 226.” – The use of different layers to get input and the use of a cascading unit is the use of channels.)
“and generate, by the ANN model, an inference based on the set of attend features of the mixture of channel features.” (Paragraph [0082]: “At step 1350, a speech waveform corresponding to the text input may be generated based on the phone feature and context features. The speech waveform may be generated in a variety of ways.” In paragraph [0083]: “In an implementation, as shown in FIG. 2, the phone feature and the context features may be combined into mixed features, and the mixed features may be provided successively to an attention unit and a decoder to generate acoustic features corresponding to the mixed features. The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” In paragraph [0084]: “In another implementation, as shown in FIG. 11, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to an attention unit to obtain first attended mixed features... The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” And in paragraph [0085]: “In yet another implementation, as shown in FIG. 12, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to the attention unit to obtain first attended mixed features... The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.”)
But Zhang does not appear to explicitly recite “the spatial mixture of the set of features being received by the FFN block as output from the token interaction block; determine, by an attention block of the ANN model, a set of attended features of the mixture of channel features according to a set of attention weights;”
However, Yuan teaches “determine, by an attention block of the ANN model, a set of attended features of the mixture of channel features according to a set of attention weights;” (Page 561, column 3, paragraph 3: “MSA. For a self-attention (SA) module, the sequence of input tokens xt ∈ R(N+1)× C are linear transformed into qkv spaces, i.e., queries Q ∈ R(N+1)×C, keys K ∈ R(N+1)×C and values V ∈ R(N+1)×C. Then a weighted sum over all values in the sequence is computed through: Attention(Q,K,V) = softmax(QKT√C )V (2) And a linear transformation is performed to the weighted values.”)
And Mihal teaches “the spatial mixture of the set of features being received by the FFN block as output from the token interaction block;” (Column 7, line 60-Column 8, line 4: “the neural networks of other embodiments may be types of networks other than feed forward networks (e.g., recurrent networks, regulatory feedback networks, radial basis function networks, aggregation networks, SRRNs, etc.). The illustrated network 100 is a fully-connected network in which each node in a particular layer receives as inputs all of the outputs from the previous layer. However, the neural networks of some embodiments are convolutional neural networks, in which the intermediate layers (referred to as “hidden” layers) may include convolutional layers, pooling layers, fully-connected layers, and normalization layers.” And in column 22, lines 10-22: “during the course of pooling and convolutional operations, the resolution of the input data is reduced. For example, a video frame of size 1000×1000 may be reduced to feature maps of 10×10 size. The tree structure of the aggregation nodes continuously mixes local information from previous feature maps (e.g., high spatial resolution) with global information (e.g., lower spatial resolution). The aggregation nodes thus serve in some embodiments to mix local spatial decisions with global spatial decisions, and keep mixing that back with what was previously computed. Through these aggregation nodes that pass the higher resolution information through to the end of the network, the network is able to output a full resolution output (de-noised) video frame.”)
Zhang, Yuan, and Mihal are analogous in the arts because Zhang, Yuan, and Mihal describe models with feature mixing.
Therefore, it would be obvious to one of ordinary skill in the art at the filing date of the instant application, having the teachings of Zhang, Yuan, and Mihal before him or her, to modify the teachings of Zhang to include the teachings of Yuan in order to support the use of images as a feature input (see Yuan page 561, column 2, paragraph 7) and thereby increase marketability of Zhang; and to further include the teachings of Mihal in order to support passing higher resolution information through the a neural network (see Mihal column 22, lines 19-22) of Zhang and Yuan and thereby increase efficiency.
Regarding claims 2, 10, 18, and 25, taking claim 10 as exemplary:
Zhang, Yuan, and Mihal teach the method, apparatus, non-transitory computer-readable medium, and apparatus of claims 1, 9, 17, and 24 as claimed and specified above.
And Yuan teaches “in which the at least one processor is further configured to process the set of tokens corresponding to the input using a depth-wise convolution to generate the spatial mixture of the set of features.” (Page 562, Paragraph 1: "Figure 3: Illustration of the Locally-enhanced Feed-Forward module. First, patch tokens are projected into a higher dimension. Second, they are restored to “images” in the spatial dimension based on the original positions. Third, a depth-wise convolution is performed on the restored tokens as shown in the yellow region. Then the patch tokens are flattened and projected to the initial dimension. Besides, the class token conducts an identical mapping.")
Regarding claims 3, 11, 19, and 26, taking claim 11 as exemplary:
Zhang, Yuan, and Mihal teach the method, apparatus, non-transitory computer-readable medium, and apparatus of claims 1, 9, 17, and 24 as claimed and specified above.
And Yuan teaches “in which the at least one processor is further configured to process the spatial mixture of the set of features using a point-wise convolution to generate the mixture of channel features.” (Page 561, column 2, paragraphs 6-7: "FFN. FFNperforms point-wise operations, which are applied to each token separately. It consists of two linear transformations with a non-linear activation in between: FFN(x) = σ(xW1 +b1)W2 +b2 (3) where W1 ∈ RC× K is the weight of the first layer, projecting each token into a higher dimension K. And W2 ∈ RK×C is the weight of the second layer. b1 ∈ RK and b2 ∈ RC are the biases. And σ(·) is the non-linear activation of GELU [13] in ViT. Complementary to the MSA module, the FFN module performs dimensional expansion/reduction and non-linear transformation on each token, thereby enhancing the representation ability of tokens. However, the spatial relationship among tokens, which is important in vision, is not considered. This leads that the original ViT needs a mass of training data to learn these inductive biases.")
Regarding claims 4, 12, 20, and 27, taking claim 12 as exemplary:
Zhang, Yuan, and Mihal teach the method, apparatus, non-transitory computer-readable medium, and apparatus of claims 1, 9, 17, and 24 as claimed and specified above.
And Zhang shows “in which the ANN model comprises a vision transformer, a bi-directional encoder representations from transformers (BERT), a robustly optimized BERT approach (RoBERTa)-based transformer, an XLNet-based transformer, Transformer-XL-based transformer, or a generative pre-trained transformer (GPT).” (Paragraph [0048]: “The word embedding model 412 may be based on Natural Language Processing (NLP) techniques, such as Neural Machine Translation (NMT). Both the word embedding model and the neural TTS system have similar sequence-to-sequence encoder-decoder frameworks, this would benefit to network convergence. In one embodiment, a Bidirectional Encoder Representations from Transformers (BERT) model may be employed as the word embedding model. A word embedding vector sequence may be generated based on the word sequence 404 through the word embedding model 412, wherein each word has a corresponding embedding vector, and all of these embedding vectors form the word embedding vector sequence. The word embedding vector contains meaning and semantic context information of a word, which will facilitate improvement of naturalness of a generated speech.” – Note that the claim is written in the alternative and not all claim elements (i.e. a robustly optimized BERT approach (RoBERTa)-based transformer, an XLNet-based transformer, Transformer-XL-based transformer, or a generative pre-trained transformer (GPT)) must be recited for teaching by the reference to be satisfied.)
Regarding claims 5, 13, 21, and 28, taking claim 13 as exemplary:
Zhang, Yuan, and Mihal teach the method, apparatus, non-transitory computer-readable medium, and apparatus of claims 1, 9, 17, and 24 as claimed and specified above.
And Zhang shows “in which the input comprises an image, an audio signal, or a textual input.” (Paragraph [0082]: “At step 1350, a speech waveform corresponding to the text input may be generated based on the phone feature and context features. The speech waveform may be generated in a variety of ways.” And in paragraph [0083]: “In an implementation, as shown in FIG. 2, the phone feature and the context features may be combined into mixed features, and the mixed features may be provided successively to an attention unit and a decoder to generate acoustic features corresponding to the mixed features. The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” In paragraph [0084]: “In another implementation, as shown in FIG. 11, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to an attention unit to obtain first attended mixed features... The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” In paragraph [0085]: “In yet another implementation, as shown in FIG. 12, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to the attention unit to obtain first attended mixed features... The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” – Note that the claim is written in the alternative and not all claim elements (i.e. image input) needs to be recited for teaching by the reference to be satisfied.)
Regarding claims 6, 14, 22, and 29, taking claim 14 as exemplary:
Zhang, Yuan, and Mihal teach the method, apparatus, non-transitory computer-readable medium, and apparatus of claims 5, 13, 21, and 28 as claimed and specified above.
And Zhang shows “in which the at least one processor is further configured to divide ... into a set of patches, the set of tokens corresponding to the set of patches of…” (Paragraph [0086]: “Since mute phones are added as the begin token and/or the end token at the beginning and/or the end of the phone sequence of the text input, respectively, the generated speech waveform corresponding to the text input has a period of mute at the beginning and/or the end, respectively. Since the context features are taken into account when generating the speech waveform, the mute at the beginning and/or the end of the speech waveform is related to the context features, which may vary with the context features. For example, the mute at the end of the speech waveform of the text input and the mute at the beginning of a speech waveform of a next sentence of the text input constitute a pause between the text input and the next sentence. The duration of the pause may accordingly vary with the context features, such that the rhythm of the generated speech corresponding to a set of sentences is more abundant and natural.” – the use of periods is the dividing into patches.)
But Zhang does not appear to explicitly recite the use of “when the input comprises an image… images”.
However, Yuan teaches the use of “when the input comprises an image… images”. (Page 561, column 2, paragraph 7: "To solve the above-mentioned problems in tokenization, we propose a simple but effective module named as Image to-Tokens (I2T) that extracts patches from feature maps instead of raw input images." – the patches from feature maps associated with tokens is the use of dividing into patches associated with images.)
Regarding claims 7 and 15, taking claim 15 as exemplary:
Zhang, Yuan, and Mihal teach the method and apparatus of claims 1 and 9 above.
And Zhang shows “in which the inference is a classification of the input.” (Paragraph [0079]: “step 1320, a phone sequence may be identified from the text input, such as a current sentence, by various techniques such as LTS.” In paragraph [0080]: “At step 1330, the phone sequence may be updated by adding a begin token and/or an end token to the beginning and/or the end of the phone sequence, respectively. In an implementation, a mute phone may be used as the begin token and the end token.” In paragraph [0081]: “At step 1340, a phone feature may be generated based on the updated phone sequence. For example, the updated phone sequence may be converted into a phone embedding vector sequence through a phone embedding model. The phone embedding vector sequence may then be provided to an encoder, such as the encoder 110 in FIG. 1, to generate the phone feature corresponding to the updated phone sequence.” And in paragraph [0082]: “At step 1350, a speech waveform corresponding to the text input may be generated based on the phone feature and context features. The speech waveform may be generated in a variety of ways.” And in paragraph [0083]: “ In an implementation, as shown in FIG. 2, the phone feature and the context features may be combined into mixed features, and the mixed features may be provided successively to an attention unit and a decoder to generate acoustic features corresponding to the mixed features. The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.”)
Regarding claims 8, 16, 23, and 30, taking claim 16 as exemplary:
Zhang, Yuan, and Mihal teach the method, apparatus, non-transitory computer-readable medium, and apparatus of claims 1, 9, 17, and 24 as claimed and specified above.
And Zhang shows “in which the at least one processor is further configured to reshape the mixture of channel features to form a … sequence of mixed features.” (Paragraph [0082]: “At step 1350, a speech waveform corresponding to the text input may be generated based on the phone feature and context features. The speech waveform may be generated in a variety of ways.” And in paragraph [0083]: “In an implementation, as shown in FIG. 2, the phone feature and the context features may be combined into features, and the mixed features mixed may be provided successively to an attention unit and a decoder to generate acoustic features corresponding to the mixed features. The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” In paragraph [0084]: “In another implementation, as shown in FIG. 11, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to an attention unit to obtain first attended mixed features... The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.” In paragraph [0085]: “In yet another implementation, as shown in FIG. 12, the phone feature and the context features may be combined into first mixed features, and the first mixed features may be provided to the attention unit to obtain first attended mixed features... The acoustic features may then be provided to a vocoder to generate the speech waveform corresponding to the text input.”)
And Yuan teaches “one-dimensional” (Page 562, Paragraph 1: "Figure 3: Illustration of the Locally-enhanced Feed-Forward module. First, patch tokens are projected into a higher dimension. Second, they are restored to “images” in the spatial dimension based on the original positions. Third, a depth-wise convolution is performed on the restored tokens as shown in the yellow region. Then the patch tokens are flattened and projected to the initial dimension. Besides, the class token conducts an identical mapping.")
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE D WOOLWINE whose telephone number is (571)272-4138. The examiner can normally be reached M-F 9:30-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA HUANG can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHANE D. WOOLWINE
Primary Examiner
Art Unit 2124
/SHANE D WOOLWINE/Primary Examiner, Art Unit 2124
