DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim(s) 1-8, 10, 11, and 14-23 is/are pending and has/have been examined.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d).
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/07/2024, 10/27/2025, 04/13/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to because of the following informalities: Fig. 4 - elements 603, 604, 605, 606, and 607 are not in the spec.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 2, 5, 7, 8, 14, 17, and 19-21 are objected to because of the following informalities:
Claims 2, 14, and 21 recite “prosody feature information of the target text” in lines 3, 3, and 3, respectively. The Examiner suggests amending the claim(s) to recite --prosody feature information of the target text-- in order to maintain clear antecedent basis.
Claims 5 and 17 recite “respective loss values” and “respective feature prediction networks” in lines 3-4 and 3-4, respectively. The Examiner suggests amending the claim(s) to recite --the respective loss values—and –the respective feature prediction networks--, respectively, in order to maintain clear antecedent basis.
Claims 7, 8, 19, and 20 recite “prosody feature information of the respective text identifications” in lines 4-5, 4-5, 4-5, and 5-6, respectively. The Examiner suggests amending the claim(s) to recite –the prosody feature information of the respective text identifications-- in order to maintain clear antecedent basis.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4-6, 16-18, and 23 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 4, 16, and 23, recite “the respective feature prediction networks” in lines 7, 7, and 7, respectively. There is insufficient antecedent basis for this limitation in the claim.
Claims 5, 6, 17, and 18 are rejected as being dependent upon a rejected base claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim(s) 1, 10, and 11, the limitation(s) of obtaining and determining, as drafted, are processes that, under broadest reasonable interpretation, covers performance of the limitation in the mind and/or with pen and paper but for the recitation of generic computer components, as well as mathematical calculations in prose. More specifically, the mental process of a human reading text written on a piece of paper and using different sets of rules to identify features of the text and the desired vocalization characteristics of the text when spoken aloud. The prosody prediction model, feature extraction network, and feature prediction networks read on sets of learned rules for how a human should perform different parts of the process. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind and/or with pen and paper but for the recitation of generic computer components, as well as mathematical calculations in prose, then it falls within the --Mental Processes— and –Mathematical Concepts-- groupings of abstract ideas. Accordingly, the claim(s) recite(s) an abstract idea.
This judicial exception is not integrated into a practical application because the recitation of a storage medium and processor in claim 10, and an electronic device, storage device, and processing unit of claim 11, reads to generalized computer components, based upon the claim interpretation wherein the structure is interpreted using [0069-80] in the specification. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim(s) is/are directed to an abstract idea.
The claim(s) do(es) not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using generalized computer components to obtain and determine amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim(s) is/are not patent eligible.
With respect to claim(s) 2, 14, and 21, the claim(s) recite(s) converting, determining, inputting, and determining, which reads on a human re-writing the text into a specific format using a reference document, recognizing the new format as the format to be further utilized, using the learned rules to identify a set of vocalization characteristics with a set of values representing a confidence in the characteristics, and using the highest values to select the final vocalization characteristics. No additional limitations are present.
With respect to claim(s) 3, 15, and 22, the claim(s) recite(s) obtaining, inputting, determining, determining, updating, and determining, which reads on a human looking at information for learning how to identify vocalization characteristics for specific text, using the current learned rules to make a prediction, stopping the learning process when the result is better than a specific goal, and continuing the learning process when the result is below a specific goal, where continuing the learning process includes determining an error score using specific input, changing features of the rules based on the score, and continuing the process until the result is better than the specified goal. No additional limitations are present.
With respect to claim(s) 4, 16, and 23, the claim(s) recite(s) determining, performing, and performing, which reads on a human performing a specific series of calculations on a set of error scores in order to determine a final error score for the purposes of learning how to select vocalization characteristics. No additional limitations are present.
With respect to claim(s) 5 and 17, the claim(s) recite(s) determining, determining, and determining, which reads on a human identifying values related to the training data and performing a specific series of calculations using those values to determine a final error score for the purposes of learning how to select vocalization characteristics. No additional limitations are present.
With respect to claim(s) 6 and 18, the claim(s) recite(s) the characteristics of a calculation weight, which reads on characteristics of a value used during the learning process. No additional limitations are present.
With respect to claim(s) 7, 8, 19, and 20, the claim(s) recite(s) determining steps, which reads on a human identifying a specific set of vocalization characteristics to be applied to the text when spoken allowed based on values representing confidence in the characteristics, and for claims 7 and 19, additionally based on the relationships between the different characteristics. No additional limitations are present.
These claims further do not remedy the judicial exception being integrated into a practical application and further fail to include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 10, and 11, is/are rejected under 35 U.S.C. 103 as being unpatentable over Fructuoso et al. (U.S. PG Pub No. 2015/0186359), as found in the IDS, hereinafter Fructuoso, in view of Bae et al. (“Hierarchical Context-Aware Transformers for Non-Autoregressive Text to Speech”, arXiv:2106.15144v1, 29 Jun 2021), hereinafter Bae.
Regarding claims 1, 10, and 11, Fructuoso teaches
(claim 1) A method for prosody prediction ([0006]), the method comprising:
(claim 10) A non-transitory computer readable storage medium having a computer program stored thereon, the computer program, when executed by a processor (a computer program configured to cause the data processing apparatus to perform the actions encoded on computer storage devices [0007]), implements acts comprising:
(claim 11) An electronic device (a system of one or more computers [0007]), comprising:
(claim 11) a storage device having at least one computer program stored thereon (a computer program having instructions encoded on computer storage devices [0007]);
(claim 11) at least one processing unit configured to execute the at least one computer program in the storage device (a computer program configured to cause the data processing apparatus to perform the actions encoded on computer storage devices [0007]), implementing acts comprising:
obtaining a target text to be processed (the computing system obtains text for which synthesized speech should be generated [0030]); and
determining prosody feature information of the target text based on the target text and a pre-trained prosody prediction model, the prosody feature information comprising prosody features corresponding to a plurality of predetermined prosody dimensions (linguistic features are selected by the system and the language identifier of the text are provided to a trained neural network, i.e. based on the target text and a pre-trained prosody prediction model, where the neural network outputs prosody information for the linguistic features, such as duration, energy level, and fundamental frequency coefficients for each of the linguistic features, i.e. determining prosody feature information of the target text…the prosody feature information comprising prosody features corresponding to a plurality of predetermined prosody dimensions [0030-5]);
wherein the prosody prediction model comprises a feature extraction network and a … feature prediction networks, the feature extraction network being configured to extract linguistic information of the target text, the … feature prediction networks each connected to the feature extraction network …, and each of the feature prediction networks being configured to predict, based on the linguistic information extracted by the feature extraction network, a prosody feature corresponding to a predetermined prosody dimension (the system obtains text, accesses a lexicon to identify a sequence of phonetic units, and selecting linguistic features from a phonetic alphabet, i.e. feature extraction network being configured to extract linguistic information of the target text, and the neural network, i.e. feature prediction network, receives the linguistic features, i.e. connected to the feature extraction network, to produce outputs indicating prosody information, such as duration, energy level, and fundamental frequency coefficients, i.e. each of the feature prediction networks being configured to predict…a prosody feature corresponding to a predetermined prosody dimension, for each of the linguistic features, i.e. based on the linguistic information extracted by the feature extraction network [0030-5]).
While Fructuoso provides selecting linguistic features from text and using a neural network to determine prosody information, Fructuoso does not specifically teach the use of a plurality of feature prediction networks corresponding to predetermined prosody dimensions, and thus does not teach
plurality of feature prediction networks each connected to the feature extraction network and being corresponding to the predetermined prosody dimensions, respectively, and each of the feature prediction networks being configured to predict, based on the linguistic information extracted by the feature extraction network, a prosody feature corresponding to a predetermined prosody dimension.
Bae, however, teaches plurality of feature prediction networks each connected to the feature extraction network and being corresponding to the predetermined prosody dimensions, respectively, and each of the feature prediction networks being configured to predict, based on the linguistic information extracted by the feature extraction network, a prosody feature corresponding to a predetermined prosody dimension (a hierarchical context-aware transformer, where an encoder receives a text embedding and processes the embedding to emphasize different portions of the text as a hidden representation sequence, i.e. feature extraction network…linguistic information extracted by the feature extraction network, and the output of the encoder is fed to a pitch predictor and a duration predictor, i.e. plurality of feature prediction networks each connected to the feature extraction network and being corresponding to the predetermined prosody dimensions respectively, which output pitch and duration values, respectively, i.e. each of the feature prediction networks being configured to predict based on the linguistic information…a prosody feature corresponding to a predetermined prosody dimension Fig. 2,(Intro para 5, Sec. 3.1)).
Fructuoso and Bae are analogous art because they are from a similar field of endeavor in identifying prosody for TTS systems. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the selecting linguistic features from text and using a neural network to determine prosody information teachings of Fructuoso with providing the output of a text embedding encoder to respective pitch and duration predictors as taught by Bae. It would have been obvious to combine the references to improve the performance of a TTS model, including improving the pitch modeling accuracy (Bai Abstract).
Claim(s) 2-4, 7, 8, 14-16, and 19-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fructuoso, in view of Bae, and further in view of Phillips et al. (U.S. Patent No. 6,961,704), hereinafter Phillips.
Regarding claims 2, 14, and 21, Fructuoso in view of Bae teaches claims 1, 10, and 11, and Fructuoso further teaches
converting the target text into a text identification sequence based on a plurality of unit texts comprised in the target text and a predetermined mapping table, and determining the text identification sequence as a target identification sequence, wherein the predetermined mapping table indicates a correlation between unit texts and text identifications (the system obtains text, i.e. target text, and accesses a lexicon, i.e. a predetermined mapping table, to identify a sequence of phonetic units in a phonetic representation of the text, i.e. the predetermined mapping table indicates a correlation between unit texts and text identifications, and selecting linguistic features from a phonetic alphabet that includes all possible sounds in all of the languages that the neural network is trained to be used with, i.e. converting the target text into a text identification sequence based on a plurality of unit texts comprised in the target text and a predetermined mapping table, and determining the text identification sequence as a target identification sequence, where a language identifier is also determined [0030-3],[0046-7]).
While Fructuoso in view of Bae provides determining different prosody values, Fructuoso in view of Bae does not specifically teach the use of probabilities, and thus does not teach
inputting the target identification sequence into the prosody prediction model to obtain a first result output by the prosody prediction model, the first result indicating respective probabilities of respective text identifications in the target identification sequence belonging to respective prosody categories in respective predetermined prosody dimensions; and
determining, based on maximum probabilities corresponding to respective text identifications in respective predetermined prosody dimensions in the first result, prosody feature information of the respective text identifications in the target identification sequence, to determine the prosody feature information of the target text.
Phillips, however, teaches inputting the target identification sequence into the prosody prediction model to obtain a first result output by the prosody prediction model, the first result indicating respective probabilities of respective text identifications in the target identification sequence belonging to respective prosody categories in respective predetermined prosody dimensions (the TTS front end takes a text as input and generates a target unit sequence with linguistic target as its output, where the target unit sequence specifies a plurality of phonetic units arranged in an order consistent with the input text, mapped based on a dictionary, i.e. target identification sequence, where the target unit sequence is fed into the linguistic prosody generation mechanism to annotate the target unit sequence to produce a linguistically annotated target sequence, i.e. inputting the target identification sequence into the prosody prediction model/feature extraction, where the unit selection mechanism receives a target unit sequence that is annotated with linguistic prosodic characteristics and selects a sequence of phonetic units according to minimized joint costs computed using linguistic prosodic models, i.e. inputting the target identification sequence into the prosody prediction model to obtain a first result output by the prosody prediction model the first result indicating respective probabilities, where each model is related to a different acoustic feature, such as pitch, energy, and duration, and the cost related to the difference between the desired linguistic prosody and achieved linguistic prosody is calculated based on the selected unit sequence, and where the cost may be based on an estimated probability, i.e. the first result indicating respective probabilities of respective text identifications in the target identification sequence belonging to respective prosody categories in respective predetermined prosody dimensions (3:52-56),(5:65-6:11),(6:23-44),(13:8-56),(14:20-25),(17:17-32)); and
determining, based on maximum probabilities corresponding to respective text identifications in respective predetermined prosody dimensions in the first result, prosody feature information of the respective text identifications in the target identification sequence, to determine the prosody feature information of the target text (the unit selection mechanism selects a sequence of phonetic units according to minimized joint costs computed using linguistic prosodic models, i.e. determining…prosody feature information of the respective text identifications in the target identification sequence to determine the prosody feature information of the target text, where each model is related to a different acoustic feature, such as pitch, energy, and duration, and the cost related to the difference between the desired linguistic prosody and achieved linguistic prosody is calculated based on the selected unit sequence, where the cost may be based on an estimated probability, and the highest probability/lowest cost unit sequence is the one that is selected, i.e. based on maximum probabilities corresponding to respective text identifications in respective predetermined prosody dimensions in the first result (3:66-4:14),(12:35-45),(13:8-65),(14:20-25),(17:17-32)).
Fructuoso, Bae, and Phillips, are analogous art because they are from a similar field of endeavor in identifying prosody for TTS systems. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the determining different prosody values teachings of Fructuoso, as modified by Bae, with the use of probability and minimizing cost for individual prosodic units using feature-specific models as taught by Phillips. It would have been obvious to combine the references to enable maximizing the similarity between the target unit sequence and the selected unit sequence measured in terms of different aspects (Phillips (12:19-45)).
Regarding claims 3, 15, and 22, Fructuoso in view of Bae and Phillips teaches claims 2, 14, and 21, and Fructuoso further teaches
obtaining a plurality of training datasets, wherein each training dataset comprises a training identification sequence and prosody label information corresponding to a training text, the training identification sequence is obtained by converting the training text via the predetermined mapping table, and the prosody label information comprises a prosody feature corresponding to a predetermined prosody dimension (training data sets including various recorded utterances are obtained, i.e. obtaining a plurality of training datasets, including speech data for utterances, a transcription, data identifying the first language, and linguistic features and prosody information determined based on the transcriptions, i.e. each training dataset comprises a training identification sequence and prosody label information corresponding to a training text, where a lexicon is used to identify a sequence of phonetic units to select linguistic features, i.e. the training identification sequence is obtained by converting the training text via the predetermined mapping table, and prosody parameters include duration, energy, and fundamental frequency contour corresponding to each linguistic feature, i.e. prosody label information comprises a prosody feature corresponding to a predetermined prosody dimension [0031],[0067-70],[0078]);
inputting a target training identification sequence in the training identification sequence into the prosody prediction model in a current round of training to obtain a second result output by the prosody prediction model in the current round of training, the second result indicating respective probabilities of each text identification in the target training identification sequence belonging to prosody categories in the respective predetermined prosody dimensions (the neural network is trained using the linguistic features and prosody information, i.e. inputting a target training identification sequence in the training identification sequence into the prosody prediction model in a current round of training, where several training iterations may be performed for each utterance in the set of training data, where training may be ended when the error between the actual prosody parameters and the prosody parameters output by the neural network is below a threshold, i.e. the second result indicating respective probabilities of each text identification in the target training identification sequence belonging to prosody categories in the respective predetermined prosody dimensions, and the target output in each iteration target output of the neural network is the encoded set of prosody parameters that were extracted from the portions of the training utterance, i.e. obtain a second result output by the prosody prediction model in the current round of training [0070-6]);
in response to a training stopping condition being satisfied, determining a prosody prediction model in the current round of training as a trained prosody prediction model (the neural network is trained using the linguistic features and prosody information, where several training iterations may be performed for each utterance in the set of training data, where training may be ended when the error between the actual prosody parameters and the prosody parameters output by the neural network is below a threshold, i.e. in response to a training stopping condition being satisfied, and where a trained neural network is obtained, i.e. determining a prosody prediction model in the current round of training as a trained prosody prediction model [0070-6],[0081]); and
in response to the training stopping condition being dissatisfied, determining a target loss value of the current round of training, updating a parameter of the prosody prediction model in the current round of training using the target loss value, and determining the updated prosody prediction model for use in a next round of training until the training stopping condition is satisfied, wherein the target loss value is determined based on the prosody label information corresponding to the target training identification sequence and the second result (the neural network is trained using the linguistic features and prosody information, where several training iterations may be performed for each utterance in the set of training data, where training may be ended when the error between the actual prosody parameters and the prosody parameters output by the neural network is below a threshold, i.e. in response to the training stopping condition being dissatisfied…until the training stopping condition is satisfied, determining a target loss value of the current round of training, where weights of the neural network are updated to reduce the error values based on the prosody comparison of the actual extracted prosody parameters and the prosody parameters output by the neural network, i.e. determining the updated prosody prediction model for use in a next round of training…the target loss value is determined based on the prosody label information corresponding to the target training identification sequence and the second result [0070-6]).
Where Phillips teaches that the unit selection is based on a calculated probability (13:43-14:25).
And where the motivation to combine is the same as previously presented.
Regarding claims 4, 16, and 23, Fructuoso in view of Bae and Phillips teaches claims 3, 15, and 22, and Phillips further teaches
the second result comprises output content of each of the feature prediction networks in the prosody prediction model in the current round of training (each model, such as pitch, energy, and duration models, may be evaluated individually to determine the difference between the target unit acoustic features and the selected unit, i.e. the second result comprises output content of each of the feature prediction networks in the prosody prediction model (13:24-14:19), where linguistic models are generated through iterative training, i.e. in the current round of training (7:5-18),(9:50-58)); and
wherein determining the target lost value of the current training comprises:
performing respective loss value calculations on respective output contents based on the prosody label information corresponding to the target training identification sequence, to obtain respective loss values corresponding to the respective feature prediction networks (the models for individual acoustic features may be trained to minimize a classification error, i.e. performing respective loss value calculations on respective output contents based on the prosody label information corresponding to the target training identification sequence, where the model identifies how closely a prosodic characteristic of a candidate unit sequence matches with a linguistic target, i.e. obtain respective loss values corresponding to the respective feature prediction networks (11:6-28),(11:52-12:12),(12:35-45)); and
performing weighted summation on the respective loss values corresponding to the respective feature prediction networks based on calculation weights of the respective feature prediction networks, to obtain the target loss value (separate evaluation results may be combined in a meaningful manner to assess the overall discrepancy, i.e. performing –a calculation-- on the respective loss values corresponding to the respective feature prediction networks based on calculation weights of the respective feature prediction networks to obtain the target loss value, such as the cost for each acoustic feature being determined, and where a weighted sum may be used to compute the overall linguistic prosody cost, i.e. performing weighted summation on the respective loss values corresponding to the respective feature prediction networks based on calculation weights of the respective feature prediction networks to obtain the target loss value (10:54-67),(13:24-42),(14:4-19),(17:33-44)).
Where Fructuoso specifically teaches that the difference between the actual prosody parameters extracted from an utterance and the prosody parameters output by the neural network during training is the error, i.e. target loss value, used to determine whether training should continue [0075].
And where the motivation to combine is the same as previously presented.
Regarding claims 7 and 19, Fructuoso in view of Bae and Phillips teaches claims 2 and 14, and Fructuoso further teaches
the predetermined prosody dimension comprises a pitch stress, a phrase stress, and a boundary tone (the system determines which phonetic units are stressed, such as in pitch, manner of articulation, length, and syllabic stress, i.e. phrase stress, and where linguistic groups may be determined on the stress patterns at the beginning and ending of an utterance, i.e. boundary tone [0046-8],[0060]); and…
Where Phillips further teaches for each text identification in the target identification sequence, determining pitch stresses, phrase stresses, and boundary tones corresponding to the prosody feature information of the text identification based on the maximum probabilities corresponding to the text identification in the respective predetermined prosody dimensions, and determining prosody feature information of a break index corresponding to the text identification based on the phrase stresses and the boundary tones corresponding to the prosody feature information of the text identification, and a predetermined correlation between respective phrase stresses, boundary tones and break indexes (the unit selection mechanism selects a sequence of phonetic units according to minimized joint costs computed using linguistic prosodic models, i.e. for each text identification in the target identification sequence determining…corresponding to the prosody feature information of the text identification, where each model is related to a different acoustic feature, such as pitch, energy, and duration, and stress may be identified based on the location of the target unit in a phrase, such as the beginning position, i.e. determining pitch stresses phrase stresses and boundary tones, and the cost related to the difference between the desired linguistic prosody and achieved linguistic prosody is calculated based on the selected unit sequence, where the cost may be based on an estimated probability, and the highest probability/lowest cost unit sequence is the one that is selected, i.e. based on the maximum probabilities corresponding to the text identification in the respective predetermined prosody dimensions, and where joint cost may also consider a syllable, pitch/accent, or phrase position cost, where words at the beginning or end of a phrase will have specific energy and duration and the selected unit should not have a mismatch with the target syllable position, i.e. determining prosody feature information of a break index corresponding to the text identification based on the phrase stresses and the boundary tones corresponding to the prosody feature information of the text identification and a predetermined correlation between respective phrase stresses boundary tones and break indexes (3:51-4:14),(12:35-45),(13:8-65),(14:20-25),(14:65-15:32),(17:17-32)).
And where the motivation to combine is the same as previously presented.
Regarding claims 8 and 20, Fructuoso in view of Bae and Phillips teaches claims 2 and 14, and Phillips further teaches
a predetermined prosody dimension comprises a break index, a pitch stress, a phrase stress, and a boundary tone (for linguistic models, each model is related to a different acoustic feature, such as pitch, energy, and duration, and stress may be identified based on the location of the target unit in a phrase, such as the beginning position, i.e. boundary tone, where phonetic units from similar locations should have similar linguistic prosodic characteristics, i.e. break index (13:42),(13:66-14:19),(14:65-15:32)); and…
for each text identification in the target identification sequence, determining break indexes, pitch stresses, phrase stresses and boundary tones corresponding to the prosody feature information of the text identification based on the maximum probabilities corresponding to the text identification in the respective predetermined prosody dimensions (the unit selection mechanism selects a sequence of phonetic units according to minimized joint costs computed using linguistic prosodic models, i.e. for each text identification in the target identification sequence determining…corresponding to the prosody feature information of the text identification, where each model is related to a different acoustic feature, such as pitch, energy, and duration, and stress may be identified based on the location of the target unit in a phrase, such as the beginning position, where phonetic units from similar locations should have similar linguistic prosodic characteristics, i.e. determining break indexes pitch stresses phrase stresses and boundary tones, and the cost related to the difference between the desired linguistic prosody and achieved linguistic prosody is calculated based on the selected unit sequence, where the cost may be based on an estimated probability, and the highest probability/lowest cost unit sequence is the one that is selected, including where words at the beginning or end of a phrase will have specific energy and duration and the selected unit should not have a mismatch with the target syllable position, i.e. based on the maximum probabilities corresponding to the text identification in the respective predetermined prosody dimensions (3:51-4:14),(12:35-45),(13:8-65),(14:20-25),(14:65-15:32),(17:17-32)).
Where the motivation to combine is the same as previously presented.
Claim(s) 5 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fructuoso, in view of Bae, in view of Phillips, and further in view of Fernando et al. (“Dynamically Weighted Balanced Loss: Class Imbalanced Learning and Confidence Calibration of Deep Neural Networks”, IEEE, January 14, 2021), hereinafter Fernando.
Regarding claims 5 and 17, Fructuoso in view of Bae and Phillips teaches claims 4 and 16, and Phillips further teaches
determining the respective feature prediction networks as respective target feature prediction network (each model, such as pitch, energy, and duration models, may be evaluated individually to determine the difference between the target unit acoustic features and the selected unit, i.e. determining the respective feature prediction networks as respective target feature prediction network (13:24-14:19), where linguistic models are generated through iterative training, i.e. respective target feature prediction network (7:5-18),(9:50-58)), and performing the following:
While Fructuoso in view of Bae and Phillips provides calculating errors for training and weighting cost sums, Fructuoso in view of Bae and Phillips does not specifically teach weighting based on the number of times a prosody dimension appears in a training dataset, and thus does not teach
determining, based on the plurality of training datasets, respective calculation weights corresponding to prosody categories comprised in a target prosody dimension, the target prosody dimension being a predetermined prosody dimension corresponding to a target feature prediction network, and the more times a prosody category appears in the plurality of training datasets, the smaller a calculation weight corresponding to the prosody category is; and
determining a loss value corresponding to a target feature prediction network based on the prosody label information corresponding to the target training identification sequence, the output content of the target feature prediction network and the respective calculation weights corresponding to the respective prosody categories of the target prosody dimensions.
Fernando, however, teaches determining, based on the plurality of training datasets, respective calculation weights corresponding to prosody categories comprised in a target prosody dimension, the target prosody dimension being a predetermined prosody dimension corresponding to a target feature prediction network, and the more times a prosody category appears in the plurality of training datasets, the smaller a calculation weight corresponding to the prosody category is (class weights, i.e. respective calculation weights corresponding to prosody categories comprised in a target prosody dimension, are handled as a hyperparameter that is set proportional to inverse class frequency, i.e. determining…respective calculation weights, where the class frequency is computed over the training data set, such that a misclassification error for a minority class will have a greater penalty than an error for a majority class, i.e. based on the plurality of training datasets…the more times a prosody category appears in the plurality of training datasets the smaller a calculation weight corresponding to the prosody category is Fig. 2,(Sec. IV(A)(2) para 1-4)); and
determining a loss value corresponding to a target feature prediction network based on the prosody label information corresponding to the target training identification sequence, the output content of the target feature prediction network and the respective calculation weights corresponding to the respective prosody categories of the target prosody dimensions (loss functions are calculated for each class, i.e. determining a loss value corresponding to a target feature prediction network, where the neural network is being trained to classify data, i.e. output content of the target feature prediction network, and the loss function includes a class weight set proportional to the inverse class frequency in the training data set, i.e. the respective calculation weights corresponding to the respective prosody categories of the target prosody dimensions Fig. 2,(Sec. IV(A)(2) para 1-4)).
Where Phillips teaches that the linguistic models are trained to minimize a classification error, and each model may be trained according to a specific acoustic feature with labeled training data (7:5-28),(9:50-58),(13:24-14:19).
Fructuoso, Bae, Phillips, and Fernando are analogous art because they are from a similar field of endeavor in training models to accurately classify data. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the calculating errors for training and weighting cost sums teachings of Fructuoso, as modified by Bae and Phillips, with the weighting of loss values based on class frequency in the training data set as taught by Fernando. It would have been obvious to combine the references to improve training of classes to balance the training contributions of minority and majority frequency classes (Fernando (Sec. IV(A)(2))).
Claim(s) 6 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fructuoso, in view of Bae, in view of Phillips, and further in view of Heydari et al. (“SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions”, arXiv:1912.12355v1, 27 Dec 2019), hereinafter Heydari.
Regarding claims 6 and 18, Fructuoso in view of Bae and Phillips teaches claims 4 and 16.
While Fructuoso in view of Bae and Phillips provides calculating errors for training and weighting cost sums, Fructuoso in view of Bae and Phillips does not specifically teach calculating weighting as inversely related to a loss value, and thus does not teach
a calculation weight corresponding to a feature prediction network is inversely related to a loss value corresponding to the feature prediction network.
Heydari, however, teaches a calculation weight corresponding to a feature prediction network is inversely related to a loss value corresponding to the feature prediction network (a tunable hyperparameter can be adapted to assign more weight to the pest performing losses, i.e. a calculation weight…is inversely related to a loss value (Sec. 3.1.1)).
Where Phillips teaches that the linguistic models are trained to minimize a classification error, and each model may be trained according to a specific acoustic feature with labeled training data (7:5-28),(9:50-58),(13:24-14:19).
Fructuoso, Bae, Phillips, and Heydari are analogous art because they are from a similar field of endeavor in training models to accurately classify data. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the calculating errors for training and weighting cost sums teachings of Fructuoso, as modified by Bae and Phillips, with the weighting of loss functions according to performance as taught by Heydari. It would have been obvious to combine the references to better favor the gradient of a function according to its recent performance (Heydari (Sec. 3.1.1)).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICOLE A K SCHMIEDER whose telephone number is (571)270-1474. The examiner can normally be reached 8:00 - 5:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICOLE A K SCHMIEDER/Primary Examiner, Art Unit 2659
Prosecution Insights