DETAILED ACTION
This office action is in response to Applicant’s Amendment/Request for Reconsideration, received on 04/15/2026. Claims 1 and 5 have been amended. Claims 12-15 have been added. Claims 1, 3, 5-10, 12-15 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119
(a)-(d). The certified copy has been filed for the parent Application No. KR10-2020-0102500, filed on 08/14/2020.
Response to Arguments
Applicant’s arguments, see pgs. 8-9, filed 04/15/2026, with respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 (Kaszczuk in view of Sung) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Pollet et al. (US-20190108830-A1), hereinafter Pollet. Pollet discloses “Techniques for performing multi-style speech synthesis. The techniques include using at least one computer hardware processor to perform: obtaining input comprising text and an identification of a desired speaking style to use in rendering the text as speech; identifying a plurality of speech segments for use in rendering the text as speech, the identifying comprising identifying a first speech segment recorded and/or synthesized in a first speaking style that is different from the desired speaking style based at least in part on a measure of similarity between the desired speaking style and the first speaking style; synthesizing speech from the text in the desired speaking style at least in part by using the first speech segment; and outputting the synthesized speech” (abstract). See updated rejections below.
Applicant’s arguments, see pgs. 10-11, filed 04/15/2026, with respect to the rejection(s) of claim(s) 5 under 35 U.S.C. 103 (Kaszczuk in view of Sung) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Pollet et al. (US-20190108830-A1), hereinafter Pollet. Pollet discloses “Techniques for performing multi-style speech synthesis. The techniques include using at least one computer hardware processor to perform: obtaining input comprising text and an identification of a desired speaking style to use in rendering the text as speech; identifying a plurality of speech segments for use in rendering the text as speech, the identifying comprising identifying a first speech segment recorded and/or synthesized in a first speaking style that is different from the desired speaking style based at least in part on a measure of similarity between the desired speaking style and the first speaking style; synthesizing speech from the text in the desired speaking style at least in part by using the first speech segment; and outputting the synthesized speech” (abstract). See updated rejections below.
Applicant’s arguments with respect to newly added claim(s) 12-15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk et al. (US-9978359-B1), hereinafter Kaszczuk, in view of Sung et al. (US-20200234693-A1), hereinafter Sung, further in view of Pollet et al. (US-20190108830-A1), hereinafter Pollet.
Regarding claim 1, Kaszczuk discloses: a method for performing a synthetic speech generation operation on text ([Col. 1, Lines 9-11] In such text-to-speech (TTS) systems, a device converts text into an acoustic waveform that is recognizable as speech corresponding to the input text [TTS tracks to a method of speech synthesis]), comprising:
receiving, from a first terminal of a first user ([Col. 2, Lines 3-4] device 106 receives text 108 for TTS processing… [Col. 3, Lines 60-65] textual data may originate from an internal component of the TTS device 202 or may be received by the TTS device 202 from an input device such as a keyboard [Receiving text from an input device such as a keyboard indicates a user to enter that text]), a plurality of sentences ([Col. 3, Lines 65-67] The text may be in the form of sentences including text, numbers, and/or punctuation for conversion by the TTS module 214 into speech); and,
receiving, from the first terminal of the first user ([In view of the previously disclosed first terminal and user]), a plurality of speech style characteristics for the plurality of sentences ([Col. 4, Lines 1-3] The input text may also include special annotations for processing by the TTS module 214 to indicate how particular text is to be pronounced when spoken aloud [Information regarding pronunciation tracks to speech style characteristics]).
selecting, by the processor of the information processing system ([Fig. 2, controller/processor 208], [Wherein the system the processor is used in is for TTS, i.e. information processing]), at least one sentence associated with the at least one synthetic speech as an inspection target from among the plurality of sentences based on a result of analyzing at least one of the plurality of speech style characteristics or the plurality of synthetic speeches ([Fig. 7B, Speech Waveform 704], [Col. 10, Lines 24-28] allow a user to edit the preliminary TTS results by adjusting acoustic features of the preliminary TTS results, by substituting phonetic units in the preliminary TTS results, or by making other edits… As the user feedback is taken as an input to the TTS system, the second round of TTS results is likely to be more desirable than the preliminary TTS results [In view of the previously disclosed input being comprised of multiple sentences, indicating selection of at least one sentence in view of the user interface of Fig. 7B, i.e. sentence “Hello”, wherein that is “selected” and sent to a user from a system based on results of a preliminary TTS operation, wherein generation of a speech waveform resembling input text indicates a required analysis of speech style characteristics, i.e. the selection is “based on” a required first waveform generation which inherently will require speech style analysis. This method could be extended to more than one speech without a change in functionality]);
instructing, by the processor of the information processing system ([In view of the previously disclosed processor of an information processing system of Kaszczuk]), a second terminal of a second user to output ([Fig. 1, User 102, Display 110], [In view of the input text being received by device 106 through a different medium, indicating the user 102 is distinct, i.e. second, from a user with a keyboard entering original text as previously disclosed, further indicating that keyboard input system is distinct from the display 110 on device 106 meaning a second terminal/user could be implemented here without a change in functionality to Kaszczuk. Disclosing “input device(s)”, [Fig. 2 206], indicates multiple terminals with associated users]), on a screen of the second terminal ([Fig. 7B, 700b], [Col. 13, Lines 40-45] if the user selects portions of TTS results to edit, the user interface then displays a screen 700b showing the text 702 corresponding to the selected portion of TTS results to edit), a visual representation indicating the inspection target in an area corresponding to the selected at least one sentence ([Col. 11, Lines 8-10] The user may then select a portion of the TTS results to edit. The user interface may then display the selected results to be altered in a separate screen, 700 [In view of the previously disclosed input text containing multiple sentences indicating inspection of at least one sentence, i.e. “Hello”, a waveform tracks to a visual representation of the inspection target]); and,
receiving, from the second terminal of the second user ([In view of the previously disclosed second terminal and user of Kaszczuk]), a response to at least one of the plurality of synthetic speeches including a request to change at least one speech style characteristic corresponding to the at least one sentence ([Fig. 6, 608], [Fig. 7B, Select Characteristic 706b], [Col. 10, Lines 37-39] to receive feedback from a user on those results, and to incorporate that feedback to alter the speech synthesis results [Selecting/changing emotional characteristics of input text indicates a response to at least one synthetic speech 704 including a request to change a speech style characteristic, i.e. tone, received by TTS system at step 608 of Fig. 6]).
Kaszczuk does not disclose:
inputting, by a processor of an information processing system, the plurality of sentences and the plurality of speech style characteristics into an artificial neural network text-to-speech synthesis model, so as to generate a plurality of synthetic speeches for the plurality of sentences that reflect the plurality of speech style characteristics; and,
Sung discloses:
inputting, by a processor of an information processing system ([In view of the previously disclosed processor of an information processing system of Kaszczuk]), the plurality of sentences and the plurality of speech style characteristics into an artificial neural network text-to-speech synthesis model ([0250] Here, the artificial intelligence model may be an artificial intelligence model learned using at least one of artificial intelligence algorithms such as machine learning, neural networks, genes, deep learning, and classification algorithms, and in particular, may include at least one artificial neural network, [In view of Fig. 5A speech synthesizer]), so as to generate a plurality of synthetic speeches for the plurality of sentences that reflect the plurality of speech style characteristics ([0065] In addition, the same text may be output as a ‘calm style’ speech and a ‘friendly style’ speech, respectively [Generating multiple styles for the same sentence tracks to generation of sentences reflecting a plurality of speech style characteristics in view of the TTS synthesis model disclosed in Kaszczuk indicating that this method can be applied to multiple sentences simultaneously with no change in functionality]).
Kaszczuk and Sung are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk to incorporate the teachings of Sung, because of the novel way to synthesize a plurality of response sentence or feelings (Sung, [0005]).
Kaszczuk further discloses:
determining whether the plurality of speech characteristics are revealed by the plurality of synthetic speeches ([Col. 11, Lines 50-59] the user may provide audio feedback to the TTS device to provide the TTS device with an example of how certain selected portions of the preliminary TTS results should sound, similar to a line reading in an acting context. Thus the user may speak the input text with the same stresses, mannerisms, and other characteristics the user wishes the TTS device to emulate in the voice of the synthesized speech, [Providing feedback based on how TTS results, i.e. synthesized speech, should sound indicates a user determination that the plurality of speech characteristics are not revealed by the synthetic speeches; therefore, should a user have not feedback, i.e. accepting a preliminary TTS results (see [Col. 11, Lines 17-20]), this is indicative that the speech characteristics are revealed, i.e. the amount of feedback tracks to a measure of “revealed-ness”, as determined by the user]).
Kaszczuk in view of Sung does not disclose:
determining whether an emotion characteristic of at least one synthetic speech of the plurality of synthetic speeches is different from emotion characteristics of synthetic speeches adjacent to the at least one synthetic speech;
determining whether a sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech; and,
selecting, by the processor of the information processing system, at least one sentence associated with the at least one synthetic speech as an inspection target from among the plurality of sentences in response to determining that the emotion characteristic of the at least one synthetic speeches of the plurality of synthetic speeches is different from the emotion characteristics of synthetic speeches adjacent to the at least one synthetic speech or determining that the sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech.
Pollet discloses:
determining whether an emotion characteristic of at least one synthetic speech of the plurality of synthetic speeches is different from emotion characteristics of synthetic speeches adjacent to the at least one synthetic speech ([Fig. 2B, Speech Segments in Styles A-C 216a-c], [0021] the extent to which acoustic and/or prosodic characteristics of a speech segment having one style match those of another style may be obtained based on a measure of similarity between speech segments of different speaking styles. Similarity between speech segments of speaking styles may be estimated, [Wherein storing all speech style segments on a server 210 ([0028]) indicates the segments to be adjacent with respect to the server space]);
determining whether a sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech ([0073] results of applying a phoneme recognizer (or any suitable ASR technique) to a contiguous speech segment sequence may be processed using one or more rules to identify phonetic anomalies, as described in more detail below. As yet another example, a contiguous sequence of one or more speech segments having a phonetic anomaly may be identified by performing forced alignment of transcriptions of TTS speech data to the speech data, [0079] process 500 proceeds to act 506, where output of the phoneme recognizer is used to identify phonetic anomalies in one or more contiguous speech segment sequences. This may be done in any suitable way based on the output of the recognizer and the phonetic transcription of the TTS speech data. For example, contiguous sequences of speech segments that were incorrectly recognized may be identified as containing phonetic anomalies, [Identifying a phonetic anomaly based on a forced alignment of transcriptions of TTS speech to speech data indicates the phonetic anomaly to be a difference between sentence input and synthetic speech output associated with the speech]); and,
selecting, by the processor of the information processing system ([Considering the previously disclosed processor of Kaszczuk]), at least one sentence associated with the at least one synthetic speech as an inspection target from among the plurality of sentences in response to determining that the emotion characteristic of the at least one synthetic speeches of the plurality of synthetic speeches is different from the emotion characteristics of synthetic speeches adjacent to the at least one synthetic speech ([The examiner would like to note that due to the disjunctive construction of the claim, this element does not require a mapping]) or determining that the sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech ([0081] Next, process 500 proceeds to act 510, where the calculated anomaly scores are stored for subsequent use in speech synthesis, after which process 500 completes. As described above, the calculated anomaly scores may be used to modulate the way in which a TTS system selects speech segments for use in generating speech so that the TTS system avoids concatenating a contiguous speech segment sequence having a phonetic anomaly with non-adjacent speech segments, [Indicating selection of sentences, i.e. segments, based on determinations that the segments do not contain phonetic anomalies (which disclose misalignments between ASR output and a transcript as previously disclosed) suggesting the non-used segments to be selected as ‘inspection targets’ and not used. The examiner respectfully asserts that an inspection target is not currently defined to be anything other than a sentence in view of the GUI of Kaszczuk which displays sentences as inspection targets]).
Kaszczuk, Sung, and Pollet are considered analogous art within multi-style speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung to incorporate the teachings of Pollet, because of the novel way to adapt existing speech segments in one style to another style based on similarities between the styles of speaking, reducing the cost of implementing TTS systems configured to perform multi-style synthesis (Pollet, [0018]).
Regarding claim 10, Kaszczuk in view of Sung, further in view of Pollet discloses: the method according to claim 1.
Kaszczuk further discloses:
A non-transitory computer-readable recording medium storing instructions that, when executed by one or more processors, cause performance of the method according to claim 1 ([Col. 14, Lines 30-35] Aspects of the present disclosure may be implemented as a computer implemented method, a system, or as an article of manufacture such as a memory device or non-transitory computer readable storage medium [In view of the rejection of claim 1, see above]).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk in view of Sung, further in view of Pollet, further in view of Kang (US-20140088970-A1).
Regarding claim 3, Kaszczuk in view of Sung, further in view of Pollet discloses: the method according to claim 1.
Kaszczuk in view of Sung, further in view of Pollet does not disclose:
wherein the receiving the response to the at least one of the plurality of synthetic speeches further includes receiving a request to change at least one sentence associated with the at least one synthetic speech.
Kang discloses:
wherein the receiving the response to the at least one of the plurality of synthetic speeches further includes receiving a request to change at least one sentence associated with the at least one synthetic speech ([Fig. 7], [0028] FIG. 7 illustrates a method through which the user inputs an editing command for the selected range of the text, [0037] The editing command may correspond to one of a copy command, a cut command, an edit command, a transmit command and a search command for the selected range of the text [An edit command tracks to a request to change at least one sentence associated with the synthetic speech output from the Kaszczuk in view of Sung, further in view of Pollet synthesizing system]).
Kaszczuk, Sung, Pollet, and Kang are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet to incorporate the teachings of Kang, because of the novel way to incorporate a multi-modal environment allowing for users to edit text from synthesized output through an intuitive interface for increased accessibility (Kang, [0021]).
Sung further discloses:
inputting the changed at least one speech style characteristic and the changed at least one sentence into the artificial neural network text-to- speech synthesis model ([0250] Here, the artificial intelligence model may be an artificial intelligence model learned using at least one of artificial intelligence algorithms such as machine learning, neural networks, genes, deep learning, and classification algorithms, and in particular, may include at least one artificial neural network, [In view of the style editing of Kaszczuk, the sentence editing of Kang, and the text-to-speech synthesis model of Kaszczuk in view of Sung]), so as to generate at least one synthetic speech for the changed at least one sentence that reflects the changed at least one speech style characteristic ([Fig. 4B], [Fig. 4B indicates a scenario where two responses are generated varying in both wording and prosody, i.e. speech style characteristics, shown as ‘slowly with high tone’, in view of the speech style characteristic tracking/collection of Kaszczuk, further in view of the text editing of Kang]).
Claim(s) 5-7, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk in view of Sung, further in view of Pollet, further in view of Czuczman et al. (US-9679554-B1), hereinafter Czuczman.
Regarding claim 5, Kaszczuk discloses: a method for performing a synthetic speech generation operation on text ([Col. 1, Lines 9-11] In such text-to-speech (TTS) systems, a device converts text into an acoustic waveform that is recognizable as speech corresponding to the input text [TTS tracks to a method of speech synthesis]), comprising:
receiving, from a first terminal of a first user ([Col. 2, Lines 3-4] device 106 receives text 108 for TTS processing… [Col. 3, Lines 60-65] textual data may originate from an internal component of the TTS device 202 or may be received by the TTS device 202 from an input device such as a keyboard [Receiving text from an input device such as a keyboard indicates a user to enter that text]), a plurality of sentences ([Col. 3, Lines 65-67] The text may be in the form of sentences including text, numbers, and/or punctuation for conversion by the TTS module 214 into speech);
receiving, from the first terminal of the first user ([In view of the previously disclosed first terminal and user]), a plurality of speech style characteristics for the plurality of sentences ([Col. 4, Lines 1-3] The input text may also include special annotations for processing by the TTS module 214 to indicate how particular text is to be pronounced when spoken aloud [Information regarding pronunciation tracks to speech style characteristics]);
instructing, by the processor of the information processing system ([In view of the previously disclosed processor of an information processing system of Kaszczuk]), a second terminal of a second user to output ([Fig. 1, User 102, Display 110], [In view of the input text being received by device 106 through a different medium, indicating the user 102 is distinct, i.e. second, from a user with a keyboard entering original text as previously disclosed, further indicating that keyboard input system is distinct from the display 110 on device 106 meaning a second terminal/user could be implemented here without a change in functionality to Kaszczuk. Disclosing “input device(s)”, [Fig. 2 206], indicates multiple terminals with associated users]), on a screen of the second terminal ([Fig. 7B, 700b], [Col. 13, Lines 40-45] if the user selects portions of TTS results to edit, the user interface then displays a screen 700b showing the text 702 corresponding to the selected portion of TTS results to edit), a visual representation indicating the inspection target in an area corresponding to the selected at least one sentence ([Col. 11, Lines 8-10] The user may then select a portion of the TTS results to edit. The user interface may then display the selected results to be altered in a separate screen, 700 [In view of the previously disclosed input text containing multiple sentences indicating inspection of at least one sentence, i.e. “Hello”, a waveform tracks to a visual representation of the inspection target]); and,
receiving, from the second terminal of the second user ([In view of the previously disclosed second terminal and user of Kaszczuk]), a response to at least one of the plurality of synthetic speeches including a request to change at least one speech style characteristic corresponding to the at least one sentence ([Fig. 6, 608], [Fig. 7B, Select Characteristic 706b], [Col. 10, Lines 37-39] to receive feedback from a user on those results, and to incorporate that feedback to alter the speech synthesis results [Selecting/changing emotional characteristics of input text indicates a response to at least one synthetic speech 704 including a request to change a speech style characteristic, i.e. tone, received by TTS system at step 608 of Fig. 6]).
Kaszczuk does not disclose:
inputting, by a processor of an information processing system, the plurality of sentences and the plurality of speech style characteristics into an artificial neural network text-to-speech synthesis model, so as to generate a plurality of synthetic speeches for the plurality of sentences that reflect the plurality of speech style characteristics.
Sung discloses:
inputting, by a processor of an information processing system ([In view of the previously disclosed processor of an information processing system of Kaszczuk]), the plurality of sentences and the plurality of speech style characteristics into an artificial neural network text-to-speech synthesis model ([0250] Here, the artificial intelligence model may be an artificial intelligence model learned using at least one of artificial intelligence algorithms such as machine learning, neural networks, genes, deep learning, and classification algorithms, and in particular, may include at least one artificial neural network, [In view of Fig. 5A speech synthesizer]), so as to generate a plurality of synthetic speeches for the plurality of sentences that reflect the plurality of speech style characteristics ([0065] In addition, the same text may be output as a ‘calm style’ speech and a ‘friendly style’ speech, respectively [Generating multiple styles for the same sentence tracks to generation of sentences reflecting a plurality of speech style characteristics in view of the TTS synthesis model disclosed in Kaszczuk indicating that this method can be applied to multiple sentences simultaneously with no change in functionality]).
Kaszczuk and Sung are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk to incorporate the teachings of Sung, because of the novel way to synthesize a plurality of response sentences in differing styles to more accurately respond to a user's situational intent or feelings (Sung, [0005]).
Kaszczuk in view of Sung does not disclose:
wherein the at least one sentence is selected as the inspection target in response to determining that selection of the speech style characteristics for the at least one sentence is made in less than a predetermined time.
Pollet discloses:
wherein the at least one sentence is selected as the inspection target in response to determining that selection of the speech style characteristics for the at least one sentence is made in less than a predetermined time ([0065] the average voice style model and the transformations may be re-estimated a predefined number of times (e.g., the training algorithm is performed using a predefined number of iterations). In this case, it may be determined that the average voice style model and the transformations are to be re-estimated when they have been re-estimated fewer than the predefined number of times, [Determining to re-estimate, i.e. inspect, a feature transformation (comprising speech style characteristics) when they have been re-estimated fewer than a predefined number of times indicates an estimated transform, i.e. speech style characteristic for speech data, taking less than a predetermined time, i.e. that which corresponds to the time it would take for the satisfactory number of re-estimations to be performed]).
Kaszczuk, Sung, and Pollet are considered analogous art within multi-style speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung to incorporate the teachings of Pollet, because of the novel way to adapt existing speech segments in one style to another style based on similarities between the styles of speaking, reducing the cost of implementing TTS systems configured to perform multi-style synthesis (Pollet, [0018]).
Kaszczuk in view of Sung, further in view of Pollet does not disclose:
selecting, by the processor of the information processing system, at least one sentence as an inspection target from among the plurality of sentences by analyzing a behavior pattern of the first user that selects the plurality of speech style characteristics for the plurality of sentences; and,
receiving, from the second terminal of the second user, a request to change at least one speech style characteristic corresponding to the at least one sentence.
Czuczman discloses:
selecting, by the processor of the information processing system ([In view of the previously disclosed processor of an information processing system of Kaszczuk]), at least one sentence as an inspection target from among the plurality of sentences by analyzing a behavior pattern of the first user that selects the plurality of speech style characteristics for the plurality of sentences ([Col. 6, Lines 55-62] Sentences may be prioritized in a number of different ways, for example how many different prosodic units are included in each sentence, the linguistic coverage of the sentence, the length of the sentence, how easy the sentence is to pronounce (which may be represented by a readability score), the number of new units in the sentence not already in the corpus, and/or a number of other factors. The overall priority may be a combination of such factors [Determining priority sentences, i.e. to be sent to proofreaders, based on different prosodic units, tracks to determining inspection sentences based on the behavior pattern of the first user (output from server 112), i.e. the speech style characteristics or prosodic units known in the system at the time of determining priority sentences are used to determine which sentences are different, and therefore have new prosodic units, i.e. behavior patterns, which could be determined via the feature estimations/transformation of Pollet for determining inspection targets as disclosed in Czuczman]); and,
receiving, from the second terminal of the second user ([In view of the second user and terminal of Kaszczuk]), a request to change at least one speech style characteristic corresponding to the at least one sentence ([Col. 8, Lines 25-30] In another aspect, the present system may offer proofreaders the ability to accept or reject text segments and may also offer the ability to edit text segments or to otherwise indicate why a text segment is rejected. For example, a proofreader may mark words that are incorrect, from another language, have confusing pronunciations, etc. [A proofreader, i.e. a second user, indicating reasons why a text segment is rejected tracks to a request to change at least one speech style, i.e. pronunciation, corresponding to the text segment which can be a sentence so the text segment is not rejected]).
Kaszczuk, Sung, Pollet, and Czuczman are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet to incorporate the teachings of Czuczman, because of the novel way to introduce proofreaders to identify appropriately synthesized text and using the approved sentences as templates for other similarly formatted sentence, reducing training time consumption (Czuczman, [Col. 1, Lines 45-60]).
Regarding claim 6, Kaszczuk in view of Sung, further in view of Pollet discloses: the method according to claim 1.
Kaszczuk in view of Sung, further in view of Pollet does not disclose:
wherein the receiving, from the second terminal of the second user, the response to the at least one synthetic speech further includes receiving a marker indicating whether or not to use the at least one synthetic speech, in an area displaying at least one sentence associated with the at least one synthetic speech.
Czuczman discloses:
wherein the receiving, from the second terminal of the second user, the response to the at least one synthetic speech further includes receiving a marker indicating whether or not to use the at least one synthetic speech, in an area displaying at least one sentence associated with the at least one synthetic speech ([Fig. 5], [A second user 10 responding to whether or not speech is accepted, in view of the synthesized speech output of Kaszczuk in view of Sung, further in view of Pollet, indicating whether or not to use that speech, i.e. yes or no, in view of the display of Kaszczuk determining whether input text needs to be edited or accepted. The results from the proofreader of Czuczman could be sent to the display of Kaszczuk with no change in functionality. Consider a situation in which the preliminary TTS results are that sent from the proofreader with the accompanying yes/no]).
Kaszczuk, Sung, Pollet, and Czuczman are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet to incorporate the teachings of Czuczman, because of the novel way to introduce proofreaders to identify appropriately synthesized text and using the approved sentences as templates for other similarly formatted sentence, reducing training time consumption (Czuczman, [Col. 1, Lines 45-60]).
Regarding claim 7, Kaszczuk in view of Sung, further in view of Pollet, further in view of Czuczman discloses: the method according to claim 6.
Czuczman further discloses:
if the marker indicates that the at least one synthetic speech is not to be used, providing information on the at least one sentence associated with the at least one synthetic speech to the first terminal of the first user ([Col. 8, Lines 25-30] In another aspect, the present system may offer proofreaders the ability to accept or reject text segments and may also offer the ability to edit text segments or to otherwise indicate why a text segment is rejected [A rejected text segment tracks to one which should not be used, information sent to server 112 which could then reasonably be sent to the speech synthesis system of Kaszczuk in view of Sung, further in view of Pollet where the first user would be alerted not to use it]).
Regarding claim 15, Kaszczuk in view of Sung, further in view of Pollet, further in view of Czuczman discloses: the method according to claim 5.
Czuczman further discloses:
wherein the at least one sentence is selected as the inspection target in response to determining that selection of the speech style characteristics for the at least one sentence is frequently changed from the first terminal of the first user ([Col. 6, Lines 55-62] Sentences may be prioritized in a number of different ways, for example how many different prosodic units are included in each sentence, the linguistic coverage of the sentence, the length of the sentence, how easy the sentence is to pronounce (which may be represented by a readability score), the number of new units in the sentence not already in the corpus, and/or a number of other factors. The overall priority may be a combination of such factors [Determining priority sentences, i.e. to be sent to proofreaders, based on different prosodic units, tracks to determining inspection sentences based on the behavior pattern of the first user (output from server 112), i.e. the speech style characteristics or prosodic units known in the system at the time of determining priority sentences are used to determine which sentences are different, and therefore have new prosodic units, e.g. behavior patterns describing style characteristic changes, necessarily having a frequency, which could be determined via the feature estimations/transformation of Pollet for determining inspection targets as disclosed in Czuczman]).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk in view of Sung, further in view of Pollet, further in view of Morita et al. (US-20200066250-A1), hereinafter Morita.
Regarding claim 8, Kaszczuk in view of Sung, further in view of Pollet discloses: the method according to claim 1.
Kaszczuk further discloses:
receiving a response for selecting at least one speech style characteristic from among the plurality of speech style characteristic candidates ([Col. 10, Lines 37-39] to receive feedback from a user on those results, and to incorporate that feedback to alter the speech synthesis results [Where feedback in Kaszczuk is given based on synthesized speech 124 having speech style characteristics]).
Kaszczuk in view of Sung, further in view of Pollet does not disclose:
outputting a plurality of speech style characteristic candidates for each of the plurality of sentences.
Morita discloses:
outputting a plurality of speech style characteristic candidates for each of the plurality of sentences ([Fig. 6], [0050] The screen 100 illustrated in FIG. 6 includes a text box 101 for inputting the text information to be subjected to speech synthesis; a pulldown menu 102 for selecting the speaker individuality to be used; slide bars 103a, 103b, and 103c for setting general voice quality parameters [Having a display consisting of a plurality of speaker identities 102, and a plurality of voice settings 103, indicates that the text output when synthesizer button 104 is hit will have a speech style characteristic candidate for a sentence, wherein there is no change to functionality in producing more samples with differing style or words, see store button 105 for this purpose]).
Kaszczuk, Sung, Pollet, and Morita are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet to incorporate the teachings of Morita, because of the novel way to enable creation of a plurality of speaker individualities while keeping track of existing profiles to prevent voice theft for increased security (Morita, [0004]).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk in view of Sung, further in view of Pollet, further in view of Morita, further in view of Knight et al. (US-9788777-B1), hereinafter Knight.
Regarding claim 9, Kaszczuk in view of Sung, further in view of Pollet, further in view of Morita discloses: the method according to claim 8.
Kaszczuk in view of Sung, further in view of Pollet, further in view of Morita does not disclose:
wherein the plurality of speech style characteristic candidates includes a recommended speech style characteristic candidate that is determined based on a result of analyzing the plurality of sentences.
Knight discloses:
wherein the plurality of speech style characteristic candidates includes a recommended speech style characteristic candidate that is determined based on a result of analyzing the plurality of sentences ([Col. 12, 35-45] In some examples, mood and/or emotion-based recommendation is used during a telephone call (e.g., as on-hold music). The music may be recommended by the example recommendation engine 155 based on a mood of a caller (as measured by the tone of voice and/or sensors), a type of caller (e.g., work colleague, family member, friend, etc.), the type of call (e.g., irate customer support, status inquiry, etc.), an environmental context, etc. [Making a recommendation about what kind of on-hold music to play based on the mood of the caller, gathered through tone and other speech style characteristics of previous sentences of the call, tracks to recommending a speech style characteristic, i.e. type of music, based on the context of how the caller is speaking]).
Kaszczuk, Sung, Pollet, Morita, and Knight are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet, further in view of Morita to incorporate the teachings of Knight, because of the novel way to use pre-verbal utterances for building a classification model for increased emotional classification leading to an improved user experience in responses with surrounding tone knowledge (Knight, [Col. 5, Lines 1-15]).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk in view of Sung, further in view of Pollet, further in view of Kaszczuk et al. (US-20140122081-A1), hereinafter Kaszczuk2.
Regarding claim 12, Kaszczuk in view of Sung, further in view of Pollet discloses: the method according to claim 1.
Kaszczuk in view of Sung, further in view of Pollet does not discloses:
determining whether a sound quality of at least one synthetic speech of the plurality of synthetic speeches is lower than a sound quality of remaining synthetic speeches of the plurality of synthetic speeches; and
selecting, by the processor of the information processing system, at least one sentence associated with the at least one synthetic speech as an inspection target from among the plurality of sentences in response to determining that the sound quality of at least one synthetic speech of the plurality of synthetic speeches is lower than the sound quality of remaining synthetic speeches of the plurality of synthetic speeches.
Kaszczuk2 discloses:
determining whether a sound quality of at least one synthetic speech of the plurality of synthetic speeches is lower than a sound quality of remaining synthetic speeches of the plurality of synthetic speeches ([Fig. 3A, Select Test Sentences and Synthesize Speech 308], [Fig. 3B, Quality Score Exceeds Threshold? 320], [0042] the threshold may be based on the quality score that the user or group of users assigned to the recordings of human-read sentences. For example, once the average quality score for synthesized audio representations is greater than 85% of the quality score given to the recordings of human-read sentences, the language or voice may be considered ready for release, [Synthesizing speech for a plurality of sentences indicates each sentence to be a synthetic speech of a plurality. Further, determining whether a sample is below a threshold, wherein other sound samples are defined to surpass the threshold (see Release Voice? operation 326) indicating a situation in which at least one of the plurality of sentences is below the threshold compared to other synthetic speeches, i.e. the remaining sentences]); and
selecting, by the processor of the information processing system ([In view of the previously disclosed processor of Kaszczuk]), at least one sentence associated with the at least one synthetic speech as an inspection target from among the plurality of sentences in response to determining that the sound quality of at least one synthetic speech of the plurality of synthetic speeches is lower than the sound quality of remaining synthetic speeches of the plurality of synthetic speeches ([Fig. 3B, Determine Modification to Address Issue 322], [Determining how to correct a synthesized sentence indicates that sentence to be selected as an inspection target from the plurality of sentences being compared to the threshold]).
Kaszczuk, Sung, Pollet, and Kaszczuk2 are considered analogous art within speech synthesis techniques. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet to incorporate the teachings of Kasczuk2, because of the novel way to determine modifications to language/voice rules and speech units of TTS systems through recursive synthesis of audio based on feedback from testing users, improving the final quality of synthesized output (Kaszczuk, [0013]).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk in view of Sung, further in view of Pollet, further in view of Zhao et al. (US-20140257815-A1), hereiafter Zhao.
Regarding claim 13, Kaszczuk in view of Sung, further in view of Pollet discloses: the method according to claim 1.
Kaszczuk in view of Sung, further in view of Pollet does not disclose:
determining whether a sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech; and
selecting, by the processor of the information processing system, at least one sentence as an inspection target from among the plurality of sentences in response to determining that the sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech.
Zhao discloses:
determining whether a sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech ([0019] Pronunciation issue detector 26 may compare the results determined using acoustic features on signal level by comparing the synthesized speech output from TTS flow and the recordings 242. Using the constrained text may assist in removing errors from the SR engine by adjusting for the mismatch between the recognized text of the synthesized speech and the input text by comparing the similarity of the recognized text between synthesized speech and the corresponding recording); and
selecting, by the processor of the information processing system, at least one sentence as an inspection target from among the plurality of sentences in response to determining that the sentence through speech recognition of at least one synthetic speech of the plurality of synthetic speeches is different from the sentence input from the user that is associated with the at least one synthetic speech ([0013] The pronunciation issue detector 26 evaluates results obtained at multiple levels of the TTS flow and the SR flow (e.g. phone, word, and signal level) by using the corresponding human recordings 104 as the reference for the synthesized speech generated from text 106, and outputs results 108 that list possible pronunciation issues… The pronunciation issue detector uses the different level evaluation results to output possible pronunciation issue candidates as results 108 that may be used by a user to adjust parameters of the TTS engine, [Providing candidate pronunciation issues to a user indicates the candidate results to be inspection targets associated with an incorrectly pronounced (mismatch between text and synthesized speech) sentence (consider text level to be sentence as disclosed in [0020] of Zhao)]).
Kaszczuk, Sung, Pollet, and Zhao are considered analogous art within speech synthesis techniques. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet to incorporate the teachings of Zhao, because of the novel way to evaluate results at multiple levels of TTS flows without the need for human listening tests to evaluate pronunciation errors, reducing the time and cost of training resources required for TTS systems (Zhao, [0001]-[0003]).
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaszczuk in view of Sung, further in view of Pollet, further in view of Czuczman, further in view of Rosen et al. (US-9460703-B2), hereinafter Rosen.
Regarding claim 14, Kaszczuk in view of Sung, further in view of Pollet, further in view of Czuczman discloses: the method according to claim 5.
Kaszczuk in view of Sung, further in view of Pollet, further in view of Czuczman does not disclose:
wherein the at least one sentence is selected as the inspection target in response to determining that the speech style characteristics selected for the at least one sentence is different from recommended speech style characteristics ([Fig. 1, Modification Module 107 based on Input and/or suggestions 101/103], [Col. 8, Lines 50-60] Such a modification muddle may act to modify a directive dispatched by a selection module before passing the directive on to a speech synthesis module. In certain embodiments, the modification could be in accordance with input received from a user, [Receiving user input for how a speech should be modified in view of a selected suggested by selection module 105 indicates the user input to be overriding a recommended, i.e. suggested, style, wherein the synthesized speech from text corresponds to a sentence, i.e. inspection target, in view of the sentence-level analysis of Kaszczuk]).
Kaszczuk, Sung, Pollet, Czuczman, and Rosen are considered analogous art within speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kaszczuk in view of Sung, further in view of Pollet, further in view of Czuczman to incorporate the teachings of Rosen, because of the novel way to take into account the environment where the speech is presented, allowing for the determination of ambient noise presented in signals to be removed before synthesis of a modified speech, improving audibility and/or intelligibility of synthesized speech (Rosen, [Col. 2, Lines 5-20]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yang et al. (US-20210074261-A1) discloses “A method includes generating first synthesized speech by using text and a first emotion vector configured for the text, extracting a second emotion vector included in the first synthesized speech, determining whether correction of the second emotion information vector is needed by comparing a loss value calculated by using the first emotion information vector and the second emotion information vector with a preconfigured threshold, re-performing speech synthesis by using a third emotion information vector generated by correcting the second emotion information vector, and outputting the generated synthesized speech, thereby configuring emotion information of speech in a more effective manner. A speech synthesis apparatus may be associated with an artificial intelligence module, drone (unmanned aerial vehicle, UAV), robot, augmented reality (AR) devices, virtual reality (VR) devices, devices related to 5G services, and the like” (abstract). See entire document.
Rossano et al. (US-20160021334-A1) discloses “A system and method for automatically dubbing a video in a first language into a second language, comprising: an audio/video pre-processor configured to provide separate original audio and video files of the same media; a text analysis unit configured to receive a first text file of the video's subtitles in the first language and a second text file of the video's sub-titles in the second language, and re-divide them into text sentences; a text-to-speech unit configured to receive the text sentences in the first and second languages from the text analysis unit and produce therefrom first and second standard TTS spoken sentences; a prosody unit configured to receive the first and second spoken sentences, the separated audio file and timing parameters and produce therefrom dubbing recommendations; and a dubbing unit configured to receive the second spoken sentence and the recommendations and produce therefrom an automatically dubbed sentence in the second language.” (abstract). See entire document.
Biadsy et al. (US-20220068257-A1) discloses “A method for training a speech conversion model personalized for a target speaker with atypical speech includes obtaining a plurality of transcriptions in a set of spoken training utterances and obtaining a plurality of unspoken training text utterances. Each spoken training utterance is spoken by a target speaker associated with atypical speech and includes a corresponding transcription paired with a corresponding non-synthetic speech representation. The method also includes adapting, using the set of spoken training utterances, a text-to-speech (TTS) model to synthesize speech in a voice of the target speaker and that captures the atypical speech. For each unspoken training text utterance, the method also includes generating, as output from the adapted TTS model, a synthetic speech representation that includes the voice of the target speaker and that captures the atypical speech. The method also includes training the speech conversion model based on the synthetic speech representations.” (abstract). See entire document.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached on (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights