DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Terminal Disclaimer
The terminal disclaimer filed on 07/20/2026 disclaiming the terminal portion of any patent granted on this application which would extend beyond the expiration date of 03/19/2034 has been reviewed and is accepted. The terminal disclaimer has been recorded.
Response to Arguments
Applicants’ arguments with respect to claim(s) 1-3, 7, 11, 12 ,17-19 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3,7, 11, 12, 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Suendermann U.S. PAP 2010/0268536 A1.
Regarding claim 1 Deisher teaches a method implemented by one or more processors of a computing device of a user (The present description pertains to speech recognition for connected devices and in particular to speech recognition using both local and remote resources, see par. [0001]), the method comprising:
receiving, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user (a speech utterance is received. This is typically in the form of digitized audio from a local microphone., see par. [0024]);
processing, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device (the system performs a local ASR and at 330 waits for the local ASR to finish. After the local ASR is finished, then this result is applied locally to the appropriate application at 332, see par. [0026]);
selecting, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values (After the local ASR is finished, then this result is applied locally to the appropriate application, see par. [0026]);
causing the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user ( The local ASR converts the speech utterance to text and this text is used by the active application as data, command, or in any other way, depending on the nature or the active application, see par. [0028]);
storing the plurality of speech hypotheses in the on-device memory of the computing device (The cloud service may then generate the new models and then transmit them to the client when the network allows, see par. [0046]);
and transmitting, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network (If a network connection is available and there are no restrictions on use of the network, then at 310, the system sends the audio to one or more network connected ASR's. These systems will be referred to as the cloud ASR's, see par. [0029]; The ASR systems that provide the best results may be returned to for later utterances. In this way the local device is optimized for the particular users and for the available resources which may change over time, see par. [0030]; the system may still determine whether all of the cloud ASR results have been received at 316. If they have all been received, then the system may proceed to determine whether they all match at 318. If the results have not all been received at 316 and the timer has not expired at 312, then the system continues to wait until the timer expires before operating on the cloud ASR results. After the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance. While the selected remote ASRs are working, the local ASR may also generate a text result corresponding to the utterance at 334, see par. [0032-0033].
However Deisher does not teach wherein the given speech hypothesis incorporated as the portion of the transcription is modifiable to one of the additional speech hypotheses based on one or more user interactions with the additional computing device.
In the same field of endeavor Suendermann teaches method and apparatus for continuously improving the performance of semantic classifiers in the scope of spoken dialog systems are disclosed. Rule-based or statistical classifiers are replaced with better performing rule-based or statistical classifiers and/or certain parameters of existing classifiers are modified. Automated quality assurance leads to more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior, see abstract. The utterances are transcribed 116. This can be done manually or partially in an automated fashion. An exemplary transcription component can be, for instance, a program that plays back an utterance and gives the speech recognizer's hypothesis as a suggestion that the transcriber can modify if necessary. The automated transcription can be based on models trained on data formerly processed by the continuous optimization cycle 125, as described below, see par. [0063].
It would have been obvious to one of ordinary skill in the art to combine the Deisher invention with the teachings of Suendermann for the benefit of more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior, see abstract.
Regarding claim 2 Deisher teaches the method of claim 1, further comprising:
determining a respective confidence level associated with each of the plurality of speech hypotheses, for the part of the spoken utterance, based on the values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device, wherein selecting the given speech hypothesis, from among the plurality of speech hypotheses, predicted to correspond to the part of the spoken utterance is based on the respective confidence level associated with each of the plurality of speech hypotheses (he acoustic confidence of the original speech utterance may be assessed at 502. If the confidence is too low, then the process ends at 518. If the confidence is sufficiently high, then at 504, a string matching algorithm is used to identify words in each text string that may be OOV words. As mentioned above, high or low may be evaluated using numerical thresholds or in any other desired way, see par. [0047]; the newly modified local phone lattice is rescored for each variation within the lattice. Alternatively, various candidates may be generated by substitution and each scored using an acoustic confidence score, see par. [0050]).
Regarding claim 3 Deisher teaches the method of claim 2, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device is in response to determining that the respective confidence level for two or more of the plurality of speech hypotheses, for the part of the spoken utterance, are within a threshold range of confidence levels (based on this scoring, a text string can be selected as the best candidate. This may be a particular text string received from the cloud or it may be a combination of different text strings received from different cloud ASRs., see par. [0050]).
Regarding claim 7 Deisher teaches the method of claim 2, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device comprises storing each the plurality of speech hypotheses in association with the respective confidence level in the memory that is accessible by at least the computing device ( if the acoustic confidence score is high, either with a single text string or with one selected from multiple candidates, then the local ASR may be improved, see par. [0045]).
Regarding claim 11 Deisher teaches the method of claim 1, further comprising:
generating a finite state decoding graph that includes a respective confidence level associated with each of the plurality of speech hypotheses based on the values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device, wherein selecting the given speech hypothesis, from among the plurality of speech hypotheses, is based on the finite state decoding graph (The client lexicon and the language model may be augmented using a cached LM technique or by interpolation. The lexicon may be updated directly, for example by rebuilding a HCL (Hidden Markov model Context model Lexicon grammar) in a WFST-based (Weighted Finite State Transducer) system, see par. [0055]).
Regarding claim 12 Deisher teaches the method of claim 11, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device comprises storing the finite state decoding graph in the on-device memory of the computing device (The client lexicon may be updated directly, for example by rebuilding a HCL (Hidden Markov model Context model Lexicon grammar) in a WFST-based (Weighted Finite State Transducer) system, see par. [0055]).
Regarding claim 17 Deisher teaches the method of claim 1, wherein transmitting the plurality of speech hypotheses to the additional computing device comprises: subsequent to causing the given speech hypothesis to be incorporated as the portion of the transcription associated with the software application: determining the transcription associated with the software application is subsequently accessed at the additional computing device (after the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance, see par. [0033]);
and causing the plurality of speech hypotheses, for the part of the spoken utterance, to be transmitted to the additional computing device and from the memory that is accessible by at least the computing device (These results may be combined based on the comparison to determine a final text result for the utterance at 320. The final result may then be used by a client application at 322 as a command, as data, or in a variety of other ways, see par. [0033]).
Regarding claim 18 Deisher teaches a computing device of a user, the computing device comprising:
at least one processor (a processor 4 , see par. [0058]);
and memory storing instructions that, when executed (volatile memory (e.g., DRAM) 8, non-volatile memory (e.g., ROM) 9, flash memory (not shown), a graphics processor 12, a digital signal processor (not shown), see par. [0059]), cause the at least one processor to be operable to:
receive, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user (a speech utterance is received. This is typically in the form of digitized audio from a local microphone., see par. [0024]);
process, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device (the system performs a local ASR and at 330 waits for the local ASR to finish. After the local ASR is finished, then this result is applied locally to the appropriate application at 332, see par. [0026]);
select, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values (After the local ASR is finished, then this result is applied locally to the appropriate application, see par. [0026]);
cause the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user ( The local ASR converts the speech utterance to text and this text is used by the active application as data, command, or in any other way, depending on the nature or the active application, see par. [0028]);
store the plurality of speech hypotheses in the on-device memory of the computing device (The cloud service may then generate the new models and then transmit them to the client when the network allows, see par. [0046]);
and transmit, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network (If a network connection is available and there are no restrictions on use of the network, then at 310, the system sends the audio to one or more network connected ASR's. These systems will be referred to as the cloud ASR's, see par. [0029]; The ASR systems that provide the best results may be returned to for later utterances. In this way the local device is optimized for the particular users and for the available resources which may change over time, see par. [0030]; the system may still determine whether all of the cloud ASR results have been received at 316. If they have all been received, then the system may proceed to determine whether they all match at 318. If the results have not all been received at 316 and the timer has not expired at 312, then the system continues to wait until the timer expires before operating on the cloud ASR results. After the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance. While the selected remote ASRs are working, the local ASR may also generate a text result corresponding to the utterance at 334, see par. [0032-0033]).
However Deisher does not teach wherein the given speech hypothesis incorporated as the portion of the transcription is modifiable to one of the additional speech hypotheses based on one or more user interactions with the additional computing device.
In the same field of endeavor Suendermann teaches method and apparatus for continuously improving the performance of semantic classifiers in the scope of spoken dialog systems are disclosed. Rule-based or statistical classifiers are replaced with better performing rule-based or statistical classifiers and/or certain parameters of existing classifiers are modified. Automated quality assurance leads to more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior, see abstract. The utterances are transcribed 116. This can be done manually or partially in an automated fashion. An exemplary transcription component can be, for instance, a program that plays back an utterance and gives the speech recognizer's hypothesis as a suggestion that the transcriber can modify if necessary. The automated transcription can be based on models trained on data formerly processed by the continuous optimization cycle 125, as described below, see par. [0063].
It would have been obvious to one of ordinary skill in the art to combine the Deisher invention with the teachings of Suendermann for the benefit of more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior, see abstract.
Regarding claim 19 Deisher teaches a non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor of a computing device of a user to perform operations (volatile memory (e.g., DRAM) 8, non-volatile memory (e.g., ROM) 9, flash memory (not shown), a graphics processor 12, a digital signal processor (not shown), see par. [0059]), the operations comprising:
receiving, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user (a speech utterance is received. This is typically in the form of digitized audio from a local microphone., see par. [0024]);
processing, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device (the system performs a local ASR and at 330 waits for the local ASR to finish. After the local ASR is finished, then this result is applied locally to the appropriate application at 332, see par. [0026]);
selecting, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values (After the local ASR is finished, then this result is applied locally to the appropriate application, see par. [0026]);
causing the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user ( The local ASR converts the speech utterance to text and this text is used by the active application as data, command, or in any other way, depending on the nature or the active application, see par. [0028]);
storing the plurality of speech hypotheses in the on-device memory of the computing device (The cloud service may then generate the new models and then transmit them to the client when the network allows, see par. [0046]);
and transmitting, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network (If a network connection is available and there are no restrictions on use of the network, then at 310, the system sends the audio to one or more network connected ASR's. These systems will be referred to as the cloud ASR's, see par. [0029]; The ASR systems that provide the best results may be returned to for later utterances. In this way the local device is optimized for the particular users and for the available resources which may change over time, see par. [0030]; the system may still determine whether all of the cloud ASR results have been received at 316. If they have all been received, then the system may proceed to determine whether they all match at 318. If the results have not all been received at 316 and the timer has not expired at 312, then the system continues to wait until the timer expires before operating on the cloud ASR results. After the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance. While the selected remote ASRs are working, the local ASR may also generate a text result corresponding to the utterance at 334, see par. [0032-0033]).
However Deisher in view of Suendermann does not teach wherein the given speech hypothesis incorporated as the portion of the transcription is modifiable to one of the additional speech hypotheses based on one or more user interactions with the additional computing device.
In the same field of endeavor Suendermann teaches method and apparatus for continuously improving the performance of semantic classifiers in the scope of spoken dialog systems are disclosed. Rule-based or statistical classifiers are replaced with better performing rule-based or statistical classifiers and/or certain parameters of existing classifiers are modified. Automated quality assurance leads to more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior, see abstract. The utterances are transcribed 116. This can be done manually or partially in an automated fashion. An exemplary transcription component can be, for instance, a program that plays back an utterance and gives the speech recognizer's hypothesis as a suggestion that the transcriber can modify if necessary. The automated transcription can be based on models trained on data formerly processed by the continuous optimization cycle 125, as described below, see par. [0063].
It would have been obvious to one of ordinary skill in the art to combine the Deisher in view of Suendermann invention with the teachings of Suendermann for the benefit of more accurate training and testing data, higher classification performance, and feedback into the design of the spoken dialog system by suggesting changes to improve system behavior, see abstract.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Suendermann U.S. PAP 2010/0268536 A1, further in view of Pandey U.S. Patent No. 11,289,075 B1.
Regarding claim 4 Deisher in view of Suendermann does not teach the method of claim 2, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device is in response to determining that the respective confidence level for each of the plurality of speech hypotheses, for the part of the spoken utterance, fail to satisfy a threshold confidence level.
IN a similar field of endeavor Pandey teaches if there is confidence in the feedback prediction model 170's positive or negative result (e.g., a confidence value for positive/negative predicted implicit user feedback is above a confidence value threshold), the predicted implicit feedback may be stored in a non-transitory computer-readable memory at action 179. Conversely, if the confidence is below a confidence threshold and/or a determination is made that the feedback prediction model 170 is not confident in the predicted implicit user feedback for the current hypothesis, a speechlet may be used to solicit manual annotation 192. The manual annotation may be provided manually, offline, by one or more individuals. The manual annotation may indicate positive or negative user feedback for a given hypothesis sent to a given skill. The manual annotation may be stored in association with the hypothesis in a non-transitory computer-readable memory at action 191, see col. 12 lines 58-67.
It would have been obvious to one of ordinary skill in the art to combine the Deisher in view of Suendermann invention with the teachings of Pandey for the benefit of allowing implicit user feedback about the hypotheses, see col. 12 lines 58-67.
Claim(s) 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Suendermann U.S. PAP 2010/0268536 A1, further in view of Duan U.S. Patent No. 6,223,150 B1.
Regarding claim 5 Deisher in view of Suendermann does not teach the method of claim 4, further comprising:
graphically demarcating the portion of the transcription that includes the part of the spoken utterance corresponding to the given speech hypothesis, wherein graphically demarcating the portion of the transcription is in response to determining that the respective confidence level for each of the plurality of speech hypotheses, for the part of the spoken utterance, fail to satisfy a threshold confidence level.
In the same field of endeavor Duan teaches An embodiment of the user interface 1298 of FIG. 12 comprises a display screen on which utterance hypotheses are displayed for the user. FIG. 13 is an illustration of one embodiment of a display screen. The best utterance hypothesis 1302 is displayed. In this case, the best utterance hypothesis is the sentence "I want to recognize speech." In addition to forming alternative utterance hypotheses and displaying the best utterance hypothesis, the present invention recognizes segments of the best utterance hypothesis that may have alternative hypotheses. These segments are highlighted, in this embodiment, to indicate to the user that the segment 1304 is one of a group of hypotheses. In one embodiment, if there are multiple segments that have alternative hypotheses, the largest segment is chosen as the highlighted segment. The user may activate the highlighted segment 1304 by, for example, moving a cursor to the highlighted segment 1304 and clicking a mouse button. When the highlighted segment 1304 is activated, alternative hypotheses for the segment are displayed., see col. 17 lines 4-36.
It would have been obvious to one of ordinary skill in the art to combine the Deisher in view of Suendermann invention with the teachings of Duan for the benefit of allowing a user to activate a highlighted segment by clicking to display alternatives, see col. 17 lines 19-36.
Regarding claim 6 Duan teaches the method of claim 5, wherein graphically demarcating the portion of the transcription that includes the part of the spoken utterance corresponding to the given speech hypothesis comprises one or more of:
highlighting the portion of the transcription, underlining the portion of the transcription, italicizing the portion of the transcription, or providing a selectable graphical element that, when selected, causes one or more additional speech hypotheses, from among the plurality of speech hypotheses, and that are in addition to the given speech hypothesis, to be visually rendered along with the portion of the transcription (these segments are highlighted, in this embodiment, to indicate to the user that the segment 1304 is one of a group of hypotheses, see col. 17 lines 4-18).
Claim(s) 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Suendermann U.S. PAP 2010/0268536 A1, further in view of Skobeltsyn U.S. PAP 2016/0063994 A1.
Regarding claim 8 Deisher in view of Suendermann does not teach the method of claim 2, further comprising:
receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to an additional spoken utterance of the user; processing, using an additional corresponding on-device ASR model that is stored in additional on-device memory of the additional computing device, the additional audio data corresponding to the additional spoken utterance to generate, for an additional part of the additional spoken utterance, a plurality of additional speech hypotheses based on additional values generated using the additional corresponding on-device ASR model that is stored in the additional on-device memory of the additional computing device; and modifying the given speech hypothesis, for the part of the spoken utterance, incorporated as the portion of the transcription based on the plurality of additional speech hypotheses.
In the same field of endeavor Skobeltsyn teaches receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to an additional spoken utterance of the user (second voice input, see par. [0071]);
processing, using an additional corresponding on-device ASR model that is stored in additional on-device memory of the additional computing device, the additional audio data corresponding to the additional spoken utterance to generate, for an additional part of the additional spoken utterance, a plurality of additional speech hypotheses based on additional values generated using the additional corresponding on-device ASR model that is stored in the additional on-device memory of the additional computing device ( Additional recognition hypotheses for the second voice input can then be used to generate additional candidate correctio… In the above example, additional hypotheses for recognizing the second voice input of “no I said France” can include [no I said Franz], [no I said France], [no I sat France], [noah sad friends] and so on, see par. [0071-0072]);
and modifying the given speech hypothesis, for the part of the spoken utterance, incorporated as the portion of the transcription based on the plurality of additional speech hypotheses (These hypotheses, generated e.g., from speech-to-text recognition, are used, as described with the recognition from the correction request 408, to generate candidate corrections by substituting n-grams of the various hypotheses into the misrecognized output for the first voice query including, for example, [who is the president of Franz], [who is the president of France], etc., see par. [0072]).
It would have been obvious to one of ordinary skill in the art to combine the Deisher in view of Suendermann invention with the teachings of Skobeltsyn for the benefit of correcting voice queries with a corrected query improves voice search versatility, see par. [0006].
Regarding claim 9 Skobeltsyn teaches the method of claim 8, wherein modifying the given speech hypothesis incorporated as the portion of the transcription based on the plurality of additional speech hypotheses comprises:
selecting an alternate speech hypothesis, from among the plurality of speech hypotheses, based on the respective confidence level associated with each of the plurality of speech hypotheses and based on the plurality of additional speech hypotheses (he system provides a highest scoring candidate correction as a corrected recognition output if a threshold score value is satisfied, see par. [0084]);
and replacing the given speech hypothesis with the alternate speech hypothesis, for the part of the spoken utterance, in the transcription (The highest scoring candidate correction becomes the corrected recognition output and can be provided for display on the client device, see par. [0085]).
Regarding claim 10 Skobeltsyn teaches the method of claim 9, further comprising:
selecting, from among one or more of the additional speech hypotheses, an additional given speech hypothesis, the additional given speech hypothesis being predicted to correspond to the additional part of the additional spoken utterance (ach candidate correction is assigned a score value based on one or more applied scoring techniques. Each candidate correction includes information on the replacement source and target as well as an interpretation of the correction request based, e.g., on the grammar used, see par. [0077]);
and causing the additional given speech hypothesis to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription (Candidate corrections can be generated using a recognition from the correction request 408, i.e., the second voice query 404. In particular, a candidate correction can be obtained by substituting the misrecognition with the correction from the second voice input. For example, if the recognition output of the first voice query is “who is president of friends” and the second voice input is recognized as [no I meant France] the portion following the correction prefix [no I meant] is used as the candidate substitution n-gram into the first recognition output. In this example, the candidate substitution n-gram “France” can be substituted into the original recognition., see par. [0070]).
Claim(s) 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Suendermann U.S. PAP 2010/0268536 A1, further in view of Thomson U.S. Patent No. 10,573,312 B1.
Regarding claim 13 Deisher in view of Suendermann does not teach the method of claim 11, further comprising: receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to an additional spoken utterance of the user; processing, using an additional corresponding on-device ASR model that is stored in additional on-device memory of the additional computing device, the additional audio data corresponding to the additional spoken utterance to generate one or more additional speech hypotheses based on additional values generated using the additional corresponding on-device ASR model that is stored in the additional on-device memory of the additional computing device; and modifying the given speech hypothesis, for the part of the spoken utterance, incorporated as the portion of the transcription based on one or more of the additional speech hypotheses.
In the same field of endeavor Thomson teaches FIG. 35 is a schematic block diagram illustrating an example environment 3500 for editing by a CA 3518, in accordance with some embodiments of the present disclosure. In some embodiments, the CA 3518 may monitor multiple audio signals, such as multiple communication sessions, simultaneously and make corrections to transcriptions of the audio signals as needed. FIG. 35 illustrates audio signals 1-4. Each of the audio signals 1-4 are provided to one of four ASR systems, 3520a-3520d, collectively the ASR systems 3520. The ASR systems 3520 may generate transcriptions of the audio signals 1-4 and provide the transcriptions to an editor 3502. The editor 3502 may buffer the text, segment words into phrases, and deliver transcriptions to the respective user devices to be displayed, see figure 35, col. 135 line 63- col 136 line 9.
It would have been obvious to combine the Deisher in view of Suendermann invention with the teachings of Thomson for the benefit of improving an ability of the ASR system to recognize words in speech, see col. 5 lines 42-47.
Regarding claim 14 Thomson teaches the method of claim 13, wherein modifying the given speech hypothesis incorporated as the portion of the transcription based on one or more of the additional speech hypotheses comprises: adapting the finite state decoding graph based on one or more of the additional speech hypotheses to select an alternate speech hypothesis from among the plurality of speech hypotheses (In some embodiments, the decoder 510 receives a series of phonemes and their associated probabilities. In some embodiments, the phonemes and their associated probabilities may be determined at regular intervals such as every 5, 7, 10, 15, or 20 milliseconds. In these and other embodiments, the decoder 510 may also read a language model 511 (generated by an LM trainer 519) such as a statistical language model or finite state grammar and, in some configurations, a pronunciation model 513 (generated by a lexicon trainer 521) or lexicon, see col. 39 lines 39-60); and replacing the given speech hypothesis with the alternate speech hypothesis, for the part of the spoken utterance, in the transcription (he rescorer 512 analyzes the multiple hypotheses and reevaluates or reorders them and may consider additional information such as application information or a language model other than the language model used by the decoder 510, such as a rescoring language model, see col. 39 lines 61-65).
Regarding claim 15 Thomson teaches the method of claim 13, further comprising: selecting, from among one or more of the additional speech hypotheses, an additional given speech hypothesis, the additional given speech hypothesis being predicted to correspond to an additional portion of the additional spoken utterance (the decoder 510 may output a structure in a rich format, representing multiple hypotheses or alternative transcriptions, such as a word confusion network (WCN), lattice (a connected graph showing possible word combinations and, in some cases, their associated probabilities), or n-best list (a list of hypotheses in descending order of likelihood, where “n” is the number of hypotheses, see col. 39, lines 50-60);
and causing the additional given speech hypothesis to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription (Each time a new token or sequence of tokens is received from one of the transcription generation processes 1402, the new token or sequence of tokens may be appended to the previously created input hypothesis to create an updated input hypothesis. The updated input hypothesis may then be fused with other hypotheses from other transcription generation processes 1402, and the fused output becomes the fused output hypothesis., see col. 80, lines 56-67).
Regarding claim 16 Thomson teaches the method of claim 13, further comprising: causing the computing device to visually render one or more graphical elements that indicate the given speech hypothesis, for the part of the spoken utterance, was modified (A transcription of the audio generated by the transcription system may be provided back to the device for display to a user of the device. The transcription may assist the user to better understand what is being said during the communication session., see col. 4 lines 30-38).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Pertinent prior art available on form 892.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael Ortiz-Sanchez whose telephone number is (571)270-3711. The examiner can normally be reached Monday- Friday 9AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ORTIZ-SANCHEZ/Primary Examiner, Art Unit 2656
Prosecution Insights