DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 8, 9, 10, 18 and 19 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1-3, 7 and 13 of U.S. Patent No. 11,935,519. Although the claims at issue are not identical, they are not patentably distinct from each other because In re Karlson, 136 USPQ 184 (1963): “Omission of an element and its function is an obvious expedient if the remaining elements perform the same functions as before”.
Patent No. 11,935,519 B2
18/430,196
1. A method implemented by one or more processors of a computing device of a user, the method comprising:
receiving, via one or more microphones of the computing device of the user, first audio data corresponding to a first spoken utterance of the user;
processing, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the first audio data corresponding to the first spoken utterance to generate, for a given first part of the first spoken utterance, a plurality of first speech hypotheses based on first values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device;
selecting, from among the plurality of first speech hypotheses, a given first speech hypothesis, the given first speech hypothesis being predicted to correspond to the given first part of the first spoken utterance based on the first values;
causing the given first speech hypothesis to be incorporated as a first portion of a transcription, the transcription being visually rendered at a user interface of the computing device of the user;
determining that the first spoken utterance is complete;
in response to determining that the first spoken utterance is complete,
storing one or more first alternate speech hypotheses in the on-device memory of the computing device, the one or more first alternate speech hypotheses including a subset of the plurality of first speech hypotheses that excludes at least the given first speech hypothesis;
receiving, via one or more of the microphones of the computing device, second audio data corresponding to a second spoken utterance of the user; and in response to receiving the second audio data:
loading one or more of the first alternate speech hypotheses from the on-device memory of the computing device;
processing, using the corresponding on-device ASR model that is stored in on-device memory of the computing device, the second audio data corresponding to the second spoken utterance to generate, for a given second part of the second spoken utterance, a plurality of second speech hypotheses based on second values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device; selecting, from among the plurality of second speech hypotheses, a given second speech hypothesis, the given second speech hypothesis being predicted to correspond to the given second part of the second spoken utterance based on the second values; causing the given second speech hypothesis to be incorporated as a second portion of the transcription;
determining, based on the given second speech hypothesis that is incorporated into the transcription as the second portion of the transcription, whether to modify the first portion of the transcription; and in response to determining to modify the first portion of the transcription: modifying the first portion of the transcription, that was initially predicted to correspond to the given first part of the first spoken utterance, to include a given alternate first speech hypothesis, from among the one or more alternate first speech hypotheses, that is subsequently predicted to correspond to the given first part of the first spoken utterance.
2. The method of claim 1, wherein modifying the first portion of the transcription to include the given alternate first speech hypothesis comprises:
supplanting, in the transcription that is visually rendered at the user interface of the client device, the given first speech hypothesis with the given alternate first speech hypothesis.
3. The method of claim 2, wherein supplanting the given first speech hypothesis with the given alternate first speech hypothesis comprises: automatically supplanting, in the transcription that is visually rendered at the user interface of the client device, the given first speech hypothesis with the given alternate first speech hypothesis.
7. A computing device of a user, the computing device comprising: one or more hardware processors; and on-device memory storing at least instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to: receive, via one or more microphones of the computing device, first audio data corresponding to a first spoken utterance of the user; process, using a corresponding on-device automatic speech recognition (ASR) model that is stored in the on-device memory of the computing device, the first audio data corresponding to the first spoken utterance to generate, for a given first part of the first spoken utterance, a plurality of first speech hypotheses based on first values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device; select, from among the plurality of first speech hypotheses, a given first speech hypothesis, the given first speech hypothesis being predicted to correspond to the given first part of the first spoken utterance based on the first values; cause the given first speech hypothesis to be incorporated as a first portion of a transcription, the transcription being visually rendered at a user interface of the computing device of the user; determine that the first spoken utterance is complete; in response to determining that the first spoken utterance is complete, store one or more first alternate speech hypotheses in the on-device memory of the computing device, the one or more first alternate speech hypotheses including a subset of the plurality of first speech hypotheses that excludes at least the given first speech hypothesis; receive, via one or more of the microphones of the computing device, second audio data corresponding to a second spoken utterance of the user; and in response to receiving the second audio data: load one or more of the first alternate speech hypotheses from the on-device memory of the computing device; process, using the corresponding on-device ASR model that is stored in the on-device memory of the computing device, the second audio data corresponding to the second spoken utterance to generate, for a given second part of the second spoken utterance, a plurality of second speech hypotheses based on second values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device; select, from among the plurality of second speech hypotheses, a given second speech hypothesis, the given second speech hypothesis being predicted to correspond to the given second part of the second spoken utterance based on the second values; cause the given second speech hypothesis to be incorporated as a second portion of the transcription; determine, based on the given second speech hypothesis that is incorporated into the transcription as the second portion of the transcription, whether to modify the first portion of the transcription; and in response to determining to modify the first portion of the transcription: modify the first portion of the transcription, that was initially predicted to correspond to the given first part of the first spoken utterance, to include a given alternate first speech hypothesis, from among the one or more alternate first speech hypotheses, that is subsequently predicted to correspond to the given first part of the first spoken utterance.
13. A system, the system comprising: one or more hardware processors; and memory storing at least instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to: receive, via one or more microphones of a computing device of a user, first audio data corresponding to a first spoken utterance of the user; process, using a corresponding on-device automatic speech recognition (ASR) model that is stored in the on-device memory of the computing device, the first audio data corresponding to the first spoken utterance to generate, for a given first part of the first spoken utterance, a plurality of first speech hypotheses based on first values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device; select, from among the plurality of first speech hypotheses, a given first speech hypothesis, the given first speech hypothesis being predicted to correspond to the given first part of the first spoken utterance based on the first values; cause the given first speech hypothesis to be incorporated as a first portion of a transcription, the transcription being visually rendered at a user interface of the computing device of the user; determine that the first spoken utterance is complete; in response to determining that the first spoken utterance is complete, store one or more first alternate speech hypotheses in the on-device memory of the computing device, the one or more first alternate speech hypotheses including a subset of the plurality of first speech hypotheses that excludes at least the given first speech hypothesis; receive, via one or more of the microphones of the computing device, second audio data corresponding to a second spoken utterance of the user; and in response to receiving the second audio data: load one or more of the first alternate speech hypotheses from the on-device memory of the computing device; process, using the corresponding on-device ASR model that is stored in the on-device memory of the computing device, the second audio data corresponding to the second spoken utterance to generate, for a-given second part of the second spoken utterance, a plurality of second speech hypotheses based on second values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device; select, from among the plurality of second speech hypotheses, a given second speech hypothesis, the given second speech hypothesis being predicted to correspond to the given second part of the second spoken utterance based on the second values; cause the given second speech hypothesis to be incorporated as a second portion of the transcription; determine, based on the given second speech hypothesis that is incorporated into the transcription as the second portion of the transcription, whether to modify the first portion of the transcription; and in response to determining to modify the first portion of the transcription: modify the first portion of the transcription, that was initially predicted to correspond to the given first part of the first spoken utterance, to include a given alternate first speech hypothesis, from among the one or more alternate first speech hypotheses, that is subsequently predicted to correspond to the given first part of the first spoken utterance.
1. A method implemented by one or more processors of a computing device of a user, the method comprising:
receiving, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user;
processing, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device;
selecting, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values;
causing the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user;
storing the plurality of speech hypotheses in the on-device memory of the computing device;
and transmitting, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network.
8. The method of claim 2, further comprising:
receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to an additional spoken utterance of the user;
processing, using an additional corresponding on-device ASR model that is stored in additional on-device memory of the additional computing device, the additional audio data corresponding to the additional spoken utterance to generate, for an additional part of the additional spoken utterance, a plurality of additional speech hypotheses based on additional values generated using the additional corresponding on-device ASR model that is stored in the additional on-device memory of the additional computing device;
and modifying the given speech hypothesis, for the part of the spoken utterance, incorporated as the portion of the transcription based on the plurality of additional speech hypotheses.
9. The method of claim 8, wherein modifying the given speech hypothesis incorporated as the portion of the transcription based on the plurality of additional speech hypotheses comprises:
selecting an alternate speech hypothesis, from among the plurality of speech hypotheses, based on the respective confidence level associated with each of the plurality of speech hypotheses and based on the plurality of additional speech hypotheses;
and replacing the given speech hypothesis with the alternate speech hypothesis, for the part of the spoken utterance, in the transcription.
10. The method of claim 9, further comprising:
selecting, from among one or more of the additional speech hypotheses, an additional given speech hypothesis, the additional given speech hypothesis being predicted to correspond to the additional part of the additional spoken utterance;
and causing the additional given speech hypothesis to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription.
18. A computing device of a user, the computing device comprising: at least one processor; and memory storing instructions that, when executed, cause the at least one processor to be operable to: receive, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user; process, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device; select, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values; cause the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user; store the plurality of speech hypotheses in the on-device memory of the computing device; and transmit, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network.
19. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor of a computing device of a user to perform operations, the operations comprising: receiving, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user; processing, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device; selecting, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values; causing the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user; storing the plurality of speech hypotheses in the on-device memory of the computing device; and transmitting, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3,7, 11, 12, 17-19 is/are rejected under 35 U.S.C. 102(a)(!) as being anticipated by Deisher U.S. PAP 2016/0379626 A1.
Regarding claim 1 Deisher teaches a method implemented by one or more processors of a computing device of a user (The present description pertains to speech recognition for connected devices and in particular to speech recognition using both local and remote resources, see par. [0001]), the method comprising:
receiving, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user (a speech utterance is received. This is typically in the form of digitized audio from a local microphone., see par. [0024]);
processing, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device (the system performs a local ASR and at 330 waits for the local ASR to finish. After the local ASR is finished, then this result is applied locally to the appropriate application at 332, see par. [0026]);
selecting, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values (After the local ASR is finished, then this result is applied locally to the appropriate application, see par. [0026]);
causing the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user ( The local ASR converts the speech utterance to text and this text is used by the active application as data, command, or in any other way, depending on the nature or the active application, see par. [0028]);
storing the plurality of speech hypotheses in the on-device memory of the computing device (The cloud service may then generate the new models and then transmit them to the client when the network allows, see par. [0046]);
and transmitting, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network (If a network connection is available and there are no restrictions on use of the network, then at 310, the system sends the audio to one or more network connected ASR's. These systems will be referred to as the cloud ASR's, see par. [0029]; The ASR systems that provide the best results may be returned to for later utterances. In this way the local device is optimized for the particular users and for the available resources which may change over time, see par. [0030]; the system may still determine whether all of the cloud ASR results have been received at 316. If they have all been received, then the system may proceed to determine whether they all match at 318. If the results have not all been received at 316 and the timer has not expired at 312, then the system continues to wait until the timer expires before operating on the cloud ASR results. After the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance. While the selected remote ASRs are working, the local ASR may also generate a text result corresponding to the utterance at 334, see par. [0032-0033].
Regarding claim 2 Deisher teaches the method of claim 1, further comprising:
determining a respective confidence level associated with each of the plurality of speech hypotheses, for the part of the spoken utterance, based on the values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device, wherein selecting the given speech hypothesis, from among the plurality of speech hypotheses, predicted to correspond to the part of the spoken utterance is based on the respective confidence level associated with each of the plurality of speech hypotheses (he acoustic confidence of the original speech utterance may be assessed at 502. If the confidence is too low, then the process ends at 518. If the confidence is sufficiently high, then at 504, a string matching algorithm is used to identify words in each text string that may be OOV words. As mentioned above, high or low may be evaluated using numerical thresholds or in any other desired way, see par. [0047]; the newly modified local phone lattice is rescored for each variation within the lattice. Alternatively, various candidates may be generated by substitution and each scored using an acoustic confidence score, see par. [0050]).
Regarding claim 3 Deisher teaches the method of claim 2, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device is in response to determining that the respective confidence level for two or more of the plurality of speech hypotheses, for the part of the spoken utterance, are within a threshold range of confidence levels (based on this scoring, a text string can be selected as the best candidate. This may be a particular text string received from the cloud or it may be a combination of different text strings received from different cloud ASRs., see par. [0050]).
Regarding claim 7 Deisher teaches the method of claim 2, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device comprises storing each the plurality of speech hypotheses in association with the respective confidence level in the memory that is accessible by at least the computing device ( if the acoustic confidence score is high, either with a single text string or with one selected from multiple candidates, then the local ASR may be improved, see par. [0045]).
Regarding claim 11 Deisher teaches the method of claim 1, further comprising:
generating a finite state decoding graph that includes a respective confidence level associated with each of the plurality of speech hypotheses based on the values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device, wherein selecting the given speech hypothesis, from among the plurality of speech hypotheses, is based on the finite state decoding graph (The client lexicon and the language model may be augmented using a cached LM technique or by interpolation. The lexicon may be updated directly, for example by rebuilding a HCL (Hidden Markov model Context model Lexicon grammar) in a WFST-based (Weighted Finite State Transducer) system, see par. [0055]).
Regarding claim 12 Deisher teaches the method of claim 11, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device comprises storing the finite state decoding graph in the on-device memory of the computing device (The client lexicon may be updated directly, for example by rebuilding a HCL (Hidden Markov model Context model Lexicon grammar) in a WFST-based (Weighted Finite State Transducer) system, see par. [0055]).
Regarding claim 17 Deisher teaches the method of claim 1, wherein transmitting the plurality of speech hypotheses to the additional computing device comprises: subsequent to causing the given speech hypothesis to be incorporated as the portion of the transcription associated with the software application: determining the transcription associated with the software application is subsequently accessed at the additional computing device (after the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance, see par. [0033]);
and causing the plurality of speech hypotheses, for the part of the spoken utterance, to be transmitted to the additional computing device and from the memory that is accessible by at least the computing device (These results may be combined based on the comparison to determine a final text result for the utterance at 320. The final result may then be used by a client application at 322 as a command, as data, or in a variety of other ways, see par. [0033]).
Regarding claim 18 Deisher teaches a computing device of a user, the computing device comprising:
at least one processor (a processor 4 , see par. [0058]);
and memory storing instructions that, when executed (volatile memory (e.g., DRAM) 8, non-volatile memory (e.g., ROM) 9, flash memory (not shown), a graphics processor 12, a digital signal processor (not shown), see par. [0059]), cause the at least one processor to be operable to:
receive, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user (a speech utterance is received. This is typically in the form of digitized audio from a local microphone., see par. [0024]);
process, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device (the system performs a local ASR and at 330 waits for the local ASR to finish. After the local ASR is finished, then this result is applied locally to the appropriate application at 332, see par. [0026]);
select, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values (After the local ASR is finished, then this result is applied locally to the appropriate application, see par. [0026]);
cause the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user ( The local ASR converts the speech utterance to text and this text is used by the active application as data, command, or in any other way, depending on the nature or the active application, see par. [0028]);
store the plurality of speech hypotheses in the on-device memory of the computing device (The cloud service may then generate the new models and then transmit them to the client when the network allows, see par. [0046]);
and transmit, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network (If a network connection is available and there are no restrictions on use of the network, then at 310, the system sends the audio to one or more network connected ASR's. These systems will be referred to as the cloud ASR's, see par. [0029]; The ASR systems that provide the best results may be returned to for later utterances. In this way the local device is optimized for the particular users and for the available resources which may change over time, see par. [0030]; the system may still determine whether all of the cloud ASR results have been received at 316. If they have all been received, then the system may proceed to determine whether they all match at 318. If the results have not all been received at 316 and the timer has not expired at 312, then the system continues to wait until the timer expires before operating on the cloud ASR results. After the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance. While the selected remote ASRs are working, the local ASR may also generate a text result corresponding to the utterance at 334, see par. [0032-0033]).
Regarding claim 19 Deisher teaches a non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor of a computing device of a user to perform operations (volatile memory (e.g., DRAM) 8, non-volatile memory (e.g., ROM) 9, flash memory (not shown), a graphics processor 12, a digital signal processor (not shown), see par. [0059]), the operations comprising:
receiving, via one or more microphones of the computing device of the user, audio data corresponding to a spoken utterance of the user (a speech utterance is received. This is typically in the form of digitized audio from a local microphone., see par. [0024]);
processing, using a corresponding on-device automatic speech recognition (ASR) model that is stored in on-device memory of the computing device, the audio data corresponding to the spoken utterance to generate, for a part of the spoken utterance, a plurality of speech hypotheses based on values generated using the corresponding on-device ASR model that is stored in the on-device memory of the computing device (the system performs a local ASR and at 330 waits for the local ASR to finish. After the local ASR is finished, then this result is applied locally to the appropriate application at 332, see par. [0026]);
selecting, from among the plurality of speech hypotheses, a given speech hypothesis, the given speech hypothesis being predicted to correspond to the part of the spoken utterance based on the values (After the local ASR is finished, then this result is applied locally to the appropriate application, see par. [0026]);
causing the given speech hypothesis to be incorporated as a portion of a transcription, the transcription being associated with a software application that is accessible by the computing device, and the transcription being visually rendered at a user interface of the computing device of the user ( The local ASR converts the speech utterance to text and this text is used by the active application as data, command, or in any other way, depending on the nature or the active application, see par. [0028]);
storing the plurality of speech hypotheses in the on-device memory of the computing device (The cloud service may then generate the new models and then transmit them to the client when the network allows, see par. [0046]);
and transmitting, over a local area network, the plurality of speech hypotheses, including the given speech hypothesis that was incorporated as the portion of the transcription and additional speech hypotheses included in the plurality of speech hypotheses, to an additional computing device of the user, wherein transmitting the plurality of speech hypotheses to the additional computing device causes the plurality of speech hypotheses to be loaded at the additional computing device when the transcription associated with the software application is subsequently accessed by the user at the additional computing device, wherein the additional computing device is in addition to the computing device, and wherein the computing device and the additional computing device are communicatively coupled over the local area network (If a network connection is available and there are no restrictions on use of the network, then at 310, the system sends the audio to one or more network connected ASR's. These systems will be referred to as the cloud ASR's, see par. [0029]; The ASR systems that provide the best results may be returned to for later utterances. In this way the local device is optimized for the particular users and for the available resources which may change over time, see par. [0030]; the system may still determine whether all of the cloud ASR results have been received at 316. If they have all been received, then the system may proceed to determine whether they all match at 318. If the results have not all been received at 316 and the timer has not expired at 312, then the system continues to wait until the timer expires before operating on the cloud ASR results. After the timer expires at 312 or if all of the results have been received at 316, then all of the cloud ASR results that have been received may be compared at 318. The text results received from the remote speech recognition system correspond to the original user speech utterance. While the selected remote ASRs are working, the local ASR may also generate a text result corresponding to the utterance at 334, see par. [0032-0033]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Pandey U.S. Patent No. 11,289,075 B1.
Regarding claim 4 Deisher does not teach the method of claim 2, wherein storing the plurality of speech hypotheses in the on-device memory of the computing device is in response to determining that the respective confidence level for each of the plurality of speech hypotheses, for the part of the spoken utterance, fail to satisfy a threshold confidence level.
IN a similar field of endeavor Pandey teaches if there is confidence in the feedback prediction model 170's positive or negative result (e.g., a confidence value for positive/negative predicted implicit user feedback is above a confidence value threshold), the predicted implicit feedback may be stored in a non-transitory computer-readable memory at action 179. Conversely, if the confidence is below a confidence threshold and/or a determination is made that the feedback prediction model 170 is not confident in the predicted implicit user feedback for the current hypothesis, a speechlet may be used to solicit manual annotation 192. The manual annotation may be provided manually, offline, by one or more individuals. The manual annotation may indicate positive or negative user feedback for a given hypothesis sent to a given skill. The manual annotation may be stored in association with the hypothesis in a non-transitory computer-readable memory at action 191, see col. 12 lines 58-67.
It would have been obvious to one of ordinary skill in the art to combine the Deisher invention with the teachings of Pandey for the benefit of allowing implicit user feedback about the hypotheses, see col. 12 lines 58-67.
Claim(s) 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Duan U.S. Patent No. 6,223,150 B1.
Regarding claim 5 Deisher does not teach the method of claim 4, further comprising:
graphically demarcating the portion of the transcription that includes the part of the spoken utterance corresponding to the given speech hypothesis, wherein graphically demarcating the portion of the transcription is in response to determining that the respective confidence level for each of the plurality of speech hypotheses, for the part of the spoken utterance, fail to satisfy a threshold confidence level.
In the same field of endeavor Duan teaches An embodiment of the user interface 1298 of FIG. 12 comprises a display screen on which utterance hypotheses are displayed for the user. FIG. 13 is an illustration of one embodiment of a display screen. The best utterance hypothesis 1302 is displayed. In this case, the best utterance hypothesis is the sentence "I want to recognize speech." In addition to forming alternative utterance hypotheses and displaying the best utterance hypothesis, the present invention recognizes segments of the best utterance hypothesis that may have alternative hypotheses. These segments are highlighted, in this embodiment, to indicate to the user that the segment 1304 is one of a group of hypotheses. In one embodiment, if there are multiple segments that have alternative hypotheses, the largest segment is chosen as the highlighted segment. The user may activate the highlighted segment 1304 by, for example, moving a cursor to the highlighted segment 1304 and clicking a mouse button. When the highlighted segment 1304 is activated, alternative hypotheses for the segment are displayed., see col. 17 lines 4-36.
It would have been obvious to one of ordinary skill in the art to combine the Deisher invention with the teachings of Duan for the benefit of allowing a user to activate a highlighted segment by clicking to display alternatives, see col. 17 lines 19-36.
Regarding claim 6 Duan teaches the method of claim 5, wherein graphically demarcating the portion of the transcription that includes the part of the spoken utterance corresponding to the given speech hypothesis comprises one or more of:
highlighting the portion of the transcription, underlining the portion of the transcription, italicizing the portion of the transcription, or providing a selectable graphical element that, when selected, causes one or more additional speech hypotheses, from among the plurality of speech hypotheses, and that are in addition to the given speech hypothesis, to be visually rendered along with the portion of the transcription (these segments are highlighted, in this embodiment, to indicate to the user that the segment 1304 is one of a group of hypotheses, see col. 17 lines 4-18).
Claim(s) 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Skobeltsyn U.S. PAP 2016/0063994 A1.
Regarding claim 8 Deisher does not teach the method of claim 2, further comprising:
receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to an additional spoken utterance of the user; processing, using an additional corresponding on-device ASR model that is stored in additional on-device memory of the additional computing device, the additional audio data corresponding to the additional spoken utterance to generate, for an additional part of the additional spoken utterance, a plurality of additional speech hypotheses based on additional values generated using the additional corresponding on-device ASR model that is stored in the additional on-device memory of the additional computing device; and modifying the given speech hypothesis, for the part of the spoken utterance, incorporated as the portion of the transcription based on the plurality of additional speech hypotheses.
In the same field of endeavor Skobeltsyn teaches receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to an additional spoken utterance of the user (second voice input, see par. [0071]);
processing, using an additional corresponding on-device ASR model that is stored in additional on-device memory of the additional computing device, the additional audio data corresponding to the additional spoken utterance to generate, for an additional part of the additional spoken utterance, a plurality of additional speech hypotheses based on additional values generated using the additional corresponding on-device ASR model that is stored in the additional on-device memory of the additional computing device ( Additional recognition hypotheses for the second voice input can then be used to generate additional candidate correctio… In the above example, additional hypotheses for recognizing the second voice input of “no I said France” can include [no I said Franz], [no I said France], [no I sat France], [noah sad friends] and so on, see par. [0071-0072]);
and modifying the given speech hypothesis, for the part of the spoken utterance, incorporated as the portion of the transcription based on the plurality of additional speech hypotheses (These hypotheses, generated e.g., from speech-to-text recognition, are used, as described with the recognition from the correction request 408, to generate candidate corrections by substituting n-grams of the various hypotheses into the misrecognized output for the first voice query including, for example, [who is the president of Franz], [who is the president of France], etc., see par. [0072]).
It would have been obvious to one of ordinary skill in the art to combine the Deisher invention with the teachings of Skobeltsyn for the benefit of correcting voice queries with a corrected query improves voice search versatility, see par. [0006].
Regarding claim 9 Skobeltsyn teaches the method of claim 8, wherein modifying the given speech hypothesis incorporated as the portion of the transcription based on the plurality of additional speech hypotheses comprises:
selecting an alternate speech hypothesis, from among the plurality of speech hypotheses, based on the respective confidence level associated with each of the plurality of speech hypotheses and based on the plurality of additional speech hypotheses (he system provides a highest scoring candidate correction as a corrected recognition output if a threshold score value is satisfied, see par. [0084]);
and replacing the given speech hypothesis with the alternate speech hypothesis, for the part of the spoken utterance, in the transcription (The highest scoring candidate correction becomes the corrected recognition output and can be provided for display on the client device, see par. [0085]).
Regarding claim 10 Skobeltsyn teaches the method of claim 9, further comprising:
selecting, from among one or more of the additional speech hypotheses, an additional given speech hypothesis, the additional given speech hypothesis being predicted to correspond to the additional part of the additional spoken utterance (ach candidate correction is assigned a score value based on one or more applied scoring techniques. Each candidate correction includes information on the replacement source and target as well as an interpretation of the correction request based, e.g., on the grammar used, see par. [0077]);
and causing the additional given speech hypothesis to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription (Candidate corrections can be generated using a recognition from the correction request 408, i.e., the second voice query 404. In particular, a candidate correction can be obtained by substituting the misrecognition with the correction from the second voice input. For example, if the recognition output of the first voice query is “who is president of friends” and the second voice input is recognized as [no I meant France] the portion following the correction prefix [no I meant] is used as the candidate substitution n-gram into the first recognition output. In this example, the candidate substitution n-gram “France” can be substituted into the original recognition., see par. [0070]).
Claim(s) 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Deisher U.S. PAP 2016/0379626 A1 in view of Thomson U.S. Patent No. 10,573,312 B1.
Regarding claim 13 Deisher does not teach the method of claim 11, further comprising: receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to an additional spoken utterance of the user; processing, using an additional corresponding on-device ASR model that is stored in additional on-device memory of the additional computing device, the additional audio data corresponding to the additional spoken utterance to generate one or more additional speech hypotheses based on additional values generated using the additional corresponding on-device ASR model that is stored in the additional on-device memory of the additional computing device; and modifying the given speech hypothesis, for the part of the spoken utterance, incorporated as the portion of the transcription based on one or more of the additional speech hypotheses.
In the same field of endeavor Thomson teaches FIG. 35 is a schematic block diagram illustrating an example environment 3500 for editing by a CA 3518, in accordance with some embodiments of the present disclosure. In some embodiments, the CA 3518 may monitor multiple audio signals, such as multiple communication sessions, simultaneously and make corrections to transcriptions of the audio signals as needed. FIG. 35 illustrates audio signals 1-4. Each of the audio signals 1-4 are provided to one of four ASR systems, 3520a-3520d, collectively the ASR systems 3520. The ASR systems 3520 may generate transcriptions of the audio signals 1-4 and provide the transcriptions to an editor 3502. The editor 3502 may buffer the text, segment words into phrases, and deliver transcriptions to the respective user devices to be displayed, see figure 35, col. 135 line 63- col 136 line 9.
It would have been obvious to combine the Deisher invention with the teachings of Thomson for the benefit of improving an ability of the ASR system to recognize words in speech, see col. 5 lines 42-47.
Regarding claim 14 Thomson teaches the method of claim 13, wherein modifying the given speech hypothesis incorporated as the portion of the transcription based on one or more of the additional speech hypotheses comprises: adapting the finite state decoding graph based on one or more of the additional speech hypotheses to select an alternate speech hypothesis from among the plurality of speech hypotheses (In some embodiments, the decoder 510 receives a series of phonemes and their associated probabilities. In some embodiments, the phonemes and their associated probabilities may be determined at regular intervals such as every 5, 7, 10, 15, or 20 milliseconds. In these and other embodiments, the decoder 510 may also read a language model 511 (generated by an LM trainer 519) such as a statistical language model or finite state grammar and, in some configurations, a pronunciation model 513 (generated by a lexicon trainer 521) or lexicon, see col. 39 lines 39-60); and replacing the given speech hypothesis with the alternate speech hypothesis, for the part of the spoken utterance, in the transcription (he rescorer 512 analyzes the multiple hypotheses and reevaluates or reorders them and may consider additional information such as application information or a language model other than the language model used by the decoder 510, such as a rescoring language model, see col. 39 lines 61-65).
Regarding claim 15 Thomson teaches the method of claim 13, further comprising: selecting, from among one or more of the additional speech hypotheses, an additional given speech hypothesis, the additional given speech hypothesis being predicted to correspond to an additional portion of the additional spoken utterance (the decoder 510 may output a structure in a rich format, representing multiple hypotheses or alternative transcriptions, such as a word confusion network (WCN), lattice (a connected graph showing possible word combinations and, in some cases, their associated probabilities), or n-best list (a list of hypotheses in descending order of likelihood, where “n” is the number of hypotheses, see col. 39, lines 50-60);
and causing the additional given speech hypothesis to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription (Each time a new token or sequence of tokens is received from one of the transcription generation processes 1402, the new token or sequence of tokens may be appended to the previously created input hypothesis to create an updated input hypothesis. The updated input hypothesis may then be fused with other hypotheses from other transcription generation processes 1402, and the fused output becomes the fused output hypothesis., see col. 80, lines 56-67).
Regarding claim 16 Thomson teaches the method of claim 13, further comprising: causing the computing device to visually render one or more graphical elements that indicate the given speech hypothesis, for the part of the spoken utterance, was modified (A transcription of the audio generated by the transcription system may be provided back to the device for display to a user of the device. The transcription may assist the user to better understand what is being said during the communication session., see col. 4 lines 30-38).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Pertinent prior art available on form 892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael Ortiz-Sanchez whose telephone number is (571)270-3711. The examiner can normally be reached Monday- Friday 9AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ORTIZ-SANCHEZ/Primary Examiner, Art Unit 2656
Prosecution Insights