DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/13/26 has been entered.
Response to Arguments
Applicant’s arguments, see pages 13-22, filed 3/13/26, with respect to the rejection(s) of claim(s) 18 and 32 under 35 U.S.C. 101 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn.
However, upon further consideration, a new ground(s) of rejection is made for all pending claims in view of newly identified patent and non-patent literature. See the new 35 U.S.C. 103 rejections below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 18, 19, and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Fritsch et al. (US 8,918,317, herein “Fritsch”) in view of "A low latency sequential model and its user-focused evaluation for automatic punctuation of ASR closed captions", authored by Tündik, Máté Ákos, Balázs Tarján, and György Szaszák, published in Computer Speech & Language 63: 101076, available online on February 12, 2020, herein “Tundik” and further in view of "Streamed Punctuation Annotation using Transformers", authored by Christoph Minixhofer, Fourth Year Project Report, School of Informatics, University of Edinburgh, 2021, Retrieved from the Internet: <URL: https://project-archive.inf.ed.ac.uk/ug4/20212434/ug4_proj.pdf>. Note: Wayback Machine shows publication at above URL on June 24, 2022, herein “Minixhofer”.
Regarding Claim 1, Fritsch teaches:
A computing system for generating transcriptions of language utterances occurring in streaming audio data, the computing system comprising:
a processor (col. 11 lns. 46-51: “The techniques described above may be implemented in one or more computer programs executing on a programmable computer including a processor, a storage medium readable by the processor (including, for example, volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.” (emphasis added)); and
a hardware storage device storing computer-executable instructions that are executable by the processor (col. 11 lns. 46-51: “The techniques described above may be implemented in one or more computer programs executing on a programmable computer including a processor, a storage medium readable by the processor (including, for example, volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.” (emphasis added)) to cause the computing system to:
obtain streaming audio data comprising language utterances from a speaker (col. 6 lns. 1-9: “The system 100 includes a speech decoder 102. In general, the decoder 102 derives a sequence of tokens 104a-c, also referred to herein as a token stream 104, such as a sequence of words and/or punctuation marks, from speech 106. The speech 106 may take any form, such as a live audio signal generated using a microphone and transmitted by the microphone to a computer or other recording device, or a recorded audio signal stored on an analog or digital recording medium and subsequently read by the decoder 102.” (emphasis added));
cause a decoder of the computing system to continuously decode the streaming audio data in order to generate decoded streaming audio data (col. 6 lns. 53-60: “In general, the decoder 102 decodes a first portion of the speech 106 into a first word (or other token) (FIG. 2, step 202). Assume for purposes of example that in step 202 the decoder 102 decodes portion 106a of speech 106 to produce first word 104a. Such decoding may be performed using any speech decoding technique. The decoder 102, or other component of the system 100, inserts the first word 104a into the data structure 104 (e.g., document) (step 204).” (emphasis added));
determine whether a linguistic boundary exists within an initial segment of decoded streaming audio data (col. 6 ln. 61 – col. 6 ln. 4: “The decoder 102 then predicts the occurrence of a non-verbalized punctuation symbol 104b in the speech stream 106, where the predicted punctuation symbol 104b occurs in sequence after the first token 104a in the sequence of tokens 104 (step 206). As will be described in more detail below, this prediction may include considering a plurality of possible tokens (possibly including a plurality of distinct punctuation symbols, such as a period, comma, and question mark). The decoder 102, or other component of the system 100, inserts the predicted punctuation symbol 104b into the token stream 104 at a position after that of the first word 104a (step 208).” (emphasis added));
when a linguistic boundary is determined to exist, (i) identify a first portion of the initial segment located temporally prior to the linguistic boundary and a second portion of the initial segment located temporally subsequent to the linguistic boundary and (ii) cause a punctuator of the computing system to apply a punctuation corresponding to the first portion of the initial segment at the linguistic boundary (col. 7 lns. 5-27: “When creating a document from the token stream, whitespace and other formatting may be applied to the token stream 104 in any manner. For example, if the token stream 104 is used at some point to create a text document, the position of the punctuation symbol 104b within such a text document may, for example, be immediately after that of the first word 104a within the text document (such as by inserting a period after "day" to produce the text "day."). As another example, the punctuation symbol 104b may be inserted into the text document such that the first word 104a and the punctuation symbol 104b are separated by whitespace (and/or other inconsequential characters) but by no other words or characters (such as by inserting a dash after a space to produce the text "day--").
The decoder 102 then decodes a second portion of the speech 106 into a second word (or other token) (step 210), making use of the fact that a non-verbalized punctuation was predicted immediately before the second word. Assume for purposes of example that the decoder 102 decodes portion 106b of speech 106 to produce word 104c. The decoder 102, or other component of the system 100, inserts the second word 104c into the token stream 104 at a position in the token stream 104 after that of the punctuation mark 104b (step 212).” (emphasis added)).
Fritsch doesn’t describe a system or method to determine a number of newly decoded words to wait based on a type of audio device;
cause an orchestrator of the computing system to validate the first portion of the initial segment using the number of newly decoded words to wait, and if validated, output the validated first portion of the initial segment including the corresponding punctuation, while refraining from outputting the second portion of the initial segment; and
cause the validated first portion of the initial segment including the corresponding punctuation to be displayed at a user device.
However, Tundik describes a system and method to determine a number of newly decoded words to wait based on a type of audio device (See section 2.2 “The MaxEnt model”, page 4, third paragraph: “The radius parameter of the MaxEnt tagger determines the size of the context considered before predicting a punctuation mark. By default, left (past) and right (future) context is taken into account. We will refer to this setup as off-line mode. As taking future context into account increases latency, we have to limit the future context. Operation by limited future context will be referred to as on-line mode. In the experiments we use round brackets to specify left and right context, respectively. Hence (5,1) means that we are considering 5 past and 1 future tokens actually. In on-line mode, we usually have to limit future context for a single token, which, depending on the length of the word and the speech rate, will result in a latency of several hundred milliseconds when spoken.” (emphasis added) Here, the on-line audio device and off-line audio device are types of audio devices that determine the number of newly decoded words wait.
Also see section 2.3 “The recurrent neural network model”, page 5, first paragraph: “RNN models build up from Long-Short Term Memory (LSTM) cells. LSTM networks contain cells with a memory unit, preserving some information about past states of the cell Hochreiter and Schmidhuber (1997). The memory unit itself, as well as the output of the cell combined from a weighted contribution of the current input and the memory unit, are regulated by the data flow. These regulating weights are learned during the training phase. Connecting LSTM cells sequentially leads to powerful sequential models, whereby typically each cell receives the features at a given time frame. It is common to incorporate future features into the processing framework, that is, the output of the network at time t depends on inputs ranging from t – k..t..t + k. This is usually more effective if we allow for a bidirectional (from past to future and from future to past) flow of the information within the network (e.g. Bidirectional LSTM, BiLSTM Schuster and Paliwal (1997)). Obviously, the future is not known, so technically such networks wait until future samples are available, and delay their output accordingly. For on-line operation, as seen with the MaxEnt models, future again has to be limited to a single word. Adapting this framework to the word sequence we consider a token each time frame, that is word by word by indexing the words by t.” (emphasis added).
Finally, see section 4.1 “Hungarian overall results”, page 4 first paragraph, and Table 3 on page 5: “First, we compare the performance of the RNN-based punctuation recovery system (see Section 2.3) to the baseline MaxEnt sequence tagger (see Section 2.2) on the Hungarian broadcast dataset. Both approaches are presented in two configurations. While in the off-line mode, the future word context is also exploited to achieve the best result with the given features and architecture, we also focus on the performance degradation when switching to on-line mode, where the punctuation restoration depends only on the past word context and the target word (in case of MaxEnt, we allowed future context involving only a single word). This configuration is intended to fit to real-time closed captioning requirements with low latency. In both modes, punctuations are predicted for the slot preceding the target word (at t) in the input sequence. The hyperparameters of all presented approaches and their configurations were optimized on the validation set as explained earlier (see Section 2).
PNG
media_image1.png
246
892
media_image1.png
Greyscale
The obtained results are presented in Table 3 for the manual transcripts and in Table 4 for the automatic (ASR) transcripts, respectively. We remind the reader, that in the notation of MaxEnt models (i, j), i stands for the backward (past), whereas j stands for the forward (future) radius.” (emphasis added).
Note that Table 3 illustrates that in off-line mode the MaxEnt model waited for 19 future words before predicting punctuation, while in on-line mode the MaxEnt model only waited for 1 future word.);
cause an orchestrator of the computing system to [output] the first portion of the initial segment using the number of newly decoded words to wait, and output the first portion of the initial segment including the corresponding punctuation, while refraining from outputting the second portion of the initial segment (See section 2.3 “The recurrent neural network model”, page 5, first paragraph: “RNN models build up from Long-Short Term Memory (LSTM) cells. LSTM networks contain cells with a memory unit, preserving some information about past states of the cell Hochreiter and Schmidhuber (1997). The memory unit itself, as well as the output of the cell combined from a weighted contribution of the current input and the memory unit, are regulated by the data flow. These regulating weights are learned during the training phase. Connecting LSTM cells sequentially leads to powerful sequential models, whereby typically each cell receives the features at a given time frame. It is common to incorporate future features into the processing framework, that is, the output of the network at time t depends on inputs ranging from t – k..t..t + k. This is usually more effective if we allow for a bidirectional (from past to future and from future to past) flow of the information within the network (e.g. Bidirectional LSTM, BiLSTM Schuster and Paliwal (1997)). Obviously, the future is not known, so technically such networks wait until future samples are available, and delay their output accordingly. For on-line operation, as seen with the MaxEnt models, future again has to be limited to a single word. Adapting this framework to the word sequence we consider a token each time frame, that is word by word by indexing the words by t.” (emphasis added).); and
cause the first portion of the initial segment including the corresponding punctuation to be displayed at a user device (See section 2.1 “Tolerated maximal latency”, page 4, first two paragraphs: “The closed captioning use-case we are optimizing for is as follows: an ASR decoder is providing the transcription in real-time on a quasi word-by-word basis, concatenating the words left-to-right. Once a row fills up with words, the lines are scrolled up. At the same time up to 3 lines can be visible, and a single line can contain up to 35 – 40 characters including spaces. Average word length was found 7.1 characters for the highly agglutinating Hungarian and 5.4 characters for English for the used datasets overall (see Section 3 for description of the used corpora).
Once a word or a group of words is put on the screen, the proper punctuation should already be in place, as readers do not have the chance of reading anything twice. Therefore, very low latency of punctuation is required, as already the ASR introduces some latency – considering a back-tracking triggered right after a word ends, we even delay the output by the length of the last word (a reader can start reading the word at this point by when a listener has just finished listening to it).” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch a system and method to determine a number of newly decoded words to wait based on a type of audio device; cause an orchestrator of the computing system to [output] the first portion of the initial segment using the number of newly decoded words to wait, and output the first portion of the initial segment including the corresponding punctuation, while refraining from outputting the second portion of the initial segment; and cause the first portion of the initial segment including the corresponding punctuation to be displayed at a user device, as taught by Tundik, in order to enable more accurate punctuation of audio data converted to text by delaying the punctuation decisions as much as possible, given the time limitations associated with the audio being punctuated, which enables the system to use as much forward context as possible when determining which punctuation should be added to the final textual output.
Fritsch in view of Tundik doesn’t describe a system or method to cause an orchestrator of the computing system to validate the first portion of the initial segment using the number of newly decoded words to wait, and if validated, output the validated first portion of the initial segment including the corresponding punctuation, while refraining from outputting the second portion of the initial segment.
However, Minixhofer describes a system and method to cause an orchestrator of the computing system to validate the first portion of the initial segment using the number of newly decoded words to wait, and if validated, output the validated first portion of the initial segment including the corresponding punctuation, while refraining from outputting the second portion of the initial segment (See “Abstract”, page i: “To improve readability, punctuation prediction is typically performed on text output by an Automatic Speech Recognition (ASR) model. We introduce a Transformer-based model to predict punctuation marks on unpunctuated text suitable for text streamed word-for-word, as is often the case for ASR models. We propose a decoding strategy that delays punctuation marks’ insertion in case of uncertainty until a specific threshold is reached. […] To make the model viable for real-time use in combination with an ASR system and on low-resource devices, we evaluate input truncation and weight quantization. We show these techniques lead to faster-than-real-time inference speeds and a significant reduction in model size.”
Also see section 3.2 “Varying Lookahead and Decoding”, page 18: “To predict punctuation in a streamed setting, we need a way to progressively feed the model more right-side context should it fail to predict punctuation with the context it is given initially, and make it robust to predicting sequences with varying lengths of this context.
Training
Using the [PUNCT] token described above, this can be achieved when training the model: We set the minimum and maximum lookahead (lmin and lmax), and then insert the [PUNCT] token at n – l for each sample, where n is the sequence length and l is the lookahead. The lookahead can be cycled through or drawn randomly from [lmin; lmax].
Inference
For inference, a decoding strategy utilising varying lookahead is needed, which we propose in Algorithm 1. The first question that presents itself is how we decide if the system is predicting punctuation with reasonable confidence, or if more context is needed. We solve this by computing the Shannon-Entropy (Shannon, 1948) H over the set of probabilities pi,…,pk [in the set of] P assigned to each of the k punctuation marks after the softmax step.
PNG
media_image2.png
62
354
media_image2.png
Greyscale
This value can be understood as the uncertainty of the model, and will be lower when the model is more certain of a prediction, being 0 when one probability is 1 and all others are 0. Given the four possible outcomes of comma, period, question mark and no punctuation, the maximum value is reached when all probabilities are ¼ which corresponds with H(P) = 2. For decoding, we set an entropy threshold h and wait for more right-side context and repeat inference if the computed entropy H(P)>h. This is repeated until H(P) [less than or equal to] h or lmax is reached. As words are streamed into the system, we potentially do inference on multiple punctuation positions at the same time step, with the maximum number of inferences conducted at the same time being lmax – lmin + 1, as shown in Figure 3.3.”). (emphasis added).
Here, the system only adds punctuation to the ASR output if computed entropy is below a threshold value (which means the confidence level is above a threshold level) for the current amount of right-side context, where the system will wait for more right-side context when the computed entropy currently above the threshold value, where the computed entropy is used to validate the first portion of the initial segment using the number of newly decoded words to wait, and if validated, output the validated first portion of the initial segment including the corresponding punctuation, while refraining from outputting the second portion of the initial segment.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch in view of Tundik a system and method to cause an orchestrator of the computing system to validate the first portion of the initial segment using the number of newly decoded words to wait, and if validated, output the validated first portion of the initial segment including the corresponding punctuation, while refraining from outputting the second portion of the initial segment, as taught by Minixhofer, in order to improve the accuracy of the final punctuated text by varying the lookahead based on the system’s confidence in the current punctuation prediction, which leads to more accurate punctuation determinations while delaying the final output as little as possible.
Regarding claim 18, Fritsch describes a method implemented by a computing system for generating throttled closed captions of streaming audio data, the method comprising:
obtaining streaming audio data comprising language utterances from a speaker (col. 6 lns. 1-9: “The system 100 includes a speech decoder 102. In general, the decoder 102 derives a sequence of tokens 104a-c, also referred to herein as a token stream 104, such as a sequence of words and/or punctuation marks, from speech 106. The speech 106 may take any form, such as a live audio signal generated using a microphone and transmitted by the microphone to a computer or other recording device, or a recorded audio signal stored on an analog or digital recording medium and subsequently read by the decoder 102.” (emphasis added));
causing a decoder of the computing system to continuously decode the streaming audio data in order to generate decoded streaming audio data (col. 6 lns. 53-60: “In general, the decoder 102 decodes a first portion of the speech 106 into a first word (or other token) (FIG. 2, step 202). Assume for purposes of example that in step 202 the decoder 102 decodes portion 106a of speech 106 to produce first word 104a. Such decoding may be performed using any speech decoding technique. The decoder 102, or other component of the system 100, inserts the first word 104a into the data structure 104 (e.g., document) (step 204).” (emphasis added));
determining whether a linguistic boundary exists within an initial segment of decoded streaming audio data (col. 6 ln. 61 – col. 6 ln. 4: “The decoder 102 then predicts the occurrence of a non-verbalized punctuation symbol 104b in the speech stream 106, where the predicted punctuation symbol 104b occurs in sequence after the first token 104a in the sequence of tokens 104 (step 206). As will be described in more detail below, this prediction may include considering a plurality of possible tokens (possibly including a plurality of distinct punctuation symbols, such as a period, comma, and question mark). The decoder 102, or other component of the system 100, inserts the predicted punctuation symbol 104b into the token stream 104 at a position after that of the first word 104a (step 208).” (emphasis added));
when a linguistic boundary is determined to exist, causing a punctuator of the computing system to apply an initial punctuation at the linguistic boundary (col. 7 lns. 5-27: “When creating a document from the token stream, whitespace and other formatting may be applied to the token stream 104 in any manner. For example, if the token stream 104 is used at some point to create a text document, the position of the punctuation symbol 104b within such a text document may, for example, be immediately after that of the first word 104a within the text document (such as by inserting a period after "day" to produce the text "day."). As another example, the punctuation symbol 104b may be inserted into the text document such that the first word 104a and the punctuation symbol 104b are separated by whitespace (and/or other inconsequential characters) but by no other words or characters (such as by inserting a dash after a space to produce the text "day--").
The decoder 102 then decodes a second portion of the speech 106 into a second word (or other token) (step 210), making use of the fact that a non-verbalized punctuation was predicted immediately before the second word. Assume for purposes of example that the decoder 102 decodes portion 106b of speech 106 to produce word 104c. The decoder 102, or other component of the system 100, inserts the second word 104c into the token stream 104 at a position in the token stream 104 after that of the punctuation mark 104b (step 212).” (emphasis added));
Fritsch doesn’t describe a system or method that includes determining a number of newly decoded words to wait based on a language associated with the streaming audio data;
subsequent to the initial punctuation, causing an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait;
if validated, outputting for display on a user display the validated first portion of the initial segment of the streaming audio data ending at the linguistic boundary while refraining from outputting a second portion of the initial segment which is located temporally subsequent to the first portion of the initial segment.
However, Tundik describes a system and method method that includes determining a number of newly decoded words to wait based on a language associated with the streaming audio data (See Tables 3 and 4, page 8, which show that for Hungarian the MaxEnt Model uses a future context of 19 words in off-line mode.
PNG
media_image1.png
246
892
media_image1.png
Greyscale
Paragraph below Table 3: “The obtained results are presented in Table 3 for the manual transcripts and in Table 4 for the automatic (ASR) transcripts, respectively. We remind the reader, that in the notation of MaxEnt models (i, j), i stands for the backward (past), whereas j stands for the forward (future) radius.” (emphasis added).
PNG
media_image3.png
212
842
media_image3.png
Greyscale
Further see Tables 7 and 8, page 10, which show that for English the MaxEnt model only uses a future context of 6 words in off-line mode.
PNG
media_image4.png
583
823
media_image4.png
Greyscale
Also see section 6 “Conclusions”, page 18, second paragraph: “Summarizing the important contributions of our work done in punctuation restoration: (1) we use a lightweight and fast RNN model by closely maintained performance compared to more complex models; (2) we target real-time operation with low latency; (3) we use the approach for the highly agglutinating Hungarian which has a much less constrained word order than English, as grammatical functions depend much less on the word order than on suffixes (case endings), which makes sequence modelling more difficult due to higher variation seen in the data (4) confirmed by subjective tests that users benefit from automatic punctuation and (5) confirmed this for the primary audience, that is DHH subjects. Currently we are continuing this research to combine our model with the prosody inspired approach proposed in Moro and Szaszak (2017), and to involve adult DHH people in subjective testing.” (emphasis added).);
(See section 2.3 “The recurrent neural network model”, page 5, first paragraph: “RNN models build up from Long-Short Term Memory (LSTM) cells. LSTM networks contain cells with a memory unit, preserving some information about past states of the cell Hochreiter and Schmidhuber (1997). The memory unit itself, as well as the output of the cell combined from a weighted contribution of the current input and the memory unit, are regulated by the data flow. These regulating weights are learned during the training phase. Connecting LSTM cells sequentially leads to powerful sequential models, whereby typically each cell receives the features at a given time frame. It is common to incorporate future features into the processing framework, that is, the output of the network at time t depends on inputs ranging from t – k..t..t + k. This is usually more effective if we allow for a bidirectional (from past to future and from future to past) flow of the information within the network (e.g. Bidirectional LSTM, BiLSTM Schuster and Paliwal (1997)). Obviously, the future is not known, so technically such networks wait until future samples are available, and delay their output accordingly. For on-line operation, as seen with the MaxEnt models, future again has to be limited to a single word. Adapting this framework to the word sequence we consider a token each time frame, that is word by word by indexing the words by t.” (emphasis added).
Also see section 2.1 “Tolerated maximal latency”, page 4, first two paragraphs: “The closed captioning use-case we are optimizing for is as follows: an ASR decoder is providing the transcription in real-time on a quasi word-by-word basis, concatenating the words left-to-right. Once a row fills up with words, the lines are scrolled up. At the same time up to 3 lines can be visible, and a single line can contain up to 35 – 40 characters including spaces. Average word length was found 7.1 characters for the highly agglutinating Hungarian and 5.4 characters for English for the used datasets overall (see Section 3 for description of the used corpora).
Once a word or a group of words is put on the screen, the proper punctuation should already be in place, as readers do not have the chance of reading anything twice. Therefore, very low latency of punctuation is required, as already the ASR introduces some latency – considering a back-tracking triggered right after a word ends, we even delay the output by the length of the last word (a reader can start reading the word at this point by when a listener has just finished listening to it).” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch a system and method that includes determining a number of newly decoded words to wait based on a language associated with the streaming audio data; [and] as taught by Tundik, in order to enable more accurate punctuation of audio data converted to text by determining an optimal number of words to wait for each language, which reduces delay as much as possible while still providing accurate punctuation predictions for each language.
Fritsch in view of Tundik doesn’t describe a system or method wherein subsequent to the initial punctuation, causing an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; [and] if validated, outputting for display on a user display the validated first portion of the initial segment
However, Minixhofer describes a system and method wherein subsequent to the initial punctuation, causing an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; [and] if validated, outputting for display on a user display the validated first portion of the initial segment (See “Abstract”, page i: “To improve readability, punctuation prediction is typically performed on text output by an Automatic Speech Recognition (ASR) model. We introduce a Transformer-based model to predict punctuation marks on unpunctuated text suitable for text streamed word-for-word, as is often the case for ASR models. We propose a decoding strategy that delays punctuation marks’ insertion in case of uncertainty until a specific threshold is reached. […] To make the model viable for real-time use in combination with an ASR system and on low-resource devices, we evaluate input truncation and weight quantization. We show these techniques lead to faster-than-real-time inference speeds and a significant reduction in model size.”
Also see section 3.2 “Varying Lookahead and Decoding”, page 18: “To predict punctuation in a streamed setting, we need a way to progressively feed the model more right-side context should it fail to predict punctuation with the context it is given initially, and make it robust to predicting sequences with varying lengths of this context.
Training
Using the [PUNCT] token described above, this can be achieved when training the model: We set the minimum and maximum lookahead (lmin and lmax), and then insert the [PUNCT] token at n – l for each sample, where n is the sequence length and l is the lookahead. The lookahead can be cycled through or drawn randomly from [lmin; lmax].
Inference
For inference, a decoding strategy utilising varying lookahead is needed, which we propose in Algorithm 1. The first question that presents itself is how we decide if the system is predicting punctuation with reasonable confidence, or if more context is needed. We solve this by computing the Shannon-Entropy (Shannon, 1948) H over the set of probabilities pi,…,pk [in the set of] P assigned to each of the k punctuation marks after the softmax step.
PNG
media_image2.png
62
354
media_image2.png
Greyscale
This value can be understood as the uncertainty of the model, and will be lower when the model is more certain of a prediction, being 0 when one probability is 1 and all others are 0. Given the four possible outcomes of comma, period, question mark and no punctuation, the maximum value is reached when all probabilities are ¼ which corresponds with H(P) = 2. For decoding, we set an entropy threshold h and wait for more right-side context and repeat inference if the computed entropy H(P)>h. This is repeated until H(P) [less than or equal to] h or lmax is reached. As words are streamed into the system, we potentially do inference on multiple punctuation positions at the same time step, with the maximum number of inferences conducted at the same time being lmax – lmin + 1, as shown in Figure 3.3.”). (emphasis added).
Here, the system only adds punctuation to the ASR output if computed entropy is below a threshold value (which means the confidence level is above a threshold level) for the current amount of right-side context, where the system will wait for more right-side context when the computed entropy currently above the threshold value, and where the computed entropy is used to, subsequent to the initial punctuation, cause an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; [and] if validated, outputting for display on a user display the validated first portion of the initial segment.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch in view of Tundik a system and method wherein subsequent to the initial punctuation, causing an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; [and] if validated, outputting for display on a user display the validated first portion of the initial segment, as taught by Minixhofer, in order to improve the accuracy of the final punctuated text by varying the lookahead based on the system’s confidence in the current punctuation prediction, which leads to more accurate punctuation determinations while delaying the final output as little as possible.
Regarding claim 19, Fritsch describes a computing system for generating transcriptions of language utterances occurring in streaming audio data, the computing system comprising:
a processor (col. 11 lns. 46-51: “The techniques described above may be implemented in one or more computer programs executing on a programmable computer including a processor, a storage medium readable by the processor (including, for example, volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.” (emphasis added)); and
a hardware storage device storing computer-executable instructions that are executable by the processor (col. 11 lns. 46-51: “The techniques described above may be implemented in one or more computer programs executing on a programmable computer including a processor, a storage medium readable by the processor (including, for example, volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.” (emphasis added)) to cause the computing system to:
obtain streaming audio data comprising language utterances from a speaker (col. 6 lns. 1-9: “The system 100 includes a speech decoder 102. In general, the decoder 102 derives a sequence of tokens 104a-c, also referred to herein as a token stream 104, such as a sequence of words and/or punctuation marks, from speech 106. The speech 106 may take any form, such as a live audio signal generated using a microphone and transmitted by the microphone to a computer or other recording device, or a recorded audio signal stored on an analog or digital recording medium and subsequently read by the decoder 102.” (emphasis added));
cause a decoder of the computing system to continuously decode the streaming audio data in order to generate decoded streaming audio data (col. 6 lns. 53-60: “In general, the decoder 102 decodes a first portion of the speech 106 into a first word (or other token) (FIG. 2, step 202). Assume for purposes of example that in step 202 the decoder 102 decodes portion 106a of speech 106 to produce first word 104a. Such decoding may be performed using any speech decoding technique. The decoder 102, or other component of the system 100, inserts the first word 104a into the data structure 104 (e.g., document) (step 204).” (emphasis added));
determine whether a linguistic boundary exists within an initial segment of decoded streaming audio data (col. 6 ln. 61 – col. 6 ln. 4: “The decoder 102 then predicts the occurrence of a non-verbalized punctuation symbol 104b in the speech stream 106, where the predicted punctuation symbol 104b occurs in sequence after the first token 104a in the sequence of tokens 104 (step 206). As will be described in more detail below, this prediction may include considering a plurality of possible tokens (possibly including a plurality of distinct punctuation symbols, such as a period, comma, and question mark). The decoder 102, or other component of the system 100, inserts the predicted punctuation symbol 104b into the token stream 104 at a position after that of the first word 104a (step 208).” (emphasis added));
when a linguistic boundary is determined to exist, cause a punctuator of the computing system to apply an initial punctuation at the linguistic boundary (col. 7 lns. 5-27: “When creating a document from the token stream, whitespace and other formatting may be applied to the token stream 104 in any manner. For example, if the token stream 104 is used at some point to create a text document, the position of the punctuation symbol 104b within such a text document may, for example, be immediately after that of the first word 104a within the text document (such as by inserting a period after "day" to produce the text "day."). As another example, the punctuation symbol 104b may be inserted into the text document such that the first word 104a and the punctuation symbol 104b are separated by whitespace (and/or other inconsequential characters) but by no other words or characters (such as by inserting a dash after a space to produce the text "day--").
The decoder 102 then decodes a second portion of the speech 106 into a second word (or other token) (step 210), making use of the fact that a non-verbalized punctuation was predicted immediately before the second word. Assume for purposes of example that the decoder 102 decodes portion 106b of speech 106 to produce word 104c. The decoder 102, or other component of the system 100, inserts the second word 104c into the token stream 104 at a position in the token stream 104 after that of the punctuation mark 104b (step 212).” (emphasis added)).
Fritsch doesn’t describe a system or method to determine a number of newly decoded words to wait based on a context of output application associated with the streaming audio data; subsequent to the initial punctuation, cause an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; upon determining that the initial punctuation is correct, cause the orchestrator to output for display on a user device a first portion of the initial segment of the streaming audio data ending at the validated initial punctuation.
However, Tundik describes a system and method to determine a number of newly decoded words to wait based on a context of output application associated with the streaming audio data (See section 2.2 “The MaxEnt model”, page 4, third paragraph: “The radius parameter of the MaxEnt tagger determines the size of the context considered before predicting a punctuation mark. By default, left (past) and right (future) context is taken into account. We will refer to this setup as off-line mode. As taking future context into account increases latency, we have to limit the future context. Operation by limited future context will be referred to as on-line mode. In the experiments we use round brackets to specify left and right context, respectively. Hence (5,1) means that we are considering 5 past and 1 future tokens actually. In on-line mode, we usually have to limit future context for a single token, which, depending on the length of the word and the speech rate, will result in a latency of several hundred milliseconds when spoken.” (emphasis added) Here, the on-line audio mode and off-line audio mode are contexts of output applications that determine the number of newly decoded words wait.
Also see section 2.3 “The recurrent neural network model”, page 5, first paragraph: “RNN models build up from Long-Short Term Memory (LSTM) cells. LSTM networks contain cells with a memory unit, preserving some information about past states of the cell Hochreiter and Schmidhuber (1997). The memory unit itself, as well as the output of the cell combined from a weighted contribution of the current input and the memory unit, are regulated by the data flow. These regulating weights are learned during the training phase. Connecting LSTM cells sequentially leads to powerful sequential models, whereby typically each cell receives the features at a given time frame. It is common to incorporate future features into the processing framework, that is, the output of the network at time t depends on inputs ranging from t – k..t..t + k. This is usually more effective if we allow for a bidirectional (from past to future and from future to past) flow of the information within the network (e.g. Bidirectional LSTM, BiLSTM Schuster and Paliwal (1997)). Obviously, the future is not known, so technically such networks wait until future samples are available, and delay their output accordingly. For on-line operation, as seen with the MaxEnt models, future again has to be limited to a single word. Adapting this framework to the word sequence we consider a token each time frame, that is word by word by indexing the words by t.” (emphasis added).
Finally, see section 4.1 “Hungarian overall results”, page 4 first paragraph, and Table 3 on page 5: “First, we compare the performance of the RNN-based punctuation recovery system (see Section 2.3) to the baseline MaxEnt sequence tagger (see Section 2.2) on the Hungarian broadcast dataset. Both approaches are presented in two configurations. While in the off-line mode, the future word context is also exploited to achieve the best result with the given features and architecture, we also focus on the performance degradation when switching to on-line mode, where the punctuation restoration depends only on the past word context and the target word (in case of MaxEnt, we allowed future context involving only a single word). This configuration is intended to fit to real-time closed captioning requirements with low latency. In both modes, punctuations are predicted for the slot preceding the target word (at t) in the input sequence. The hyperparameters of all presented approaches and their configurations were optimized on the validation set as explained earlier (see Section 2).
PNG
media_image1.png
246
892
media_image1.png
Greyscale
The obtained results are presented in Table 3 for the manual transcripts and in Table 4 for the automatic (ASR) transcripts, respectively. We remind the reader, that in the notation of MaxEnt models (i, j), i stands for the backward (past), whereas j stands for the forward (future) radius.” (emphasis added).
Note that Table 3 illustrates that in off-line mode the MaxEnt model waited for 19 future words before predicting punctuation, while in on-line mode the MaxEnt model only waited for 1 future word.);
See section 2.3 “The recurrent neural network model”, page 5, first paragraph: “RNN models build up from Long-Short Term Memory (LSTM) cells. LSTM networks contain cells with a memory unit, preserving some information about past states of the cell Hochreiter and Schmidhuber (1997). The memory unit itself, as well as the output of the cell combined from a weighted contribution of the current input and the memory unit, are regulated by the data flow. These regulating weights are learned during the training phase. Connecting LSTM cells sequentially leads to powerful sequential models, whereby typically each cell receives the features at a given time frame. It is common to incorporate future features into the processing framework, that is, the output of the network at time t depends on inputs ranging from t – k..t..t + k. This is usually more effective if we allow for a bidirectional (from past to future and from future to past) flow of the information within the network (e.g. Bidirectional LSTM, BiLSTM Schuster and Paliwal (1997)). Obviously, the future is not known, so technically such networks wait until future samples are available, and delay their output accordingly. For on-line operation, as seen with the MaxEnt models, future again has to be limited to a single word. Adapting this framework to the word sequence we consider a token each time frame, that is word by word by indexing the words by t.” (emphasis added).
Also see section 2.1 “Tolerated maximal latency”, page 4, first two paragraphs: “The closed captioning use-case we are optimizing for is as follows: an ASR decoder is providing the transcription in real-time on a quasi word-by-word basis, concatenating the words left-to-right. Once a row fills up with words, the lines are scrolled up. At the same time up to 3 lines can be visible, and a single line can contain up to 35 – 40 characters including spaces. Average word length was found 7.1 characters for the highly agglutinating Hungarian and 5.4 characters for English for the used datasets overall (see Section 3 for description of the used corpora).
Once a word or a group of words is put on the screen, the proper punctuation should already be in place, as readers do not have the chance of reading anything twice. Therefore, very low latency of punctuation is required, as already the ASR introduces some latency – considering a back-tracking triggered right after a word ends, we even delay the output by the length of the last word (a reader can start reading the word at this point by when a listener has just finished listening to it).” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch a system and method to determine a number of newly decoded words to wait based on a context of output application associated with the streaming audio data; [and] , as taught by Tundik, in order to enable more accurate punctuation of audio data converted to text by delaying the punctuation decisions as much as possible, given the time limitations associated with the audio being punctuated, which enables the system to use as much forward context as possible when determining which punctuation should be added to the final textual output.
Fritsch in view of Tundik doesn’t describe a system or method wherein subsequent to the initial punctuation, cause an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; upon determining that the initial punctuation is correct, cause the orchestrator to output for display on a user device a first portion of the initial segment.
However, Minixhofer describes a system and method wherein subsequent to the initial punctuation, cause an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; [and] upon determining that the initial punctuation is correct, cause the orchestrator to output for display on a user device a first portion of the initial segment (See “Abstract”, page i: “To improve readability, punctuation prediction is typically performed on text output by an Automatic Speech Recognition (ASR) model. We introduce a Transformer-based model to predict punctuation marks on unpunctuated text suitable for text streamed word-for-word, as is often the case for ASR models. We propose a decoding strategy that delays punctuation marks’ insertion in case of uncertainty until a specific threshold is reached. […] To make the model viable for real-time use in combination with an ASR system and on low-resource devices, we evaluate input truncation and weight quantization. We show these techniques lead to faster-than-real-time inference speeds and a significant reduction in model size.”
Also see section 3.2 “Varying Lookahead and Decoding”, page 18: “To predict punctuation in a streamed setting, we need a way to progressively feed the model more right-side context should it fail to predict punctuation with the context it is given initially, and make it robust to predicting sequences with varying lengths of this context.
Training
Using the [PUNCT] token described above, this can be achieved when training the model: We set the minimum and maximum lookahead (lmin and lmax), and then insert the [PUNCT] token at n – l for each sample, where n is the sequence length and l is the lookahead. The lookahead can be cycled through or drawn randomly from [lmin; lmax].
Inference
For inference, a decoding strategy utilising varying lookahead is needed, which we propose in Algorithm 1. The first question that presents itself is how we decide if the system is predicting punctuation with reasonable confidence, or if more context is needed. We solve this by computing the Shannon-Entropy (Shannon, 1948) H over the set of probabilities pi,…,pk [in the set of] P assigned to each of the k punctuation marks after the softmax step.
PNG
media_image2.png
62
354
media_image2.png
Greyscale
This value can be understood as the uncertainty of the model, and will be lower when the model is more certain of a prediction, being 0 when one probability is 1 and all others are 0. Given the four possible outcomes of comma, period, question mark and no punctuation, the maximum value is reached when all probabilities are ¼ which corresponds with H(P) = 2. For decoding, we set an entropy threshold h and wait for more right-side context and repeat inference if the computed entropy H(P)>h. This is repeated until H(P) [less than or equal to] h or lmax is reached. As words are streamed into the system, we potentially do inference on multiple punctuation positions at the same time step, with the maximum number of inferences conducted at the same time being lmax – lmin + 1, as shown in Figure 3.3.”). (emphasis added).
Here, the system only adds punctuation to the ASR output if computed entropy is below a threshold value (which means the confidence level is above a threshold level) for the current amount of right-side context, where the system will wait for more right-side context when the computed entropy currently above the threshold value, where the computed entropy is used to, subsequent to the initial punctuation, cause an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; [and] upon determining that the initial punctuation is correct, cause the orchestrator to output for display on a user device a first portion of the initial segment.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch in view of Tundik a system and method to, subsequent to the initial punctuation, cause an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; [and] upon determining that the initial punctuation is correct, cause the orchestrator to output for display on a user device a first portion of the initial segment, as taught by Minixhofer, in order to improve the accuracy of the final punctuated text by varying the lookahead based on the system’s confidence in the current punctuation prediction, which leads to more accurate punctuation determinations while delaying the final output as little as possible.
Regarding claim 37, Fritsch doesn’t describe but Tundik describes the computing system of claim 19, wherein the context of output application is a final transcript (See section 1 “Introduction”, top of page 2: “On the other hand, dictating punctuation marks is not possible or tolerable in other use-cases: transcripts of meeting records, closed captioning, user friendly dictation or applications requiring speech understanding based on text analysis all involve the proper and automatic insertion of punctuation marks.” (emphasis added)
Further, see section 4.1 “Hungarian overall results”, page 4 first paragraph, and Table 3 on page 5: “First, we compare the performance of the RNN-based punctuation recovery system (see Section 2.3) to the baseline MaxEnt sequence tagger (see Section 2.2) on the Hungarian broadcast dataset. Both approaches are presented in two configurations. While in the off-line mode, the future word context is also exploited to achieve the best result with the given features and architecture, we also focus on the performance degradation when switching to on-line mode, where the punctuation restoration depends only on the past word context and the target word (in case of MaxEnt, we allowed future context involving only a single word). This configuration is intended to fit to real-time closed captioning requirements with low latency. In both modes, punctuations are predicted for the slot preceding the target word (at t) in the input sequence. The hyperparameters of all presented approaches and their configurations were optimized on the validation set as explained earlier (see Section 2).
PNG
media_image1.png
246
892
media_image1.png
Greyscale
The obtained results are presented in Table 3 for the manual transcripts and in Table 4 for the automatic (ASR) transcripts, respectively. We remind the reader, that in the notation of MaxEnt models (i, j), i stands for the backward (past), whereas j stands for the forward (future) radius.” (emphasis added)).
See the motivation to combine Tundik with Fritsch in the rejection of claim 19, as it is equally applicable here.
Claim(s) 32 is rejected under 35 U.S.C. 103 as being unpatentable over Fritsch in view of Minixhofer.
Regarding claim 32, Fritsch describes a method implemented by a computing system for generating transcriptions of audio data, the method comprising:
obtaining streaming audio data comprising language utterances from a speaker (col. 6 lns. 1-9: “The system 100 includes a speech decoder 102. In general, the decoder 102 derives a sequence of tokens 104a-c, also referred to herein as a token stream 104, such as a sequence of words and/or punctuation marks, from speech 106. The speech 106 may take any form, such as a live audio signal generated using a microphone and transmitted by the microphone to a computer or other recording device, or a recorded audio signal stored on an analog or digital recording medium and subsequently read by the decoder 102.” (emphasis added));
causing a decoder of the computing system to continuously decode the streaming audio data in order to generate decoded streaming audio data (col. 6 lns. 53-60: “In general, the decoder 102 decodes a first portion of the speech 106 into a first word (or other token) (FIG. 2, step 202). Assume for purposes of example that in step 202 the decoder 102 decodes portion 106a of speech 106 to produce first word 104a. Such decoding may be performed using any speech decoding technique. The decoder 102, or other component of the system 100, inserts the first word 104a into the data structure 104 (e.g., document) (step 204).” (emphasis added));
determining whether a linguistic boundary exists within an initial segment of decoded streaming audio data (col. 6 ln. 61 – col. 6 ln. 4: “The decoder 102 then predicts the occurrence of a non-verbalized punctuation symbol 104b in the speech stream 106, where the predicted punctuation symbol 104b occurs in sequence after the first token 104a in the sequence of tokens 104 (step 206). As will be described in more detail below, this prediction may include considering a plurality of possible tokens (possibly including a plurality of distinct punctuation symbols, such as a period, comma, and question mark). The decoder 102, or other component of the system 100, inserts the predicted punctuation symbol 104b into the token stream 104 at a position after that of the first word 104a (step 208).” (emphasis added));
when a linguistic boundary is determined to exist, causing a punctuator of the computing system to apply an initial punctuation at the linguistic boundary (col. 7 lns. 5-27: “When creating a document from the token stream, whitespace and other formatting may be applied to the token stream 104 in any manner. For example, if the token stream 104 is used at some point to create a text document, the position of the punctuation symbol 104b within such a text document may, for example, be immediately after that of the first word 104a within the text document (such as by inserting a period after "day" to produce the text "day."). As another example, the punctuation symbol 104b may be inserted into the text document such that the first word 104a and the punctuation symbol 104b are separated by whitespace (and/or other inconsequential characters) but by no other words or characters (such as by inserting a dash after a space to produce the text "day--").
The decoder 102 then decodes a second portion of the speech 106 into a second word (or other token) (step 210), making use of the fact that a non-verbalized punctuation was predicted immediately before the second word. Assume for purposes of example that the decoder 102 decodes portion 106b of speech 106 to produce word 104c. The decoder 102, or other component of the system 100, inserts the second word 104c into the token stream 104 at a position in the token stream 104 after that of the punctuation mark 104b (step 212).” (emphasis added));
Fritsch doesn’t describe a system or method that includes determining a number of newly decoded words to wait based on a speaking style associated with the speaker represented in the streaming audio data; subsequent to the initial punctuation, causing an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; and upon determining that the initial punctuation is correct, outputting for display on a user display a first portion of the initial segment of the streaming audio data ending at the initial punctuation.
However, Minixhofer describes a system and method including
determining a number of newly decoded words to wait based on a speaking style associated with the speaker represented in the streaming audio data (See “Abstract”, page i: “To improve readability, punctuation prediction is typically performed on text output by an Automatic Speech Recognition (ASR) model. We introduce a Transformer-based model to predict punctuation marks on unpunctuated text suitable for text streamed word-for-word, as is often the case for ASR models. We propose a decoding strategy that delays punctuation marks’ insertion in case of uncertainty until a specific threshold is reached. […] To make the model viable for real-time use in combination with an ASR system and on low-resource devices, we evaluate input truncation and weight quantization. We show these techniques lead to faster-than-real-time inference speeds and a significant reduction in model size.”
Also see section 3.2 “Varying Lookahead and Decoding”, page 18: “To predict punctuation in a streamed setting, we need a way to progressively feed the model more right-side context should it fail to predict punctuation with the context it is given initially, and make it robust to predicting sequences with varying lengths of this context.
Training
Using the [PUNCT] token described above, this can be achieved when training the model: We set the minimum and maximum lookahead (lmin and lmax), and then insert the [PUNCT] token at n – l for each sample, where n is the sequence length and l is the lookahead. The lookahead can be cycled through or drawn randomly from [lmin; lmax].
Inference
For inference, a decoding strategy utilising varying lookahead is needed, which we propose in Algorithm 1. The first question that presents itself is how we decide if the system is predicting punctuation with reasonable confidence, or if more context is needed. We solve this by computing the Shannon-Entropy (Shannon, 1948) H over the set of probabilities pi,…,pk [in the set of] P assigned to each of the k punctuation marks after the softmax step.
PNG
media_image2.png
62
354
media_image2.png
Greyscale
This value can be understood as the uncertainty of the model, and will be lower when the model is more certain of a prediction, being 0 when one probability is 1 and all others are 0. Given the four possible outcomes of comma, period, question mark and no punctuation, the maximum value is reached when all probabilities are ¼ which corresponds with H(P) = 2. For decoding, we set an entropy threshold h and wait for more right-side context and repeat inference if the computed entropy H(P)>h. This is repeated until H(P) [less than or equal to] h or lmax is reached. As words are streamed into the system, we potentially do inference on multiple punctuation positions at the same time step, with the maximum number of inferences conducted at the same time being lmax – lmin + 1, as shown in Figure 3.3.”). (emphasis added).
Finally, see the “Summary” section, page 23: “We have introduced a novel way to create a classification punctuation annotation system using a special [PUNCT] token. We have proposed a training procedure utilising this approach to train a model specifically on samples with little or no right-side context, to create a model well-suited for the streamed punctuation annotation task. For inference, we have introduced Entropy Threshold Decoding, which varies the lookahead needed based on model certainty. We have hypothesised that this will lead to easier samples being predicted early while the model will wait for more context for harder samples.” (emphasis added)
Here, the sequences with varying lengths, as discussed in section 3.2, and the easier samples and harder samples, as discussed in the Summary, indicate a speaking style associated with the speaker represented in the streaming audio data, where the system determines a number of newly decoded words to wait based on the speaking style);
subsequent to the initial punctuation, causing an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait (See section 3.2 “Varying Lookahead and Decoding”, page 18: “To predict punctuation in a streamed setting, we need a way to progressively feed the model more right-side context should it fail to predict punctuation with the context it is given initially, and make it robust to predicting sequences with varying lengths of this context.
Training
Using the [PUNCT] token described above, this can be achieved when training the model: We set the minimum and maximum lookahead (lmin and lmax), and then insert the [PUNCT] token at n – l for each sample, where n is the sequence length and l is the lookahead. The lookahead can be cycled through or drawn randomly from [lmin; lmax].
Inference
For inference, a decoding strategy utilising varying lookahead is needed, which we propose in Algorithm 1. The first question that presents itself is how we decide if the system is predicting punctuation with reasonable confidence, or if more context is needed. We solve this by computing the Shannon-Entropy (Shannon, 1948) H over the set of probabilities pi,…,pk [in the set of] P assigned to each of the k punctuation marks after the softmax step.
PNG
media_image2.png
62
354
media_image2.png
Greyscale
This value can be understood as the uncertainty of the model, and will be lower when the model is more certain of a prediction, being 0 when one probability is 1 and all others are 0. Given the four possible outcomes of comma, period, question mark and no punctuation, the maximum value is reached when all probabilities are ¼ which corresponds with H(P) = 2. For decoding, we set an entropy threshold h and wait for more right-side context and repeat inference if the computed entropy H(P)>h. This is repeated until H(P) [less than or equal to] h or lmax is reached. As words are streamed into the system, we potentially do inference on multiple punctuation positions at the same time step, with the maximum number of inferences conducted at the same time being lmax – lmin + 1, as shown in Figure 3.3.”). (emphasis added)); and
upon determining that the initial punctuation is correct, outputting for display on a user display a first portion of the initial segment of the streaming audio data ending at the initial punctuation (See section 3.1 “Masked Punctuation Prediction”, page 16, first paragraph: “In real-time settings, there is a lack of right-side context: an ASR streams words to the punctuation system and punctuation is added continuously. The less right-side context a model needs, the earlier can a punctuation mark be inserted into the final output.” (emphasis added).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch a system and method that includes determining a number of newly decoded words to wait based on a speaking style associated with the speaker represented in the streaming audio data; subsequent to the initial punctuation, causing an orchestrator of the computing system to validate that the initial punctuation is correct using the number of newly decoded words to wait; and upon determining that the initial punctuation is correct, outputting for display on a user display a first portion of the initial segment of the streaming audio data ending at the initial punctuation, as taught by Minixhofer, in order to enable accurate punctuation with minimal delay by varying the amount of right-side context based on the complexity of the sentences being spoken, which ensures accurate punctuation for a wide range of speakers.
Claim 2-7, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Fritsch in view of Tundik in view of Minixhofer, as applied to claim 1 above, and further in view of Stefani [U.S. 10777186 B1].
Regarding Claim 2, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: store the initial segment of decoded streaming audio data in a cache; and after the orchestrator outputs the first portion of the initial segment, clear the cache of the first portion of the initial segment of the decoded streaming audio data.
Stefani discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: store the initial segment of decoded streaming audio data in a cache; and [Stefani: col. 8 line 45-58, col. 3, line 45-48 — “…the frontend service 112 can buffer audio data being streamed in audio buffer 206. Audio buffer 206 may be implemented in a storage service 208 that provides data storage to frontend service 112 and/or using local disk storage space and/or memory available to frontend service 112. The audio buffer 206 a pre-connection audio buffer which may be used to buffer audio data from the data stream while a connection to decoder host 204 is being established.….” [col. 8 line 45-58] Additionally, “…the frontend service 112 may begin buffering audio from the associated audio stream when the request is received and prior to a connection being made with audio decoder service 118…” [col. 3, line 45-48]
after the orchestrator outputs the first portion of the initial segment, clear the cache of the first portion of the initial segment of the decoded streaming audio data. [Stefani: col. 4, line 6-10— “…the streaming engine can reset the processing chain, including all buffers in between, ensuring that any data that was stored from a previous session has been purged and is not mixed with data from a new streaming session...….” This indicates the intent to manage and clear data efficiently during audio processing [col. 4, line 6-10]
Fritsch et al. and Stefani are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Stefani, which provides a method for real-time automated speech recognition (ASR) that efficiently processes audio streams, generating accurate transcript by analyzing audio data using acoustic and language models [Stefani: col. 2, line 1-20].
Regarding Claim 3, Fritsch et al. does not teach the computing system of claim 2, the computer-executable instructions further executable by the processor to cause the computing system to: while clearing the cache of the first portion of the initial segment of decoded streaming audio data, cause the orchestrator to retain the second portion of the segment of decoded streaming audio data in the cache.
Stefani discloses:
the computing system of claim 2, the computer-executable instructions further executable by the processor to cause the computing system to: while clearing the cache of the first portion of the initial segment of decoded streaming audio data, cause the orchestrator to retain the second portion of the segment of decoded streaming audio data in the cache.
[Stefani: col. 8, line 46-56, col. 4, line 6-10— “… Audio buffer 206 may be implemented in a storage service 208 that provides data storage to frontend service 112 and/or using local disk storage space and/or memory available to frontend service 112. The audio buffer 206 a pre-connection audio buffer which may be used to buffer audio data from the data stream while a connection to decoder host 204 is being established. Audio buffer 206 may additionally, or alternatively, be used to backup audio data from the audio data stream in the event of decoder host failure after a connection to the decoder host has been made.….” [col. 8, line 46-56]. “…the streaming engine can reset the processing chain, including all buffers in between, ensuring that any data that was stored from a previous session has been purged and is not mixed with data from a new streaming session...….” This indicates the intent to manage and clear data efficiently during audio processing [col. 4, line 6-10]
Fritsch et al. and Stefani are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Stefani, which provides a method for real-time automated speech recognition (ASR) that efficiently processes audio streams, generating accurate transcript by analyzing audio data using acoustic and language models [Stefani: col. 2, line 1-20].
Regarding Claim 4, Fritsch et al. doesn’t teach but Stefani teaches:
The computing system of claim 3, the computer-executable instructions further executable by the processor to cause the computing system to:
store a subsequent segment of decoded streaming audio data in the cache, wherein the orchestrator appends the subsequent segment of decoded streaming audio data to the second portion of the initial segment of decoded streaming audio data to form a new segment of decoded streaming audio data; [Stefani: col. 6, line 59-62, FIG. 2, col. 10, line 10-17— “… the audio data stream is forwarded by the frontend service to the decoder host, allowing the user's audio stream to be sent directly to the decoder host performing the transcription,” meaning, ability to combine segments [col. 6, line 59-62] “…the frontend service 112 can buffer audio data being streamed in audio backup 206. Audio backup 206 may be implemented in a storage service that provides data storage to frontend service 112 and/or using local disk storage space and/or memory available to frontend service 112. The audio backup 206 may be used to buffer audio data from the data stream while a connection to decoder host 204 is being established.….” This implies the temporary storage of audio segments [col. 10, line 10-17
PNG
media_image5.png
580
826
media_image5.png
Greyscale
].
determine whether a subsequent linguistic boundary exists within the new segment of decoded streaming audio data; [Stefani: col. 4. line 38-55— “…models 126 can include an acoustic model and a language model. The streaming ASR engine 124 break the audio stream into portions of audio data (also referred to herein as “chunks”). In some embodiments, the streaming ASR engine 124 can identify silence within the audio data stream and use the silence as a boundary between chunks. The streaming ASR engine may be configured with a limit as to the length of a given chunk (e.g., 30 seconds). For each chunk of audio data, the streaming ASR engine can use the acoustic model to break the audio data into a series of words. The output of the acoustic model can be passed through the language model to identify phrases and/or sentences corresponding to the series of words identified by the acoustic model.….”, highlighting the identification of boundaries in the audio stream. [col. 4. line 38-55]. Additionally, “…The language model includes grammar rules, language constructs, and other language-specific nuances. In some embodiments, the language model can include a custom dictionary that includes jargon or other domain specific words or phrases,” which plays a role in detailing linguistic boundaries within the decoded segments [col. 4. line 38-55].
when a subsequent linguistic boundary is determined to exist, cause the punctuator to apply a new punctuation at the subsequent linguistic boundary; and [Stefani: col. 4, line 48-55— “… The output of the acoustic model can be passed through the language model to identify phrases and/or sentences corresponding to the series of words identified by the acoustic model. The language model includes grammar rules, language constructs, and other language-specific nuances. In some embodiments, the language model can include a custom dictionary that includes jargon or other domain specific words or phrases.” This indicates processing and punctuating based on linguistic context in real-time transcription. [col. 4, line 48-55]
cause the orchestrator to validate a first portion of the new segment of the streaming audio data ending at the subsequent linguistic boundary, and if validated, output the validated first portion of the new segment for display on the user device while refraining from outputting a second portion of the new segment located temporally subsequent to the second portion of the initial segment. [Stefani: col. 4 line 39-51— “…The streaming ASR engine 124 break the audio stream into portions of audio data (also referred to herein as “chunks”),” It’s using silence as a boundary, indicating the identification of segments a linguistic boundary. Additionally, it explains that “the output of the acoustic model can be passed through the language model to identify phrases and/or sentences corresponding to the series of words identified by the acoustic model,” which suggest that outputs are managed around linguistic boundaries, which indicate that portions of audio data can be output while excluding subsequent segment based on temporal consideration. [col. 4 line 39-51]
Fritsch et al. and Stefani are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Stefani, which provides a method for real-time automated speech recognition (ASR) that efficiently processes audio streams, generating accurate transcript by analyzing audio data using acoustic and language models [Stefani: col. 2, line 1-20].
Regarding Claim 5, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: apply the punctuation prior to outputting the first portion of the initial segment of decoded streaming audio data.
Stefani discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: apply the punctuation prior to outputting the first portion of the initial segment of decoded streaming audio data.
[Stefani: col. 4, line 48-51, col. 2, line 15-20— “…The output of the acoustic model can be passed through the language model to identify phrases and/or sentences corresponding to the series of words identified by the acoustic model.….” This indicates that punctuation is applied as part of the processing before the output of the transcription [col. 4, line 48-51]. “The acoustic model can be trained using short audio sentence data (e.g., on the order of 30 seconds to a few minutes), enabling the transcription service to accurately transcribe short chunks of audio data. The results are then punctuated and normalized. The resulting transcript is then streamed back to the user over the bi-directional connection.” [col. 2, line 15-20]
Fritsch et al. and Stefani are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Stefani, which provides a method for real-time automated speech recognition (ASR) that efficiently processes audio streams, generating accurate transcript by analyzing audio data using acoustic and language models [Stefani: col. 2, line 1-20].
Claim 6, Fritsch et al. in view of Stefani does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: apply the punctuation while outputting the first portion of the initial segment of decoded streaming audio data.
Stefani discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: apply the punctuation while outputting the first portion of the initial segment of decoded streaming audio data.
[Stefani: col. 4, line 48-51, col. 2, line 15-20— “…The output of the acoustic model can be passed through the language model to identify phrases and/or sentences corresponding to the series of words identified by the acoustic model.….” This indicates that punctuation is applied as part of the processing before the output of the transcription [col. 4, line 48-51]. “The acoustic model can be trained using short audio sentence data (e.g., on the order of 30 seconds to a few minutes), enabling the transcription service to accurately transcribe short chunks of audio data. The results are then punctuated and normalized. The resulting transcript is then streamed back to the user over the bi-directional connection.” [col. 2, line 15-20]
Fritsch et al. and Stefani are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Stefani, which provides a method for real-time automated speech recognition (ASR) that efficiently processes audio streams, generating accurate transcript by analyzing audio data using acoustic and language models [Stefani: col. 2, line 1-20].
Claim 7, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: apply the punctuation after outputting the first portion of the initial segment of decoded streaming audio data.
Stefani discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: apply the punctuation after outputting the first portion of the initial segment of decoded streaming audio data.
[Stefani: col. 4, line 48-51, col. 2, line 15-20— “…The output of the acoustic model can be passed through the language model to identify phrases and/or sentences corresponding to the series of words identified by the acoustic model.….” This indicates that punctuation is applied as part of the processing before the output of the transcription [col. 4, line 48-51]. “The acoustic model can be trained using short audio sentence data (e.g., on the order of 30 seconds to a few minutes), enabling the transcription service to accurately transcribe short chunks of audio data. The results are then punctuated and normalized. The resulting transcript is then streamed back to the user over the bi-directional connection.” [col. 2, line 15-20]
Fritsch et al. and Stefani are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Stefani, which provides a method for real-time automated speech recognition (ASR) that efficiently processes audio streams, generating accurate transcript by analyzing audio data using acoustic and language models [Stefani: col. 2, line 1-20].
Regarding Claim 13, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: identify an action item within the first portion of the initial segment of decoded streaming audio data; and
Stefani discloses:
identify an action item within the first portion of the initial segment of decoded streaming audio data; and [Stefani: col. 13, line 6-10— “the operations may further include decoding a first portion of the audio stream, identifying an audio stream type based on the first portion of the audio stream, adding an identifier associated with the audio stream type to an entry associated with the request in a metadata service, and wherein the decoder host is configured to decode the audio stream type.” This indicates the capability of recognizing specific elements or actions from the decoded audio data.
apply a visual formatting corresponding to the action item associated with the first portion of the initial segment of decoded streaming audio data to the first portion of the initial segment of decoded streaming audio data, wherein different visual formatting is applied to decoded streaming audio data associated with a different action item at the display of the user device [Stefani: col. 2, line 9-20— “A bi-directional connection is established with the audio decoder host which includes a streaming ASR engine. The streaming ASR engine can analyze chunks of the audio data stream using an acoustic model to divide the audio data into words, and a language model to identify sentences made of the words spoken in the audio file. The acoustic model can be trained using short audio sentence data (e.g., on the order of 30 seconds to a few minutes), enabling the transcription service to accurately transcribe short chunks of audio data. The results are then punctuated and normalized. The resulting transcript is then streamed back to the user over the bi-directional connection.” This implies the possibility of distinct visual formatting based on the actions associated with the audio data displayed to the user [col. 2, line 9-20].
Fritsch et al. and Stefani are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Stefani, which provides a method for real-time automated speech recognition (ASR) that efficiently processes audio streams, generating accurate transcript by analyzing audio data using acoustic and language models [Stefani: col. 2, line 1-20].
Claim(s) 8 and 34 are rejected under 35 U.S.C. 103 as being unpatentable over Fritsch in view of Tundik in view of Minixhofer, as applied to claim 1 above, and further in view of Arivazhagan et al. (US 11,562,152, herein “Arivazhagan”).
Regarding Claim 8, Fritsch in view of Tundik in view of Minixhofer doesn’t describe, but Arivazhagan describes the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to:
when a linguistic boundary is determined not to exist within the initial segment of decoded streaming audio data, cause the orchestrator to refrain from outputting the initial segment of decoded streaming audio data (col. 9 lns. 22-49: “In some implementations, the computer system 110 performs post-processing operations 138 on the ASR results to improve stability of the ASR output before providing transcribed text to the MT module 140 for translation. The post-processing 138 can include one or more of (1) suppressing final punctuation, (2) suppressing tailing tokens, and (3) forcing all ASR updates to be append-only. The first two are heuristics designed to relieve pressure on the final technique, which actually produces the guarantee of stability for the cumulative transcription 136.
First, the computer system can suppress final (e.g., sentence-ending) punctuation (e.g., a period, question mark, exclamation mark, etc.) until the end of a speech utterance is detected. The punctuation predictions of an ASR model or a punctuation model are not always accurate, and may in some cases a final punctuation element may be output before the speaker 102 has actually finished the sentence. Punctuation predictions made based on language models and the recognized words, even if they have high confidence for the current set of received speech, are subject to change as additional speech is received.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch in view of Tundik in view of Minixhofer a system and method wherein when a linguistic boundary is determined not to exist within the initial segment of decoded streaming audio data, refrain from outputting the initial segment of decoded streaming audio data, as taught by Arivazhagan, in order to improve the accuracy of the transcription by delaying the output of a segment until the end of a sentence is detected, which helps to ensure that the final punctuation applied accurately depicts what was spoken in the streaming audio data.
Regarding claim 34, Fritsch in view of Tundik in view of Minixhofer doesn’t describe, but Arivazhagan describes the computing system of claim 1, the punctuator being an integrated punctuator of the orchestrator (col. 8 lns. 42-48: “The computer system 110 can also include a punctuation model 131, which may be integrated with the ASR module 130 or be separate, to determine where to place punctuation in the incrementally received transcribed text 133 and/or in the cumulative transcription 136 that includes the accumulated outputs from a series of ASR outputs based on different audio data from the audio data stream 108.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Fritsch in view of Tundik in view of Minixhofer a system and method including the punctuator being an integrated punctuator of the orchestrator, as taught by Aravazhagan, in order to enable fast communication between the orchestrator and the punctuator, while also reducing the size of the corresponding hardware.
Claim 9, 29 are rejected under 35 U.S.C. 103 as being unpatentable over Fritsch in view of Tundik in view of Minixhofer, as applied to claims 1 and 19 above, and further in view of Rangarajan [U.S. 20150134320].
Regarding Claim 9, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: determine whether the linguistic boundary exists based on a user command OR a detected keyword recognized within the initial segment of decoded streaming audio data.
Rangarajan discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: determine whether the linguistic boundary exists based on a user command OR a detected keyword recognized within the initial segment of decoded streaming audio data.
[Rangarajan: Section 0009— “…method and computer-readable media are disclosed which balance latency and accuracy of machine translations by segmenting the speech upon locating a conjunction. Exemplary conjunctions include "or" and "and." As an example, the system, upon receiving speech, will buffer the speech until an "and" or an "or" is detected. Upon detecting a conjunction, the speech received until that point is segmented to yield a segment. The system then continues performing speech recognition on the segment, searching for the next conjunction, while simultaneously initiating translation of the segment….” [Section 0009]
Fritsch et al. and Rangarajan are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Rangarajan, which optimizes real-time speech translation through effective segmentation based on conjunctions, balancing latency and translation accuracy Rangarajan: Section 0002-0004].
Regarding Claim 29, Fritsch et al. does not teach the computing system of claim 19, the computer-executable instructions being further executable by the processor to cause the computing system to: determine a number of newly decoded words to wait based on a pre-determined accuracy of the computing system in applying the initial punctuation.
Rangarajan discloses:
the computing system of claim 19, the number of newly decoded words to wait being based on a pre-determined accuracy of the computing system in applying the initial punctuation. [Rangarajan: Section 0050— “…Regarding accuracy and latency segmentation strategies, these strategies can work in conjunction with the conjunction strategy to ensure the user is receiving speech within an accuracy threshold and a latency threshold. For example, if the system 100 determines the latency is above a threshold (meaning the second user is not receiving the translated speech output within a determined amount of time) the segments will be shortened in length. Such shortening can occur by increasing the amount of punctuation searched for, the conjunctions searched for, or by decreasing the maximum number of words received before segmentation occurs. Likewise, if the system 100 determines the accuracy of the translation output is below a threshold, the system can increase the segment lengths by reducing punctuation/conjunctions/words searched for and increasing a maximum number of words per segment ….” [Section 0050]
Fritsch et al. and Rangarajan are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Rangarajan, which optimizes real-time speech translation through effective segmentation based on conjunctions, balancing latency and translation accuracy Rangarajan: Section 0002-0004].
Claim 11-12, 14-17, 21-23, 27, and 31 rejected under 35 U.S.C. 103 as being unpatentable over Fritsch in view of Tundik in view of Minixhofer, as applied to claims 1 and 19 above, and further in view of Thomson [U.S. 20200175961].
Regarding Claim 11, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: recognize a sentiment associated the first portion of the initial segment of decoded streaming audio data; and
Thomson discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to:
recognize a sentiment associated the first portion of the initial segment of decoded streaming audio data; and [Thomson: Section 0479— “…the enhanced transcription generator 1702 may be configured to analyze the audio of the speaker and/or the transcriptions generated by the transcription system 1708 to identify emotion, word emphasis, key words, and/or phrases, among other aspects of a conversation.” “The detector 1720 may also be configured to associate the determined emotion and word emphasis with associated words in the transcription.” [Section 0479]
apply a visual formatting corresponding to the sentiment associated with the first portion of the initial segment of decoded streaming audio data to the first portion of the initial segment of decoded streaming audio data, wherein a different visual formatting is applied to a different sentiment detected for a different portion of decoded streaming audio data. [Thomson: Section 0479 “The detector 1720 may also be configured to associate the determined emotion and word emphasis with associated words in the transcription.” And provide “symbols, such as words, characters, or other symbols that may be added to the transcription to attempt to convey the determined emotion and word emphasis.” This shows the general idea of associating visual attributes with audio sentiment [Section 0479]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 12, Fritsch et al. doesn’t teach but Thomson teaches:
The computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to:
identify a speaker role attributed to a speaker of the first portion of the initial segment of decoded streaming audio data; and [Thomson: section 1399-1405— “The information may include text and/or caller audio from the first communication session. [1400] b. The information may include data extracted from text and/or caller audio from the first communication session. [1401] c. The information may include at least part of an ASR model. [1402] d. The information may include features extracted from an ASR model. [1403] 3. The adapter 7412 may use the first caller profile to adapt a speech model to the first caller's voice, creating a first adapted speech model. [1404] 4. The adapter 7412 may use a second caller profile (obtained from a second caller's voice sample on a second communication session in a manner similar to that of the first caller profile) to adapt a speech model to the second caller's voice, creating a second adapted speech model. [1405] 5. An ASR system may use the first adapted speech model to transcribe audio from the first caller to create a transcription.”]
apply a visual formatting corresponding to the speaker role attributed to the speaker of the first portion of the initial segment of decoded streaming audio data, wherein a different visual formatting is applied to decoded streaming audio data attributed to a different speaker role. [Thomson: Section 0779— “the transcription unit 3714 may also send indicators to a user device so that the user device can display speaker information such as “New speaker:” or “Female Speaker:” or “Party 3:” to the subscriber. Such indicators may also be sent to the CA by, for example, providing a panel light, a lit region on the captioning assistance (CA) screen, or a displayed text advisement to notify the CA which speaker is talking and when the speaker changes. The visual indicator may be a signal for the CA to resume revoicing. For example, a region of the screen may dim or change color when the subscriber is speaking,” This indicates the concept of different speakers in visual formatting based on the role of different speakers in the decoded audio data. [Section 0779]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 14, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: identify external content associated with the first portion of the initial segment of decoded streaming audio data;
Thomson discloses
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: identify external content associated with the first portion of the initial segment of decoded streaming audio data;
[Thomson: Section 0497— “when words are presented on the display 1705 that may be associated with data for which other aspects of the device 1704 may use to perform functions, such as telephone numbers, email addresses, references to additional information, URLs or other Internet addresses, etc., links to allow usage of the data may be created. “a word may be displayed with a color change or with underlining to indicate that the word is associated with a link.” [Section 0497]
embed a link within the first portion of the initial segment of decoded streaming audio data to the first portion of the initial segment of decoded streaming audio data, wherein the link directs a user to the external content; and [Thomson: Section 0497— “when words are presented on the display 1705 that may be associated with data for which other aspects of the device 1704 may use to perform functions, such as telephone numbers, email addresses, references to additional information, URLs or other Internet addresses, etc., links to allow usage of the data may be created. “a word may be displayed with a color change or with underlining to indicate that the word is associated with a link.” [Section 0497]
display the link as a selectable object at the display of the user device. [Thomson: Section 0497— “when words are presented on the display 1705 that may be associated with data for which other aspects of the device 1704 may use to perform functions, such as telephone numbers, email addresses, references to additional information, URLs or other Internet addresses, etc., links to allow usage of the data may be created. “a word may be displayed with a color change or with underlining to indicate that the word is associated with a link.” [Section 0497]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 15, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: obtain streaming audio data comprising language utterances from multiple speakers and multiple audio input devices, wherein the streaming audio data is separated into a plurality of audio data streams according to each audio input device such that each audio data stream and each audio stream is analyzed by a different orchestrator.
Thomson discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to:
obtain streaming audio data comprising language utterances from multiple speakers and multiple audio input devices, wherein the punctuator separates the streaming audio data into a plurality of audio data streams according to each audio input device such that each audio data stream and each audio stream is analyzed by a different orchestrator.
[Thomson: Section 0827, 0830-0831— “The multiple input ASR system 4002 may be configured to process multiple audio inputs. The multiple audio inputs may include received audio and revoiced audio. the multiple audio inputs include audio and revoiced audio from each of three audio interfaces 4018a-c, collectively audio interfaces 4108.” [Section 0827]. “. the multiple input ASR system 4002 may be configured to receive input from the audio interfaces 4018 and from the regular audio. The multiple input ASR system 4002 may include feature extractors 4004a and 4004b, collectively the feature extractors 4004, for extracting features from the revoiced audio and regular audio, respectively.” [Section 0830]. “The outputs of the feature extractors 4004 may be communicated to the joint processor 4010. The joint processor 4010 may include components of an ASR system as described above with reference to FIG. 5, including to a feature transformer, probability calculator, rescorer, capitalizer, punctuator, and scorer, among others,” which indicates separate analysis streams for different audio inputs. [FIG. 5, Section 0831
PNG
media_image6.png
621
845
media_image6.png
Greyscale
]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 16, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: obtain multiple outputs and combine the multiple outputs into a paragraph prior to transmitting the paragraph to an output for display.
Thomson discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: obtain multiple outputs from one or more orchestrators and combine the multiple outputs into a paragraph prior to transmitting the paragraph to an output for display. [Thomson: Section 0681, 0477— “the display of the text editor 3126 or audio signals provided by the audio interface 3122 may be configured to draw the CA's attention to areas most likely to need correction. For example, the display may indicate ASR system confidence via color coding, highlighting, changes in font, brightness changes, or by other variations in the visual presentation.” This shows a method visually formatting data according to different parameters. Additionally, “… the presentation of the transcription such that the presented transcription provides context regarding emotion and word emphasis from the audio.” [Section 0681]. “..The presentation of the transcription may be adjusted by adjusting the presentation of certain words in the transcription or adding symbols to the transcription,” which imply different visual formats for varying action or context.” [Section 0477].
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 17, Fritsch et al. does not teach the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: obtain multiple outputs from different speakers; and
Thomson discloses:
the computing system of claim 1, the computer-executable instructions further executable by the processor to cause the computing system to: obtain multiple outputs from different speakers; and
[Thomson: Section 1029, 0827 and 0623 — “the audio inputs to the transcription units 5414 may be substantially identical, being derived from a common source such as audio from a communication session. Alternatively, audio inputs may be derived from multiple sources.” [Section 1029] “The multiple input ASR system 4002 may combine information from the multiple audio streams to create a transcription.” [Section 0827]. “The outputs of the transcription units may be fused.” [Section 0623]. All these collectively illustrate processes for obtaining outputs from multiple speakers.
combine outputs according to each of the different speakers. [Thomson: Section 0827, 0562 — “The multiple input ASR system 4002 may combine information from the multiple audio streams to create a transcription,” which indicates integration of audio outputs based on different speakers.” [Section 0827]. “the comparison between transcriptions may be used for accuracy estimation purposes of ASR systems, for determining difficulty of transcribing the audio, for determining which transcription to select when fusing outputs from multiple transcription units, or for classification, among other purposes.” [Section 0562]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 21, Fritsch et al. does not teach the computing system of claim 19, the computer-executable instructions being further executable by the processor to cause the computing system to: upon determining that the initial punctuation is not correct, remove the initial punctuation from the initial segment of decoded streaming audio data; and refrain from outputting the initial segment of decoded streaming audio data.
Thomson discloses:
the computing system of claim 19, the computer-executable instructions being further executable by the processor to cause the computing system to: upon determining that the initial punctuation is not correct, remove the initial punctuation from the initial segment of decoded streaming audio data; and cause and orchestrator to refrain from outputting the initial segment of decoded streaming audio data
[Thomson: Section 0441-0442—Capitalization and punctuation may be provided in transcriptions that are provided to a user device for presentation. In some embodiments, a transcription may be sent with the incorrect capitalization and punctuation or that lacks capitalization and punctuation. In these and other embodiments, corrected capitalization and punctuation for a transcription may be provided to the user device for presentation. In some embodiments, the transcription unit 1514 may obtain the corrections of capitalization and punctuation based on input from a CA or from other processes performed by the ASR system 1520.” [Section 0441]. In some embodiments, the punctuator 1504 may be configured to use a punctuation model 1506 to punctuate a transcription generated by the ASR system 1520. In these and other embodiments, the ASR system 1520 may generate the transcription without punctuation as described previously. The punctuator 1504 may use the punctuation model 1506 to add the punctuation to the transcription.” [Section 0442]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 22, Fritsch et al. does not teach the computing system of claim 19, the computer-executable instructions being further executable by the processor to cause the computing system to: while outputting the first portion of the initial segment, refraining from outputting a leftover portion of the initial segment which is located temporally subsequent to the first portion of the initial segment.
Thomson discloses:
the computing system of claim 19, the computer-executable instructions being further executable by the processor to cause the computing system to: while the orchestrator outputs the first portion of the initial segment, the orchestrator refrains from outputting a leftover portion of the initial segment which is located temporally subsequent to the first portion of the initial segment. [Thomson: Section 0422, 0898— “partial results may be obtained by evaluating substantially all input, including text input, confidence estimates, endpoints, etc., received from the start of a communication session, or start of the transcription session, from the point where a transcribed party begins speaking to the current point in time, or from the point where a transcribed party begins speaking and has continuously spoke to the current point in time….” [Section 0422]. “The audio may be delayed between subsequent stages of editing the transcription so that the audio presented by the audio interface 4744 may be more closely synchronized to the portion of text being displayed for and/or edited by a CA through the text editor 4742. The delay time may be constant or variable and may be responsive to endpoints and text from a previous stage.” This reflects the notion of outputting only specific segments of the audio data at given time while withholding subsequent portions. [Section 0898]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 23, Fritsch et al. does not teach the computing system of claim 19, the computer-executable instructions being further executable by the processor to cause the computing system to: determine a number of newly decoded words to wait based on a context of the streaming audio data.
Thomson discloses:
the computing system of claim 19, the number of newly decoded words to wait being based on a context of the streaming audio data.
[Thomson: Table 5. Section 93-96— “93. An analysis of communication session audio and/or transcriptions to determine how much of the conversation a first user is understanding. The communication session audio and/or transcription may, for example, be input to a machine learning system trained to estimate a first user's level of comprehension. 94. “The number of words in a hypothesis transcription,” indicating words quantities in relation to the audio stream. 95. The number of words in a reference transcription. 96. An alignment length, which may be the total number of token columns created by an alignment between two or more transcriptions. For example, if the hypothesis “the quick brown “is aligned with “quick brown fox “so that the tokens “quick “are matched and the tokens “brown “are matched, the alignment length may be the number of words in the string “the quick brown fox, “which is four.….” [Section 93-96]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 27, Fritsch et al. does not teach the computing system of claim 19, wherein the context of output application is identified as a closed captioning of streaming audio data.
Thomson discloses:
the computing system of claim 19, wherein the context of output application is identified as a closed captioning of streaming audio data. [Thomson: Section 0091— the user may not understand everything being said during the phone call from the audio of the phone. However, the audio may be provided to a transcription system. The transcription system may generate a transcription of the audio in real-time during the phone call and provide the transcription to a device of the user. The device may present the transcription to the user. Having a transcription of the audio may assist the hard of hearing user to better understand the audio and thereby better participate in the phone call.” This highlights the process of transcribing audio data similar to closed capturing. [Section 0091]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Regarding Claim 31, Fritsch et al. does not teach the computing system of claim 19, wherein the audio data is a pre-recorded set of audio data.
Thomson discloses:
the computing system of claim 19, wherein the audio data is a pre-recorded set of audio data. [Thomson: Section 1596— “…When transcriptions of audio are not generated in real time because the initial language assumption is incorrect or due to other delays, and a second selected language may be changed during the communication session, recorded communication session audio from before the change in the selected language may be processed by an ASR system configured to transcribe the spoken language so that the missing content, or a summary thereof, may be transcribed and displayed.….” [Section 1596]
Fritsch et al. and Thomson are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Thomson, which Improves transcription accuracy through advanced automatic speech recognition technique in audio communication sessions [Thomson: Section 0002-0003].
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over Fritsch in view of Tundik in view of Minixhofer, as applied to claims 1 and 19 above, and further in view of Matusov [U.S. 20200226327].
Regarding Claim 30, Fritsch et al. does not teach the computing system of claim 19, wherein the orchestrator is configured to hold a portion of the transcription until the initial punctuation is validated.
Matusov discloses:
the computing system of claim 19, wherein the orchestrator is configured to hold a portion of the transcription until the initial punctuation is validated. [Matusov: Section 0051, 0038— “…Pre-train the multilingual RNN encoder on a sequence of pairs (source language speech, source language sentence) using the usual human-transcribed speech training data for all supported source languages. If real sentence segmentation is not available (i.e. the human transcript consists of words only without punctuation marks), use automatic punctuation prediction method to predict sentence-final punctuation and thus define real sentence boundaries. As part of the pre-training, also train the predictor of the sentence boundaries that does not rely on the labels of the predicted source words. This may be done by assigning the reference sentence boundary label to the vector of the last RNN encoder layer that corresponds to the word end of the source word that ends the sentence….” [Section 0051]. “…the direct speech translation component is also used to, explicitly or implicitly, predict sentence boundaries so that it can generate translations of full target sentences with punctuation. Here, by sentences we mean syntactic/semantic units which are usually the units of processing in machine translation….” [ Section 0038]
Fritsch et al. and Matusov are considered analogous art because they were in the similar field related to Automatic Speech Recognition (ASR) systems that process and transcribe audio data into text, it focuses on generating transcriptions from spoken language by employing methods that enhances segmentation, punctuation and readability for application like real-time captioning and video conferencing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Fritsch et al. to combine the teaching of Matusov, which create an efficient multilingual speech translation system that accurately translates spoken language while maintaining speaker voice characteristics [Matusov: Section 0002-0003].
Claims 35 and 36 are rejected under 35 U.S.C. 103 as being unpatentable over Fritsch in view of Tundik in view of Minixhofer, as applied to claim 1 above, and further in view of "Automated production of true-cased punctuated subtitles for weather and news broadcasts," authored by Joris Driesen, Alexandra Birch, Simon Grimsey, Saeid Safarfashandi, Juliet Gauthier, Matt Simpson, and Steve Renals, published in INTERSPEECH 2014 15th Annual Conference of the International Speech Communication Association, pp. 2146-2147. International Speech Communication Association, in 2014, herein “Driesen”.
Regarding claim 35, Tundik et al. doesn’t describe but Driesen describes the computing system of claim 1, wherein the type of audio device is a multi-speaker audio device (See section 1. “Introduction”, page 2 (marked as 2146), second column: “In this paper, we present a modular end-to-end system which takes a video as input and produces readable subtitles in a fully automated way. A schematic overview of this system is shown in figure 1. The video content that serves as input to this system comes from two different domains: ‘Weatherview’ and ‘Sky News’. Weatherview is the daily weather forecast for the UK on BBC. From an ASR perspective, this data is the easier of the two to handle. It consists of 3-minute segments, uttered by a single speaker in isolation, employing a restricted vocabulary. For more details, see [1]. The other input type consists of broadcast news, taken from the news channel ‘Sky News’. This data poses a bigger challenge, as it contains a multitude of different speakers, recording and noise conditions, a much larger vocabulary which tends to change over time as words and expressions rise and fade in popularity, etc. All inputs in this paper consists of data that was aired recently. As such, the proposed system is designed to cope with inputs of real-world scale and complexity.” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Tundik et al. a system and method wherein the type of audio device is a multi-speaker audio device, as suggested by Driesen, in order to adjust the number of words to wait based on a multi-speaker audio environment, which ensures accurate punctuation in the more challenging situation of tracking sentences from multiple speakers who are taking turns speaking.
Regarding claim 36, Tundik et al. doesn’t describe but Driesen describes the computing system of claim 1, wherein the type of audio device is a personal audio device (See section 1. “Introduction”, page 2 (marked as 2146), second column: “In this paper, we present a modular end-to-end system which takes a video as input and produces readable subtitles in a fully automated way. A schematic overview of this system is shown in figure 1. The video content that serves as input to this system comes from two different domains: ‘Weatherview’ and ‘Sky News’. Weatherview is the daily weather forecast for the UK on BBC. From an ASR perspective, this data is the easier of the two to handle. It consists of 3-minute segments, uttered by a single speaker in isolation, employing a restricted vocabulary. For more details, see [1]. The other input type consists of broadcast news, taken from the news channel ‘Sky News’. This data poses a bigger challenge, as it contains a multitude of different speakers, recording and noise conditions, a much larger vocabulary which tends to change over time as words and expressions rise and fade in popularity, etc. All inputs in this paper consists of data that was aired recently. As such, the proposed system is designed to cope with inputs of real-world scale and complexity.” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Tundik et al. a system and method wherein the type of audio device is a personal audio device, as suggested by Driesen, in order to adjust the number of words to wait based on a single-speaker audio environment, which ensures accurate punctuation and reduced latency in the more predictable situation of tracking sentences from a single speaker.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure – see additional NPL references cited on PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daniel C Washburn whose telephone number is (571)272-5551. The examiner can normally be reached Monday-Friday 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights