Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
2. Applicant's arguments filed 02/02/2026 have been fully considered but they are not persuasive. The newly added limitations directed to “identifying a first span within the ASR transcription and identifying a second span within the ground truth transcription; determining a first entity in the first span and determining a second entity in the second span; matching the first entity to the second entity based on the tying of the second sentence embeddings to the third sentence embeddings resulting in a match; and generating a predicted entity output and a predicted non-entity output using the trained natural language processing model based on the match,” as previously set forth, Li teaches generating and aligning embeddings for automated speech recognition (see ¶29-33) and corresponding ground truth transcriptions using a cosine based loss in a common embedding space (see ¶44). Ganea teaches generating and training entity span embeddings using margin-based triplet style objective. The Yamada and Kolitsas references further teach identifying entity spans and classifying them within text.
More specifically, the additional limitations of identifying spans, determining entities and matching entities based on embedding similarity represent a direct and predictable use of the trained embeddings space to associate corresponding semantic elements across aligned inputs are by this combination as follows:
Li teaches that the correction model generates higher order textual feature representations for candidate text and compares those representations to the higher order audio feature representations using cosine distance and selecting the nearest neighbor textual representation based on the lowest cosine distance (see ¶65-68). Ganea teaches that entity disambiguation is performed by scoring candidate entity embeddings against the local context representation and selecting the candidate entity based on that score, including use of mention/entity (spans) candidate sets and margin based training of correct entity assignments over alternatives (see Ganea section 3, equations 5-7). Yamada further teaches treating entities as independent tokens and outputting contextualized representations and both words and entities in the same transformer model, including masked entity prediction using entity representations (Yamada, Fig. 1 Section 3.1 and 3.3). Accordingly, applying the already trained embedding space to identify corresponding spans/entities across ASR and ground truth transcriptions would have been a predictable us of the combined teachings
3. Claims 1-3, 5, 7, 8, 14-16 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (US 2025/0006217) (herein Li), in view of Ganea et al. “Deep Joint Entity Disambiguation with Local Neural Attention” (herein Ganea) and further in view of Kolitsas et al “End-to-End Neural Entity Linking” (herein Kolitsas).
Regarding Claim 1:
Li discloses a device (Li: Fig. 5 displays a device), comprising:
a processing system including a processor (Li: ¶[0076] displays a processor);
and a memory (Li: ¶[0076] discloses memory) that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
r
receiving a second text input that represents Automatic Speech Recognition (ASR) transcription at a text encoder (Li: ¶[0029-0033] and ¶[0057] discloses “ In these implementations, the shared audio-text encoder 210 is further configured to receive, as input, a corresponding transcription 509 of the corresponding transcribed speech utterance 508 and generate, as output, an encoded textual representation”, in other words, the duel encoder architecture contains a speech encoder that produces ASR transcriptions which then inputs the ASR output into text encoders to produce encoded text representation);
receiving a third text input that represents ground truth transcription at the text encoder (Li: ¶[0044] discloses an “unspoken textual utterance” in which the alignment output is generated from also serves as a ground-truth transcription to train the encoder);
performing embeddings of (Li: ¶[0043] discloses textual embeddings via the text encoder. The text encoder generates encoded text representations corresponding to a ground truth textual utterance; ¶[0045] discloses the audio text encoder is trained (emphasis added) on the ground truth labels, meaning it uses these ground truth labels to learn how to embed the textual data properly);
training a natural language processing model resulting in a trained natural language processing model comprising:
and tying second sentence embeddings in the second text input and third sentence embeddings in the third text input on same utterances by applying a cosine loss in the common embedding space ((Li: ¶[0065] teaches minimizing the distance between the embeddings of ASR transcription text and corresponding manual transcription text during model training);
(Li: ¶[0060] teaches determining a loss between the higher order audio feature representation and higher order textual representation for the same respective transcribed speech utterance. ¶[0065] discloses generating textual feature representations with a minimal cosine distance from corresponding audio features representations for the same spoken utterance. ¶[0067]-[0068] further teaches selecting the nearest neighbor textual representation based on the lowest cosine distance, resulting in a match between the ASR derived representation and corresponding textual representation);
Li does not explicitly disclose:
receiving a first text input that represents entity description at an entity encoder;
tying first sentence embeddings in the first text input with entity span embeddings in the second text input by applying triplet loss in a common vector space;
based on a triplet loss value, adjusting a distance between the first sentence embeddings and the entity span embeddings in the second text input
and identifying a first span within the ASR transcription and identifying a second span within the ground truth transcription;
determining a first entity in the first span and determining a second entity in the second span;
matching the first entity to the second entity based on the tying of the second sentence embeddings to the third sentence embeddings resulting in a match; and
and generating a predicted entity output and a predicted non-entity output using the trained natural language processing model based on the match.
However, Ganea discloses:
receiving a first text input that represents entity description at an entity encoder (Ganea: Section 2 and Section 3 discloses embedding entities and words into the same vector space, these words are the entity description);
tying first sentence embeddings in the first text input with entity span embeddings in the second text input by applying triplet loss in a common vector space (Ganea: Section 4, Learning the Local Model Heading, compares embeddings to span (mention) embeddings in a shared embedding space, pushing correct entity-span pairs closer and incorrect pairs farther, which is triplet contrastive loss tying the first and second embeddings);
based on a triplet loss value, adjusting a distance between the first sentence embeddings and the entity span embeddings in the second text input (Ganea: Section 4, Learning the Local Model Heading, “adjust a distance” based on the margin loss outcome. The embedding space is actively reshaped to separate entity spans vs. non-entity spans);
and identifying a first span within the ASR transcription and identifying a second span within the ground truth transcription (Ganea: Section 5 mentions “a document as consisting of a set of mentions m = m1,…, mn, along with their context windows c = c1,…cn. Ganea further discloses in Section 6 that “candidate selection, i.e., construction of Γ (e), is done for each input mention” and “the top 30 candidates are selected based on the prior p^(e|m).” A mention is a span of text. Therefore, when applied to Li’s ASR transcription and Li’s ground truth transcription, the first span corresponds to a mention/entity span in the ASR transcription and the second span corresponds to the matching mention/entity span in the ground truth transcription);
determining a first entity in the first span and determining a second entity in the second span (Ganea: Section 3 discloses that “for each mention m, a pruned candidate set Γ (m) of at most S entities has been identified” and that the “model computes a score for each e in the set of Γ (m) on the K-word local context c = {w1,…, 1K} surrounding m}” Ganea further discloses that “prediction is done independently for each mention mi and context ci by maximizing the Ψ(e,mi,ci) score.” Therefore, for the first span in the ASR transcription, Ganea determines a first entity by scoring candidate entities and selecting the best scoring entity. Likewise, for the second span in the ground truth transcription, Ganea determines a second entity by scoring candidate entities and selecting the best scoring entity);
matching the first entity to the second entity based on the tying of the second sentence embeddings to the third sentence embeddings resulting in a match (Ganea: Section 4 discloses “Prediction is done independently for each mention mi and context ci by maximizing the Ψ(e,mi,ci) score” this shows the first entity determined for the first span is matched to the second entity determining for the second span / ground truth span, tied with Li this can apply to the ASR transcriptions); and
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Ganea explicitly teaches determining whether a text span should be associated with an entity or treated as a non-entity based on similarity thresholds that use triple loss. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose relating textual input through triplet loss to updating a distance between two representations, and applying cosine similarity to pull together descriptions and entities. The motivation for doing so is “Our approach thereby combines benefits of deep learning with more traditional approaches such as graphical models and probabilistic mention-entity maps. Extensive experiments show that we are able to obtain competitive or state of-the-art accuracy at moderate computational costs” as disclosed in the Abstract of Ganea.
The combination of Li and Ganea do not disclose generating a predicted entity output and a predicted non-entity output using the trained natural language processing model. However, Kolitsas discloses and generating a predicted entity output and a predicted non-entity output using the trained natural language processing model (Kolitsas: Section 3, Inference Heading, teaches the final output is a set of annotations for every single possible entity, so for every candidate text space the model computes a score for each possible entity e, the span m could refer to. Then it compares the best score to a learned threshold, if the score is greater than the threshold, the system links the span to predicted entity output, if the score is less than the threshold then the system drops the span (no link) and it is non entity output because it is explicit prediction that such spans are non-entity, therefore producing both entity and non-entity outputs).
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Li discloses providing a score in order to classify an entity or not. It merely does not disclose outputting simultaneously a predicted entity and predicted non-entity. Kolitsas explicitly teaches and output linking for entities and no linking for non-entities by actively dropping a span if it is predicted to be a non-entity. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose outputting both the entities and determined non-entities. The motivation for doing so is “we emphasize the importance of the mutual dependency between [mention dependency] and [entity disambiguation]. First, numerous and more informative linkable spans found by MD obviously offer more contextual cues for ED. Second, finding the true entities appearing in a specific context encourages better mention boundaries, especially for multi-word mentions” As disclosed in Kolitsas’ introduction.
Regarding Claim 2:
The combination of Li, Ganea and Kolitsas further discloses the device of claim 1, wherein the training the natural language processing model further comprises determining a distance between the second sentence embeddings of the second text input and the third sentence embeddings of the third text input (Li: ¶[0049] discloses the transcribed non-synthetic speech (which corresponds to ASR transcription) which is the claimed second text, the alignment output (paired manual text) this corresponds to ground truth transcriptions (claims third text input) and that the consistency loss is computed from embeddings, which requires determining the distance between those embeddings).
Regarding Claim 3:
The combination of Li, Ganea and Kolitsas further discloses the device of claim 2, wherein the training the natural language processing model further comprises minimizing the distance between the second sentence embeddings of the second text input and the third sentence embeddings of the third text input via the cosine loss (Li: ¶[0065] teaches minimizing the distance between the embeddings of ASR transcription text and corresponding manual transcription text during model training).
Regarding Claim 5:
The combination of Li, Ganea and Kolitsas further discloses the device wherein the operations further comprises tying embeddings of entity spans from the ASR transcription and embeddings of entity spans from the ground truth transcription via the cosine loss (Li: ¶[0043] discloses that the text encoder receives ASR transcription text and produces embeddings, ¶[0060] discloses using cosine loss between higher order audio, higher order text and the previously mentioned ground truth labels; Ganea: Section 4, Local Score Combination Heading, Section 7, ranks candidate entities for a span and selecting the best match, Ganea outputs entity vs non-entity decisions producing a predicted entity output and treating low-scoring spans as non-entity prediction).
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Ganea explicitly teaches determining whether a text span should be associated with an entity or treated as a non-entity based on similarity thresholds that use triplet loss. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose relating textual input through triplet loss to updating a distance between two representations, and applying cosine similarity to pull together descriptions and entities. The motivation for doing so is “Our approach thereby combines benefits of deep learning with more traditional approaches such as graphical models and probabilistic mention-entity maps. Extensive experiments show that we are able to obtain competitive or state of-the-art accuracy at moderate computational costs” As disclosed in the Abstract of Ganea.
Regarding Claim 7:
The combination of Li, Ganea and Kolitsas further discloses the device of claim 1, wherein the tying the first sentence embeddings in the first text input with entity span embeddings in the second text input further comprises:
setting the first sentence embeddings in the first text input as a reference point in a common vector space (Li: ¶[0043] discloses generating encoded textual and audio embeddings in a shared latent space, such that one embedding serves as the comparison reference);
and determining a dynamic prediction threshold(Li: ¶[0066]-[0068] cosine distances are computed for each utterance, so these distances differ. [0071] discloses using embedding distance to determine a threshold based selection which correspond to determining a prediction threshold between a reference and an ASR textual transcription. This threshold is dynamic because it depends on the actual cosine distance computed for the specific utterance at inference time not a fixed preset threshold, instead of a globally fixed threshold, it computes cosine distances fresh for each utterance transcription and determines based off of this whether to update).
Regarding Claim 8:
The combination of Li, Ganea and Kolitsas further discloses the device of claim 7,
wherein the tying the first sentence embeddings in the first text input with the entity span embeddings in the second text input further comprises determining whether the entity span embeddings are within the dynamic prediction threshold (Li: ¶[0071] discloses computing a threshold to decide whether a candidate transcription representation lies close enough to a reference, which satisfies the determination step);
and wherein the adjusting the distance between the first sentence embeddings and the entity span embeddings further comprises: upon determination that the entity span embeddings are within the dynamic prediction threshold, minimizing a second distance between the entity span embeddings and the first sentence embeddings (Ganea: Section 4: Learning the Local Model, discloses the margin γ controls the decision boundary, if the score difference meets or exceeds γ, it is treated as within the threshold, and the model minimizes the distance between embeddings (positive pair tightening));
and upon determination that the entity span embeddings are outside of the dynamic prediction threshold, maximizing a third distance between the entity span embeddings and the first sentence embeddings (Ganea: Section 4: Learning the Local Model, if the score fails to meet γ it is treated as outside the threshold and the model maximizes the distance).
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Ganea explicitly teaches determining whether a text span should be associated with an entity or treated as a non-entity based on similarity thresholds that use triplet loss. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose relating textual entity spans through loss functions and triplet loss. The motivation for doing so is “Our approach thereby combines benefits of deep learning with more traditional approaches such as graphical models and probabilistic mention-entity maps. Extensive experiments show that we are able to obtain competitive or state of-the-art accuracy at moderate computational costs” As disclosed in the Abstract of Ganea.
Regarding Claim 14:
Li discloses a method, comprising:
receiving, by a processing system including a processor, a manual transcription at a text encoder (Li: ¶[0044] discloses an “unspoken textual utterance” in which the alignment output is generated from also serves as a ground-truth transcription to train the encoder, interpreted functionally as manual transcription data);
receiving, by the processing system, an automatic speech recognition (ASR) transcription at the text encoder (Li: ¶[0029-0033] and ¶[0057] discloses “ In these implementations, the shared audio-text encoder 210 is further configured to receive, as input, a corresponding transcription 509 of the corresponding transcribed speech utterance 508 and generate, as output, an encoded textual representation”, in other words, the duel encoder architecture contains a speech encoder that produces ASR transcriptions which then inputs the ASR output into text encoders to produce encoded text representation);
generating, by the processing system, a natural language processing model by training the text encoder to learn from both the manual transcription and the ASR transcription (Li: ¶[0049] discloses the transcribed non-synthetic speech (which corresponds to ASR transcription) which is the claimed second text, the alignment output (paired manual text) this corresponds to ground truth transcriptions (claims third text input) and that the consistency loss is computed from embeddings, which requires determining the distance between those embeddings), wherein the training comprises:
using a cosine loss, minimizing a distance between sentence embeddings of the ASR transcription and sentence embeddings of the manual transcription (Li: ¶[0043] discloses that the text encoder receives ASR transcription text and produces embeddings, ¶[0060] discloses using cosine loss between higher order audio, higher order text and the previously mentioned ground truth labels).;
and using the cosine loss (Li: ¶[0061] discloses using cosine similarity),
determining an ASR entity (Li: ¶[0043] discloses that the text encoder receives ASR transcription text and produces embeddings and manual ground truth embeddings);
and matching the ASR entity to the manual entity based on the embeddings of the ASR entity (Li: ¶[0065] teaches the ASR transcription and manual transcription correspond the same utterance and are tied by cosine distance training); and
generating a predicted entity output and a predicted non-entity output using the trained natural language processing model and the match.
Li does not explicitly disclose the crossed out limitations above. However, Ganea, discloses
receiving, by the processing system, an entity type description at an entity type encoder (Ganea: Section 2 and Section 3 discloses embedding entities and words into the same vector space, these words are the entity description);
setting a reference point based on embeddings of the entity type description (Ganea: Section 4 Learning the Local Model, discloses ground truth entities as a reference point for candidate words/entity descriptions to compare from);
using a triplet loss function, maximizing a distance between the reference point and embeddings of an ASR non-entity span and minimizing a distance between the reference point and embeddings of an ASR entity span (Ganea: Section 4, Learning the Local Model Heading, “adjust a distance” based on the margin loss outcome. The embedding space is actively reshaped to separate entity spans vs. non-entity spans);
minimizing a distance between embeddings of an ASR entity span and embeddings of a manual entity span (Ganea: Table 8, Section 7.2 discloses cosine similarity loss for tying together sentence embeddings using ground truth labels);
determining an ASR entity in the ASR span and determining a manual entity in the manual entity span (Ganea: Section 3 teaches determining entity assignments for text spans by scoring span / entity embeddings, applying Ganea’s entity assignments for text spans by scoring span / entity embeddings. Therefore, applying Ganea’s entity determination to Li’s tied ASR / manual transcription pair results in matching the ASR entity determined for the ASR span to the manual entity determined for the corresponding manual span);
and matching the ASR entity to the manual entity based on the embeddings of the ASR entity span and the embeddings of the manual entity span resulting in a match (Ganea: Section 4 discloses “Prediction is done independently for each mention mi and context ci by maximizing the Ψ(e,mi,ci) score” this shows the first entity determined for the first span is matched to the second entity determining for the second span / ground truth span, tied with Li this can apply to the ASR transcriptions).
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Ganea explicitly teaches determining whether a text span should be associated with an entity or treated as a non-entity based on similarity thresholds that use triplet loss. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose relating textual input through triplet loss to updating a distance between two representations, and then apply cosine loss already present in Li to pull together descriptions and entities. The motivation for doing so is “Our approach thereby combines benefits of deep learning with more traditional approaches such as graphical models and probabilistic mention-entity maps. Extensive experiments show that we are able to obtain competitive or state of-the-art accuracy at moderate computational costs” As disclosed in the Abstract of Ganea.
The combination of Li and Ganea do not disclose and generating a predicted entity output and a predicted non-entity output using the trained natural language processing model. However, Kolitsas discloses and generating a predicted entity output and a predicted non-entity output using the trained natural language processing model (Kolitsas: Section 3, Inference Heading, teaches the final output is a set of annotations for every single possible entity, so for every candidate text space the model computes a score for each possible entity e, the span m could refer to. Then it compares the best score to a learned threshold, if the score is greater than the threshold, the system links the span to predicted entity output, if the score is less than the threshold then the system drops the span (no link) and it is non entity output because it is explicit prediction that such spans are non-entity, therefore producing both entity and non-entity outputs).
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Li discloses providing a score in order to classify an entity or not. It merely does not disclose outputting simultaneously a predicted entity and predicted non-entity Kolitsas explicitly teaches and output linking for entities and no linking for non-entities by actively dropping a span if it is predicted to be a non-entity. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose outputting both the entities and determined non-entities. The motivation for doing so is “we emphasize the importance of the mutual dependency between [mention dependency] and [entity disambiguation]. First, numerous and more informative linkable spans found by MD obviously offer more contextual cues for ED. Second, finding the true entities appearing in a specific context encourages better mention boundaries, especially for multi-word mentions” As disclosed in Kolitsas’ introduction.
Regarding Claim 15:
Li, Ganea and Kolitsas further disclose the method of claim 14, comprising:
during an inference phase, receiving, by the processing system, an ASR utterance (Li: Fig. 1 106 discloses processing an utterance);
and in response to the ASR utterance, generating, by the processing system, a resulting predicted entity output and a resulting predicted non-entity output using the trained natural language processing model (Kolitsas: Section 3, Inference Heading, the system links the span to predicted entity output, if the score is less than the threshold then the system drops the span (no link) and it is non entity output because it is explicit prediction that such spans are non-entity, therefore producing both entity and non-entity outputs).
Regarding Claim 16:
Li, Ganea and Kolitsas further disclose the method of claim 14, comprising:
tying, by the processing system, the sentence embeddings of the ASR transcription and the sentence embeddings of the manual transcription in a common vector space (Li: ¶[0043] discloses generating encoded textual and audio embeddings in a shared latent space, such that one embedding serves as the comparison reference and “tying” together these by minimizing distance);
and tying, by the processing system, the embeddings of the ASR entity span and the embeddings of the manual entity span in the common vector space (Ganea: Section 4, Learning the Local Model Heading, “adjust a distance” based on the margin loss outcome. The embedding space is actively reshaped to separate entity spans vs. non-entity spans);
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Ganea explicitly teaches determining whether a text span should be associated with an entity or treated as a non-entity based on similarity thresholds that use triplet loss. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose entity spans in a common space. The motivation for doing so is “Our approach thereby combines benefits of deep learning with more traditional approaches such as graphical models and probabilistic mention-entity maps. Extensive experiments show that we are able to obtain competitive or state of-the-art accuracy at moderate computational costs” As disclosed in the Abstract of Ganea.
Regarding Claim 18:
Li, Ganea and Kolitsas further disclose the method of claim 14, wherein the training further comprises:
receiving, by the processing system, a manually annotated text (Li: ¶[0049] discloses the alignment output (paired manual text) this corresponds to ground truth transcriptions and that the consistency loss is computed from embeddings, which requires determining the distance between those embeddings);
receiving, by the processing system, a corresponding ASR text (Li: ¶[0049] transcribed non-synthetic speech (which corresponds to ASR transcription) which is the claimed second text);
and outputting, by the processing system, a set of input including manual entity spans, ASR entity spans and manual-ASR-entity matchings (Ganea: Table 8, Section 7.2 discloses cosine similarity loss for tying together sentence embeddings using ground truth labels, based on the margin loss outcome. The embedding space is actively reshaped to separate entity spans vs. non-entity spans as output).
Regarding Claim 19:
Li, Ganea and Kolitsas further disclose the method of claim 18, wherein the manual entity spans indicate which tokens in the manually annotated text are entities and entity types thereof, the ASR entity spans indicate which tokens in the ASR text are entities and entity types thereof, and the manual-ASR-entity matchings correspond to a set of one-to-one matchings between the manual entity spans in the manually annotated text and the ASR entity spans in the corresponding ASR text transcription (Li: Section 3.1 discloses that the text encoder receives ASR transcription text and produces embeddings; Kolitsas: Section 3 teaches identifying entity spans directly from annotated text and assigning entity types using a neural mention extraction and classification model; Ganea: provides span-level representation and entity typing. Section 1 Further teaches that the same span detection and typing model operates over any textual input, including noisy or automatically transcribed input (ASR text) because the system takes raw text tokens as input and predicts spans + types. Table 2 Section 3 discloses matching/linking entities
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Li discloses providing a score in order to classify an entity or not. It merely does not disclose outputting simultaneously a predicted entity and predicted non-entity Kolitsas explicitly teaches and output linking for entities and no linking for non-entities by actively dropping a span if it is predicted to be a non-entity. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose entity matching of spans as done in Kolitsas. The motivation for doing so is “we emphasize the importance of the mutual dependency between [mention dependency] and [entity disambiguation]. First, numerous and more informative linkable spans found by MD obviously offer more contextual cues for ED. Second, finding the true entities appearing in a specific context encourages better mention boundaries, especially for multi-word mentions” As disclosed in Kolitsas’ introduction.
4. Claims 4, 6 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Ganea, further in view of Kolitsas and further in view of Mangu et al. “Finding consensus in speech recognition: word error minimization and other applications of confusion networks” (herein Mangu) .
Regarding Claim 4:
The combination of Li, Ganea and Kolitsas further discloses the device of claim 1, except wherein the operations further comprise:
prior to the training of the natural language processing model, running a character-error-rate based matching algorithm with respect to annotated spans of entities in a manual text and corresponding annotated spans of the entities in an ASR text;
and outputting matching pairs of entities from the manual text and the ASR text.
However, Mangu discloses:
prior to the training of the natural language processing model, running a character-error-rate based matching algorithm with respect to annotated spans of entities in a manual text and corresponding annotated spans of the entities in an ASR text (Mangu: Section 2.1, Section 2.4 discloses teaches applying CER/minimum edit distance alignment with hypothesis and reference text, where the hypothesis fulfills the role of the ASR text and the reference fulfills the role of the manual text)
and outputting matching pairs of entities from the manual text and the ASR text (Mangu: Section 3.1 page 8 discloses after alignment, outputting matched word spans across the reference and equivalent hypothesis that are matched through the error rate);
Li and Mangu are from the same field of endeavor, speech recognition accuracy evaluation and improvement. Li discloses obtaining paired text sequences for the same utterance, including an ASR generated transcription and a ground truth manual transcription for comparison. Mangu discloses operating over paired hypothesis text and reference text and aligns the two sequences using a consensus word-error alignment procedure to compute word-level correspondence. It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to apply Mangu’s alignment and error rate computation process to the paired ASR transcription and ground truth transcription already present in Li. The motivation for doing so is “the more accurate representation of the hypothesis space gives better estimates for word posterior probabilities and, consequently, of expected word error” as disclosed by Mangu in the Introduction.
Regarding Claim 6:
The combination of Li, Mangu and Kolitsas further discloses the device of claim 1, except wherein the operations further comprise executing a span matching algorithm to map equivalent entity spans in the ASR transcription and the ground truth transcription for training. However, Mangu discloses this limitation: (Mangu: Section 2.4 page 8 discloses the algorithm that aligns two text sequences by computing their word level correspondences, the alignment procedure identifies the spans considered equivalent across ASR and ground truth (hypothesis and reference)).
Li and Mangu are from the same field of endeavor, speech recognition accuracy evaluation and improvement. Li discloses obtaining paired text sequences for the same utterance, including an ASR generated transcription and a ground truth manual transcription for comparison. Mangu discloses operating over paired hypothesis text and reference text and aligns the two sequences using a consensus word-error alignment procedure to compute word-level correspondence. It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the alignment procedure of Mangu into the training framework of Li, because Mangu expressly teaches that aligning word spans between ASR hypothesis and ground truth transcription improves quality by providing a consistent alignment: “lattices provide a larger set of hypotheses from which to choose; second, the more accurate representation of the hypothesis space gives better estimates for word posterior probabilities and, consequently, of expected word error.”
Regarding Claim 17:
Li, Ganea and Kolitsas further disclose the method of claim 16, except further comprising:
executing, by the processing system, a span matching algorithm that maps equivalent entity spans in the ASR transcription and the manual transcription for training. However, Mangu discloses this limitation: (Mangu: Section 2.4 page 8 discloses the algorithm that aligns two text sequences by computing their word level correspondences, the alignment procedure identifies the spans considered equivalent across ASR and ground truth (hypothesis and reference)).
Li and Mangu are from the same field of endeavor, speech recognition accuracy evaluation and improvement. Li discloses obtaining paired text sequences for the same utterance, including an ASR generated transcription and a ground truth manual transcription for comparison. Mangu discloses operating over paired hypothesis text and reference text and aligns the two sequences using a consensus word-error alignment procedure to compute word-level correspondence. It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the alignment procedure of Mangu into the training framework of Li, because Mangu expressly teaches that aligning word spans between ASR hypothesis and ground truth transcription improves quality by providing a consistent alignment: “lattices provide a larger set of hypotheses from which to choose; second, the more accurate representation of the hypothesis space gives better estimates for word posterior probabilities and, consequently, of expected word error.”
5. Claims 9-13 are rejected under 35 U.S.C. 103 as being unpatentable over Li, in view of Ganea and further in view of Yamada et al. “LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention” (herein Yamada).
Regarding Claim 9:
Li discloses a non-transitory machine-readable medium (Li: ¶[0077]), comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
receiving automatic speech recognition (ASR) transcription at a text encoder, (Li: ¶[0029-0033] and ¶[0057] discloses “ In these implementations, the shared audio-text encoder 210 is further configured to receive, as input, a corresponding transcription 509 of the corresponding transcribed speech utterance 508 and generate, as output, an encoded textual representation”, in other words, the duel encoder architecture contains a speech encoder that produces ASR transcriptions which then inputs the ASR output into text encoders to produce encoded text representation. These transcriptions have contexts they come from (Speech) and therefore have a span (see Fig. 1 106);
receiving a manual transcription including (Li: ¶[0044] discloses an “unspoken textual utterance” in which the alignment output is generated from also serves as a ground-truth transcription to train the encoder);
generating second embeddings of the ASR (Li: ¶[0029-0033] and ¶[0057] discloses “ In these implementations, the shared audio-text encoder 210 is further configured to receive, as input, a corresponding transcription 509 of the corresponding transcribed speech utterance 508 and generate, as output, an encoded textual representation”, in other words, the duel encoder architecture contains a speech encoder that produces ASR transcriptions which then inputs the ASR output into text encoders to produce encoded text representation);
generating third embeddings of the manual (Li: ¶[0044] discloses an “unspoken textual utterance” in which the alignment output is generated from also serves as a ground-truth transcription to train the encoder);
adjusting a first distance of the second embeddings of the ASR special token from the third embeddings of the manual (Li: ¶[0049] discloses the transcribed non-synthetic speech (which corresponds to ASR transcription) which is the claimed second text, the alignment output (paired manual text) this corresponds to ground truth transcriptions (claims third text input) and that the consistency loss is computed from embeddings, which requires determining the distance between those embeddings);
matching the ASR entity ASR transcription, wherein the third embedding is associated with the manual transcription (Li: ¶[0060] teaches determining a loss between the higher order audio feature representation and higher order textual representation for the same respective transcribed speech utterance. ¶[0065] discloses generating textual feature representations with a minimal cosine distance from corresponding audio features representations for the same spoken utterance. ¶[0067]-[0068] further teaches selecting the nearest neighbor textual representation based on the lowest cosine distance, resulting in a match between the ASR derived representation and corresponding textual representation); and
generating a predicted entity output (Li: Fig. 6 120b finalized updated output)
Li does not explicitly discloses each portion of the limitation reciting matching the ASR entity span to the manual entity
However, Ganea discloses these missing portions matching the ASR entity span to the manual entity span based on the second embeddings and the third embeddings resulting in a match, wherein the second embeddings is associated with the ASR transcription, wherein the third embedding is associated with the manual transcription (Ganea: Section 4 discloses “Prediction is done independently for each mention mi and context ci by maximizing the Ψ(e,mi,ci) score” this shows the first entity determined for the first span is matched to the second entity determining for the second span / ground truth span, tied with Li this can apply to the ASR transcriptions);
Li discloses producing a shared latent text/speech embedding space for accurate semantic interpretation even for poor quality ASR text. Ganea explicitly teaches determining whether a text span should be associated with an entity or treated as a non-entity based on similarity thresholds that use triplet loss. It would have been obvious to one of ordinary skill in the art before the effective filing date in the claimed invention to disclose entity spans in a common space. The motivation for doing so is “Our approach thereby combines benefits of deep learning with more traditional approaches such as graphical models and probabilistic mention-entity maps. Extensive experiments show that we are able to obtain competitive or state of-the-art accuracy at moderate computational costs” As disclosed in the Abstract of Ganea.
Li and Ganea do not explicitly disclose receiving an entity description including an entity type special token at an entity type encoder;
wherein the ASR transcription further comprises an ASR special token, an ASR non- entity span and an ASR entity span,
generating first embeddings of the entity type special token in a common vector space token;
and a predicted non-entity output based on the first embeddings of the entity type special token and the second embeddings of ASR special token, the ASR non-entity span and the ASR entity span and the.
However, Yamada discloses
receiving an entity description including an entity type special token at an entity type encoder (Yamada: Fig. 1 Section 3.1, discloses entity descriptions being input to an encoder, entity spans represented as their own token embedding and entity type special token functionality);
wherein the ASR transcription further comprises an ASR special token, an ASR non- entity span and an ASR entity span (Yamada: Fig. 1 Section 3.1, discloses “we insert special tokens [CLS] and [SEP] into the word sequence as the first and last words, respectively” operates on all text and is not limited to ASR transcriptions especially in view of Li which explicitly discloses ASR, section 4.4 discloses entity spans) ,
generating first embeddings of the entity type special token in a common vector space token (Yamada: Fig. 1 Section 3.1, discloses outputs contextualized representations for each word and entity in the text using the same transformer architecture, meaning this is all done in a common vector space for first, second, third embeddings etc.);
and a predicted non-entity output based on the first embeddings of the entity type special token and the second embeddings of ASR special token, the ASR non-entity span and the ASR entity span and the match (Yamada: Fig. 1 Section 4.2, discloses classifying each span using a linear classifier, to avoid overlapping span Yamada select spans predicted to be of an entity type and classify all remain spans as non-entity type).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify Li and Ganea’s span and sentence architecture to use Yamada’s entity aware tokenization and embedding mechanism in order to improve handling of entity spans in ASR transcription. Yamada explicitly teaches treating entity mentions as separate input tokens with dedicated entity embeddings and using these embeddings within the same contextual vector space as surrounding text. This directly corresponds to special entity tokens and span-level embeddings. Because both Li and Yamada concern representation learning for natural language and the training of neural encoders to distinguish entities from non-entities, the combination is from the same field od endeavor and yields predictable results. The motivation for doing so is “it extends the transformer using our entity-aware self-attention mechanism” as disclosed by Yamada in the introduction.
Regarding Claim 10:
The combination of Li, Ganea and Yamada further discloses the non-transitory machine-readable medium of claim 9, wherein the adjusting the first distance further comprises minimizing the first distance between the second embeddings of the ASR special token and the third embeddings of the manual special token (Li: ¶[0060]-[0065] teaches minimizing the distance between the embeddings of ASR transcription text and corresponding manual transcription text during model training).
Regarding Claim 11:
The combination of Li, Ganea and Yamada further discloses the non-transitory machine-readable medium of claim 9, wherein the operations further comprise:
generating fourth embeddings of a manual entity span from the manual transcription token (Yamada: Introduction 6443, Section 3 LUKE, Section 3.3 discloses generating embedding specifically for entity spans in manually transcribed text from sources such as Wikipedia. The system treats entities in the text as separate input tokens and computes a contextualized embedding for each entity token in the same embedding space as the surrounding words, therefore Yamada teaches generating fourth embeddings of a manual entity span from the manual transcription);
and adjusting a second distance between the second embeddings of the ASR entity span from the fourth embeddings of the manual entity span via a cosine loss (Yamada: Section 3.3-4.2, computes similarity between span embeddings using loss functions during training, Li ¶[0060]-[0065] discloses determining and minimizing cosine distance between embeddings derived from ASR outputs and corresponding ground truth textual embeddings during training).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify Li’s span and sentence architecture to use Yamada’s entity aware tokenization and embedding mechanism in order to improve handling of entity spans in ASR transcription. Yamada explicitly teaches entity-span level embeddings that correspond to text and are not limited from text transcriptions. Li discloses the cosine distance based objective and text derived directly from ASR. Because both Li and Yamada concern representation learning for natural language and the training of neural encoders to distinguish entities from non-entities, the combination is from the same field of endeavor and yields predictable results. The motivation for doing so is “it extends the transformer using our entity-aware self-attention mechanism” as disclosed by Yamada in the introduction.
Regarding Claim 12:
The combination of Li, Ganea and Yamada further discloses the non-transitory machine-readable medium of claim 9, wherein the operations further comprise training the text encoder based on both the ASR transcription and the manual transcription substantially simultaneously (Li: ¶[0043]-[0046] and ¶[0059]-[0061] teaches training the text encoder using both ASR transcription and corresponding manual/ground truth transcription in the same training process. Li explains that each spoken utterance is paired with its ground-truth transcription, and the shared audio-text encoder and correction model text encoder optimized based on loss computed between the representation derived from the ASR output and the manual transcription).
Regarding Claim 13:
The combination of Li, Ganea and Yamada further discloses the non-transitory machine-readable medium of claim 9, wherein the operations further comprise:
determining a third distance between the first embeddings of the entity type special token and the second embeddings of the ASR special token (Li: ¶[0065] teaches minimizing the distance between the embeddings of ASR transcription text and corresponding manual transcription text during model training; Yamada: Fig. 1 Section 3.1, discloses special tokens);
setting a dynamic prediction threshold based on the third distance (Li ¶[0067]-[0068] discloses a biasing module selecting the nearest neighbor based on lowest corresponding cosine distance, and the rescorer replaces the transcription when the distance satisfies a threshold);
and with reference to the dynamic prediction threshold (Li: ¶[0066]-[0068], [0071] ¶[0065] teaches minimizing the distance between the embeddings of ASR transcription text and corresponding manual transcription text during model training with the dynamic threshold), determining each distance between the second embeddings of the ASR non-entity span and the first embeddings of the entity type special token and between the second embeddings of the ASR entity span and the first embeddings of the entity type special token (Yamada: Section 3.1 and 3.3 discloses representing both entity spans and non-entity spans as contextualized embeddings within a shared vector space).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify Li’s span and sentence architecture to use Yamada’s entity aware tokenization and embedding mechanism in order to improve handling of entity spans in ASR transcription. Yamada explicitly teaches entity-span level embeddings that correspond to text and are not limited from text transcriptions. Li discloses the cosine distance based objective and text derived directly from ASR. Because both Li and Yamada concern representation learning for natural language and the training of neural encoders to distinguish entities from non-entities, and Li already discloses cosine distance and threshold mechanism along with Yamada’s span level embeddings constitute a predictable substitution of a known scoring mechanism to operate on the span level rather than sentence or token level the combination yields predictable results. The motivation for doing so is “it extends the transformer using our entity-aware self-attention mechanism” as disclosed by Yamada in the introduction.
6. Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Li et al. in view of Ganea et al., further in view of Kolitsas et al, and further in view of Yamada.
Regarding Claim 20:
Li, Ganea and Kolitsas further disclose the method of claim 14, comprising:
determining a dynamic prediction based on a distance (Li: ¶[0071] disclose biasing/rescoring selection threshold based on cosine distance between embeddings), wherein the embeddings of the ASR non-entity span are positioned outside of the dynamic prediction threshold and the embeddings of the ASR entity span(Li: ¶[0067]-[0069] discloses rejecting candidate outputs when distance does not meet threshold conditions; Ganea: provides span-level representation and entity typing).
The combination of Li, Ganea and Kolitsas does not disclose from the embeddings of a special token associated with the entity type description to the embeddings of a special token. However, Yamada discloses this limitation: (Yamada: Fig. 1, Section 3.1 provides treating entities as special tokens with entity type embeddings inserted into the token sequence and represented in the same vector space as surrounding text tokens. Further discloses representing entity spans a contextualized span level embeddings using entity aware self-attention to separate entity spans and non-entity spans)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to modify Li’s span and sentence architecture to use Yamada’s entity aware tokenization and embedding mechanism in order to improve handling of entity spans in ASR transcription. Yamada explicitly teaches treating entity mentions as separate input tokens with dedicated entity embeddings and using these embeddings within the same contextual vector space as surrounding text. This directly corresponds to special entity tokens and span-level embeddings. Because both Li and Yamada concern representation learning for natural language and the training of neural encoders to distinguish entities from non-entities, the combination is from the same field od endeavor and yields predictable results. The motivation for doing so is “it extends the transformer using our entity-aware self-attention mechanism” as disclosed by Yamada in the introduction.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IAN SCOTT MCLEAN whose telephone number is (703)756-4599. The examiner can normally be reached "Monday - Friday 8:00-5:00 EST, off Every 2nd Friday".
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IAN SCOTT MCLEAN/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights