DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 31 January 2025 and 24 February 2026 is/are being considered by the examiner.
Regarding the reference described in IDS communication filed 31 January 2025, the IDS and the IDS communication fails to comply with 37 CFR 1.98(a)(3)(i) because it does not include a concise explanation of the relevance, as it is presently understood by the individual designated in 37 CFR 1.56(c) most knowledgeable about the content of the information, of each reference listed that is not in the English language. Applicant’s statement of relevance is directed to the citation of said reference in a different office for a different patent application. Respectfully, the word relevance in “concise statement of the relevance… of each reference listed that is not in the English language” is the relevance to the instant application, not the relevance to other applications. It has been placed in the application file, but the information referred to therein has not been considered.
Claim Objections
Claim 3, 6, and 8 is objected to because of the following informalities:
Regarding claims 3, 6, and 8, “at least one entity” should read as “the at least one entity”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10-11 and 13-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 10, the phrases “at least one respective entity” and “a respective representation” lack clarity. Claim 10 recites “wherein a determination whether at least one respective entity within a respective representation…” at line 1. Wherein clauses provide further modification of existing claim components. However claim 10 seeks to modify “a determination whether at least one respective entity within a respective representation”, where said limitation does not occur in the claim from which it depends. Claim 1 recites 3 distinct representations: “a representation”, “an adjusted representation”, and “a modified and adjusted representation”, where each of these representations are created by an adjustment to the representation and a modification to the adjusted representation. As claim 1 only recites “at least one entity”, not “at least one respective entity.” Further, neither the entity nor the representation is provided with antecedent basis which might indicate a relationship between the components. As there is no “respective entity” or “respective representation” described in claim 1, the wherein clause in claim 10 fails to describe a modification of an existing claim component. Therefore, claim 10 is indefinite and is therefore rejected.
Regarding claim 11, the limitation “in accordance with a determination that at least one entity within the representation requires disambiguation… forgoing providing the adjusted representation to the language model” lacks clarity. Claim 11 recites “in accordance with a determination that at least one entity within the representation requires disambiguation… forgoing providing the adjusted representation to the language model” at lines 2-4. Claim 1, from which claim 11 depends, recites “in accordance with a determination that at least one entity within the representation requires disambiguation… providing the adjusted representation to the language model” at lines 9-12. It is unclear how the model can simultaneously provide and forgo providing the adjusted representation to the language model. Therefore, claim 11 is indefinite and is therefore rejected.
Regarding claim 13-15, the phrase “a respective representation” lacks clarity. Claim 13 recites “wherein a determination whether a respective representation requires modification includes…” at line 1. As discussed above, wherein clauses provide further modification of existing claim components. However claims 13-15 seek to modify “a determination whether a respective representation requires modification includes…”, where said limitation does not occur in the claim from which it depends. Claim 1 recites 3 distinct representations: “a representation”, “an adjusted representation”, and “a modified and adjusted representation”, where each of these representations are created by an adjustment to the representation and a modification to the adjusted representation. Claim 1 only recites “at least one entity”, not “at least one respective entity.” Further, neither the entity nor the representation is provided with antecedent basis which might indicate a relationship between the components. As there is no “respective representation” described in claim 1, the wherein clause in claims 13-15 fails to describe a modification of an existing claim component. Therefore, claims 13-15 are indefinite and is therefore rejected.
Regarding claim 16, the limitation “determining that a reference resolution system is unable to obtain a reference for at least one entity within the respective representation” lacks clarity. Claim 1 establishes the “at least one entity” as a component of “the representation”. Claim 15, from which claim 16 depends establishes “a respective representation” as being a distinct claim part from the “representation”. As claim 16 recites “at least one entity” as a component of the “respective representation”, the relationship between the “respective representation” of claim 16 and the “representation” of claim 1 is unclear. Therefore, claim 16 is indefinite and is rejected.
Appropriate correction is required.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 11 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Claim 11 includes limitations which directly conflict with the limitations of the claim from which it depends. Claim 11 recites “in accordance with a determination that at least one entity within the representation requires disambiguation and based on a reference resolution setting: forgoing providing the adjusted representation to the language model” at lines 2-4. Claim 1, from which claim 11 depends, recites “in accordance with a determination that at least one entity within the representation requires disambiguation: adjusting the representation by adding an indication to the at least one entity; and providing the adjusted representation to a language model” at line 9-12. The limitation “in accordance with a determination that at least one entity within the representation requires disambiguation” is the condition of claim 1 which the claim requires to result in “providing the adjusted representation to a language model.” The above determination cannot simultaneously result in “providing the adjusted representation to a language model” as recited in claim 1 and “forgoing providing the adjusted representation to the language model” as recited in claim 11. The introduction of a limitation in a dependent claim which changes or otherwise cannot coincide with a required limitation in an independent claim is improper. Therefore, claim 11 fails to further limit the subject matter of the claim from which it depends (claim 1) and is rejected for being in improper dependent form under 35 U.S.C. 112(d).
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. §101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea without significantly more.
To determine subject matter eligibility for each of the recited claims above, we turn to the subject matter eligibility test, also referred to as the Alice/Mayo test, described in MPEP 2106. Regarding step 1 of the subject matter eligibility test, we first determine if the claims are directed to a statutory category. The independent claim(s) 1, and mutatis mutandis claim(s) 19 and 20, recite “receiving a speech input from a user; obtaining a representation of the speech input; in accordance with a determination that at least one entity within the representation requires disambiguation: adjusting the representation by adding an indication to the at least one entity; and providing the adjusted representation to a language model; in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation; and providing an output based on the modified and adjusted representation” As the claims recites a process as embodied in a device and media, the claim is directed to one of the statutory categories under step 1 of the subject matter eligibility test.
In Step 2A of the test, which is a Two Prong analysis, we then determine if the claim is directed to a judicial exception. For Step 2A, Prong One, we first ask if the claim recites an abstract idea, Law of Nature, or Natural Phenomena. Regarding claim(s) 1, 19, and 20, the limitations of “receiving…”, “obtaining…”, “adjusting…”, “modifying…”, and “providing…” as drafted cover managing personal behavior or relationships or interactions between people, which is a method of organizing human activity. More specifically, a first person in communication with a second person (e.g., a secretary receiving dictation from a corporate executive), where the first person provides speech to the second person, such as for a letter. The second person writes the speech down exactly as it is received, where the speech includes, for example, “I look forward to seeing a representative from your company at the picnic, next Friday, I mean Saturday” The second person determines that “your company” is ambiguous. In accordance with the determination, the phrase “your company” is marked by the second person for later correction (or he corrects it now, for example, to “Widget Co.,” if the intended company is currently known). The second person further detects a disfluency and determines that the intended date of the picnic is Saturday, not Friday, based on the interregnum “I mean” between the reparandum “Friday” and the repair “Saturday”. The second person then further modifies the adjusted representation “next Friday, I mean Saturday” to “next Saturday”. The adjusted and modified representation “I look forward to seeing a representative from Widget Co. at the picnic, next Saturday,” is output as part of the letter, such as for the first person to review. Therefore, the claims are directed to human activity, and, thus, directed to an abstract idea which is a judicial exception.
In Step 2A, Prong Two of the analysis, we next determine if the claim recites additional elements which integrate the judicial exception into a practical application. The judicial exception recited in claims 1, 19, and 20 is not integrated into a practical application. In particular, claim(s) 1, 19, and 20 recite additional elements of a “computer-readable storage medium,” an “electronic device,” “memory,” “processor,” and “providing the adjusted representation to a language model”, as per the independent claims. However, these are general-purpose computer components with no provisions for the practical application of the abstract idea. The “computer readable storage medium,” the “computing device,” the “memory,” and the “processor,” is/are not meaningfully integrated into the practical application of the abstract ideas recited in claim(s) 1, 19, and 20. The electronic device is described in the context of a processor performing a generic function in light of instructions stored in the memory. Processor and memory, as integrated in and implemented through the electronic device, are recited at a high-level of generality (i.e., as a generic processor of an electronic device performing a generic computer function based on instructions stored in a generic memory, the function being the automatic correction of recognized text) such that it amounts no more than mere instructions to apply the exception using a generic computing device and/or a generic computer component. Regarding “providing the adjusted representation to a language model”, this is no more than the use of a generic model to perform a generic function. The claim fails to provide sufficient detail such that the language model could be anything more than a platform upon which to perform the described human activity. Accordingly, the additional elements fail to integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Regarding Step 2B of the analysis, we next determine if the claim recites additional elements which amount to substantially more than the judicial exception. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using a “processor,” a “memory,” or “computer readable storage medium” to perform the correction of human speech amounts to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using a generic computing device or general purpose computer component cannot provide an inventive concept. (See Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 223, 110 USPQ2d 1976, 1982-84 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). As well, the remaining claim limitations are well-known, routine, and conventional such as to not qualify as an inventive concept. Specifically, transmission of data over a network is well known, as is evidenced by OIP Techs, Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015). Further, “merely identifying a [graphic] user interface” with relation to determinations which “can be performed in the human mind or using a pencil and paper” has been deemed insufficient to render otherwise abstract claims as non-abstract, where the court further indicated that the resulting device, system and/or method was “still missing” an “improved structure or function.” See Broadband iTV, Inc. v. Amazon.com, Inc., 113 F.4th 1359, 1367-68 (Fed. Cir. 2024). The court has consistently held that “[s]teps that do nothing more than spell out what it means to ‘apply it on a computer’ cannot confer patent-eligibility.” Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1371-72 (Fed.Cir. 2015)(citing Alice, 134 S.Ct. at 2359 (warning against a § 101 analysis that turns on the draftsman's art (citing Parker v. Flook, 437 U.S. 584, 593, 98 S.Ct. 2522, 57 L.Ed.2d 451 (1978))).
Therefore, and in light of the preceding analysis, the claims do not amount to significantly more than the judicial exception. For these reasons, claims 1, 19, and 20 are not patent eligible.
With respect to claim 2, the claim relates to the timing of the determination that the representation requires modification. As performed by a person, these steps appear to refer to the mental process of receiving information before processing the information. No additional limitation is present.
With respect to claim 3, the claim relates to determining that disambiguation is not required and foregoing performing disambiguation in response. As performed by a person, these steps appear to refer to the mental process of determining that disambiguation is not necessary and subsequently not performing disambiguation. No additional limitation is present.
With respect to claim 4, the claim relates to determining that modification is required and performing said modification in response to the determination. As performed by a person, these steps appear to refer to the mental process of determining that a modification is necessary and subsequently performing the modification. No additional limitation is present.
With respect to claim 5, the claim relates to determining that modification is not required and foregoing performing modification in response. As performed by a person, these steps appear to refer to the mental process of determining that a modification is not necessary and subsequently not performing the modification. No additional limitation is present.
With respect to claim 6, the claim relates to determining that at least one entity requires disambiguation, determining the information is available, performing disambiguation in response. As performed by a person, these steps appear to refer to the mental process of determining that disambiguation is necessary, that the information to perform said disambiguation is available, and subsequently performing disambiguation. No additional limitation is present.
With respect to claim 7, the claim relates to determining that at least one entity requires disambiguation, determining the information is not available, and forwarding the response without disambiguation. As performed by a person, these steps appear to refer to the mental process of determining that disambiguation is necessary, determining that the information to perform said disambiguation is not available, and subsequently performing not performing disambiguation (e.g., in the above example, presenting the text including “your company,” and possibly asking the first person what the name of the company is). No additional limitation is present.
With respect to claim 8, the claim relates to determining context information as part of determining whether the disambiguation is necessary. As performed by a person, these steps appear to refer to the mental process of using known context information (e.g., about the target audience, the first person, etc.) to determine if the disambiguation can be made with sufficient accuracy, whether said correction is desired, etc. No additional limitation is present.
With respect to claim 9, the context information being both displayed and not displayed. As performed by a person, these steps appear to refer to the mental process of using known context information (e.g., about the target audience, the first person, etc.) as well as visible information (such as in the dictated document) to determine whether disambiguation is appropriate. Further relying on the above example, if the first person’s history (non-displayed) indicated that the first person wouldn’t want to include redundant uses of the company name and the prior dictated sentence (displayed) was “Widget Co. had a very strong first quarter, of which were pleased to be a part!”, the second person may forego the disambiguation. No additional limitation is present.
With respect to claim 10, the claim relates to determining that at least one entity requires disambiguation, based on whether a resolution system can obtain information regarding the entity which is sufficiently accurate. As performed by a person, these steps appear to refer to the mental process of conditioning the determination that disambiguation is necessary on whether sufficient reliable information is available. No additional limitation is present.
With respect to claim 11, the claim relates to determining that at least one entity requires disambiguation and, based on a reference resolution setting, modifying and adjusting the representation at the language model. As performed by a person, these steps appear to refer to the mental process of performing revisable steps in any order, in light of an indicated preference. No additional limitation is present.
With respect to claim 12, the claim relates to determining that modification is not required and foregoing performing modification in response. As performed by a person, these steps appear to refer to the mental process of determining that a modification is not necessary and subsequently not performing the modification and producing an output based on the unmodified text. No additional limitation is present.
With respect to claim 13, the claim relates to the determination that the representation requires modification being based on a detected disfluency. As performed by a person, these steps appear to refer to the mental process of detecting disfluency in speech and correcting the speech based on detecting said disfluency. No additional limitation is present.
With respect to claim 14, the claim relates to the determination that the representation requires modification being based on a detected correction. As performed by a person, these steps appear to refer to the mental process of detecting a correction in speech and correcting the speech based on detecting said correction, where the part of a disfluency after the interregnum can be referred to as the repair or correction. No additional limitation is present.
With respect to claim 15, the claim relates to the determination that the representation requires modification being based on an ambiguity above an ambiguity threshold. As performed by a person, these steps appear to refer to the mental process of detecting an ambiguity in speech and correcting the speech based on detecting said ambiguity above a certain confidence level. No additional limitation is present.
With respect to claim 16, the claim relates to determining whether the reference resolution system can obtain a reference for an entity before the determination that the representation requires modification being based on an ambiguity above an ambiguity threshold. As performed by a person, these steps appear to refer to the mental process of determining the level of available information before detecting an ambiguity in speech (i.e., is this ambiguous relative to the amount of information I have available?). No additional limitation is present.
With respect to claim 17, the claim relates to the modification being a syntax correction. As performed by a person, these steps appear to refer to the mental process of performing the appropriate modification such that required modification is completed. No additional limitation is present.
With respect to claim 18, the claim relates to the modification being a grammatical error correction. As performed by a person, these steps appear to refer to the mental process of performing the appropriate modification such that required modification is completed. No additional limitation is present.
These claims further do not remedy the judicial exception being integrated into a practical application and further fail to include additional elements that are sufficient to amount to significantly more than the judicial exception. As such, for the same reasons as described above with reference to independent claim(s) 1, 19, and 20, dependent claim(s) 2-18 are not patent eligible.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10 and 12-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Printz (U.S. Pat. App. Pub. No. 2023/0317079, hereinafter Printz) in view of Inbavaluthi (U.S. Pat. App. Pub. No. 2024/0127812, hereinafter Inbavaluthi).
Regarding claim 1, Printz discloses An electronic device (Systems and methods described with reference to a system comprising an “open dictation ASR component”,; Printz, ¶ [0109]), comprising: one or more processors; a memory (the system can be implemented using “computing system 1800” including “one or more central processing units (“processors”) 1805” and “memory 1810”/”storage devices 1820”; Printz, ¶ [0380]); and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs (“The memory 1810 and storage devices 1820 are computer-readable storage media that may store instructions that implement at least portions of the various embodiments” and “instructions stored in memory 1810 can be implemented as software and/or firmware to program the processor 1805 to carry out actions described above.”; Printz, ¶ [0381]-[0382]) including instructions for: receiving a speech input from a user (“Various embodiments accept as input an audio signal comprising fluent, natural human speech, which notably may contain one or several proper names, or unorthodox sequences of otherwise ordinary words.”; Printz, ¶ [0114]); obtaining a representation of the speech input (“The primary recognizer may include a conventional open dictation automatic speech recognition (ASR) system” which “accepts as input an audio signal comprising human speech” and produces “a textual transcription of this input, labeled with the start time and end time, within the input audio signal, of each transcribed word. “; Printz, ¶ [0116]); in accordance with a determination that at least one entity within the representation requires disambiguation (“the natural language understanding module may accept as input the transcription and word timings generated by the primary recognizer” and “may identify a particular word or word sequence in the input transcription that potentially comprises a proper name entity”; Printz, ¶ [0118]-[0119]): adjusting the representation by adding an indication to the at least one entity (In response to identifying “a particular word or word sequence in the input transcription that potentially comprises a proper name entity” the module can then “label this word or word sequence with a putative type (for instance, a person’s name, a street intersection, a numbered street address, and so on)” where “each such word or word sequence is called a proper name entity acoustic span, or the acoustic span of a proper name entity, or just an acoustic span.”; Printz, ¶ [0118]-[0119]); and providing the adjusted representation to a language model (“The hypotheses may be received at a secondary recognizer 860” where “if the hypotheses do include acoustic spans (e.g., in the request “Tell me how to get to Guddu du Karahi”), the secondary recognizer 860 may consult adaptation object generator 865 to identify an appropriate grammar of proper names from the various components 840 a-e of the user device” and the secondary recognizer can include “neural networks including deep neural networks... [and] language models” which “may apply various of the identified proper entities and determine corresponding confidence levels.”; Printz, ¶ [0124], [0199]-[0200]). However, Printz fails to expressly recite in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation; and providing an output based on the modified and adjusted representation.
Inbavaluthi teaches systems and methods “for auto-correction of an ongoing speech command or transcription.” (Inbavaluthi, ¶ [0009]). Regarding claim 1, Inbavaluthi teaches in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation (“determining a connection between the audio and the text based on an individual contextual embedding and a fused contextual embedding at the context tokens level by a multi modal unified attention sequence tagger 406” and “tagging a plurality of replacement, cue and correction words sequentially {modifying, by the language model, the adjusted representation} based on the determined connection between the audio and the text {a determination that the adjusted representation requires modification} by the multi modal unified attention sequence tagger 406,” where the multi-modal unified attention sequence tagger can be a language model, as described, for example, with reference to FIG. 7E; Inbavaluthi, ¶ [0090], [0106]-[0108]; FIG. 7E); and providing an output based on the modified and adjusted representation (“displaying the decoded revised text on a user interface by the on-the-fly decoder 408” and/or “send the decoded revised text to an NLP to generate a response corresponding to the input speech”; Inbavaluthi, ¶ [0109]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation; and providing an output based on the modified and adjusted representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 2, the rejection of claim 1 is incorporated. Printz discloses all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite wherein the determination that the adjusted representation requires modification occurs after the adjusted representation is provided to the language model.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 2, Inbavaluthi teaches wherein the determination that the adjusted representation requires modification occurs after the adjusted representation is provided to the language model (Discloses receipt by the “multi modal unified attention sequence tagger 406 {language model}” and “tagging a plurality of replacement, cue and correction words sequentially based on the determined connection between the audio and the text by the multi modal unified attention sequence tagger 406.”; Inbavaluthi, ¶ [0106], [0107]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include wherein the determination that the adjusted representation requires modification occurs after the adjusted representation is provided to the language model. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 3, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses comprising: in accordance with a determination that at least one entity within the representation does not require disambiguation, providing the representation to the language model. (“Suppose that no proper name entity acoustic spans are identified and that only a single meaning hypothesis is generated” thus when a determination is made that no entities require disambiguation, “This yields a symbolic representation of the command’s meaning” and the representation is not adjusted.; Printz, ¶ [0129]).
Regarding claim 4, the rejection of claim 3 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite comprising: in accordance with a determination that the representation requires modification, modifying, by the language model, the representation; and providing the output based on the modified representation.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 4, Inbavaluthi teaches comprising: in accordance with a determination that the representation requires modification, modifying, by the language model, the representation (“determining a connection between the audio and the text based on an individual contextual embedding and a fused contextual embedding at the context tokens level by a multi modal unified attention sequence tagger 406” and “tagging a plurality of replacement, cue and correction words sequentially based on the determined connection between the audio and the text {a determination that the adjusted representation requires modification} by the multi modal unified attention sequence tagger 406,” where the tagging and where the multi-modal unified attention sequence tagger can be a language model, as described, for example, with reference to FIG. 7E; Inbavaluthi, ¶ [0090], [0106]-[0108]; FIG. 7E); and providing the output based on the modified representation. (“displaying the decoded revised text on a user interface by the on-the-fly decoder 408” and/or “send the decoded revised text to an NLP to generate a response corresponding to the input speech”; Inbavaluthi, ¶ [0109]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include comprising: in accordance with a determination that the representation requires modification, modifying, by the language model, the representation; and providing the output based on the modified representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 5, the rejection of claim 3 is incorporated. Printz discloses all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite comprising: in accordance with a determination that the representation does not require modification: forgoing modifying, by the language model, the representation; and providing an output based on the representation.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 5, Inbavaluthi teaches comprising: in accordance with a determination that the representation does not require modification: forgoing modifying, by the language model, the representation (The system includes “tagging a plurality of replacement, cue and correction words sequentially based on the determined connection between the audio and the text by the multi modal unified attention sequence tagger,” as such the determined connection is a condition either met or not met, upon which the decision to tag or not tag is based, which is the determination of “[the existence of] the disfluency in the dictation”. As such, if the condition of the “determined connection” is not met, the system does not tag “a plurality of replacement, cue and correction words sequentially” as there is no determined connection upon which to base the tagging.; Inbavaluthi, ¶ [0107], [0111]-[0114], [0117]); and providing an output based on the representation. (As shown, for example, in FIG. 11, the real time audio received which does not include a “determined connection” does not correspond to and therefore does not receive tags (the “tagging a plurality of replacement, cue and correction words sequentially based on the determined connection” is both sequential and are directly associated with the determined connection of replacement, cue, and correction). As such, as shown in “Today I woke up early and saw the food menu,” the system provides the output based on the representation alone.; Inbavaluthi, ¶ [0107], [0112], [0117]; FIG. 11).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include comprising: in accordance with a determination that the representation does not require modification: forgoing modifying, by the language model, the representation; and providing an output based on the representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 6, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses comprising: in accordance with a determination that at least one entity within the representation requires disambiguation (“the natural language understanding module may accept as input the transcription and word timings generated by the primary recognizer” and “may identify a particular word or word sequence in the input transcription that potentially comprises a proper name entity”; Printz, ¶ [0118]-[0119]): in accordance with a determination that a reference for the at least one entity can be obtained, adjusting the representation by adding the indication to the at least one entity (In response to identifying “a particular word or word sequence in the input transcription that potentially comprises a proper name entity” the module can then “label this word or word sequence with a putative type (for instance, a person’s name, a street intersection, a numbered street address, and so on)” where “each such word or word sequence is called a proper name entity acoustic span, or the acoustic span of a proper name entity, or just an acoustic span.”; Printz, ¶ [0118]-[0119]).
Regarding claim 7, the rejection of claim 6 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses comprising: in accordance with a determination that a reference for the at least one entity cannot be obtained: forgoing adjusting the representation by adding the indication to the at least one entity; and providing the representation to the language model. (“the system may determine which of the proposed proper entities for the acoustic spans (and/or the confidence levels associated with a hypothesis without acoustic spans) best corresponds to the utterance” where “if no appropriate resolutions are found at block 1055 {in accordance with a determination that a reference for the at least one entity cannot be obtained},...the system may instead attempt fulfillment with the words having deficient probabilities {forgoing adjusting the representation...} or with the closest approximates {providing the representation to the language model}.”; Printz, ¶ [0217]-[0219]).
Regarding claim 8, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses comprising: identifying context information associated with the electronic device, (As disclosed with reference to an example, the “a hypothesis 1205 having an acoustic span between 1520 and 1750 may be generated (one will recognize that the values 1520 and 1750 are merely exemplary and other representations, e.g., milliseconds, may be used). The NLU may infer that this is a ‘Location inquiry’ and ascribe a corresponding potential meaning, based upon the ‘Where’ and ‘parking for’ portions of the utterance” and “the NLU may infer based upon the phrase ‘restaurant’ that the span is of type ‘Business name’” all of which “may be used to localize the search for proper names on the client device”; Printz, ¶ [0222]-[0224]) wherein a determination whether at least one entity within the representation requires disambiguation is (The search for proper names corresponding to the acoustic span of hypothesis 1205 is a determination of whether a corresponding proper name is reflected in the acoustic span, which is a determination whether the named entity {at least one entity} indicated in the acoustic span of the hypothesis {the representation} is correct {requires disambiguation}; Printz, ¶ [0224]) based on the context information associated with the electronic device. (The ascribed potential meaning is used to “localize the search for proper names on the client device {based on the context information associated with the electronic device}”; Printz, ¶ [0224]).
Regarding claim 9, the rejection of claim 8 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses wherein the context information includes at least one of displayed content and non-displayed content. (the contextual information can include “one of the displayed business names” {displayed content} and “the user’s contact list” which can be searched for “a suitable phone number or email address” {non-displayed content} which corresponds to the entity in the acoustic span.; Printz, ¶ [0147], [0254]-[0257]).
Regarding claim 10, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses wherein a determination whether at least one respective entity within a respective representation requires disambiguation includes determining whether a reference resolution system can obtain a reference for the at least one respective entity with a confidence level exceeding a threshold confidence level. (“the system may determine if one or more word confidence values are deficient, e.g., have confidence levels falling below a threshold, or if the modified systems have otherwise identified one or more potential proper names” {candidates “...with a confidence level exceeding a threshold confidence level”} and where “no proper name candidates {a reference for the at least one respective entity...} are otherwise identified {a reference for the at least one respective entity...}” then “the system may complete processing to generate a symbolic representation” and “may attempt fulfilment using the symbolic representation and return any results to the user.”; Printz, ¶ [0118], [0217]).
Regarding claim 12, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite comprising: in accordance with a determination that the adjusted representation does not require modification, providing the output based on the adjusted representation.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 12, Inbavaluthi teaches comprising: in accordance with a determination that the adjusted representation does not require modification (The system includes “tagging a plurality of replacement, cue and correction words sequentially based on the determined connection between the audio and the text by the multi modal unified attention sequence tagger,” as such the determined connection is a condition either met or not met, upon which the decision to tag or not tag is based, which is the determination of “[the existence of] the disfluency in the dictation”. As such, if the condition of the “determined connection” is not met, the system does not tag “a plurality of replacement, cue and correction words sequentially” as there is no determined connection upon which to base the tagging.; Inbavaluthi, ¶ [0107], [0111]-[0114], [0117]), providing the output based on the adjusted representation. (As shown, for example, in FIG. 11, the real time audio received which does not include a “determined connection” does not correspond to and therefore does not receive tags (the “tagging a plurality of replacement, cue and correction words sequentially based on the determined connection” is both sequential and are directly associated with the determined connection of replacement, cue, and correction). As such, as shown in “Today I woke up early and saw the food menu,” the system provides the output based on the representation alone.; Inbavaluthi, ¶ [0107], [0112], [0117]; FIG. 11).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include comprising: in accordance with a determination that the adjusted representation does not require modification, providing the output based on the adjusted representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 13, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite wherein a determination whether a respective representation requires modification includes detecting whether the respective representation includes a disfluency.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 13, Inbavaluthi teaches wherein a determination whether a respective representation requires modification includes detecting whether the respective representation includes a disfluency. (Regarding the example, ““what’s the weather in Mumbai oh sorry Delhi”, the “voice assistant” detects “the disfluency in the command and it provides accurate results” by modifying the original text with the disfluency correction “Thus, the user does not have to repeat the command as the voice assistant provides accurate results by detecting the cue words i.e., “oh sorry” and replacing the replacement word i.e., Mumbai with the correct word i.e., Delhi.”; Inbavaluthi, ¶ [0111]; FIG. 10).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include wherein a determination whether a respective representation requires modification includes detecting whether the respective representation includes a disfluency. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 14, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite wherein a determination whether a respective representation requires modification includes detecting whether the respective representation includes a correction.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 14, Inbavaluthi teaches wherein a determination whether a respective representation requires modification includes detecting whether the respective representation includes a correction (Regarding the example, “Today I woke up early and saw the food menu. Recently I had seen a show about top 100 I mean top 10 foods to eat during breakfast. I think I saw it somewhere on YouTube sorry Netflix. I should watch it again,” the “voice assistant” detects “the disfluency in the dictation and it provides accurate results by writing the notes correctly as, ‘Today I woke up early and saw the food menu. Recently I had seen a show about top 10 foods to eat during breakfast. I think I saw it somewhere on Netflix. I should watch it again’,” based on “detecting the cue words i.e., “I mean and sorry” and replacing the replacement words with the correct words.” where said cue words indicate a correction by the user.; Inbavaluthi, ¶ [0112]; FIG. 11).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include wherein a determination whether a respective representation requires modification includes detecting whether the respective representation includes a correction. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 15, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses wherein a determination whether a respective representation requires modification includes detecting whether the respective representation includes an ambiguity exceeding a threshold ambiguity. (“ where low confidence values are generated, but words are identified anyway, may correspond to the path through blocks 1015 and block 1045 discussed above. These confidence values may be lower than a threshold, e.g., 300, indicating an incorrect association. In this example, “goose” mismatches “Guddu” and “karate” mismatches “Karahi” as the words are superficially similar. Accordingly, the 110 and 150 confidence levels reflect an unlikely match (e.g., because the spectral character of the waveform doesn’t agree with the expected character of the phonemes in these words) {ambiguity exceeding a threshold ambiguity}.”; Printz, ¶ [0221]).
Regarding claim 16, the rejection of claim 15 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. Printz further discloses wherein prior to detecting whether the respective representation includes an ambiguity exceeding a threshold ambiguity (“where low confidence values are generated, but words are identified anyway, may correspond to the path through blocks 1015 and block 1045 discussed above. These confidence values may be lower than a threshold, e.g., 300, indicating an incorrect association. In this example, “goose” mismatches “Guddu” and “karate” mismatches “Karahi” as the words are superficially similar. Accordingly, the 110 and 150 confidence levels reflect an unlikely match (e.g., because the spectral character of the waveform doesn’t agree with the expected character of the phonemes in these words) {ambiguity exceeding a threshold ambiguity}.”; Printz, ¶ [0221]), determining that a reference resolution system is unable to obtain a reference for at least one entity within the respective representation based on the ambiguity. (Prior to the above step, “the system may determine if one or more word confidence values are deficient, e.g., have confidence levels falling below a threshold, or if the modified systems have otherwise identified one or more potential proper names. Where all of the confidence values exceed a threshold, or where no proper name candidates are otherwise identified, the system may transition to block 1035.”; Printz, ¶ [0221]).
Regarding claim 17, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite wherein modifying, by the language model, the adjusted representation includes correcting a syntax of the adjusted representation.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 17, Inbavaluthi teaches wherein modifying, by the language model, the adjusted representation includes correcting a syntax of the adjusted representation (Regarding the example, ““what’s the weather in Mumbai oh sorry Delhi”, the “voice assistant” detects “the disfluency in the command and it provides accurate results” by modifying the original text with the disfluency correction “Thus, the user does not have to repeat the command as the voice assistant provides accurate results by detecting the cue words i.e., “oh sorry” and replacing the replacement word i.e., Mumbai with the correct word i.e., Delhi.” As the original sentence included a structural discontinuity, “oh sorry” which breaks the syntactic rules in English. The operation corrects the structural sequence, which corrects the syntax of the adjusted representation, as received from the processes of Printz.; Inbavaluthi, ¶ [0111]; FIG. 10).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include wherein modifying, by the language model, the adjusted representation includes correcting a syntax of the adjusted representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 18, the rejection of claim 1 is incorporated. Printz and Inbavaluthi disclose all of the elements of the current invention as stated above. However, Printz fail(s) to expressly recite wherein modifying, by the language model, the adjusted representation includes correcting a grammatical error within the adjusted representation.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 18, Inbavaluthi teaches wherein modifying, by the language model, the adjusted representation includes correcting a grammatical error within the adjusted representation (Regarding the example, “Today I woke up early and saw the food menu. Recently I had seen a show about top 100 I mean top 10 foods to eat during breakfast. I think I saw it somewhere on YouTube sorry Netflix. I should watch it again,” the “voice assistant” detects “the disfluency in the dictation and it provides accurate results by writing the notes correctly as, ‘Today I woke up early and saw the food menu. Recently I had seen a show about top 10 foods to eat during breakfast. I think I saw it somewhere on Netflix. I should watch it again’,” based on “detecting the cue words i.e., “I mean and sorry” and replacing the replacement words with the correct words.” where said cue words indicate a correction by the user. As the original sentence included an interrupted noun phrase and an orphaned modifier that renders the sentence grammatically incoherent. The operation corrects the structural and generates a grammatically compliant sentence, which corrects a grammar error in the adjusted representation, as received from the processes of Printz.; Inbavaluthi, ¶ [0112]; FIG. 11).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include wherein modifying, by the language model, the adjusted representation includes correcting a grammatical error within the adjusted representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 19, Printz discloses A computer-implemented method (Systems and methods described with reference to a system comprising an “open dictation ASR component”; Printz, ¶ [0109]), comprising: at an electronic device with one or more processors and memory (the system can be implemented using “computing system 1800” including “one or more central processing units (“processors”) 1805” and “memory 1810”/”storage devices 1820”; Printz, ¶ [0380]): receiving a speech input from a user (“Various embodiments accept as input an audio signal comprising fluent, natural human speech, which notably may contain one or several proper names, or unorthodox sequences of otherwise ordinary words.”; Printz, ¶ [0114]); obtaining a representation of the speech input (“The primary recognizer may include a conventional open dictation automatic speech recognition (ASR) system” which “accepts as input an audio signal comprising human speech” and produces “a textual transcription of this input, labeled with the start time and end time, within the input audio signal, of each transcribed word. “; Printz, ¶ [0116]); in accordance with a determination that at least one entity within the representation requires disambiguation (“the natural language understanding module may accept as input the transcription and word timings generated by the primary recognizer” and “may identify a particular word or word sequence in the input transcription that potentially comprises a proper name entity”; Printz, ¶ [0118]-[0119]): adjusting the representation by adding an indication to the at least one entity (In response to identifying “a particular word or word sequence in the input transcription that potentially comprises a proper name entity” the module can then “label this word or word sequence with a putative type (for instance, a person’s name, a street intersection, a numbered street address, and so on)” where “each such word or word sequence is called a proper name entity acoustic span, or the acoustic span of a proper name entity, or just an acoustic span.”; Printz, ¶ [0118]-[0119]); and providing the adjusted representation to a language model (“The hypotheses may be received at a secondary recognizer 860” where “if the hypotheses do include acoustic spans (e.g., in the request “Tell me how to get to Guddu du Karahi”), the secondary recognizer 860 may consult adaptation object generator 865 to identify an appropriate grammar of proper names from the various components 840 a-e of the user device” and the secondary recognizer can include “neural networks including deep neural networks... [and] language models” which “may apply various of the identified proper entities and determine corresponding confidence levels.”; Printz, ¶ [0124], [0199]-[0200]). However, Printz fails to expressly recite in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation; and providing an output based on the modified and adjusted representation.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 19, Inbavaluthi teaches in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation (“determining a connection between the audio and the text based on an individual contextual embedding and a fused contextual embedding at the context tokens level by a multi modal unified attention sequence tagger 406” and “tagging a plurality of replacement, cue and correction words sequentially {modifying, by the language model, the adjusted representation} based on the determined connection between the audio and the text {a determination that the adjusted representation requires modification} by the multi modal unified attention sequence tagger 406,” where the multi-modal unified attention sequence tagger can be a language model, as described, for example, with reference to FIG. 7E; Inbavaluthi, ¶ [0090], [0106]-[0108]; FIG. 7E); and providing an output based on the modified and adjusted representation (“displaying the decoded revised text on a user interface by the on-the-fly decoder 408” and/or “send the decoded revised text to an NLP to generate a response corresponding to the input speech”; Inbavaluthi, ¶ [0109]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation; and providing an output based on the modified and adjusted representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Regarding claim 20, Printz discloses A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device (Systems and methods described with reference to a system comprising an “open dictation ASR component” which can be implemented using “computing system 1800” including “one or more central processing units (“processors”) 1805” and “memory 1810”/”storage devices 1820” and “The memory 1810 and storage devices 1820 are computer-readable storage media that may store instructions that implement at least portions of the various embodiments” and “instructions stored in memory 1810 can be implemented as software and/or firmware to program the processor 1805 to carry out actions described above.”; Printz, ¶ [0109], [0380]-[0382]) , the one or more programs including instructions for: receiving a speech input from a user (“Various embodiments accept as input an audio signal comprising fluent, natural human speech, which notably may contain one or several proper names, or unorthodox sequences of otherwise ordinary words.”; Printz, ¶ [0114]); obtaining a representation of the speech input (“The primary recognizer may include a conventional open dictation automatic speech recognition (ASR) system” which “accepts as input an audio signal comprising human speech” and produces “a textual transcription of this input, labeled with the start time and end time, within the input audio signal, of each transcribed word. “; Printz, ¶ [0116]); in accordance with a determination that at least one entity within the representation requires disambiguation (“the natural language understanding module may accept as input the transcription and word timings generated by the primary recognizer” and “may identify a particular word or word sequence in the input transcription that potentially comprises a proper name entity”; Printz, ¶ [0118]-[0119]): adjusting the representation by adding an indication to the at least one entity (In response to identifying “a particular word or word sequence in the input transcription that potentially comprises a proper name entity” the module can then “label this word or word sequence with a putative type (for instance, a person’s name, a street intersection, a numbered street address, and so on)” where “each such word or word sequence is called a proper name entity acoustic span, or the acoustic span of a proper name entity, or just an acoustic span.”; Printz, ¶ [0118]-[0119]); and providing the adjusted representation to a language model (“The hypotheses may be received at a secondary recognizer 860” where “if the hypotheses do include acoustic spans (e.g., in the request “Tell me how to get to Guddu du Karahi”), the secondary recognizer 860 may consult adaptation object generator 865 to identify an appropriate grammar of proper names from the various components 840 a-e of the user device” and the secondary recognizer can include “neural networks including deep neural networks... [and] language models” which “may apply various of the identified proper entities and determine corresponding confidence levels.”; Printz, ¶ [0124], [0199]-[0200]). However, Printz fails to expressly recite in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation; and providing an output based on the modified and adjusted representation.
The relevance of Inbavaluthi is described above with relation to claim 1. Regarding claim 20, Inbavaluthi teaches in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation (“determining a connection between the audio and the text based on an individual contextual embedding and a fused contextual embedding at the context tokens level by a multi modal unified attention sequence tagger 406” and “tagging a plurality of replacement, cue and correction words sequentially {modifying, by the language model, the adjusted representation} based on the determined connection between the audio and the text {a determination that the adjusted representation requires modification} by the multi modal unified attention sequence tagger 406,” where the multi-modal unified attention sequence tagger can be a language model, as described, for example, with reference to FIG. 7E; Inbavaluthi, ¶ [0090], [0106]-[0108]; FIG. 7E); and providing an output based on the modified and adjusted representation (“displaying the decoded revised text on a user interface by the on-the-fly decoder 408” and/or “send the decoded revised text to an NLP to generate a response corresponding to the input speech”; Inbavaluthi, ¶ [0109]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the open dictation ASR systems of Printz to incorporate the teachings of Inbavaluthi to include in accordance with a determination that the adjusted representation requires modification: modifying, by the language model, the adjusted representation; and providing an output based on the modified and adjusted representation. Printz discloses an open dictation ASR system which can “recognize and understand completely novel proper names” as part of a real-time process. Further, Printz discloses the use of a language model for the named entity process. However, Printz fails to expressly recite the language model also correcting disfluencies which commonly occur during speech recognition processes. Inbavaluthi discloses systems and methods for “auto-correction of an ongoing speech command” such that disfluencies are addressed in commands, dictation, and other speech recognition processes, which would be understood by one of ordinary skill in the art to interact synergistically with a named entity recognition system, where ambiguous entities and disfluencies are well known problems in the art and proper recognition of named entities is of known importance in both references. The two-stage system created by the combination of Printz and Invabaluthi would have been obvious to a person having ordinary skill in the art, to handle real-time data such that disambiguation and disfluency resolution are performed in a time sensitive manner (i.e., “real-time”) and provides the known benefit of the time sensitive generation of corrected speech for a wide variety of speech processes, as recognized based on the disclosure of Inbavaluthi. (Inbavaluthi, ¶ [0009], [0070]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tablan (U.S. Pat. No. 10963497) discloses a query parsing system which uses a multi-stage process to parse the text of incoming queries before attempting to answer the queries.
Zhu (U.S. Pat. App. Pub. No. 2021/0375289) an automated transcription system where transcription of audio speech included in electronic content associated with a meeting is created by an ASR model trained on speech-to-text data, and post-processed by modifying text included in the transcription, for example, by modifying punctuation, grammar, or formatting introduced by the ASR model and by changing or omitting one or more words that were included in both the audio speech and the transcription.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sean E. Serraguard whose telephone number is (313)446-6627. The examiner can normally be reached 07:00-17:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel C. Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Sean E Serraguard/Primary Examiner, Art Unit 2657
