DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-6 are pending and have been examined. Claims 1, 5, and 6 are independent.
This Application was published as U.S. 2025/0356853 A1.
Apparent priority: May 20, 2024.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on May 1, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
a. In paragraph [0060], “a I-length subword token sequence” should apparently read “an I-length subword token sequence.”
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. - An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “an acquisition unit configured to acquire audio data of an utterance” and “a speech recognition unit configured to generate text from the audio data using an automatic speech recognition model” in claim 1, and “the speech recognition unit registers any one or a combination of the words, phrases, and sentences input by the user to the input interface as the first token” in claim 4.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The corresponding structure for the acquisition unit and the speech recognition unit is the processing unit 140 implemented by a processor executing a program stored in the storage unit 150, or by hardware such as an LSI circuit, an ASIC, an FPGA, or an SoC: “The processing unit 140 includes, for example, an acquisition unit 141, a speech recognition unit 142, an output control unit 143, and a machine learning unit 144. Constituent elements of the processing unit 140 are implemented by a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) executing a program stored in the storage unit 150. Moreover, the constituent elements of the processing unit 140 may be implemented by hardware such as a large-scale integration (LSI) circuit, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a system on chip (SOC) or may be implemented by software and hardware in cooperation.” (published specification, paragraph [0034]), performing the algorithm of FIG. 2: “First, the acquisition unit 141 acquires audio data of an utterance from the microphone 110 (step S100).” (paragraph [0038]), and “Subsequently, the speech recognition unit 142 generates text data from the audio data using the E2E-ASR model (step S102).” (paragraph [0039]), the E2E-ASR model being the audio encoder ENC1, bias encoder ENC2, and bias decoder DEC1 of FIG. 3 (paragraph [0057]). The registration recited in claim 4 is performed by the same processor with the input interface 120: “For example, the user may input any one or a combination of words, phrases, and sentences to the input interface 120. These are registered as dynamic bias tokens to be described below.” (paragraph [0032]) and “The bias list B is, for example, a list in which any one or a combination of words, phrases, and sentences input to the input interface 120 is registered as a dynamic bias token.” (paragraph [0059]).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
The terms “first encoder,” “second encoder,” and “decoder” in claims 1, 5, and 6 are not interpreted under 35 U.S.C. 112(f). Each names a neural network structure recognized in the art and is defined in the claim by its recited inputs and outputs. The term “input interface” in claim 4 is likewise not interpreted under 35 U.S.C. 112(f); the specification identifies it as “a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch panel, or the like” (paragraph [0031]).
The following express claim constructions are applied in this action. The recited “first token” is a registered word, phrase, or sentence treated as a single output unit of the decoder, and the recited “second token” is an ordinary output unit of the decoder (a character, subword, or word of the fixed vocabulary) that is not a first token. The recited “second token sequence” is the sequence of tokens previously estimated by the decoder, which may contain first tokens, second tokens, or both. The recited “decoder expanded to correspond to the first token” is a decoder whose output units include one unit for each registered first token in addition to the units for the second tokens.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 (Statutory Category). Claim 1 is directed to a speech recognition device (a machine), claim 5 is directed to a speech recognition method (a process), and claim 6 is directed to a non-transitory storage medium storing a program (a manufacture). Claims 1-6 are therefore directed to a statutory category of invention.
Step 2A, Prong One (Recitation of a Judicial Exception). Claim 1 recites, in pertinent part, a first encoder configured to convert a first feature sequence in which features of the audio data are arranged into a second feature sequence; a second encoder configured to register any one or a combination of pre-registered words, phrases, and sentences as a first token and convert a first token sequence in which first tokens are arranged into a third feature sequence; and a decoder expanded to correspond to the first token and configured to estimate the second token or the first token following a second token sequence in which at least one of the first token and a second token different from the first token previously estimated as the text is arranged on the basis of the second feature sequence, the third feature sequence, and the second token sequence. Under their broadest reasonable interpretation, the converting, converting, and estimating limitations recite mathematical concepts, namely mathematical relationships and calculations that map one sequence of numerical features to another and compute which token follows a sequence from those features (the specification describes each of them as an equation: Eqs. (9)-(14)), and they recite mental processes, namely observation, evaluation, and judgment. A person holding a list of registered names and phrases, listening to an utterance, and writing down its text carries out these steps mentally or with pen and paper: the person notes the features of the sounds heard, keeps the list of registered phrases in mind as a whole, and, given what has been written so far, judges whether the next unit of the text is an ordinary word or one of the registered phrases and writes it down as a single unit. Claim 3 further recites calculating a first score on the basis of an inner product, calculating a second score, and calculating a probability on the basis of the first score and the second score, which are mathematical calculations as such. The claims therefore recite an abstract idea.
Step 2A, Prong Two (Integration into a Practical Application). The judicial exception is not integrated into a practical application. Beyond the converting and estimating limitations, claim 1 recites an acquisition unit configured to acquire audio data of an utterance, a speech recognition unit configured to generate text from the audio data using an automatic speech recognition model, and the first encoder, second encoder, and decoder as components of that model. Acquiring the audio data is insignificant extra-solution activity in the nature of gathering the data on which the mathematical and mental steps operate. The automatic speech recognition model and its encoders and decoder are recited at a high level of generality as neural network components with no more than their inputs and outputs, and the specification describes them as a generic end-to-end model whose audio encoder “is the same as the encoder of the general E2E-ASR model” (paragraph [0057]), implemented by a processor such as a CPU or a GPU executing a program stored in a memory (paragraph [0034]). Registering words, phrases, and sentences as tokens and expanding the decoder to output them is a change to the mathematical model itself, that is, to the abstract idea, and not to the functioning of a computer or to another technology: the asserted advance is that infrequent words can be registered without retraining the model (paragraphs [0004]-[0005]), which is an improvement to the abstract recognition scheme. Considered individually and in combination, the additional elements amount to collecting audio data, analyzing it with a generic model, and outputting text, generally linked to a computing environment, and they do not impose meaningful limits on practicing the abstract idea. The claim is directed to the abstract idea.
Step 2B (Inventive Concept). Claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the acquisition unit, the speech recognition unit, and the encoder-decoder model are recited generically and perform the well-understood, routine, and conventional functions of receiving audio data, processing it with a neural network, and outputting text; the specification acknowledges that end-to-end speech recognition models with an encoder and a decoder are conventional (paragraphs [0003]). Considered individually and as an ordered combination, the additional elements amount to no more than mere instructions to apply the mathematical and mental steps using generic components and therefore do not provide an inventive concept. Claim 1 is not patent eligible.
Independent claims 5 and 6. Claim 5 recites the operations of claim 1 as a method using a computer, and claim 6 recites them as a program stored on a non-transitory storage medium for causing a computer to perform them. Beyond the operations addressed above, claim 5 adds a computer and claim 6 adds a non-transitory storage medium storing a program, which are generic computer components described at a high level of generality (paragraphs [0034] and [0036]) and do not integrate the abstract idea into a practical application or provide an inventive concept, for the reasons given for claim 1. Claims 5 and 6 are not patent eligible.
The dependent claims have been considered and do not cure the deficiencies of the independent claims. Each dependent claim is addressed separately below.
Claim 2 recites that the embedding layer determines whether the second token sequence includes the first token, converts the sequence into a fourth feature sequence, and generates a fifth feature sequence by concatenating a third feature with the fourth feature sequence. This relates to a person checking whether a registered phrase has already been written and combining its representation with the representation of the rest of the text, which is a mental process, and to concatenating feature vectors, which is a mathematical concept. No additional limitations are present. Claim 2 neither integrates the exception into a practical application nor amounts to significantly more.
Claim 3 recites that the output layer converts the fourth or fifth feature sequence into a sixth feature sequence and calculates a first score on the basis of an inner product, a second score, and a probability on the basis of the two scores. These are mathematical calculations as such. No additional limitations are present. Claim 3 neither integrates the exception into a practical application nor amounts to significantly more.
Claim 4 recites an input interface capable of being manipulated by a user, through which the words, phrases, and sentences to be registered as the first token are input. The input interface is a generic component, described in the specification as “a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch panel, or the like” (paragraph [0031]), and receiving the list of phrases to register through it is insignificant extra-solution activity in the nature of gathering the data on which the abstract idea operates. Claim 4 neither integrates the exception into a practical application nor amounts to significantly more.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless -
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 5, and 6 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Xiong (U.S. 2023/0186912), hereinafter Xiong.
Regarding claim 1, Xiong discloses:
1. A speech recognition device comprising:
“A speech recognition device is provided according to a third aspect of the present disclosure. The device includes: a memory, configured to store a program; and a processor, configured to execute the program, to perform the speech recognition method as described above.” (Xiong, paras. [0039]-[0041].)
“The speech recognition method may be implemented by an electronic device, such as a mobile phone, a translator, a computer, a server and other devices with data processing capabilities.” (Xiong, para. [0061]; see also FIG. 7.)
an acquisition unit configured to acquire audio data of an utterance; and
“In step S100, a to-be-recognized speech and a configured hot word library are acquired.” (Xiong, para. [0063]; see also FIG. 3.)
“a speech of the user in the scenario of human-computer interaction may be acquired as the to-be-recognized speech” (Xiong, para. [0080].)
“For convenience of description, the audio feature of the to-be-recognized speech is defined as X=[x1, x2, . . . , xk], where xk represents an audio feature vector of a k-th frame, and k represents the total number of frames of the to-be-recognized speech.” (Xiong, para. [0099].)
The speech of the user acquired at step S100 is the audio data of an utterance of the claim. Xiong performs the acquisition with a data acquiring unit 11 “configured to acquire a to-be-recognized speech and a configured hot word library” (Xiong, FIG. 6), which is the acquisition unit of the claim under the interpretation set out in the Claim Interpretation section above.
a speech recognition unit configured to generate text from the audio data using an automatic speech recognition model, wherein the automatic speech recognition model includes
“processing by a pre-trained speech recognition model, the to-be-recognized speech and the hot word library, to obtain the recognition result of the to-be-recognized speech outputted by the speech recognition model” (Xiong, para. [0014].)
“The audio-related feature and the hot word-related feature are decoded and classified to obtain a recognition text corresponding to an input speech.” (Xiong, para. [0057].)
“The speech recognition model may include an audio encoding module, a hot word encoding module, a joint attention module, a decoding module and a classifying module. The modules cooperatively process hot words in the hot word library and the audio feature of the to-be-recognized speech, to output the recognition result.” (Xiong, para. [0098]; see also FIG. 4.)
Xiong’s pre-trained end-to-end speech recognition model is the automatic speech recognition model of the claim, and the recognition text it outputs for the input speech is the text generated from the audio data. The recognition result acquiring unit 14 of FIG. 6, implemented by the speech recognition model (Xiong, FIG. 6), is the speech recognition unit of the claim under the interpretation set out in the Claim Interpretation section above.
a first encoder configured to convert a first feature sequence in which features of the audio data are arranged into a second feature sequence;
“The audio encoding module may encode the audio feature X of the to-be-recognized speech, to obtain an encoded audio feature vector sequence formed by encoded audio feature vectors of all frames.” (Xiong, para. [0102].)
“The audio feature vector sequence obtained after encoding is expressed as: Hx=[h1x, h2x, . . . , hkx], where hkx represents an encoded audio feature vector of the k-th frame, and hkx corresponds to a result of encoding xk by the audio encoding module.” (Xiong, para. [0103].)
The audio feature X=[x1, x2, . . . , xk], in which the audio feature vectors of the frames of the speech are arranged, is the first feature sequence of the claim; the encoded audio feature vector sequence Hx output by the audio encoding module is the second feature sequence; and the audio encoding module that converts X into Hx is the first encoder.
a second encoder configured to register any one or a combination of pre-registered words, phrases, and sentences as a first token and convert a first token sequence in which first tokens are arranged into a third feature sequence; and
“Before the speech is recognized, the configured hot word library may be acquired and multiple hot words are stored in the hot word library. It can be understood that the hot word library may be formed by hot words related to the speech recognition task including all potential hot words, such as technology terminologies, in the to-be-recognized speech.” (Xiong, para. [0064].)
“Therefore, operation keywords in the voice commands of the user may form a hot word library according to the present disclosure.” (Xiong, para. [0079].)
“Specifically, the hot word encoding module encodes each hot word in the hot word library separately, to obtain a hot word feature vector sequence formed by hot word feature vectors obtained by encoding respective hot words.” (Xiong, para. [0108].)
“The hot word library includes N+1 hot words expressed as Z=[z0, z1, . . . , zN], where zN represents an N-th hot word.” (Xiong, para. [0109].)
“If the total number of the hot words is N+1, the hot word encoding module encodes each of the hot words separately, to obtain the hot word feature vector sequence expressed as: Hz=[h0z, h1z, . . . , hNz], where hNz represents a hot word feature vector obtained by encoding the N-th hot word by the hot word encoding module.” (Xiong, para. [0110].)
“It can be understood that different hot words include different numbers of characters. For example, Chinese expressions “ZHONG KE DA” and “KE DA XUN FEI”, if defined as hot words, include three and four Chinese characters, respectively.” (Xiong, para. [0111]; see also FIG. 5.)
Each hot word of the configured hot word library is a pre-registered word or phrase, and defining a word or a multi-character expression as a hot word of the library is registering it as a first token of the claim. The hot word library Z=[z0, z1, . . . , zN], in which the hot words are arranged, is the first token sequence, the hot word feature vector sequence Hz is the third feature sequence, and the hot word encoding module that encodes each hot word into one vector of Hz is the second encoder.
a decoder expanded to correspond to the first token and configured to estimate the second token or the first token following a second token sequence in which at least one of the first token and a second token different from the first token previously estimated as the text is arranged on the basis of the second feature sequence, the third feature sequence, and the second token sequence.
“The decoding module may calculate the state vector dt at the current decoding time instant t by using the combined feature ct-1 used at a decoding time instant t−1 prior to the current decoding time instant t and a result character recognized at the previous decoding time instant t−1.” (Xiong, para. [0150].)
“It is defined that, t represents the current decoding time instant, dt represents the state vector outputted by the decoding module at the current decoding time instant t, and the first attention model determines the audio-related feature ctx used at the current decoding time instant from the audio feature vector sequence Hx based on the state vector dt and the hot word feature vector sequence Hz.” (Xiong, para. [0126].)
“The ctx and ctz are combined into the combined feature ct used at the current decoding time instant, and then the combined feature ct is transmitted to the decoding module.” (Xiong, para. [0146].)
“The classification nodes of the classifying module include not only fixed commonly-used character nodes but also dynamically expandable hot word nodes, so as to recognize the entire hot word by units of word, rather than recognizing and assigning incentives by units of characters in the hot words as in the conventional technology.” (Xiong, para. [0161].)
“In a case that N hot words are in the hot word library, the number of the hot word nodes in the classifying module may be N, that is, the total number of classification nodes of the classifying module is V+N.” (Xiong, para. [0162].)
“For the i-th (i ranges from 1 to N, inclusive) hot word node among the N hot words, a score of the i-th hot word in the probability distribution outputted by the static classifier is 0, and a probability score of the i-th hot word in the hot word classifier is bit, and thus the final probability score of the i-th hot word is Pbt*bit. For yt among the V commonly-used characters, a score of yt in the probability distribution outputted by the static classifier is Pv(yt), and a probability score of yt in the hot word classifier is Pv (yt)*b0t, and thus the final probability score of yt is (1−pbt)*Pv(yt)+Pbt*Pv(yt)*b0t.” (Xiong, para. [0176].)
The decoding module together with the classifying module is the decoder of the claim. Its V fixed commonly-used character nodes are the output units for the second tokens, and its N dynamically expandable hot word nodes, one per hot word of the library, are the output units added for the first tokens, so the decoder is expanded to correspond to the first token. At each decoding time instant the result recognized at the previous time instant, which is either a commonly-used character (a second token) or an entire hot word (a first token), is fed back to the decoding module, so the results previously recognized as the text are the second token sequence of the claim; the state vector dt is computed from that fed-back result, the audio-related feature is determined from Hx (the second feature sequence) based on dt and Hz (the third feature sequence), and the classifying module then scores every character node and every hot word node to determine the recognition result at the current time instant. Estimating whether the next output is a commonly-used character or an entire hot word from Hx, Hz, and the previously recognized results is estimating the second token or the first token following the second token sequence on the basis of the second feature sequence, the third feature sequence, and the second token sequence.
Claim 5 is a method claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Additionally:
Regarding claim 5, Xiong discloses:
5. A speech recognition method using a computer, comprising: acquiring audio data of an utterance; and generating text from the audio data using an automatic speech recognition model, wherein the automatic speech recognition model includes a first encoder configured to convert a first feature sequence in which features of the audio data are arranged into a second feature sequence; a second encoder configured to register any one or a combination of pre-registered words, phrases, and sentences as a first token and convert a first token sequence in which first tokens are arranged into a third feature sequence; and a decoder expanded to correspond to the first token and configured to estimate the second token or the first token following a second token sequence in which at least one of the first token and a second token different from the first token previously estimated as the text is arranged on the basis of the second feature sequence, the third feature sequence, and the second token sequence.
“The speech recognition method may be implemented by an electronic device, such as a mobile phone, a translator, a computer, a server and other devices with data processing capabilities.” (Xiong, para. [0061].)
“Next, the speech recognition method according to the present disclosure is described in combination with the flow shown in FIG. 3. The method includes the following steps S100 to S130.” (Xiong, para. [0062]; see also FIG. 3.)
The speech recognition method of Xiong performed by an electronic device such as a computer is the speech recognition method using a computer of the claim; the acquiring step and the generating step, and the automatic speech recognition model with the first encoder, the second encoder, and the decoder, are set forth in the claim 1 mapping above.
Claim 6 is a storage medium claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Additionally:
Regarding claim 6, Xiong discloses:
6. A non-transitory storage medium storing a program for causing a computer to: acquire audio data of an utterance; and generate text from the audio data using an automatic speech recognition model, wherein the automatic speech recognition model includes a first encoder configured to convert a first feature sequence in which features of the audio data are arranged into a second feature sequence; a second encoder configured to register any one or a combination of pre-registered words, phrases, and sentences as a first token and convert a first token sequence in which first tokens are arranged into a third feature sequence; and a decoder expanded to correspond to the first token and configured to estimate the second token or the first token following a second token sequence in which at least one of the first token and a second token different from the first token previously estimated as the text is arranged on the basis of the second feature sequence, the third feature sequence, and the second token sequence.
“A readable storage medium is provided according to a fourth aspect of the present disclosure. The readable storage medium stores a computer program. The computer program, when being executed by a processor, performs the speech recognition method as described above.” (Xiong, para. [0042]; see also FIG. 7.)
The readable storage medium storing the computer program, read by the processor 1001 from the memory 1003 of the electronic device of FIG. 7, is a non-transitory storage medium, and the stored program that a processor executes to perform the speech recognition method is the program for causing a computer to perform the acquiring and generating of the claim, which are set forth in the claim 1 mapping above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Xiong (U.S. 2023/0186912) in view of Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., and Xiang, B., “Abstractive text summarization using sequence-to-sequence RNNs and beyond,” Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), Berlin, Germany, August 2016, pp. 280-290 (arXiv:1602.06023v5, posted August 26, 2016, https://arxiv.org/abs/1602.06023), hereinafter Nallapati.
The examiner reads claim 2 as follows. The recited concatenating of a third feature corresponding to the first token and the fourth feature sequence is read as joining the two into the one feature sequence that is input to the decoder, so that the position of a first token in the second token sequence is represented by its third feature, taken from the third feature sequence output by the second encoder, and the positions of the remaining tokens are represented by the fourth feature sequence converted from them. The specification describes the expanded embedding layer in this way (published specification, FIG. 4 and Eq. (12)).
Regarding claim 2, Xiong discloses:
2. The speech recognition device according to claim 1,
Xiong discloses the speech recognition device of claim 1, as set forth in the rejection of claim 1 above.
Xiong does not teach the embedding-layer aspects of the claim. Nallapati teaches:
wherein the decoder has an embedding layer expanded to correspond to the first token, and (“E[oi−1] is the embedding vector of the emission from the previous time step” (Nallapati, Sec. 2.3.) “When the pointer is activated, the embedding from the source is used as input for the next time-step as shown by the arrow from the encoder to the decoder at the bottom.” (Nallapati, Sec. 2.3, Fig. 2.) Nallapati’s decoder input, which is the embedding of the emission from the previous time step or, when that emission was copied by the pointer, the embedding from the source, reads on an embedding layer expanded to correspond to the first token, and a word that the pointer copies from the source reads on the first token.)
wherein the embedding layer determines whether or not the second token sequence includes the first token, (“In this model, the decoder is equipped with a ‘switch’ that decides between using the generator or a pointer at every time-step.” (Nallapati, Sec. 2.3.) “When the pointer is activated, the embedding from the source is used as input for the next time-step as shown by the arrow from the encoder to the decoder at the bottom.” (Nallapati, Sec. 2.3, Fig. 2.) Nallapati selects the input for the next time step according to whether the previous emission was copied by the pointer, which reads on determining whether or not the second token sequence includes the first token.)
converts the second token sequence into a fourth feature sequence when the second token sequence does not include the first token, (“If the switch is turned on, the decoder produces a word from its target vocabulary in the normal fashion.” (Nallapati, Sec. 2.3.) “On the target side, we continue to use only word-based embeddings as the representation.” (Nallapati, Sec. 2.2.) The word-based embeddings of the words that the decoder produces from its target vocabulary read on the fourth feature sequence converted from a second token sequence that includes no copied word.)
and converts the remaining second token sequence, excluding the first token, into the fourth feature sequence when the second token sequence includes the first token, (“Since the vocabulary of the decoder is fixed at training time, it cannot emit these unseen words.” (Nallapati, Sec. 2.3.) “When the pointer is activated, the embedding from the source is used as input for the next time-step as shown by the arrow from the encoder to the decoder at the bottom.” (Nallapati, Sec. 2.3, Fig. 2.) A copied word, which is outside the decoder’s fixed vocabulary, is input as the embedding from the source, which reads on converting the remaining second token sequence, excluding the first token, into the fourth feature sequence.)
and generates a fifth feature sequence by concatenating a third feature corresponding to the first token included in the second token sequence among a plurality of third features included in the third feature sequence and the fourth feature sequence after conversion of the remaining second token sequence, excluding the first token. (“When the pointer is activated, the embedding from the source is used as input for the next time-step as shown by the arrow from the encoder to the decoder at the bottom.” (Nallapati, Sec. 2.3, Fig. 2.) Under the reading of claim 2 stated above, the decoder inputs of Nallapati’s successive time steps, the word-based embeddings of the generated words joined with the embedding from the source for the copied word, read on the fifth feature sequence, and the representation of the copied word taken from the encoder side, one among the representations of the source words, reads on the third feature corresponding to the first token among the plurality of third features.)
Xiong and Nallapati are considered to be analogous to the claimed invention because they are in the same field of attention-based encoder-decoder neural networks whose decoder outputs, as a single output unit, an item that is not in the decoder’s fixed vocabulary and takes each output back as the input of the next decoding step. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiong’s decoding module, whose first long short-term memory layer takes the result recognized at the previous decoding time instant as its input (“taking the result character recognized at the time instant t−1 and the combined feature ct-1 outputted by the joint attention module as inputs of a first long short-term memory layer” (Xiong, para. [0154])), so that a result that is an entire hot word, output from one of Xiong’s dynamically expandable hot word nodes, is input to that layer as the hot word feature vector hiz obtained by encoding that hot word in the hot word encoding module (Xiong, para. [0110]), while a result that is a commonly-used character continues to be input as a character, in the manner of Nallapati’s decoder, which inputs a word copied from the source as the embedding from the source and a generated word as its word-based embedding. One of ordinary skill in the art would have been motivated to make this modification in order to give the decoder an input for a previous output that its fixed vocabulary does not contain, which is the problem Nallapati’s pointer and source-side input address: “Since the vocabulary of the decoder is fixed at training time, it cannot emit these unseen words.” (Nallapati, Sec. 2.3.) Xiong’s hot words are such outputs, produced by nodes added to the fixed character nodes: “The classification nodes of the classifying module include not only fixed commonly-used character nodes but also dynamically expandable hot word nodes” (Xiong, para. [0161]).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Xiong (U.S. 2023/0186912) in view of Nallapati and further in view of Gu, J., Lu, Z., Li, H., and Li, V. O. K., “Incorporating copying mechanism in sequence-to-sequence learning,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, August 2016, pp. 1631-1640 (arXiv:1603.06393v3, posted June 8, 2016, https://arxiv.org/abs/1603.06393), hereinafter Gu.
The examiner reads the recited inner product of the sixth feature sequence and the first token sequence as an inner product of the sixth feature sequence with the encoded representations of the first tokens of the first token sequence, that is, with the third feature sequence output by the second encoder, which is how the specification computes the first score (published specification, FIG. 5 and Eq. (13)).
Regarding claim 3, Xiong discloses:
3. The speech recognition device according to claim 2, wherein the decoder has an output layer expanded to correspond to the first token, and
“The classification nodes of the classifying module include not only fixed commonly-used character nodes but also dynamically expandable hot word nodes, so as to recognize the entire hot word by units of word” (Xiong, para. [0161].)
“In a case that N hot words are in the hot word library, the number of the hot word nodes in the classifying module may be N, that is, the total number of classification nodes of the classifying module is V+N.” (Xiong, para. [0162].)
Xiong’s classifying module, with V fixed commonly-used character nodes and N dynamically expandable hot word nodes, reads on the output layer expanded to correspond to the first token.
wherein the output layer converts the fourth feature sequence or the fifth feature sequence into a sixth feature sequence,
“When decoding the to-be-decoded character at the current time instant t, the state vector dt of the decoding module at the current decoding time instant is calculated by taking the result character recognized at the time instant t−1 and the combined feature ct-1 outputted by the joint attention module as inputs of a first long short-term memory layer.” (Xiong, para. [0154].)
“The decoding module calculates the output feature htd of the decoding module by taking dt and ct as inputs of a second long short-term memory layer.” (Xiong, para. [0154].)
The second long short-term memory layer and the classifying module that follows it are read as the output layer. The state vector dt is computed from the result recognized at the previous time instant, which in the combination set forth for claim 2 is input to the first long short-term memory layer as the fourth or fifth feature sequence, and the second long short-term memory layer converts dt into the output feature htd, which reads on the sixth feature sequence.
calculates a second score that is a score of each of the second tokens included in the second token sequence, and
“The static classifier outputs a V-dimensional probability distribution, and the probability distribution is expressed as Pv(yt)=softmax(W*htd), where yt represents the to-be-decoded character at the current decoding time instant t, and matrix W represents a model parameter of the static classifier.” (Xiong, para. [0167].)
“The classifying module determines the probability score of each of the commonly-used character nodes and the probability score of each of the hot word nodes by using the output feature htd of the decoding module at the current decoding time instant; and then determines the final recognition result.” (Xiong, para. [0163].)
Pv(yt), computed from the output feature htd, gives a score for each of the V commonly-used characters, and so for each commonly-used character among the previously recognized results, which reads on the second score.
Xiong and Nallapati do not teach the inner-product first score aspects of the claim. Gu teaches:
calculates a first score that is a score of the first token on the basis of an inner product of the sixth feature sequence and the first token sequence, (“Copy-Mode: The score for “copying” the word xj is calculated as ψc(yt = xj) = σ(hjT Wc) st, xj ∈ X” (Gu, Sec. 3.2, Eq. (8).) “When calculating the copy-mode score, we use the hidden states {h1,...,hTS} to “represent” each of the word in the source sequence {x1,...,xTS}” (Gu, Sec. 3.2.) Gu’s decoder state st reads on the sixth feature sequence, the encoder hidden state hj that represents the copyable word xj reads on the encoded first token of the first token sequence, and the product of σ(hjT Wc) with st reads on the inner product.)
calculates a probability of the second token or the first token following the second token sequence on the basis of the first score and the second score. (“where ψg(·) and ψc(·) are score functions for generate-mode and copy-mode, respectively, and Z is the normalization term shared by the two modes” (Gu, Sec. 3.2, Eqs. (5)-(6).) “Due to the shared normalization term, the two modes are basically competing through a softmax function” (Gu, Sec. 3.2.) “Generate-Mode: The same scoring function as in the generic RNN encoder-decoder (Bahdanau et al., 2014) is used, i.e. ψg(yt = vi) = viT Wo st, vi ∈ V ∪ UNK” (Gu, Sec. 3.2, Eq. (7).) The generate-mode score ψg of each vocabulary word reads on the second score, and the probability of the next word, normalized over the copy-mode and generate-mode scores by the shared term Z, reads on the probability calculated on the basis of the first score and the second score.)
Xiong, Nallapati, and Gu are considered to be analogous to the claimed invention because they are in the same field of attention-based encoder-decoder neural networks whose decoder outputs, as a single output unit, an item that is not in the decoder’s fixed vocabulary. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiong’s classifying module, in the combination of Xiong and Nallapati set forth for claim 2, to score each of its N hot word nodes by an inner product of the projected output feature htd with the hot word feature vector of that hot word in Hz, and to normalize the hot word scores and the commonly-used character scores in one shared softmax, in the manner of Gu’s copy-mode and generate-mode scoring. One of ordinary skill in the art would have been motivated to make this modification in order to let the character outputs and the whole-hot-word outputs compete directly in a single distribution that the model learns to coordinate from the training data, as Gu expressly teaches: “Since the probabilistic model for observing any target word is a mixture of generate-mode and copy-mode, there is no need for any additional labels for modes. The network can learn to coordinate the two modes from data.” (Gu, Sec. 4.)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Xiong (U.S. 2023/0186912) in view of Wang et al. (CN 114528812 A, published May 24, 2022), hereinafter Wang.
Wang is published in Chinese. Citations to Wang are to the English machine translation provided with this Office action.
Regarding claim 4, Xiong discloses:
4. The speech recognition device according to claim 1,
Xiong discloses the speech recognition device of claim 1, as set forth in the rejection of claim 1 above.
wherein the speech recognition unit registers any one or a combination of the words, phrases, and sentences input by the user to the input interface as the first token.
“Before the speech is recognized, the configured hot word library may be acquired and multiple hot words are stored in the hot word library. It can be understood that the hot word library may be formed by hot words related to the speech recognition task including all potential hot words, such as technology terminologies, in the to-be-recognized speech.” (Xiong, para. [0064].)
“The classification nodes of the classifying module include not only fixed commonly-used character nodes but also dynamically expandable hot word nodes, so as to recognize the entire hot word by units of word, rather than recognizing and assigning incentives by units of characters in the hot words as in the conventional technology.” (Xiong, para. [0161].)
Each hot word of Xiong’s hot word library, a word or a multi-character expression, is encoded into one vector of Hz and recognized as a whole by its own hot word node, which reads on registering words and phrases as the first token, as set forth in the claim 1 mapping above.
Xiong does not teach the user input interface aspects of the claim. Wang teaches:
further comprising an input interface capable of being manipulated by a user, (“As shown in fig. 4, there are controls for setup and upload in the interactive interface 3000. Specifically, interactive interface 3000 includes one or more of the following controls: file selection control 3002, upload control 3004, similarity setting control 3008, and determination control 3010. Selecting the file control 3002 can allow the user at the service device side to provide a custom hotword file for uploading.” (Wang, FIG. 4 and description of step S2002.) “After the user at the service device side confirms that the selected file is error-free, the initial uploading of the file is completed by triggering the uploading control 3004.” (Wang, FIG. 4 and description of step S2002.) Wang’s interactive interface 3000, whose file selection and upload controls the user operates to provide custom hot words, reads on the input interface capable of being manipulated by a user.)
wherein the speech recognition unit registers any one or a combination of the words, phrases, and sentences input by the user to the input interface as the first token. (“The speech recognition system 1000 is configured to perform speech recognition on audio data input by the terminal device according to data such as a custom hotword uploaded by the service device or the terminal device.” (Wang, FIG. 1 and accompanying description.) “In some implementations, as in the application scenario shown in fig. 2A, the speech recognition system 1000 (specifically, the interaction module 1002) may collect information through a client on the service device side. Specifically, the information uploaded by the service device side includes similarity configuration and a custom hot word list.” (Wang, step S2002.) “The files and similarity configurations uploaded by the user at the service device side are stored in the speech recognition system 1000.” (Wang, step S2002.) “In some implementations, as shown in FIG. 2A, custom hotwords uploaded by a service device are stored in the speech recognition system 1000. The above-described conversion from the custom hotword to the phoneme sequence and the expansion operation of the custom hotword phoneme sequence are both performed by the speech recognition system 1000, and the obtained expanded custom hotword phoneme sequence is stored in the speech recognition system 1000.” (Wang, step S2006.) “Specifically, if the similarity meets the requirement of similarity configuration, the corresponding self-defined hotword is added to the decoded word lattice as a new path.” (Wang, step S2012.) Wang’s speech recognition system 1000 both recognizes the audio data and, through its own interaction module 1002 and processing module 1006, receives the hot words the user uploads through the interface, stores them, converts them, and adds them to its decoding, which reads on the speech recognition unit registering the words and phrases input by the user to the input interface.)
Xiong and Wang are considered to be analogous to the claimed invention because they are in the same field of speech recognition biased toward a list of hot words. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiong’s speech recognition device to include an input interface through which the user enters custom hot words, and to have its speech recognition unit receive the hot words entered and register them in the hot word library that Xiong’s hot word encoding module encodes and for which its classifying module provides hot word nodes, in the manner of Wang’s interaction module and processing module. One of ordinary skill in the art would have been motivated to make this modification in order to improve the recognition of the hot words that matter to the particular user, as Wang expressly teaches: “the probability of the hot words being recognized in the voice recognition can be increased by uploading the user-defined hot words by a user, so that the accuracy of the voice recognition is improved.” (Wang, Background.)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
a. Hori et al. (U.S. 2019/0189115): open-vocabulary end-to-end speech recognition with an encoder, an attention decoder, and a character-level and word-level recurrent neural network language model over a fixed vocabulary.
b. Zhou, S., Li, Z., Hong, Y., Zhang, M., Wang, Z., and Huai, B., “CopyNE: Better contextual ASR by copying named entities,” arXiv:2305.12839v1, posted May 22, 2023, https://arxiv.org/abs/2305.12839: contextual speech recognition that copies a whole named entity from a named-entity dictionary into the output as one unit, with a copy probability computed by dot-product attention between the decoder state and the entity representation.
c. Huber, C., Hussain, J., Stüker, S., and Waibel, A., “Instant one-shot word-learning for context-specific neural sequence-to-sequence speech recognition,” 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021 (arXiv:2107.02268, posted July 5, 2021, https://arxiv.org/abs/2107.02268): a word and phrase memory attached to a sequence-to-sequence speech recognizer, with memory-attention scores computed as inner products, whose output distribution remains over the subword vocabulary.
d. Sudo, Y., Shakeel, M., Fukumoto, Y., Peng, Y., and Watanabe, S., “Contextualized automatic speech recognition with attention-based bias phrase boosted beam search,” ICASSP 2024 (arXiv:2401.10449, posted January 2024, https://arxiv.org/abs/2401.10449): a bias encoder and bias attention with bias-phrase boosted beam search, without an expanded output vocabulary.
e. Prabhavalkar et al. (U.S. 2020/0357387): end-to-end speech recognition biased toward a set of bias phrases, such as a user’s contact names or media item names, each encoded by a bias encoder into a bias vector that a bias attention module supplies to the decoder, whose outputs remain graphemes.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YUVAL H. LEVENTAL whose telephone number is (571) 270-3130. The examiner can normally be reached Monday-Friday, 8:00 AM - 5:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, PIERRE-LOUIS DESIR, can be reached at (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YUVAL HAIM LEVENTAL/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
