Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to application 18/734,064, which was filed 06/05/24. Claims 1-12 are pending in the application and have been considered.
Foreign Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 11 and 12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to a “speech recognizer” comprising “a speech encoder” and “a decoder”. While paragraphs [0035]-[0036] on page 9 describe a processor and memory executing components of “speech recognizer training system 100”, these are not the same as the subject matter of claim 11, which is directed to a “speech recognizer” with no clearly claimed hardware components such as a processor or memory. A “speech recognizer” which includes “a speech encoder” and “a speech decoder” is described at paragraphs [0054-0058] of the specification, pages 12-13. This portion of the specification also does not describe any hardware components such as a processor or memory used to implement the claimed “speech recognizer”. The evidence above from the specification weighs against eligibility of the claimed “speech recognizer” because it does not establish that the “speech recognizer”, “speech encoder”, or “speech decoder” include any physical components, e.g. computer components, that would cause them to fall within the statutory category of “machine”, or that would rule out software-only embodiments. Further, paragraph [0036] of the specification states that software controls software components of the speech recognizer training system to perform various data processing, which suggests implementing components disclosed in the specification which are not clearly implemented in hardware as including software per se components. Given the broadest reasonable interpretation consistent with the specification, the “speech recognizer” comprising “a speech encoder” and “a decoder” are interpreted as encompassing a software per se recognizer, speech encoder, and decoder, and the claim as a whole is interpreted as encompassing software per se embodiments, which do not fall into a category of eligible subject matter.
Dependent claim 12 is also directed to the “speech recognizer of claim 11” without adding any structure that would rule out interpretations of software per se embodiments. That is, it does not remedy the deficiencies of parent claim 11 for eligibility purposes. Claim 12 is therefore also rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 4-7, and 9-11 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Rosenberg et al. (US 20230317059).
Consider claim 1, Rosenberg discloses a method of training a speech recognizer based on shared and exclusive attributes (pre-training an audio encoder of a speech recognizer to jointly learn shared representations of speech and text, using alignment outputs for unspoken textual utterances and un-transcribed non-synthetic speech utterances, i.e. exclusive attributes, [0004], [0046]), which is a method performed by a computer (computer-implemented method, [0004]), the method comprising:
inputting a parallel speech corpus constituting a labeled speech corpus and a non-parallel speech corpus into a speech encoder constituting a speech recognizer (transcribed speech 304 and un-transcribed speech 306 are input to audio encoder 210 of the ASR model 200, [0036], Fig 3A; transcribed speech 304 is a parallel corpus with pairs of textual labels and audio representations, and untranscribed speech corpus is only audio representations not paired with text, i.e. non-parallel);
outputting a representation vector representing training speech as an output of the speech encoder (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature, i.e. representation vector representing training speech, since it is used for determining the contrastive loss, [0048], Fig 3A; shared encoded of audio encoder 210 generates a second encoded shared representation 324, Fig 3B, [0046]);
inputting a parallel text corpus constituting the labeled speech corpus and a non-parallel text corpus into a text encoder (Transcribed speech 304, a parallel text and speech corpus, is input to audio encoder 210, encoded, and input to shared encoder 250, [0046], and Unspoken text 320 is input to shared encoder 250 of audio encoder 210 via alignment model and text encoder 202, Fig. 3B, [0052], [0053]; as shown in Fig 3B, audio encoder 210 includes text encoder 202 and shared encoder 250 and is therefore shared encoder 250 is considered a text encoder which receives as input transcribed speech and unspoken text encodings);
outputting a representation vector representing text as an output of the text encoder (shared encoder receives encoded textual representations and generates as output a first encoded shared representation 322, [0046], Fig. 3B); and
receiving and decoding, by a decoder, each of the representation vectors of the speech encoder and the text encoder (auxiliary decoder 390 receives, as input, first and second encoded shared representations and generates, as output, corresponding first and second probability distributions over possible speech recognition hypothesis, [0046]).
Consider claim 6, Rosenberg discloses a system for training a speech recognizer based on shared and exclusive attributes (system for pre-training an audio encoder of a speech recognizer to jointly learn shared representations of speech and text, using alignment outputs for unspoken textual utterances and un-transcribed non-synthetic speech utterances, i.e. exclusive attributes, [0004], [0009], [0046]), the system comprising:
a communication module that receives a labeled speech corpus, a non-parallel speech corpus, and a non-parallel text corpus (transcribed speech 304 and un-transcribed speech 306 are input to audio encoder 210 of the ASR model 200, [0036], Fig 3A; and Transcribed speech 304, a parallel text and speech corpus, is input to audio encoder 210, [0046], received over various communications ports, [0086]);
a memory in which a program for training the speech recognizer is stored (program for training the SR audio encoder stored on memory, [0078]; and
a processor that executes the program stored in the memory (data processing hardware executes the program from the memory, [0078]) to:
input a parallel speech corpus constituting the labeled speech corpus and the non-parallel speech corpus into a speech encoder constituting the speech recognizer transcribed speech 304 and un-transcribed speech 306 are input to audio encoder 210 of the ASR model 200, [0036], Fig 3A; transcribed speech 304 is a parallel corpus with pairs of textual labels and audio representations, and untranscribed speech corpus is only audio representations not paired with text, i.e. non-parallel) to output a representation vector representing training speech (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature, i.e. representation vector representing training speech, since it is used for determining the contrastive loss, [0048], Fig 3A; shared encoded of audio encoder 210 generates a second encoded shared representation 324, Fig 3B, [0046]);
input a parallel text corpus constituting the labeled speech corpus and the non-parallel text corpus into a text encoder (Transcribed speech 304, a parallel text and speech corpus, is input to audio encoder 210, encoded, and input to shared encoder 250, [0046], and Unspoken text 320 is input to shared encoder 250 of audio encoder 210 via alignment model and text encoder 202, Fig. 3B, [0052], [0053]; as shown in Fig 3B, audio encoder 210 includes text encoder 202 and shared encoder 250 and is therefore shared encoder 250 is considered a text encoder which receives as input transcribed speech and unspoken text encodings) to output a representation vector representing text (shared encoder receives encoded textual representations and generates as output a first encoded shared representation 322, [0046], Fig. 3B); and
receive and decode, by a decoder, each of the representation vectors of the speech encoder and the text encoder (auxiliary decoder 390 receives, as input, first and second encoded shared representations and generates, as output, corresponding first and second probability distributions over possible speech recognition hypothesis, [0046]).
Consider claim 11, Rosenberg discloses speech recognizer (speech recognition system, [0015], Fig 1) comprising:
a speech encoder that, on the basis of input speech, separately outputs a first representation vector based on shared attributes of speech and text and a second representation vector based on exclusive attributes of only the speech (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature that corresponds to a “respective one” of the transcribed non-synthetic speech utterances 304 “or” a “respective one” of the un-transcribed non-synthetic speech utterances 306, [0048], Fig 3A; the respective outputs corresponding to the transcribed utterances are considered to collectively make up a representation vector “based on shared attributes of speech and text”, and the respective outputs corresponding to the transcribed utterances, i.e. audio only training data, are collectively considered to make up “a second representation vector based on exclusive attributes of only the speech”; these are considered to be output “separately” at separate time steps); and
a decoder that receives the first representation vector and the second representation vector and outputs recognized text (auxiliary decoder 390 receives, as input, first and second encoded shared representations and generates, as output, corresponding first and second probability distributions over possible speech recognition hypothesis, [0046], and based on probability distributions, outputs possible word piece label hypotheses, i.e. recognized textual labels, [0061]).
Consider claim 2, Rosenberg discloses the outputting of a representation vector representing training speech as the output of the speech encoder includes separately outputting, by the speech encoder, a first representation vector based on shared attributes of speech and text and a second representation vector based on exclusive attributes of only the speech (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature that corresponds to a “respective one” of the transcribed non-synthetic speech utterances 304 “or” a “respective one” of the un-transcribed non-synthetic speech utterances 306, [0048], Fig 3A; the respective outputs corresponding to the transcribed utterances are considered to collectively make up a representation vector “based on shared attributes of speech and text”, and the respective outputs corresponding to the transcribed utterances, i.e. audio only training data, are collectively considered to make up “a second representation vector based on exclusive attributes of only the speech”; these are considered to be output “separately” at separate time steps).
Consider claim 4, Rosenberg discloses the receiving and decoding of, by the decoder, each of the representation vectors of the speech encoder and the text encoder includes:
performing an alignment between modalities to learn a relationship between each of the representation vectors of the speech encoder and the text encoder (alignment model generates each paired alignment output 604 using the corresponding transcription 302 that is paired with the transcribed non-synthetic speech utterance 304; paired alignment output maps the unspoken textual utterance into speech frames, [0060]);
performing decoding to generate text based on a result of the alignment between the modalities (text encoder receives each paired alignment and outputs encoded textual representation, which is received by shared encoder to generate a shared encoded representation; auxiliary decoder receives encoded shared representations and outputs a probability distribution of speech recognition hypothesizes, [0061]); and
outputting the generated text (based on probability distribution, outputting possible word piece label hypotheses, i.e. textual labels, [0061]).
Consider claim 5, Rosenberg discloses receiving and fine-tuning the parallel speech corpus and the parallel text corpus constituting the labeled speech corpus (after pre-training, training process may fine-tune audio encoder on transcribed speech utterances that may include supervised training samples of both alignment outputs corresponding to unspoken textual utterance and non-synthetic speech, [0065]; this is considered fine-tuning the parallel speech corpus and the parallel text corpus constituting the labeled speech corpus, e.g. by adding training examples).
Consider claim 7, Rosenberg discloses the processor allows the speech encoder to separately output a first representation vector based on shared attributes of speech and text and a second representation vector based on exclusive attributes of only the speech (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature that corresponds to a “respective one” of the transcribed non-synthetic speech utterances 304 “or” a “respective one” of the un-transcribed non-synthetic speech utterances 306, [0048], Fig 3A; the respective outputs corresponding to the transcribed utterances are considered to collectively make up a representation vector “based on shared attributes of speech and text”, and the respective outputs corresponding to the transcribed utterances, i.e. audio only training data, are collectively considered to make up “a second representation vector based on exclusive attributes of only the speech”; these are considered to be output “separately” at separate time steps).
Consider claim 9, Rosenberg discloses the processor is configured to:
perform an alignment between modalities to learn a relationship between each of the representation vectors of the speech encoder and the text encoder (alignment model generates each paired alignment output 604 using the corresponding transcription 302 that is paired with the transcribed non-synthetic speech utterance 304; paired alignment output maps the unspoken textual utterance into speech frames, [0060]);
perform decoding to generate text based on a result of the alignment between the modalities (text encoder receives each paired alignment and outputs encoded textual representation, which is received by shared encoder to generate a shared encoded representation; auxiliary decoder receives encoded shared representations and outputs a probability distribution of speech recognition hypothesizes, [0061]); and
output the generated text (based on probability distribution, outputting possible word piece label hypotheses, i.e. textual labels, [0061]).
Consider claim 10, Rosenberg discloses the processor receives and fine-tunes the parallel speech corpus and the parallel text corpus constituting the labeled speech corpus (after pre-training, training process may fine-tune audio encoder on transcribed speech utterances that may include supervised training samples of both alignment outputs corresponding to unspoken textual utterance and non-synthetic speech, [0065]; this is considered fine-tuning the parallel speech corpus and the parallel text corpus constituting the labeled speech corpus, e.g. by adding training examples).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 8, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Rosenberg et al. (US 20230317059) in view of Hsu et al. (“Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data”. arXiv:1709.07902v1 [cs.LG] 22 Sep 2017).
Consider claim 3, Rosenberg discloses the outputting of a representation vector representing training speech as the output of the speech encoder includes separately outputting, by the speech encoder, the first representation vector and the second representation vector (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature that corresponds to a “respective one” of the transcribed non-synthetic speech utterances 304 “or” a respective one of the un-transcribed non-synthetic speech utterances 306, [0048], Fig 3A; these are considered to be output “separately” at separate time steps).
Rosenberg does not specifically mention through an attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level.
Hsu discloses an attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level (graphical model having hierarchical priors which encourage the model to encode the sequence-level attributes and segment-level attributes into different sets of latent variables, Section 2.1, pages 4-5, Section 3, page 5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rosenberg by outputting of a representation vector representing training speech as the output of the speech encoder includes separately outputting, by the speech encoder, the first representation vector and the second representation vector, as in Rosenberg, through the attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level of Hsu in order to learn to disentangle mixed information from a surface representation, as suggested by Hsu (Section 4, page 6), predictably making speech recognition more robust against noise, as suggested by Hsu (Section 4, page 6). The references cited are analogous art in the same field of speech processing.
Consider claim 8, Rosenberg discloses the processor allows the speech encoder to separately output the first representation vector and the second representation vector (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature that corresponds to a “respective one” of the transcribed non-synthetic speech utterances 304 “or” a respective one of the un-transcribed non-synthetic speech utterances 306, [0048], Fig 3A; these are considered to be output “separately” at separate time steps).
Rosenberg does not specifically mention through an attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level.
Hsu discloses an attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level (graphical model having hierarchical priors which encourage the model to encode the sequence-level attributes and segment-level attributes into different sets of latent variables, Section 2.1, pages 4-5, Section 3, page 5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rosenberg such that the processor allows the speech encoder to separately output a representation vector representing training speech as the output of the speech encoder includes separately outputting, by the speech encoder, the first representation vector and the second representation vector, as in Rosenberg, through the attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level of Hsu for reasons similar to those for claim 3.
Consider claim 12, Rosenberg discloses the speech encoder separately outputs the first representation vector and the second representation vector (convolution subsampling block outputs, for each of a plurality of output steps, an encoded audio feature that corresponds to a “respective one” of the transcribed non-synthetic speech utterances 304 “or” a respective one of the un-transcribed non-synthetic speech utterances 306, [0048], Fig 3A; these are considered to be output “separately” at separate time steps).
Rosenberg does not specifically mention through an attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level.
Hsu discloses an attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level (graphical model having hierarchical priors which encourage the model to encode the sequence-level attributes and segment-level attributes into different sets of latent variables, Section 2.1, pages 4-5, Section 3, page 5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rosenberg such that the speech encoder separately outputs a representation vector representing training speech as the output of the speech encoder includes separately outputting, by the speech encoder, the first representation vector and the second representation vector, as in Rosenberg, through the attribute-separated latent variable inference based on a graph structure that separates attributes varying at a segment level and attributes varying at a sequence level of Hsu for reasons similar to those for claim 3.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liu et al. (“Learning Representations without Compositional Assumptions”. arXiv:2305.19726v1 [cs.LG] 31 May 2023) discloses a hierarchical graph autoencoder that learns a smaller, latent graph to aggregate information from multiple views dynamically
US 20240028829 Sainath discloses a joint speech and text streaming model for ASR
US 20230169281 Zheng discloses fused acoustic and text encoding for multimodal bilingual pretraining and speech translation
US 20220253696 Chen discloses interpretable time series representation learning with multiple-level disentanglement
US 20220019868 Klein discloses privacy-preserving representation machine learning by disentanglement
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jesse Pullias whose telephone number is 571/270-5135. The examiner can normally be reached on M-F 8:00 AM - 4:30 PM. The examiner’s fax number is 571/270-6135.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Andrew Flanders can be reached on 571/272-7516.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jesse S Pullias/
Primary Examiner, Art Unit 2655 01/14/26