Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-20 are pending. Claims 1,12, and 20 are independent and have been amended. Dependent Claims are also amended.
This Application was published as U.S. 20250037704.
Apparent priority: 17 July 2023.
Claims 9 and 11 were allowed by the Chinese patent office. Claim 9 pertains to TCNs which are taught by the primary reference Weston and used for extracting prosody of speech in Weston.
Applicant’s amendments and arguments are considered but are either unpersuasive or moot in view of the new grounds of rejection that, if presented, were necessitated by the amendments to the Claims.
This action is Final.
Response to Amendments and Arguments
Drawings
Applicant has indicated that Figures 1-2 need not be designated by a legend such as --Prior Art-- because they depict related art which is different from prior art. Arguments are persuasive and the objection to the drawings is withdrawn.
Art Rejections
Applicant’s arguments are directed to the amended language and are moot in view of the modified grounds of rejection.
Examiner had requested that the “contrastive words” which are key to the claim are defined inside the claim language. Amended claim now includes the following definition: “the contrastive words in the preset contrastive word library having specific attribute information and being words of a specific content type.” This is still quite broad and non-specific. Yet, “words of a specific content type” are being interpreted as “domain specific” words.
Gandhe was not applied to Claim 1 and is not addressed by the Applicant’s arguments. Gandhe uses domain-specific dictionaries and is applied to the amended language.
Claim 1 is amended as follows and the other independent Claims are amended similarly:
1. A voice recognition method performed by an electronic device,
the voice recognition method comprising:
performing a sliding window interception on a to-be-recognized voice signal to obtain at least a first sub-voice signal and a second sub-voice signal;
performing a first voice feature extraction on the first sub-voice signal using a pre-trained embedded feature representation system to obtain a first sub-voice embedded representation feature of the first sub-voice signal,
the pre-trained embedded feature representation system comprising a first-stage feature extraction network and a second-stage feature extraction network,
wherein the first-stage feature extraction network performs a first-stage voice feature extraction on the first sub-voice signal to obtain a first-stage voice feature,
wherein the second-stage feature extraction network performs a second-stage voice feature extraction on the first sub-voice signal based on the first-stage voice feature, and
wherein a first feature extraction precision of the first-stage voice feature extraction is less than a second feature extraction precision of the second-stage voice feature extraction;
performing a second voice feature extraction on the second sub-voice signal using the pre-trained embedded feature representation system to obtain a second sub-voice embedded representation feature of the second sub-voice signal,
wherein the first-stage feature extraction network performs the first-stage voice feature extraction on the second sub-voice signal to obtain a second first-stage voice feature,
wherein the second-stage feature extraction network performs the second-stage voice feature extraction on the second sub-voice signal based on the second first-stage voice feature;
obtaining an embedded representation feature of each contrastive word in a preset contrastive word library, the contrastive words in the preset contrasted world library having specific attribute information and being words of a specific content type;
performing a first voice recognition on the first sub-voice signal based on the first sub-voice embedded representation feature and the embedded representation feature of each contrastive word to obtain a first sub-voice recognition result, the first sub-voice recognition result indicating whether a sub-voice corresponding to the first sub-voice signal comprises a voice word having a same attribute as a contrastive word in the preset contrastive word library;
performing a second voice recognition on the second sub-voice signal based on the second sub-voice embedded representation feature and the embedded representation feature of each contrastive word to obtain a second sub-voice recognition result, the second sub-voice recognition result indicating whether a sub-voice corresponding to the second sub voice signal comprises a voice word having a same attribute as a contrastive word in the preset contrastive word library; and
determining a voice recognition result corresponding to the to-be-recognized voice signal according to the first sub-voice recognition result and the second sub-voice recognition result.
Applicant has not pointed to the location of support and Examiner identifies:
PNG
media_image1.png
552
587
media_image1.png
Greyscale
As to the definition of “contrastive words,” the Specification appears to intend “domain specific words.”
[0069] The preset contrastive word library may include a plurality of contrastive words. The contrastive words in the preset contrastive word library may have specific attribute information. In other words, the contrastive words in the preset contrastive word library may be words of a specific type, i.e. the first content type. For example, when a foul language recognition is to be performed on the to-be-recognized voice signal, the contrastive words in the preset contrastive word library may be foul language words collected and stored in advance. In other words, the preset contrastive word library may be a foul word library. When a commendation word recognition is to be performed on the to-be-recognized voice signal, the contrastive words in the preset contrastive word library are commendation words collected and stored in advance. In other words, the preset contrastive word library may be a commendation word library. When a game instruction recognition is to be performed on the to-be-recognized voice signal, the contrastive words in the preset contrastive word library may be game instruction-related words collected and stored in advance. In other words, the preset contrastive word library may be a game instruction word library.
Claim includes repetitions and is summarized as:
1. A voice recognition method … comprising:
performing a sliding window interception on a to-be-recognized voice signal to obtain at least a first sub-voice signal and a second sub-voice signal; (framing the input audio)
performing a first voice feature extraction on the first sub-voice signal using a pre-trained embedded feature representation system to obtain a first sub-voice embedded representation feature of the first sub-voice signal, (acoustic feature extraction such as MFCC extraction)
the pre-trained embedded feature representation system comprising a first-stage feature extraction network and a second-stage feature extraction network, (two-stage feature extraction)
wherein the first-stage feature extraction network performs a first-stage voice feature extraction on the first sub-voice signal to obtain a first-stage voice feature, ( MFCC extraction)
wherein the second-stage feature extraction network performs a second-stage voice feature extraction on the first sub-voice signal based on the first-stage voice feature, and (keyword extraction)
wherein a first feature extraction precision of the first-stage voice feature extraction is less than a second feature extraction precision of the second-stage voice feature extraction;
… (repeat of the same process on the second frame)
obtaining an embedded representation feature of each contrastive word in a preset contrastive word library, the contrastive words in the preset contrasted world library having specific attribute information and being words of a specific content type; (vectorizing words of a domain-specific lexicon/library)
performing a first voice recognition on the first sub-voice signal based on the first sub-voice embedded representation feature and the embedded representation feature of each contrastive word to obtain a first sub-voice recognition result, the first sub-voice recognition result indicating whether a sub-voice corresponding to the first sub-voice signal comprises a voice word having a same attribute as a contrastive word in the preset contrastive word library; (performing speech recognition on the first frame using the domain-specific library)
… (repeat of the same process on the second frame)
determining a voice recognition result corresponding to the to-be-recognized voice signal according to the first sub-voice recognition result and the second sub-voice recognition result. (generating speech recognition results)
PNG
media_image2.png
413
578
media_image2.png
Greyscale
Applicant argues:
PNG
media_image3.png
93
760
media_image3.png
Greyscale
PNG
media_image4.png
297
766
media_image4.png
Greyscale
PNG
media_image5.png
160
754
media_image5.png
Greyscale
PNG
media_image6.png
91
754
media_image6.png
Greyscale
PNG
media_image7.png
88
763
media_image7.png
Greyscale
Response 14-15.
Applicant’s arguments are persuasive to some extent. For example, Weston does teach that one of its uses is speech recognition and once speech recognition is conducted, a result is obtained. Additionally, the instant Application uses the term “contrastive” when it likely means “domain-specific” and therefore Rhode that uses the same terminology was on point.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Weston (U.S. 20230386456) in view of Gandhe (U.S. 10121467) and further in view of Lesso (U.S. 20180005636).
Certain aspects of the Claim such as the use of a sliding window to divide the audio into frames by windowing function or the smoothing of Claim 3 are well-known common steps of audio processing. (Lesso)
Otherwise, Weston is a TCN (Temporal Convolutional Network) reference which teaches the bulk of the Claim and many of the dependents. (Instant Application is includes a multi-layer convolutional neural network.)
Gandhe previously cited for Claims 5-6 includes a domain specific word concept that tracks the “contrastive word” of the instant Application and is directed to speech recognition also. Gandhe does not include the windowing which is well-known for generation of frames before ASR can be conducted and because of that Lesso had to be added.
Note that the first and second “sub-voice signals” that occur in the following Claims are mapped to consecutive frames of speech based on the definition provided in Claim 1 as shown below. Note also that some of the Claims, as shown below, express in long-hand performing the same operations on the first frame and then the second frame and repeat of the same mapping is dropped as indicated below for each Claim that has followed this format.
Regarding Claim 1, Weston teaches:
1. A voice recognition method performed by an electronic device, [Weston is directed to “speech analysis” primarily for detecting health conditions that manifest as speech impairment and not to “speech recognition” but it includes “speech recognition” as one of the uses of its method. See “[0016] … The method therefore provides representations which are well suited to use in speech analysis tasks including automatic speech recognition, text-to-speech conversion, automated spoken language understanding tasks such as detecting sentiment, emotions and sarcasm and, particularly importantly, speech analysis for detecting and monitoring health conditions.”]
the voice recognition method comprising: [Weston, the system may be used for speech recognition amongst other uses.]
performing a sliding window interception on a to-be-recognized voice signal to obtain at least a first sub-voice signal and a second sub-voice signal; [Weston does not discuss the details of dividing the input audio into frames / first sub-voice signal and second sub-voice signal but it does teach the use of TCN which is frame based: “[0143] The use of a TCN or similar model permits a large receptive field (for example 1,280 frames) and the model learns patterns in periodic signals naturally….” See also Figure 1B: “slice into audio-words 112” which breaks the input “raw audio” into slices.]
performing a first voice feature extraction on the first sub-voice signal using a pre-trained embedded feature representation system to obtain a first sub-voice embedded representation feature of the first sub-voice signal, [Weston, Figure 1A, the “vector-quantised prosody encoder 120” which is expanded in Figure 1B include the “prosody encoder 121” and in Figure 1B, “prosody encoder 121” that generates “prosody representations 122.” The TCN extracts voice features: “[0142] Since the prosody is encoded in the input audio signal, it is beneficial to use a model architecture well suited to learning patterns in raw audio in order to extract the audio features to then form initial non-quantised prosody representations which are then quantised. The method preferably implements a temporal convolutional network (TCN) to extract the audio features from the individual audio words….” See also Figure 3 showing the “N blocks” of “prosody encoder blocks 311” each of which can represent a “stage” of the “first-stage feature extraction network” and “second-stage feature extraction network” of this Claim. “[0154] More specifically the TCN comprises a stacked set of identical layers/blocks, where the input to layer 1 is the raw audio, then the input to layer 2 is the output from layer 1, and the input to layer 3 is the output of layer 2, and so on. In this way each layer can build more complex abstractions of the input but critically each layer can also learn patterns on larger timescales than the previous layer, which means the model can take a long sequence of e.g. raw audio and extract information from it.”]
the pre-trained embedded feature representation system comprising a first-stage feature extraction network and a second-stage feature extraction network, [Weston, Figure 1A, the “vector-quantised prosody encoder 120” which is expanded in Figure 1B include the “prosody encoder 121” and in Figure 1B, “prosody encoder 121” is shown in Figure 2A as the “prosody encoder 220” that includes both the first stage and the second stage of feature extraction: “[0118] The first stage of the model is the prosody encoder 220. This is a model, or series of models, configured to take one audio word as input and encode this single word as a corresponding quantised audio representation encoding the prosodic information of the audio word….” Successive figures show more and more detail.]
wherein the first-stage feature extraction network performs a first-stage voice feature extraction on the first sub-voice signal to obtain a first-stage voice feature, [Weston, “[0120] The prosody encoder 220 may have several possible different structures. As described below, in one example the prosody encoder comprises a first stage configured to encode each input audio word as a non-quantised audio representation and a second stage configured to quantise each non-quantised audio representations into one of a fixed number of quantised prosodic states (quantised prosody representations or prosody tokens)….”]
wherein the second-stage feature extraction network performs a second-stage voice feature extraction on the first sub-voice signal based on the first-stage voice feature, and [Weston, see [0120] above where the second stage works on the output of the first stage. The second stage is more precise: “[0032] … The two stage encoding process allows for a first model to be configured to effectively extract audio features from the audio data and a second model optimised for encoding these into quantised representations.”]
wherein a first feature extraction precision of the first-stage voice feature extraction is less than a second feature extraction precision of the second-stage voice feature extraction; [Weston, see [0120] above where the second stage works on the output of the first stage and therefore extract more particularized information from it. See also “[0162] As shown in FIG. 3, the features extracted using the TCN 310 are used to form a non-quantised audio representation of the input audio word 301. This is fed then into a product quantizer 320 which is configured to take a vector as input and output a quantised vector (i.e. a token). More specifically the product quantizer is trained to learn a linear mapping into a new space where its features can be split up into N parts and quantized independently, before being recombined….”]
performing a second voice feature extraction on the second sub-voice signal using the pre-trained embedded feature representation system to obtain a second sub-voice embedded representation feature of the second sub-voice signal, [This limitation repeats the above steps for a second frame of speech. In Weston “[0117] The input 210 to the model is sections of the pre-processed audio data. Preferably this comprises variable length, word-aligned audio, i.e. sections of the processed audio data which each include one spoken word. These sections of processed data are referred to as “audio words”.” The sections in Weston are not same-length frames but the feature extraction of Weston repeats for each audio-word/section the same. See Figure 2A, 210.]
wherein the first-stage feature extraction network performs the first-stage voice feature extraction on the second sub-voice signal to obtain a second first-stage voice feature, [Weston, this limitation repeats the above steps for a second frame of speech.]
wherein the second-stage feature extraction network performs the second-stage voice feature extraction on the second sub-voice signal based on the second first-stage voice feature; [Weston, this limitation repeats the above steps for a second frame of speech.]
obtaining an embedded representation feature of each contrastive word in a preset contrastive word library, the contrastive words in the preset contrasted world library having specific attribute information and being words of a specific content type; [Weston says that embedding=representation: “[0063] Preferably a representation comprises a feature vector, i.e. a vector encoding important distinguishing attributes of the input data. The term embedding is used interchangeably with the term representation. Preferably a representation captures meaningful structure of the input by placing meaningfully similar inputs close together in the representation space. A representation can be learned and reused across models or at different stages of training.” Thus the “quantized audio representation” outputs of the Weston teach embedding representation feature of the Claim. Weston, Figure 1B: shows input of the words. “[0085] The subsequent encoding stage 120 comprises encoding the audio words of the processed input audio data into vector quantised prosody representations (also referred to as prosody “tokens”)….” Weston also suggests the “contrastive word” by its “contrastive objectives” in “[0038]… Preferably the encoder model is trained using a contrastive objective wherein the model is provided with a number of possible quantised representations and is trained to predict which of the possible quantised representations is the correct quantised representation (which corresponds to the masked quantised representation). …” Yet Weston does not teach a library of words of a particular nature.]
performing a first voice recognition on the first sub-voice signal based on the first sub-voice embedded representation feature and the embedded representation feature of each contrastive word to obtain a first sub-voice recognition result, the first sub-voice recognition result indicating whether a sub-voice corresponding to the first sub-voice signal comprises a voice word having a same attribute as a contrastive word in the preset contrastive word library; [Weston, Figure 1B the output is “contextualized prosody representations 130” which can be used in a speech recognizer but does not conduct speech recognition and the interest of Weston is primarily to find the deviations from normal in the prosody that indicate an illness that manifests in speech but mentions other uses such as TTS and NLU. “[0085] … A sequence of prosody tokens is then fed into a contextualizer model 123 trained to map a prosody token 122 within a sequence of prosody tokens to a vector-quantised contextualised prosody representations 130 (contextualised prosody representations), which encodes information about its relationship with surrounding tokens within the sequence.” “[0086] Both the individual prosody tokens 122 and the contextualised prosody tokens 130 are deidentified from the speaker and can be used in downstream tasks, such as encoding speech for the monitoring and diagnosis of a health condition or expressive text-to-speech systems and spoken language understanding.” Note that the “voice recognition” of this limitation and the next is generic. There is nothing specific about it other than the generation of the “embedded representations” which is taught by Weston. Thus, the limitation is strongly suggested by Weston.]
performing a second voice recognition on the second sub-voice signal based on the second sub-voice embedded representation feature and the embedded representation feature of each contrastive word to obtain a second sub-voice recognition result, the second sub-voice recognition result indicating whether a sub-voice corresponding to the second sub voice signal comprises a voice word having a same attribute as a contrastive word in the preset contrastive word library; and [This limitation repeats the above step for a second frame of speech. The operations are conducted frame by frame and are the same for all frames.]
determining a voice recognition result corresponding to the to-be-recognized voice signal according to the first sub-voice recognition result and the second sub-voice recognition result. [Weston’s contextualization teaches the use of context, ie sequence of words. “[0085] …A sequence of prosody tokens is then fed into a contextualizer model 123 trained to map a prosody token 122 within a sequence of prosody tokens to a vector-quantised contextualised prosody representations 130 (contextualised prosody representations), which encodes information about its relationship with surrounding tokens within the sequence.” “[0121] The sequence of prosody tokens 230 is then fed into a contexualiser model 240 to encode the quantised prosody representations into contexualised prosody representations. The contextualisation model 240 is preferably a sequence-to-sequence machine learning model configured to encode contextual information of a particular prosody token 231 into a new representation. The model is configured to encode information about the relationships between a quantised prosody representation 231 and the surrounding quantised representations within the sequence 230—commonly referred to as “context”. The contextualisation model 240 is preferably an attention based model, in particular a transformer encoder.”]
Weston teaches contrastive loss but does not teach a library of contrastive terms. The contrastive words are interpreted as domain-specific words.
Ghande teaches:
obtaining an embedded representation feature of each contrastive word in a preset contrastive word library, the contrastive words in the preset contrastive world library having specific attribute information and being words of a specific content type; [Ghande, Figure 2, each of the “language models 254” has its own lexicon/dictionary of terms: “The ASR engine 258 attempts to match received feature vectors to language phonemes and words as known in the stored acoustic models 253, and language models 254. The ASR engine 258 computes recognition scores for the feature vectors based on acoustic information and language information. The acoustic information is used to calculate an acoustic score representing a likelihood that the intended sound represented by a group of feature vectors matches a language phoneme. The language information is used to adjust the acoustic score by considering what sounds and/or words are used in context with each other, thereby improving the likelihood that the ASR process will output speech results that make sense grammatically. The specific models used may be general models or may be models corresponding to a particular domain, such as music, banking, etc.” 6:23-38. Further both the “domain grammar 276a … 276n” which includes domain-specific command words and the “Domain 1 Lexicon 286a1” …”Domain n Lexicon 286an” in the “Entity Library 282” which include user-specific entity names teach the preset contrastive word library for each specific attribute/ domain or user which is a type of domain. “… The NLU process may also utilize gazetteer information (284a-284n) stored in entity library storage 282. The gazetteer information may be used for entity resolution, for example matching ASR results with different entities (such as song titles, contact names, etc.) Gazetteers may be linked to users (for example a particular gazetteer may be associated with a specific user's music collection), may be linked to certain domains (such as shopping), or may be organized in a variety of other ways.” 6:65-7:6. “A domain may represent a discrete set of activities having a common theme, such as “shopping”, “music”, “calendaring”, etc. As such, each domain may be associated with a particular language model and/or grammar database (276a-276n), a particular set of intents/actions (278a-278n), and a particular personalized lexicon (286). Each gazetteer (284a-284n) may include domain-indexed lexical information associated with a particular user and/or device. For example, the Gazetteer A (284a) includes domain-index lexical information 286aa to 286an. A user's music-domain lexical information might include album titles, artist names, and song names, for example, whereas a user's contact-list lexical information might include the names of contacts. Since every user's music collection and contact list is presumably different, this personalized information improves entity resolution.” 7:60-8:10.]
performing a first voice recognition on the first sub-voice signal based on the first sub-voice embedded representation feature and the embedded representation feature of each contrastive word to obtain a first sub-voice recognition result, the first sub-voice recognition result indicating whether a sub-voice corresponding to the first sub-voice signal comprises a voice word having a same attribute as a contrastive word in the preset contrastive word library; [Ghande, Figure 2, the “first voice recognition” may be mapped to conducting ASR at “Automatic Speech Recognition 250” using domain-specific Acoustic or Language Models 253/2541 …253/254n by matching feature vectors that are extracted from the input audio 11 and used to form embeddings. “As can be appreciated, many different characteristics of the N-gram may be represented in this manner. Many different types of feature data may be considered by the server. One example of a characteristic is a word embedding. A word embedding is a representation of how a word (or string of words) is typically used in language, as may be represented by how a word is used in the example text corpus (i.e., collection of text) or plurality of text corpuses. The word embedding is typically a high-dimensional vector (e.g., 200-500 dimensions) that represent word or word sequence usage. Thus, for N-grams, the feature values (e.g., 802-806) input into an encoder may be the words and embeddings of the words of the N-grams. Such word embedding data may enable the system to determine synonyms of words or otherwise rearrange words when processing/answering an incoming question. To determine a word embedding or word usage data, using the data from one or more data sources 180, the system may analyze individual words and their respective usages….” 23:44-62. Figure 8 and 22:60-23:25 show the generation of embeddings y (810) from feature values x1 …xn where the embeddings y of the input are compared against the embeddings of stored word to find the proper match. See Figure 9 that shows the relationships between the embeddings/vectors of words showing how writing, composing and drafting are close in distance and drinking and eating are close in distance also.]
performing a second voice recognition on the second sub-voice signal based on the second sub-voice embedded representation feature and the embedded representation feature of each contrastive word to obtain a second sub-voice recognition result, the second sub-voice recognition result indicating whether a sub-voice corresponding to the second sub voice signal comprises a voice word having a same attribute as a contrastive word in the preset contrastive word library; and [Ghande, Figure 2, the “second voice recognition” may be mapped to conducting NLU at “Natural Language Understanding 260” using domain-specific grammar 276a-n and domain specific intent 278a-n and domain specific Lexicon 286a-n. The second processing which is an NLP recognition is more accurate than the first recognition that is a simpler ASR recognition as required by the Claim.]
PNG
media_image8.png
488
694
media_image8.png
Greyscale
Weston and Ghande are directed to natural language processing including speech recognition which includes text prediction as one of its aspects and it would have been obvious to combine the use of domain-specific attributes from Ghande which with the system of Weston which includes contrastive training of its machine learning method as combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Weston and Ghande do not discuss the windowing which falls under the pre-processing of Weston.
Lesso teaches:
performing a sliding window interception on a to-be-recognized voice signal to obtain at least a first sub-voice signal and a second sub-voice signal; [Lesso, Figure 6, “[0056] The signal may then be passed to a windowing block 100, which may apply say a Hamming window or possibly some other windowing function to extract short-duration frames, say of time duration 10 ms to 50 ms, over each of which the speech may be considered stationary. The windowing block extracts a stream of short-duration frames by sliding the Hamming window along the speech signal by say half the frame length, or say sliding a 25 ms window by 10 ms, thus providing a frame of windowed data at a frame rate of 100 frames per second.” Lesso also expressly teaches “[0001] This disclosure relates to reducing the data bit rate on the interface of digital microphones, for example to minimise power consumption in Always-On Voice modes, while still passing enough information to allow downstream keyword detection or speech recognition functions.” The “sub-voice signals” are mapped to consecutive frames of speech.]
PNG
media_image9.png
240
458
media_image9.png
Greyscale
Weston/Gandhe and Lesso are directed to speech and prosody detection and feature extraction and Lesso is express in setting forth the sliding window operation that is commonly used for dividing an incoming signal and improving feature extraction as a necessary precursor to feature extraction. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 2, Lesso was cited for teaching the sliding window and teaches:
2. The voice recognition method according to claim 1, wherein the sliding window interception comprises:
framing the to-be-recognized voice signal by using a sliding window with a preset step to obtain the first sub-voice signal and the second sub-voice signal, the first sub-voice signal and the second sub-voice signal having a same frame length. [Lesso, “[0056] … The windowing block extracts a stream of short-duration frames by sliding the Hamming window along the speech signal by say half the frame length, or say sliding a 25 ms window by 10 ms, thus providing a frame of windowed data at a frame rate of 100 frames per second.” The first sub-voice and second sub-voice are consecutive frames of the same voice and would be of the same size. See also Figure 5.
PNG
media_image10.png
220
442
media_image10.png
Greyscale
]
Rationale as provided for Claim 1. This Claim further elaborates a feature that was mapped to Lesso.
Regarding Claim 3, Weston teaches:
3. The voice recognition method according to claim 1, wherein before the performing the first voice feature extraction and the second voice feature extraction, the voice recognition method further comprises:
acquiring a preset window function; and
smoothing the first sub-voice signal and the second sub-voice signal using the preset window function to correspondingly obtain a first smoothed sub-voice signal and a second smoothed sub-voice signal,
wherein the performing the first voice feature extraction and the second voice feature extraction comprises:
performing the first voice feature extraction on the first smoothed sub-voice signal to obtain the first sub-voice embedded representation feature; and [Weston, feature extraction is discussed as part of prior art: “[[0013] Generally prior art methods use heavily processed input speech data such as spectrograms or audio features….” ]
performing the second voice feature extraction on the second smoothed sub-voice signal to obtain the second sub-voice embedded representation feature. [Weston, feature extraction is discussed as part of prior art: “[[0013] Generally prior art methods use heavily processed input speech data such as spectrograms or audio features….” ]
Weston teaches that feature extraction is common but proposes a method of working on raw audio and circumvents the feature extraction.
Lesso teaches:
3. The voice recognition method according to claim 1, wherein before the performing a first voice feature extraction and the second voice feature extraction, the voice recognition method further comprises: [ Lesso teaches a step of noise removal / smoothing before feature extraction. This step is also common in the art. Lesso: “[0077] The noise reduction circuitry may act to filter the signal to suppress frequency bins where the noise, as monitored in time periods when there appears to be no voice as monitored by a Voice Activity Detector, is likely to exceed the signal at times where there is a signal. For instance a Wiener Filter set up may be used to suppress noise on a frame-by-frame basis. he Wiener filter coefficients may be updated on a frame-by-frame basis and coefficient smoothed via a Mel-frequency filter bank followed by an Inverse Discrete Cosine Transform before application to the actual signal. In some embodiments the Wiener noise reduction may comprise two stages. Each stage may incorporate some dynamic noise enhancement feature where the level of noise reduction performed is dependent on an estimated signal-to-noise ratio or other signal- or noise-related parameter or feature of the signal.”]
acquiring a preset window function; and [Lesso, Figure 6, “[0056] The signal may then be passed to a windowing block 100, which may apply say a Hamming window or possibly some other windowing function to extract short-duration frames,…”]
smoothing the first sub-voice signal and the second sub-voice signal using the preset window function to correspondingly obtain a first smoothed sub-voice signal and a second smoothed sub-voice signal, [Lesso, “[0077] … The Wiener filter coefficients may be updated on a frame-by-frame basis and coefficient smoothed via a Mel-frequency filter bank followed by an Inverse Discrete Cosine Transform before application to the actual signal. …”]
wherein the performing the first voice feature extraction and the second voice feature extraction comprises:
performing the first voice feature extraction on the first smoothed sub-voice signal to obtain the first sub-voice embedded representation feature; and [Lesso, Figure 5, both FFT 74 and Mel Filter Bank 76 are feature extraction components. “[0047] The microphone unit may comprise a packaged microphone, for example a MEMS microphone, with on-chip or co-packaged integrated speech coder circuitry or feature extraction block.” “[0064] … For example split vector quantisation to compress the MFCC vectors. In one example feature vectors of dimension 14 say may be split into pairs of sub vectors, each quantised to 5 or 6 bits say with a respective codebook at a frame period of 10 ms….”]
performing the second voice feature extraction on the second smoothed sub-voice signal to obtain the second sub-voice embedded representation feature. [Lesso, feature extraction is frame by frame and thus is performed on first and second sub-voice which are consecutive frames: “[0044] …However, as will be discussed below, the feature data to be transmitted may typically be a frame produced at nominally say 30 Hz or 10 Hz, and the design of any speech processing function, say speech recognition, may have to accommodate a wide range of pitches and spoken word rate….”]
Weston/Gandhe and Lesso are directed to speech and prosody detection and feature extraction and it would have been obvious to combine the noise removal/smoothing of Lesso which precedes feature extraction with the combination prior to feature extraction. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396. (See Prada and Sun in the Conclusion section as evidence that this step is common in the art.)
Regarding Claim 4, Weston teaches:
4. The voice recognition method according to claim 1, wherein:
the performing the first voice feature extraction comprises: [Weston, Figure 1B and Figure 2A, the prosody encoder 121/220 includes both the first and second stages of feature extraction. [0120] The prosody encoder 220 may have several possible different structures. As described below, in one example the prosody encoder comprises a first stage configured to encode each input audio word as a non-quantised audio representation and a second stage configured to quantise each non-quantised audio representations into one of a fixed number of quantised prosodic states (quantised prosody representations or prosody tokens)….”]
inputting the first sub-voice signal into the first-stage feature extraction network; [Weston, Figure 1B, “raw audio 101” as input and Figure 2A, “variable-length word aligned input 210” shown as audio waves at.]
performing a first-stage embedded feature extraction on the first sub-voice signal through the first-stage feature extraction network to obtain a first embedded representation feature with a first feature extraction precision; [Weston, “[0120] The prosody encoder 220 may have several possible different structures. As described below, in one example the prosody encoder comprises a first stage configured to encode each input audio word as a non-quantised audio representation ….” As provided above representation = embedding in Weston.]
inputting the first embedded representation feature into the second-stage feature extraction network; and [Weston, Figure 1B and Figure 2A, the prosody encoder 121/220 includes the two stages of feature extraction and the description of [120] teaches that the output of the first stage (“non-quantised audio representation” is input to the second stage to generate the “quantized prosodic states.”]
performing a second-stage embedded feature extraction on the first sub-voice signal through the second-stage feature extraction network to obtain the first embedded representation feature with a second feature extraction precision, the first feature extraction precision being less than the second feature extraction precision, the first embedded representation feature with the second feature extraction precision forming the first sub-voice embedded representation feature of the first sub-voice signal, and [Weston, Figure 1B and Figure 2A, the prosody encoder 121/220 includes both the first and second stages of feature extraction. [0120] The prosody encoder 220 may have several possible different structures…. a second stage configured to quantise each non-quantised audio representations into one of a fixed number of quantised prosodic states (quantised prosody representations or prosody tokens)….” As provided above representation = embedding in Weston.]
performing the second voice feature extraction comprises: [This part of the Claim repeats the above steps for a “second sub-voice signal” namely “a second frame.” In Weston the sections are not frame and are rather “audio words.” The processing repeats for each “audio word” nevertheless. Figure 2A, 210, “audio words” /sections shown as at-1, at, at+1.]
inputting the second sub-voice signal into the first-stage feature extraction network; [This part of the Claim is a repeat of one of the above steps.]
performing the first-stage embedded feature extraction on the second sub-voice signal through the first-stage feature extraction network to obtain a second embedded representation feature with the first feature extraction precision; [This part of the Claim is a repeat of one of the above steps.]
inputting the second embedded representation feature into the second-stage feature extraction network; and [This part of the Claim is a repeat of one of the above steps.]
performing the second-stage embedded feature extraction on the second sub-voice signal through the second-stage feature extraction network to obtain the second embedded representation feature with the second feature extraction precision, the second embedded representation feature with the second feature extraction precision forming the second sub-voice embedded representation feature of the second sub-voice signal. [This part of the Claim is a repeat of one of the above steps.]
Regarding Claim 5, Weston does not go into details of speech recognition.
Gandhe teaches:
5. The voice recognition method according to claim 1, wherein the performing the first voice recognition on the first sub-voice signal comprises: [Gandhe, Figure 2, “Automatic Speech Recognition 250.” “The device 110 and/or server 120 may include an ASR module 250. The ASR module in device 110 may be of limited or extended capabilities. The ASR module 250 may include the language models 254 stored in ASR model storage component 252, and a ASR engine 258 that performs the automatic speech recognition process. If limited speech recognition is included, the ASR module 250 may be configured to identify a limited number of words, such as keywords detected by the device, whereas extended speech recognition may be configured to recognize a much larger range of words.” 30:63-31:5.]
determining a similarity between the first sub-voice embedded representation feature and the embedded representation feature of each contrastive word; [Gandhe, Figure 1: “Determine first N-gram is used similarly to second N-gram in example text 134.” FIG. 9 illustrates representations of word usage similarity in a vector space. “The system may determine (134) that a first N-gram is used similarly to a second N-gram in the example text. This may be done, as described further below, by comparing the distance (i.e., difference) between vectors describing characteristics of each N-gram.” 3:17-21. Figure 8, “Encoder 850” which is a NN generates the “encoded feature vector y 810” as an embedding/ “embedded representation feature.” “The value y may be called an embedding of the sequence x1, . . . xN. … The encoder E may be implemented as a neural network (NN), recurrent neural network (RNN), or other model.” 23:13-20. The contrastive words of this Claim are taught by Named Entity (NER 262) of Gandhe which are Domain Specific as shown in Figure 2, 252, 273, 282 where the ASR Models 252, NLU Storage 273, and Entity Library 282 are all domain specific. (Domain specificity fits the supporting disclosure of this Application although the claim term “contrastive word” requires definition inside the Claim.) The domain specific words of Gandhe are also placed in embeddings: “One example of a characteristic is a word embedding. A word embedding is a representation of how a word (or string of words) is typically used in language, as may be represented by how a word is used in the example text corpus (i.e., collection of text) or plurality of text corpuses.” 23:45-51.]
determining, when a similarity between the first sub-voice embedded representation feature and an embedded representation feature of any contrastive word is greater than a similarity threshold, that the first sub-voice recognition result is a specific recognition result indicating that a sub-voice corresponding to the first sub-voice signal comprises a specific voice word, the specific voice word being a voice word having a same attribute as a contrastive word in the preset contrastive word library. [Gandhe finds words in a same domain/ “same attribute as the contrastive word” of the Claim by using a distance similarity and a threshold: “Words in the bottom corner of FIG. 9 exhibit similar relationships to each other as the words discussed above. The words “drink” and “eat” have different meanings, but similar usages as illustrated. … Thus, if a new word is found that is similar in usage to “eat” and/or “drink” the system may infer that the new word may have a variation of the form “x-er” or “x-ing” where x is the root of the new word. That is, if a vector corresponding to a new word is within a threshold distance to “eat” and/or “drink” the system may determine that the new word is used similarly to “eat” and/or “drink” and may create variations for the new word based on the variations of “eat” and/or “drink. …” 25:15-45. “Once N-grams are represented by a vector, the system may determine if the N-grams have a sufficiently similar usage by determining if their respective vectors are sufficiently “close” in a multi-dimensional vector space. For example, the system may determine that a first N-gram is similar to a second N-gram by determining that a first vector representation of characteristics of the first N-gram is within a threshold distance/difference of a second vector representation of characteristics of the second N-gram. (The distance may be calculated using a number of different techniques including using cosine, L2, Levenshtein distance, etc.)…” 25:66 -26:24.]
Weston and Gandhe are directed to speech and feature extraction which leads to speech recognition and it would have been obvious to combine the method of similarity detection of Gandhe which is well-known with the system of combination. Gandhe additionally includes domain specific considerations which track the “contrastive word” of the Claims. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 6, Weston pertain to or include speech recognition but Gandhe was cited for particular aspects of speech recognition.
Gandhe teaches:
6. The voice recognition method according to claim 5, wherein the determining a voice recognition result comprises:
determining that the voice recognition result corresponding to the to-be-recognized voice signal is the specific recognition result based on the first sub-voice recognition result being the specific recognition result. [Gandhe, Figure 1, the “input audio 11” / “to be recognized voice” of the Claim is being compared against text in the “text corpus 180” to find the texts that are used similarly at 134 by the “ASR 250” of Figure 2.]
Rationale as provided for Claim 5 because this Claim expands upon the limitation of Claim 6.
Regarding Claim 7, Weston pertain to or include speech recognition and but Gandhe is more specific.
Gandhe teaches:
7. The voice recognition method according to claim 1,
wherein the preset contrastive word library comprises a contrastive word voice signal of each contrastive word, and [Gandhe, Figure 1, both the keyword detection at the “keyword detection module 220” and the AFE 256 at the ASR 250 of the server 120 keep an audio /”voice signal” of the keywords and other words for comparison and keywords or domain specific words or Named entities all can teach the “contrastive words” of the Claim considering no definition is provided. “The wakeword detection module 220 works in conjunction with other components of the device, for example a microphone (not pictured) to detect keywords in audio 11. For example, the device 110 may convert audio 11 into audio data, and process the audio data with the wakeword detection module 220 to determine whether speech is detected, and if so, if the audio data comprising speech matches an audio signature and/or model corresponding to a particular keyword.” “The ASR engine 258 attempts to match received feature vectors to language phonemes and words as known in the stored acoustic models 253, and language models 254.” 6:22-37.]
wherein the obtaining comprises: performing a voice feature extraction on the contrastive word voice signal of each contrastive word through the pre-trained embedded feature representation system to obtain the embedded representation feature of each contrastive word. [Gandhe, Figure 1, AFE 256 extracts the audio features and Figure 8 shows the extracted features and the formation of the embeddings of words that will be compared including the domain specific / contrastive words of this Claim. “… The AFE may reduce noise in the audio data and divide the digitized audio data into frames representing a time intervals for which the AFE determines a number of values, called features, representing the qualities of the audio data, along with a set of those values, called a feature vector, representing the features/qualities of the audio data within the frame….” 5:54-6:10. “As can be appreciated, many different characteristics of the N-gram may be represented in this manner. Many different types of feature data may be considered by the server. One example of a characteristic is a word embedding. A word embedding is a representation of how a word (or string of words) is typically used in language, as may be represented by how a word is used in the example text corpus (i.e., collection of text) or plurality of text corpuses. The word embedding is typically a high-dimensional vector (e.g., 200-500 dimensions) that represent word or word sequence usage. Thus, for N-grams, the feature values (e.g., 802-806) input into an encoder may be the words and embeddings of the words of the N-grams. Such word embedding data may enable the system to determine synonyms of words or otherwise rearrange words when processing/answering an incoming question. To determine a word embedding or word usage data, using the data from one or more data sources 180, the system may analyze individual words and their respective usages….” 23:44-24:21.]
Weston and Gandhe are directed to speech recognition and domain specificity and it would have been obvious to combine the method of Gandhe that includes domain specific considerations which track the “contrastive word” of the Claims with the system of combination to provide the specifics of using embeddings. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 8, Weston teaches:
8. The voice recognition method according to claim 1, further comprising training the pre-trained embedded feature representation system by: [Weston includes training: “[0015] … the prosody encoder comprising a machine learning model trained using self-supervised learning to map sections of the pre-processed audio data to corresponding audio representations.”]
inputting first voice data in an unannotated voice data set into the first-stage feature extraction network; [Weston’s training is “self-supervised” which means it is not “supervised.” “Supervised training” uses labeled/annotated data and Weston’s training is not supervised. “[0031] Preferably the machine learning model is trained using a contrastive, preferably self-supervised, signal. Preferably the machine learning model is trained using raw audio, preferably with no access to linguistic information.” In this Claim the first voice data is the data used for the training of the model and the second voice data is the input that is being evaluated. “[0040] … Preferably self-supervised learning comprises learning on unlabelled data where the model creates labels using properties of the data….”]
training the first-stage feature extraction network in a contrastive learning manner to obtain a trained first-stage feature extraction network; [Weston: “[0031] Preferably the machine learning model is trained using a contrastive, preferably self-supervised, signal….”]
inputting second voice data in a single-word voice data set into the trained first-stage feature extraction network; [Weston, in this Claim the first voice data is the data used for the training of the model and the second voice data is the input that is being evaluated. Figure 2 showing the word inputs and the emphasis of the teachings of Weston that this method is preferred. “[0208] …A contrastive, self-supervised signal is used to train the model, (similar to Baevski, A., wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477, 2020), where only raw audio is used as the input and target. Unlike subtractive approaches to representing prosody the model does not rely on lexical inputs. Instead, it only has access to the downsampled raw audio signal and word-level timestamps.”] (wav2vec is the same method used by the instant Application. The use of this method is rather new in the art: “[0128] The first-stage feature extraction network may be implemented as a wav2vec model. Through the training of the wav2vec model, a trained wav2vec model may be obtained, and actual data and an interference item sample may be distinguished through the trained wav2vec model, which can assist the wav2vec model in learning data representation forms of audio data. With these data representation forms, the wav2vec model may distinguish an accurate voice or sound from interference through editing and comparison.” Instant Application.)
performing a first-stage embedded feature extraction on the second voice data through the trained first-stage feature extraction network to obtain a sample embedded representation feature with a third feature extraction precision; [Weston, in this Claim the first voice data is the data used for the training of the model and the second voice data is the input that is being evaluated. The first stage of the two-stage “Prosody encoder 220” of Figure 2A as applied to Claim 1 teaches this limitation. The first stage was mapped to a first precision. The “third … precision” is the same as the “first … precision” which is the precision associated with the operation of the “first stage.”]
inputting the sample embedded representation feature with the third feature extraction precision into the second-stage feature extraction network; [Weston, as shown with respect to Claim 1, the result of the first stage of the two-stage “prosody encoder 220” is input to a “second stage.” The second stage of the two-stage “Prosody encoder 220” of Figure 2A as applied to Claim 1 teaches this limitation.]
performing a second-stage embedded feature extraction on the second voice data through the second-stage feature extraction network to obtain a sample embedded representation feature with a fourth feature extraction precision, the third feature extraction precision being less than the fourth feature extraction precision; [Weston, as shown with respect to Claim 1, the second stage of the two-stage “prosody encoder 220” has a different and higher precision and generates the final prosody output. The fourth … precision is the same as the second … precision in Claim 1 because it is the precision associated with the output of the second stage.]
performing a voice recognition on the second voice data through a preset classification network based on the sample embedded representation feature with the fourth feature extraction precision to obtain a sample recognition result; [Weston stops at generating the prosody which can be used by an ASR and does mention that its output can be used by any of STT, ASR, or NLU but does not conduct such steps. Weston does however classifies the output: “[0052] For example the task specific machine learning model may be one or more of: [0053] a classifier trained to map the quantised audio representations to one or more categories, for example for classification of speech data as falling into a class associated with a particular health condition;” “[0130] Preferably the model is trained using a contrastive loss so that it can robustly converge to meaningful prosody representations when being trained end-to-end. Once trained, input speech data can be fed into the model and either or both of the contextual representations (post-Transformer) or the pre-Transformer non-contextualized representations (or from any layer inside the Transformer) can be used for downstream speech processing tasks.”]
inputting the sample recognition result and classification label information of the second voice data into a preset loss model; [Weston’s model is very similar to that of the instant Application (and Claims) except for the final use: {0050] … Preferably a contrastive loss is provided wherein the model is provided with a limited number of quantised audio representations, wherein one is the masked quantised audio representation within the input sequence, and the model is trained to select the correct masked quantised audio representation….”]
outputting a loss result through the preset loss model; and [Weston, once the loss is calculated it has to be output: “[0226] … We mask timesteps with a fixed probability and consider a two-part loss function: a contrastive loss and a commitment loss. The contrastive loss for selecting the true latent prosody representation amongst a set of distractors, which are uniformly sampled from other masked timesteps of the same sample. The commitment loss penalizes discrepancies between the quantizer inputs and outputs to encourage robustness. The commitment loss is averaged over the N constituent vector quantizers in our product quantizer. In lieu of a codebook loss, exponential moving average updates are used for the codebook as per (Oord et al., 2017)….”]
correcting a model parameter in the second-stage feature extraction network based on the loss result to obtain the pre-trained embedded feature representation system. [Weston, loss is discussed in the context of the “[0224] Training of Model” and “[0226] … For training on downstream tasks, a simple two layer feed-forward network (FFN) with hidden size 256, batch size 256, ReLU activations, dropout with probability 30% using the Adam optimizer (Kingma & Ba, 2014) with learning rate α=10.sup.−3 and default parameters β.sub.1=0.9, β.sub.2=0.99. A final sigmoid activation and binary cross-entropy loss is used. The input dimension varies across the different representations. We train these models for 20k steps and use the last model states to report performance on the downstream tasks.”]
Weston does not expressly teach performing speech recognition but indicates speech recognition as one of the uses of its system and Gandhe and Lesso expressly include speech recognition.
Regarding Claim 9, Weston teaches:
9. The voice recognition method according to claim 8,
wherein the first-stage feature extraction network comprises an encoder network and a context network, and [Weston, Figure 1A, the “vector-quantised prosody encoder 120” which is expanded in Figure 1B include the “prosody encoder 121” / “encoder network” and a “contextualizer 123” / “context network.”]
wherein the training the first-stage feature extraction network in a contrastive learning manner comprises: [Weston, under the “[0224]Training of Model” we have “[0226] The model is trained using a self-supervised contrastive signal, followed by assessing performance on a supervised task….”]
performing a first convolution processing on the first voice data through the encoder network to obtain a low-frequency representation feature; [Weston uses a TCN network: “[0033] Preferably the first machine learning model comprises a temporal convolutional neural network, preferably a temporal convolutional neural network with skip connections. Temporal convolutional neural networks are well suited to extracting audio features in raw audio data and have a large receptive field, configured to learn patterns in periodic signals naturally.” Figure 4, “1D causal convolution 401.” ]
performing a second convolution processing on the low-frequency representation feature through the context network to obtain an embedded representation feature with a preset dimension; [Weston: “[0034] Preferably the second machine learning model is trained to perform vector quantisation on each non-quantised audio representation. Preferably the second machine learning model is trained to perform product quantisation on each non-quantised audio representation. Product quantisation allows for efficient quantisation of large vector spaces. Product quantisers also naturally disentangle their input so facilitate explainability of the model. In particular the product quantiser encourages disentanglement of the audio features into different components of prosody.” “[0029] Preferably encoding sections of the pre-processed audio speech data into quantised audio representations comprises: encoding each section of pre-processed audio speech data into one of a fixed number of quantised audio representations, where the fixed number of quantised audio representations is preferably between 50 and 250,000, more preferably between 100 and 100,000….” Weston teaches both “fixed” and “limited” number of quantizations/dimension.]
inputting the embedded representation feature with the preset dimension into a first loss model; [Weston: “[0050] Preferably the machine learning model is trained by masking one or more of the quantised audio representations and training the model to predict the masked quantised audio representations. Preferably a contrastive loss is provided wherein the model is provided with a limited number of quantised audio representations, wherein one is the masked quantised audio representation within the input sequence, and the model is trained to select the correct masked quantised audio representation….”]
determining a first loss result corresponding to the embedded representation feature with the preset dimension through a first loss function in the first loss model; and [Weston, training involves calculating a loss function and minimizing this function and Weston teaches the “contrastive loss” of this Claim.]
correcting network parameters in the encoder network and the context network based on the first loss result to obtain the trained first-stage feature extraction network. [Weston teaches end to end training of the encoder 122 and contextualization network 123. This training yields the entire encoder including both the first stage and the second stage which teaches the “trained first-stage” of the Claim: “[0050] … The encoder and the contextualisation model are preferably trained end to end so that the encoder learns to form predictive quantised audio representations during training. In this way, during training the model converges to form quantised audio representations which encode predictive prosodic information, allowing the model to predict surrounding prosodic information. Preferably the model is trained without access to any linguistic information contained within the audio speech data.”]
Regarding Claim 10, Weston teaches:
10. The voice recognition method according to claim 8,
wherein the second-stage feature extraction network comprises a timing information extraction layer and an attention mechanism layer, and [Weston, Figure 1A: “vector quantized prosody encoder 120” expanded to include the “contextualizer 123” in Figure 1B where the “contextualizer 123” was mapped to the “second-stage feature extraction network” of this Claim and is shown in Figure 2A to include “transformer encoder 240” which is operating on a timed: t-1, t, t+1 series of inputs and includes an attention mechanism. “[0043] Preferably the probe model comprises a linear model, multi-layer perceptron, an attention-based model or a Bayesian neural network.” “[0121] … The contextualisation model 240 is preferably an attention based model, in particular a transformer encoder.”]
wherein the performing the second-stage embedded feature extraction comprises: Weston, Figure 1A: “vector quantized prosody encoder 120” expanded to include the “contextualizer 123” in Figure 1B where the “contextualizer 123” was mapped to the “second-stage feature extraction network” of this Claim.]
extracting key timing information of the sample embedded representation feature in different channels through the timing information extraction layer; [Weston, Figure 2A shows the timing information being extracted and teaches that the “channel effects” variations are considered and excluded in [0009].]
accumulating the key timing information in the different channels on a time axis through the attention mechanism layer to obtain an accumulative processing result; and [Weston, this is a function of the attention mechanism when applied to time series data such as speech: “[0146] Therefore preferably the model comprises a contextual encoder arranged to encode contextual prosodic information into output contextualised prosody representations. A suitable encoder model may be based on the Transformer architecture (see “Attention is all you need”, Vaswani 2017, arXiv:1706.03762v5).” “[0147] Prosody has relatively short-range interactions so the model may be configured to consider temporal interactions only up to 32 words apart. Prosody is strongly associated with sentences, so when feeding the input into the contextual encoder it is preferable to chop the audio-words into sentences rather than arbitrarily to preserve this structure.” This teaches “accumulating the key timing information in the different channels” where the different “channels” in Weston are the different recording environments ([0009]). “[0102] … These sections may be of variable length with a maximum duration preferably between 0.5 and 3 seconds in length, preferably around 2 seconds. This may be achieved by using word timestamps which indicate the divisions between words in the audio speech data. Alternatively word start and end timestamps may be used which indicate the bounds of the audio data around a particular word. These sections of audio data corresponding to a single word (referred to herein as “audio words”) are then input into the prosody encoder so that the encoder learns one prosodic representation for each word.” To get context for each word, the information regarding timestamps of the words in a sentence that provide context for one another are taken into account.]
performing a weighted calculation on the accumulative processing result to obtain the sample embedded representation feature with the fourth feature extraction precision. [Weston, this Claim goes through the steps performed by a TCN or a transformer both of which use a “weighted calculation.” “[0213] The max-pooled output of the TCN is passed to a product quantizer, comprising constituent vector quantizers is inspired by VQ-VAE-2 (see Razavi, A., Oord, A. v. d., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv:1906.00446, 2019) but adapted for product quantization. …” “[0215] … A decay of γ=0.99 was chosen for all quantizers and the commitment loss is weighted by α=0.5….”]
Regarding Claim 11, Weston teaches:
11. The voice recognition method according to claim 10, wherein the second-stage feature extraction network further comprises a loss calculation layer comprising a second loss function, [Weston, the training of the “second stage … network” is mapped to the training of what Weston calls the “probe model” which is the overall model including the encoder and contextualizer. “[0041] Preferably the method further comprises training a probe model to map an audio representation of the encoder model (where the encoder model comprises the prosody encoder and contextualisation model) to an independently determined measure of a prosody feature. In this way, the prosodic information encoded in the audio representation of the model may be confirmed. This can be used in application of the representations for a speech analysis task to analyse the information encoded by the representations used to make a prediction….” This comparison leads to the “second loss” of this Claim. Figure 2B shows the training and the L for loss at the top.]
the voice recognition method further comprising:
inputting the sample embedded representation feature and feature label information of the second voice data into the loss calculation layer; [Weston, “[0041] Preferably the method further comprises training a probe model to map an audio representation of the encoder model (where the encoder model comprises the prosody encoder and contextualisation model) to an independently determined measure of a prosody feature. In this way, the prosodic information encoded in the audio representation of the model may be confirmed. ….” “Audio representation” of Weston is the “sample embedded representation feature” that the encoder of Weston has output and “independently determined measure” of Weston is “feature label” of the Claim which is the true value. “[0042] Preferably training of the probe model is independent to training of the encoder model. Preferably the audio representation is one or more of: a quantised audio representation (quantised prosody representation); a contextualised audio representations (contextualised prosody representation); a representation of the product quantiser. Preferably the probe model is trained to predict an audio feature representative of one of the subcomponents of prosody: pitch, rhythm, tempo and timbre. Preferably a probe model is trained for each audio feature for each subcomponent of prosody….”]
determining a second loss result corresponding to the sample embedded representation feature with the fourth feature extraction precision through the second loss function of the loss calculation layer; and [Weston, the second loss of the Claim is taught by the difference between the true value / “independently determined measure” in [0041] of Weston and the output of the model or “audio representation of the encoder model” in [0041] of Weston based on the definitions of this Claim.]
correcting network parameters in the timing information extraction layer and the attention mechanism layer based on the second loss result to obtain a trained second-stage feature extraction network. [Weston, this is the process of training of a model. A loss function is calculated and the parameters are adjusted to minimize the loss which is the difference between the output of the model and true value that the model should have come up with. “[0044] Preferably the probe model is configured to provide a quantifiable measure of the contribution of one or more components of prosody in the audio representation. Preferably the quantifiable measure is based on one or more of (1) the accuracy of the prediction of the prosody feature provided by the audio representation; (2) the size or complexity of the probe model required to provide a given prediction accuracy, and (3) the amount of input speech data needed to train the probe to achieve a given prediction accuracy.”]
Claim 12 is an apparatus claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Hardware components taught by Figure 12 of Ghande.
Claim 13 is an apparatus claim with limitations corresponding to the limitations of Claim 2 and is rejected under similar rationale.
Claim 14 is an apparatus claim with limitations corresponding to the limitations of Claim 3 and is rejected under similar rationale.
Claim 15 is an apparatus claim with limitations corresponding to the limitations of Claim 4 and is rejected under similar rationale.
Claim 16 is an apparatus claim with limitations corresponding to the limitations of Claim 5 and is rejected under similar rationale.
Claim 17 is an apparatus claim with limitations corresponding to the limitations of Claim 6 and is rejected under similar rationale.
Claim 18 is an apparatus claim with limitations corresponding to the limitations of Claim 7 and is rejected under similar rationale.
Claim 19 is an apparatus claim with limitations corresponding to the limitations of Claim 8 and is rejected under similar rationale.
Claim 20 is a computer program product system claim with limitations corresponding to the limitations of method Claim 1 and is rejected under similar rationale. Hardware components taught by Figure 12 of Ghande.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Thornton (US 20180046721)
Thornton pertains to “content filtering” where the content may be image, video, audio, or text. “[0003] Recent years have seen an explosive growth in the number of digitally-generated text, image, video, and audio “big data” sources that are increasingly accessible via online archives….” “[0019] … The system creates a first filter using the search string and searches multiple modalities of content using the filter. For example, the modalities can be, but are not limited to, text, image, audio or video content. …” Figures 3A, 3B. “[0020] The second customizable filter is used in presenting more relevant results for second and subsequent searches of the multiple content modalities. In one embodiment, the second customizable filter takes the content retrieved by the first filter and extracts features from each content source (containing documents/images/video files, etc.) applying methods from modality-specific toolboxes. This extraction creates a wide range of feature representations for the retrieved content that contains potential indicators of relevance….” Figure 1, “Basic Keyword Filter 102” / first-stage feature extraction network and “Customizable Filter Applied 104” / second-stage feature extraction network. Figure 1, 102: “[0021] …In operation 102, the system can receive a search string from the user. The search string can be an alphanumeric string. For example, a user can search using a combination of keywords that might indicate first-hand experience with a power outage such as “power” and “out” in the same document or “no” and “power” in the same document on Twitter®. Initially the basic keyword filter is used to return a first subset of results from a first search to the user….” Figure 1, 104. “[0020] The second customizable filter is used in presenting more relevant results for second and subsequent searches of the multiple content modalities. In one embodiment, the second customizable filter takes the content retrieved by the first filter and extracts features from each content source (containing documents/images/video files, etc.) applying methods from modality-specific toolboxes. This extraction creates a wide range of feature representations for the retrieved content that contains potential indicators of relevance….”Thornton, Figure 1, 104, 106, 108. The point of Thornton is to refine the keyword spotting filter by customizing it as shown in Figure 2. Therefore, the results of the second stage 104 will be more accurate. “[0023] In other embodiments, the user can provide feedback using a sliding scale such as 1 to 10. In operation 108, the learning classifier in the customizable second filter is configured using training data based on the user's feedback to help it better select between relevant and non-relevant data in future searches. The system returns to operation 104, and applies the second filter including the trained classifier to subsequent searches.”
Molnar (U.S. 6411932):
Molnar teaches that speech recognition uses “pronunciation dictionaries” which teaches the “preset contrastive word library” and also teaches deriving rules to generate pronunciations words not in the dictionaries which also teaches a “contrastive word library” because the rules too generate the contrastive words. “Both speech synthesis and speech recognition rely on the availability of pronunciations for words or phrases. Earlier systems used pronunciation dictionaries to store word pronunciations. However, it is possible to generate word pronunciations from language-specific pronunciation rules. In fact, systems starting from the early stages have been using algorithms to generate pronunciations for words not in their pronunciation dictionary….” 1:58-64. Figure 8 shows a “contrastive pair” of school and soon. “An important factor we need to consider is that when we generate pronunciations used for speech recognition, we want to generate all acceptable (and plausible) pronunciations, so that the system will be robust.” 6:53-56. This means have a library of pronunciations. ‘Using the notion of contrasting word pairs, the slice algorithm works the following way. It looks through the words in the training set, and finds words that have the same few initial letters and phonemes in their pronunciations. It tries to find contrasting pairs that differ both in pronunciation and spelling shortly after the beginning. …” 13:23-28. Molnar emphasizes speech recognition: “Speech recognizers are becoming an important element of communication systems. These recognizers often have to recognize arbitrary phrases, especially when the information to be recognized is from an on-line, dynamic source. To make this possible, the recognizer has to be able to produce pronunciations for arbitrary words. Because of space requirements, speech systems need a compact yet robust method to make up word pronunciations.” 2:4-10. Molnar teaches the use of machine-learning and neural networks which implies the use of embeddings. “To solve this problem, more recent methods use machine-learning techniques, such as neural networks, decision trees, instance-based learning, Markov models, analogy-based techniques, or data-driven solutions to automatically extract pronunciation information for a specific language.” 2:17-23.
Prada (US 20150127594): smoothing is before feature extraction. “[0042] The deep neural network 104 provides the posterior probabilities to the posterior handling module 106. The posterior handling module 106 may smooth the posterior probabilities over a fixed time window of size w.sub.smooth to remove noise from the posterior probabilities, e.g., where posterior probabilities corresponding with multiple frames are used to determine whether a keyword was included in a window….” “[0044] The posterior handling module 106 may determine a confidence score for the j.sup.th frame x.sub.j within a sliding window of size w.sub.max using Equation (3) below. …” “[0045] In Equation (3), p'.sub.i.sub.j is the smoothed state posterior, and h.sub.max=max {1, j-w.sub.max+1} is the index of the first frame within the sliding window. …”
Sun (US 10460729):
“Referring to FIG. 2B, an embodiment of the feature extractor 140 includes a short time spectral analyzer 242, which accepts the audio signal, performs time windowing (e.g., over a window with a width of 25 milliseconds), Fourier analysis, and summation of energy over the ranges of the frequency bands….” 3:67-4:5.
Figure 1, “feature extractor 140” is performed on every frame. Sun, “Generally, the trigger detector 160 implements a timewise smoothing of the presence a sliding window in a smoother 268 and then compares the output of the smoother in a threshold detector 269 to determine when the smoothed value exceeds a pre-defined threshold. The parameters for configuring the smoother 268 and the threshold of the threshold detector 269 are stored in the detector configuration data 267.” 7:55-62.
“Referring again to FIG. 1, a second stage of processing converts the digitized audio signal to a sequence of feature values, which may be assembled in feature vectors. A feature vector is a numerical vector (e.g., an array of numbers) that corresponds to a time (e.g., a vicinity of a time instant or a time interval) in the acoustic signal and characterizes the acoustic signal at that time. In the system shown in FIG. 1, a feature extractor 140 receives the digitized audio signal and produces one feature vector for each 10 milliseconds of the audio signal. In this embodiment, elements of each feature vector represent the logarithm of the energy in corresponding audio frequency bands (“log frequency band energies” LFBE), the frequency bands (e.g., frequency bands spaced uniformly in a Mel frequency scale) together spanning the typical frequency range of speech. Other embodiments may use other representations of the audio signal, for example, using Cepstral coefficients of Linear Prediction Coding (LPC) coefficients rather than LFBEs.” 3:48-65 “Referring again to FIG. 1, the normalized feature vectors are provided to a feature analyzer 150, which generally transforms the feature vectors to a form that is directly representative of the presence of the wakeword in the original audio signal. For example, in this embodiment, the output of the feature analyzer 150 is a sequence of scalar values, referred to below as “presence values.” In the embodiment shown in FIG. 1, feature vectors are provided to the feature analyzer 150 at a rate of one feature vector every 10 milliseconds (i.e., effectively “sliding” the 25 millisecond window over the signal), and a presence value is provided as output at a rate of one presence value every 10 milliseconds….” 4:19-31.
Huang (US 20180075336):
“A convolutional neural network for classifying time series data uses a dynamic context selection. In one example a method includes receiving a plurality of inputs of different sizes at a convolutional neural network, applying convolution and pooling to each of the inputs to provide a plurality of outputs of different sizes, changing the size of each of the outputs to a selected uniform size, reshaping each of the outputs to a vector, and fully connecting the vectors.” Abstract.
Vitaladevuni (US 9443198):
“Features are disclosed for detecting an event in input data using a cascade-based detection system. Detection of the event may be triggered at any stage of the cascade, and subsequent stages of the cascade are not reached in such cases. Individual stages of the cascade may be associated with detection thresholds for use in triggering detection of the event. The sequence of stages may be selected based on some observed or desired operational characteristic, such as latency or operational cost. In addition, the cascade may be modified or updated based on data received from client devices. The data may relate to measurements and determinations made during real-world use of the cascade to detect events in input data.” Abstract.
This Application includes 199 occurrences of “contrastive word” with few examples and no definition. Providing a definition and more examples of a set of “contrastive words” would be helpful.
[0069] The preset contrastive word library may include a plurality of contrastive words. The contrastive words in the preset contrastive word library may have specific attribute information. In other words, the contrastive words in the preset contrastive word library may be words of a specific type, i.e. the first content type. For example, when a foul language recognition is to be performed on the to-be-recognized voice signal, the contrastive words in the preset contrastive word library may be foul language words collected and stored in advance. In other words, the preset contrastive word library may be a foul word library. When a commendation word recognition is to be performed on the to-be-recognized voice signal, the contrastive words in the preset contrastive word library are commendation words collected and stored in advance. In other words, the preset contrastive word library may be a commendation word library. When a game instruction recognition is to be performed on the to-be-recognized voice signal, the contrastive words in the preset contrastive word library may be game instruction-related words collected and stored in advance. In other words, the preset contrastive word library may be a game instruction word library.
[0106] The preset contrastive word library may include a plurality of contrastive words. The contrastive words in the preset contrastive word library may have attribute information. For example, the contrastive words in the preset contrastive word library are words of a particular content type. …
[0109] … In other words, the specific recognition result indicates that a sub-voice corresponding to the sub-voice signal contains a specific voice word, the specific voice word being a voice word having the same attribute as a contrastive word in the preset contrastive word library (i.e., of the same content type).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FARIBA SIRJANI whose telephone number is (571)270-1499. The examiner can normally be reached 9 to 5, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Fariba Sirjani/
Primary Examiner, Art Unit 2659
Prosecution Insights