Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Amendments
The amendment filed 07/16/2026 has been entered.
Claims 1, 7, 9, 12, 15, and 17 have been amended.
Claims 1-20 remain pending within the application.
The amendments filed 07/16/2026 are sufficient to overcome the 35 U.S.C.
102 (a)(2) rejections of claims 1-4, 7-12, and 15-20 over Thomas. The previous rejections have been withdrawn.
The amendments filed 07/16/2026 are sufficient to overcome the 35 U.S.C.
103 rejections of claims 5-6 and 13-14 over Thomas in view of Kennouche. The previous rejections have been withdrawn.
Response to Arguments
Argument 1, regarding the prior art rejections, applicant argues that Thomas does not teach “the first encoder and a third encoder for the second modality”. Applicant argues that the sequence encoder recited by Thomas is a higher level sequence encoder that operates on already encoded utterance representations output by the speech and text encoders and not on raw second-modality data. Examiner notes that the limitation recites “the first encoder and a third encoder for the second modality” and does not specify the second modality being associated with raw data. The sequence encoder recited by Thomas is associated with two different modalities, speech content and text content, with a separate speech encoder component and text encoder component (see Thomas P0039-P0040).
Applicant also argues that Thomas does not recite updating both the first and third encoder simultaneously on the same contrastive loss function during a single pre-training phase. Examiner notes that Thomas recites utilizing contrastive loss in the training process for the sequence encoder which includes both a speech encoder component and text encoder component (see Thomas P0061, P0039-P0040). Examiner also notes that Rosenburg et al (Pub. No.; US 20230103722 A1), hereafter Rosenburg, teaches pre-training an audio encoder using contrastive loss term (see Rosenburg P0046).
The full prior art rejections are outlined below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-12, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Thomas et al (US 20230298596 A1), hereafter Thomas in view of Rosenburg et al (Pub. No.; US 20230103722 A1), hereafter Rosenburg.
Regarding claims 1, 9, and 17, Thomas teaches determining, based on a model construction criterion, a first encoder for a first modality and a second encoder for a second modality (“utterance component 305 can comprise two parts, speech encoder component 410 and text encoder component 420”, P0048. Utterance component 305 is implemented in a hierarchical neural network model, P0046); constructing a first contrastive learning model, the first contrastive learning model comprising the first encoder and a third encoder for the second modality (Higher level sequence encoder may be used in conjunction with utterance component 305 to train the hierarchical model, P0039, P0061), and a model capacity of the third encoder being greater than a model capacity of the second encoder (Neural network higher level sequence encoder may be modeled as a 6-layer transformer, more complex than the speech or text encoders, P0039); performing pre-training of the first contrastive learning model based on a first training dataset for the first modality and the second modality (Training component 307 can train the hierarchical model based on the recognized spoken language understanding categories, slots and/or values for the speech-based content and/or the text-based content, P0044); and providing the pre-trained first encoder in the pre-trained first contrastive learning model for a downstream task (“speech encoder component can use the transcription network of a pretrained recurrent neural network transducer (RNN-T) based ASR model as the speech encoder, which encodes the speech signal directly to give a vector representation”, P0041. Utterance component including both speech and text encoders is provided in a hierarchical conversation model for comparing categories, slots, and values using text and proceeding to recognize the categories, slots, and values using speech, P0044), wherein the first training dataset comprises a sample pair with a first sample for the first modality and a second sample for the second modality, and wherein performing pre-training of the first contrastive learning model comprises: extracting a first feature of the first sample using the first encoder (speech encoder component extracts features from input, P0041); extracting a second feature of the second sample using the third encoder (text encoder component extracts tokens from input, P0042); determining a contrastive loss function based on a similarity between the first feature and the second feature (contrastive loss is determined based on similarity between a speech utterance and text utterance, P0044).
Thomas does not appear to explicitly teach “updating parameter values of the first encoder and the third encoder based on the contrastive loss function, to obtain the pre-trained first contrastive learning model”.
Rosenburg teaches updating parameter values of the first encoder and the third encoder based on the contrastive loss function, to obtain the pre-trained first contrastive learning model (encoder may be pre-trained based on a contrastive loss term to determine a pre-trained contrastive loss module, P0046).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Thomas and Rosenburg before them, to include Rosenburg’s specific teaching of pre-training encoders using a contrastive loss term in Thomas’s system of end-to-end integration of dialog history for spoken language understanding. One would have been motivated to make such a combination of pre-training encoders using a contrastive loss term (see Rosenburg P0046) and updating parameters of the hierarchical model being trained with contrastive loss with or without the sequence encoder g(.; ϕ) to compare performance of the different models (see Thomas P0062) for improved accuracy and latency of automatic speech recognition models (see Rosenburg P0003).
Regarding claims 2, 10, and 18, Thomas in view of Rosenburg teaches the limitations of claims 1, 9, and 17 as outlined above. Thomas further teaches wherein the downstream task comprises a single-modal downstream task for the first modality (system 201 may receive speech as input and identify an utterance within the speech as well as identify intent or a category for the speech, P0027).
Regarding claims 3, 11, and 19, Thomas in view of Rosenburg teaches the limitations of claims 1, 9, and 17 as outlined above. Thomas further teaches wherein the downstream task comprises a cross-modal downstream task for the first modality and the second modality (Cross-model functions may include comparing categories, slots, and values using text and proceeding to recognize the categories, slots, and values using speech, P0044).
Regarding claims 4, 12, and 20, Thomas in view of Rosenburg teaches the limitations of claims 3, 11, and 19 as outlined above. Thomas further teaches constructing a second contrastive learning model, the second contrastive learning model comprising the pre-trained first encoder and the second encoder (“Row 826 represents a hierarchical model as described above in relation to FIGS. 2 and 3 (e.g., a model utilizing utterance component 305 and conversation component 204) and utilizing contrastive loss in the training process. Row 827 represents a hierarchical model as described above in relation to FIGS. 2 and 3, utilizing contrastive loss in training in which the sequence encoder g(.; ϕ) utilized by conversation component 204 is replaced with a 1-layer bidirectional long short-term memory”, P0061); and performing training of the second contrastive learning model based on a second training dataset for the first modality and the second modality (Training component 307 can train the hierarchical model based on the recognized spoken language understanding categories, slots and/or values for the speech-based content and/or the text-based content, P0044. This training data may be used for training the hierarchical model being trained with contrastive loss with or without the sequence encoder g(.; ϕ)).
Regarding claims 7 and 15, Thomas in view of Rosenburg teaches the limitations of claims 1 and 9 as outlined above. Thomas further teaches wherein the model capacity of the third encoder and the model capacity of the second encoder are determined respectively based on at least one of the following: an amount of parameter values of the second encoder or the third encoder, complexity of the second encoder or the third encoder, or an amount of computation of the second encoder or the third encoder (Neural network higher level sequence encoder may be modeled as a 6-layer transformer, more complex than the speech or text encoders, P0039).
Regarding claims 8 and 16, Thomas in view of Rosenburg teaches the limitations of claims 1 and 9 as outlined above. Thomas further teaches wherein the first modality comprises any of the following modalities: an image, text, a video, audio, and the second modality comprises a further one of the modalities (First modality includes speech (audio) and second modality includes text, P0048).
Claims 5-6 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Thomas in view of Rosenburg and further in view of Kennouche et al (Pub. No.: US 20230309053 A1), hereafter Kennouche.
Regarding claims 5 and 13, Thomas in view of Rosenburg teaches the limitations of claims 4 and 12 as outlined above. Thomas does not appear to explicitly teach “wherein parameter values of the pre-trained first encoder are not updated in the training of the second contrastive learning model”.
Kennouche teaches wherein parameter values of the pre-trained first encoder are not updated in the training of the second contrastive learning model (Parameters of the first and second neural network may be updated separately, P0036).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Thomas, Rosenburg, and Kennouche before them, to include Kennouche’s specific teaching of separately updating parameters of a first and second neural network in Thomas’s system of end-to-end integration of dialog history for spoken language understanding. One would have been motivated to make such a combination of separately updating parameters of a first and second neural network (see Kennouche P0036) and updating parameters of the hierarchical model being trained with contrastive loss with or without the sequence encoder g(.; ϕ) to compare performance of the different models (see Thomas P0062) for inference accuracy and/or a training performance (see Kennouche P0023).
Regarding claims 6 and 14, Thomas in view of Rosenburg and further in view of Kennouche teaches the limitations of claims 5 and 13 as outlined above. Kennouche further teaches wherein gradient backpropagation of the parameter values of the pre-trained first encoder is blocked in the training of the second contrastive learning model (Parameters of the second model may be adjusted with gradient decent as a backpropagation mechanism while not affecting the encoder of the first network because the parameters of the first and second network may be updated separately, P0036).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISHAN MOUNDI whose telephone number is (703)756-1547. The examiner can normally be reached 8:30 A.M. - 5 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.M./ Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights