DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This action is in response to the applicant’s response filed 29 January 2026, which is in response to the USPTO office action mailed 30 October 2025. Claims 1, 5, 10, 16 and 20 are amended. Claims 2, 6, 17 and 18 are cancelled. Claims 1, 3-5, 7-16, 19 and 20 are currently pending.
Response to Arguments
With respect to the 35 USC §103 rejections of claims 1-16 and 18-20, the applicant’s arguments are moot in view of a new grounds of rejection, as necessitated by the applicant's amendments.
Claim Objections
Claim 16 is objected to because of the following informalities: There is a typo in line 14 which recites “a second second self-attention layer”. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are: “a query encoder network”, “a query projection network”, “an audio encoder network”, “an audio projection network” and “a response component” in claim 16, in claim 18, “a training component” in claim 19 and “the audio encoder network”, “the audio projection network”, “the response component” in claim 20.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 4, 8, 10-16 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over MUKHERJEE et al., US 2023/0111824 A1 (hereinafter “Murkherjee”) in view of Huang et al., by US 2024/0282294 A1 (hereinafter “Huang”).
Claim 1: Murkherjee teaches a method for cross-modal retrieval, comprising:
obtaining a query describing a sound using a query modality other than a sound modality (Murkherjee, [Fig. 6], [0064] note at 604, the computing system obtains computer-readable text that includes words. The words of the computer-readable text are associated with an emotion (e.g., happy, sad, etc.));
generating a query embedding by encoding, using a query encoder network, the query in a query embedding space for the query modality to obtain a token embedding sequence and projecting, using a query projection network comprising a first attention layer, the query token embedding sequence into a joint embedding space, wherein the joint embedding space is an embedding space for the query modality and the sound modality (Murkherjee, [0064] note At 608, the computing system obtains a textual embedding of the computer-readable text extracted from the emotional classifier model, where the textual embedding represents semantics of the text, [0053] note memory 406 further includes a projector module 428 that is configured to project text embeddings (and, as will be described below, cluster heads) from textual embedding space to audio embedding space, thereby transforming the textual embeddings 414 into projected embeddings 430, [0035] note Each encoder in the encoder(s) 212 includes a first multihead attention 214 that is connected to a first add & norm layer 216. The first multihead attention 214 takes output of the encoder pre-net 204 or output of a previous encoder in the encoder(s) 212 as input. Each encoder in the encoder(s) 212 also includes a first feed forward network (FFN) 218 that is connected to the first add & norm layer 216. Each encoder in the encoder(s) 212 further includes a second add & norm layer 220 that is connected to the first FFN 218 and the first add & norm layer 216);
generating an audio embedding by encoding, using an audio encoder network, an audio sample in an audio embedding space for the sound modality to obtain an audio token embedding sequence and projecting, using an audio projection network comprising a first attention layer, the audio token embedding sequence into the joint embedding space (Murkherjee, [0052] note memory 406 also includes an audio embedder module 424 that is configured to generate audio embeddings 426 for the audio samples in the training data 410, where the audio embeddings are in audio embedding space, [0057] note projected audio embeddings, [0035] note Each encoder in the encoder(s) 212 includes a first multihead attention 214 that is connected to a first add & norm layer 216. The first multihead attention 214 takes output of the encoder pre-net 204 or output of a previous encoder in the encoder(s) 212 as input. Each encoder in the encoder(s) 212 also includes a first feed forward network (FFN) 218 that is connected to the first add & norm layer 216. Each encoder in the encoder(s) 212 further includes a second add & norm layer 220 that is connected to the first FFN 218 and the first add & norm layer 216);
retrieving an audio sample from a database by comparing the query embedding to the audio embedding, wherein the audio sample is retrieved based on the comparison (Murkherjee, [0064] note At 610, a cluster is identified from amongst a plurality of clusters based upon the textual embedding. At 611, the computing system generates a phoneme sequence based upon content of the computer-readable text. At 612, the computing system generates, by way of an encoder of a TTS model, a phoneme encoding based upon the phoneme sequence, [0057] note training data includes a text sample 502 and a corresponding audio sample 504, wherein the text sample 502 includes a phrase and the audio sample 504 includes a person speaking the phrase in the text sample 502); and
providing a response including the audio sample, wherein the audio sample includes the sound (Murkherjee, [0064] note At 618, the computing system causes speech that includes the words to be played over a speaker based upon output of the decoder of the TTS model, where the speech expresses the emotion).
Murkherjee does not explicitly teach comprising self-attention.
However, Huang teaches this (Huang, [0038] note Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models), [0060] note The computing system can train a machine-learned audio classification model 206 (e.g., a joint audio-text embedding model) with the audio samples 202 and the corresponding sets of textual content 204. More specifically, the computing system can process the audio samples 202 with an audio embedding portion 208 of the machine-learned audio classification model 206 to obtain audio embeddings 212).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the multihead attention of Murkherjee with the multi-headed self-attention models of Huang according to known methods (i.e. generating joint audio-text embeddings based on multi-headed self-attention model). Motivation for doing so is that the computing system can evaluate the audio recordings and the corpus of textual data with the machine-learned audio classification model to select the most accurate pairs of audio recordings and descriptive sentences for training data (Huang, [0021]).
Claim 3: Murkherjee and Huang teach the method of claim 1, wherein:
the query modality comprises a text modality (Murkherjee, [Fig. 6], [0064] note at 604, the computing system obtains computer-readable text that includes words. The words of the computer-readable text are associated with an emotion (e.g., happy, sad, etc.)).
Claim 4: Murkherjee and Huang teach the method of claim 3, wherein:
the query comprises a natural language phrase (Murkherjee, [0019] note In an example, the words of the text are “I am sorry for your loss,”).
Claim 8: Murkherjee and Huang teach the method of claim 1, wherein:
the response comprises a video sample comprising the audio sample (Huang, [0020] note the computing system can extract the audio samples from the audio data of videos hosted by an audiovisual data hosting entity, and the computing system can extract the corresponding corpus of descriptive textual data from the textual content provided by users to describe the respective audio samples (e.g., a music video and the comments provided for the music video by users)).
Claim 10: Murkherjee teaches a method for cross-modal retrieval, comprising:
identifying a training dataset including an audio sample in a sound modality and a corresponding sample in a corresponding sample modality other than the sound modality (Murkherjee, [Fig. 7], [0065] note With reference to FIG. 7, an exemplary methodology 700 for training a TTS model is depicted. The methodology 700 starts at 702, and at 704 training data is received for training a TTS model, where the training data is unlabeled and includes text and corresponding audio samples);
generating a corresponding sample embedding by encoding, using a query encoder network, the corresponding sample in a corresponding sample embedding space to obtain a token embedding sequence and projecting, using a query projection network including a first attention layer, the token embedding sequence into a joint embedding space for the corresponding sample modality and the sound modality (Murkherjee, [0066] note At 706, the text samples of the training data are provided to a text embedding model. The text embedding model processes the text samples and generates text embeddings for the text samples. In some embodiments, the outputted text embeddings (or a subset of the outputted text embeddings) are projected to an audio embedding space, [0035] note Each encoder in the encoder(s) 212 includes a first multihead attention 214 that is connected to a first add & norm layer 216. The first multihead attention 214 takes output of the encoder pre-net 204 or output of a previous encoder in the encoder(s) 212 as input. Each encoder in the encoder(s) 212 also includes a first feed forward network (FFN) 218 that is connected to the first add & norm layer 216. Each encoder in the encoder(s) 212 further includes a second add & norm layer 220 that is connected to the first FFN 218 and the first add & norm layer 216);
generating an audio embedding by encoding, using an audio encoder network, the audio sample in an audio embedding space for the sound modality to obtain an audio token embedding sequence and projecting, using an audio projection network including a second attention layer, the audio token embedding sequence into the joint embedding space (Murkherjee, [0068] note At 710, audio training data is provided to an audio embedding model (e.g., a global style token (GST) model), where the audio training data is unlabeled audio training data. The audio embedding model is configured to output audio embeddings of the audio training data, where the outputted audio embeddings are in the audio embedding space, [0035] note Each encoder in the encoder(s) 212 includes a first multihead attention 214 that is connected to a first add & norm layer 216. The first multihead attention 214 takes output of the encoder pre-net 204 or output of a previous encoder in the encoder(s) 212 as input. Each encoder in the encoder(s) 212 also includes a first feed forward network (FFN) 218 that is connected to the first add & norm layer 216. Each encoder in the encoder(s) 212 further includes a second add & norm layer 220 that is connected to the first FFN 218 and the first add & norm layer 216); and
training the query projection network based on the corresponding sample embedding and the audio embedding (Murkherjee, [0071] note At 716 the TTS model is trained based upon the embedded text input. It should be appreciated that several training iterations may be performed when training the TTS model. The methodology 700 completes at 718).
Murkherjee does not explicitly teach comprising self-attention.
However, Huang teaches this (Huang, [0038] note Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models), [0060] note The computing system can train a machine-learned audio classification model 206 (e.g., a joint audio-text embedding model) with the audio samples 202 and the corresponding sets of textual content 204. More specifically, the computing system can process the audio samples 202 with an audio embedding portion 208 of the machine-learned audio classification model 206 to obtain audio embeddings 212).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the multihead attention of Murkherjee with the multi-headed self-attention models of Huang according to known methods (i.e. generating joint audio-text embeddings based on multi-headed self-attention model). Motivation for doing so is that the computing system can evaluate the audio recordings and the corpus of textual data with the machine-learned audio classification model to select the most accurate pairs of audio recordings and descriptive sentences for training data (Huang, [0021]).
Claim 11: Murkherjee and Huang teach the method of claim 10, further comprising:
training the audio projection network based on the audio embedding and the corresponding sample embedding (Murkherjee, [0071] note At 716 the TTS model is trained based upon the embedded text input. It should be appreciated that several training iterations may be performed when training the TTS model. The methodology 700 completes at 718).
Claim 12: Murkherjee and Huang teach the method of claim 10, further comprising:
computing a contrastive loss based on the audio embedding and the corresponding sample embedding; and updating parameters of the query projection network based on the contrastive loss (Huang, [0020] note The computing system can use the audio samples and corpus of descriptive textual data to train a machine-learned audio classification model using a contrastive loss function (e.g., a joint audio-text embedding model, etc.)).
Claim 13: Murkherjee and Huang teach the method of claim 10, further comprising:
identifying metadata corresponding to the audio sample; and combining the metadata with a template to obtain the corresponding sample (Murkherjee, [0057] note training data includes a text sample 502 and a corresponding audio sample 504, wherein the text sample 502 includes a phrase and the audio sample 504 includes a person speaking the phrase in the text sample 502).
Claim 14: Murkherjee and Huang teach the method of claim 10, further comprising:
identifying a pair of additional audio samples in the sound modality and a pair of additional corresponding samples in the corresponding sample modality; combining the pair of additional audio samples to obtain the audio sample; and combining the pair of additional corresponding samples to obtain the corresponding sample (Murkherjee, [0050] note data store 408 includes training data 410 that is used in connection with training the TTS model 122. In an example, the training data 410 includes text samples and audio samples that respectively correspond to the text samples… training data 410 can include thousands to millions of text sample-audio sample pairs).
Claim 15: Murkherjee and Huang teach the method of claim 14, wherein:
the corresponding sample includes a prepositional phrase (Murkherjee, [0019] note n an example, the words of the text are “I am sorry for your loss,”, [0050] note training data 410 includes the text sample “I won the lottery”).
Claim 16: Murkherjee teaches an apparatus for cross-modal retrieval, comprising: at least one processor; a memory storing instructions executable by the at least one processor;
a query encoder network configured to generate a sequence of token embeddings based on a query in a query modality other than a sound modality, wherein the query describes a sound (Note, this limitation is interpreted as the circuitry along with the algorithm described in the specification, Murkherjee, [Fig. 6], [0064] note at 604, the computing system obtains computer-readable text that includes words. The words of the computer-readable text are associated with an emotion (e.g., happy, sad, etc.)… At 608, the computing system obtains a textual embedding of the computer-readable text extracted from the emotional classifier model, where the textual embedding represents semantics of the text);
a query projection network configured to project the sequence of token embeddings into a joint embedding space for the query modality and the sound modality, wherein the query projection network includes a first attention layer (Note, this limitation is interpreted as the circuitry along with the algorithm described in the specification, Murkherjee, [0053] note memory 406 further includes a projector module 428 that is configured to project text embeddings (and, as will be described below, cluster heads) from textual embedding space to audio embedding space, thereby transforming the textual embeddings 414 into projected embeddings 430, [0035] note Each encoder in the encoder(s) 212 includes a first multihead attention 214 that is connected to a first add & norm layer 216. The first multihead attention 214 takes output of the encoder pre-net 204 or output of a previous encoder in the encoder(s) 212 as input. Each encoder in the encoder(s) 212 also includes a first feed forward network (FFN) 218 that is connected to the first add & norm layer 216. Each encoder in the encoder(s) 212 further includes a second add & norm layer 220 that is connected to the first FFN 218 and the first add & norm layer 216);
an audio encoder network configured to generate a sequence of audio token embeddings based on an audio sample (Note, this limitation is interpreted as the circuitry along with the algorithm described in the specification, Murkherjee, [0052] note memory 406 also includes an audio embedder module 424 that is configured to generate audio embeddings 426 for the audio samples in the training data 410, where the audio embeddings are in audio embedding space);
an audio projection network configured to project the sequence of audio token embeddings into the joint embedding space to obtain an audio embedding, wherein the audio projection network includes a second second attention layer (Note, this limitation is interpreted as the circuitry along with the algorithm described in the specification, Murkherjee, [0057] note projected audio embeddings, [0035] note Each encoder in the encoder(s) 212 includes a first multihead attention 214 that is connected to a first add & norm layer 216. The first multihead attention 214 takes output of the encoder pre-net 204 or output of a previous encoder in the encoder(s) 212 as input. Each encoder in the encoder(s) 212 also includes a first feed forward network (FFN) 218 that is connected to the first add & norm layer 216. Each encoder in the encoder(s) 212 further includes a second add & norm layer 220 that is connected to the first FFN 218 and the first add & norm layer 216); and
a response component configured to compare the query embedding to the audio embedding and provide a response including the audio sample based on the comparison, wherein the audio sample includes the sound (Note, this limitation is interpreted as the circuitry along with the algorithm described in the specification, Murkherjee, [0064] note At 610, a cluster is identified from amongst a plurality of clusters based upon the textual embedding. At 611, the computing system generates a phoneme sequence based upon content of the computer-readable text. At 612, the computing system generates, by way of an encoder of a TTS model, a phoneme encoding based upon the phoneme sequence, [0057] note training data includes a text sample 502 and a corresponding audio sample 504, wherein the text sample 502 includes a phrase and the audio sample 504 includes a person speaking the phrase in the text sample 502, [0064] note At 618, the computing system causes speech that includes the words to be played over a speaker based upon output of the decoder of the TTS model, where the speech expresses the emotion).
Murkherjee does not explicitly teach self-attention.
However, Huang teaches this (Huang, [0038] note Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models), [0060] note The computing system can train a machine-learned audio classification model 206 (e.g., a joint audio-text embedding model) with the audio samples 202 and the corresponding sets of textual content 204. More specifically, the computing system can process the audio samples 202 with an audio embedding portion 208 of the machine-learned audio classification model 206 to obtain audio embeddings 212).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the multihead attention of Murkherjee with the multi-headed self-attention models of Huang according to known methods (i.e. generating joint audio-text embeddings based on multi-headed self-attention model). Motivation for doing so is that the computing system can evaluate the audio recordings and the corpus of textual data with the machine-learned audio classification model to select the most accurate pairs of audio recordings and descriptive sentences for training data (Huang, [0021]).
Claim 19: Murkherjee and Huang teach the apparatus of claim 16, further comprising:
a training component configured to identify a training dataset including a training audio sample in the sound modality and a corresponding sample in in a corresponding sample modality other than the sound modality and to train the query projection network based on a training audio embedding of the training audio sample and a corresponding sample embedding of the corresponding sample (Note, this limitation is interpreted as the circuitry along with the algorithm described in the specification, Murkherjee, [Fig. 7], [0065] note With reference to FIG. 7, an exemplary methodology 700 for training a TTS model is depicted. The methodology 700 starts at 702, and at 704 training data is received for training a TTS model, where the training data is unlabeled and includes text and corresponding audio samples).
Claims 5, 7, 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Murkherjee and Huang in further view of YE et al., US 2023/0259779 A1 (hereinafter “Ye”).
Claim 5: Murkherjee and Huang do not explicitly teach the method of claim 1, further comprising: receiving an audio query; encoding the audio query to obtain an audio query embedding in the joint embedding space using the audio encoder network and the audio query projection network; and providing an additional response to the audio query, wherein the additional response comprises the query modality.
However, Ye teaches this (Ye, [0090] note The query encoder 223 may extract query features from a query that is input to the video retrieval model 121, and may output a query embedding that represents the extracted query features. The query may be obtained from a user input, such as a text input or a voice input. The voice input may be converted into text for the query encoder 223 to extract the query features, [0091] note The final video embedding and the query embedding may be projected into a joint embedding space, where semantically similar feature points are placed closer to each other in distance, [0106] note projected features xt with the positional encoding Pt may pass through the self-attention layer 112c configured to capture a relationship between a current frame and all the other frames, and then may be fed into the feed-forward neural network 112d, [0134] note the video retrieval model may output the identified video as a result of the video retrieval process).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the TTS model of Murkherjee and Huang with the method of processing multimodal tasks of Ye according to known methods (i.e. processing a user voice input) to yield predictable results; voice queries are faster and more natural to use, offering benefits like hands-free convenience for multitasking and more direct answers to questions.
Claim 7: Murkherjee and Huang do not explicitly teach the method of claim 1, further comprising: identifying timestamp information for the audio sample; and identifying an additional audio sample based on the timestamp information.
However, Ye teaches this (Ye, [0084] note sampler model 110 may transmit the selected frames themselves to the video retrieval model 121, or may provide the video retrieval model 121 with an identification of the selected frames, such as elapsed time information of each of the selected frames in the video, so that the video retrieval model 121 selects frames based on the frame identification when the entire video data is directly fed to the video retrieval model 121, [0091] note The final video embedding and the query embedding may be projected into a joint embedding space, where semantically similar feature points are placed closer to each other in distance, [0134] note the video retrieval model may output the identified video as a result of the video retrieval process).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the TTS model training based on text and audio samples of Mukherjee and Huang with the method of processing multimodal tasks of Ye according to known methods (i.e. sampling based on an elapsed time of each of a selected from of a video). Motivation for doing so is this provides optimal performance in a downstream multimodal task compared to conventional sampling approaches (Ye, [0004]).
Claim 9: Murkherjee and Huang do not explicitly teach the method of claim 8, further comprising: identifying timestamp information for the audio sample; and identifying the video sample based on the timestamp information.
However, Ye teaches this (Ye, [0084] note sampler model 110 may transmit the selected frames themselves to the video retrieval model 121, or may provide the video retrieval model 121 with an identification of the selected frames, such as elapsed time information of each of the selected frames in the video, so that the video retrieval model 121 selects frames based on the frame identification when the entire video data is directly fed to the video retrieval model 121, [0091] note The final video embedding and the query embedding may be projected into a joint embedding space, where semantically similar feature points are placed closer to each other in distance, [0108] note generate a representative video embedding for a sequence of input frames, [0134] note the video retrieval model may output the identified video as a result of the video retrieval process).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the TTS model training based on text and audio samples space of Murkherjee and Huang with the method of processing multimodal tasks of Ye according to known methods (i.e. sampling based on an elapsed time of each of a selected from of a video). Motivation for doing so is this provides optimal performance in a downstream multimodal task compared to conventional sampling approaches (Ye, [0004]).
Claim 20: Murkherjee and Huang do not explicitly teach the apparatus of claim 16, further comprising wherein: the audio encoder network is further configured to generate a sequence of audio query token embeddings based on an audio query; the audio projection network is further configured to project the sequence of audio query token embeddings into the joint embedding space to obtain an audio query embedding; and the response component is further configured to provide an additional response to the audio query based on the audio query embedding, wherein the additional response comprises the query modality.
However, Ye teaches this (Ye, [0090] note The query encoder 223 may extract query features from a query that is input to the video retrieval model 121, and may output a query embedding that represents the extracted query features. The query may be obtained from a user input, such as a text input or a voice input. The voice input may be converted into text for the query encoder 223 to extract the query features, [0091] note The final video embedding and the query embedding may be projected into a joint embedding space, where semantically similar feature points are placed closer to each other in distance, [0106] note projected features xt with the positional encoding Pt may pass through the self-attention layer 112c configured to capture a relationship between a current frame and all the other frames, and then may be fed into the feed-forward neural network 112d, [0134] note the video retrieval model may output the identified video as a result of the video retrieval process).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the TTS model of Murkherjee and Huang with the method of processing multimodal tasks of Ye according to known methods (i.e. processing a user voice input) to yield predictable results; voice queries are faster and more natural to use, offering benefits like hands-free convenience for multitasking and more direct answers to questions.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Giuseppi Giuliani whose telephone number is (571)270-7128. The examiner can normally be reached Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kavita Stanley can be reached at (571)272-8352. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GIUSEPPI GIULIANI/Primary Examiner, Art Unit 2153
Prosecution Insights