Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
Providing supporting paragraph(s) for each limitation of amended/new claim(s) in Remarks is strongly requested for clear and definite claim interpretations by Examiner (e.g., to avoid rejections under 35 U.S.C § 112(a) “Lack of written description”)
Applicant can schedule interviews (via Automated Interview Request (AIR)) at any stage of the prosecution (e.g., Non-Final, Final, and After-Final) to discuss any issues related to, for example, rejections under 35 U.S.C § 101 and § 102/103, for moving toward allowance.
Priority
Acknowledgment is made of applicant's claim for the provisional application filed on 02/05/2021.
Claim Objections
Claim(s) 1 is/are objected to because of the following informalities: it appears that “the entity” needs to read “the multimodal entity” or something else. Appropriate correction is required. In addition, claim(s) 46, 47 is/are objected to for the same reason.
Claim(s) 1 is/are objected to because of the following informalities: it appears that “multimodal entity” needs to read “multi-modal entity” or something else. Appropriate correction is required. In addition, claim(s) 46, 47 is/are objected to for the same reason.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 37-39, 43 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim(s) 37 recite(s) the limitation “the unit” (line 3). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to, since it may indicate one of “a plurality of units” (claim 28), or something else. It appears it may need to read “a unit”, or something else. For the purposes of examination, “a unit” is used. In addition, claim(s) 37 is/are rejected for the same reason (seven times more). In addition, claim(s) 38 (four times), 39 (two times) is/are rejected for the same reason.
Claim(s) 43 recite(s) the limitation “the highest selection scores” (line 4). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to. It appears it may need to read “highest selection scores”, or something else. For the purposes of examination, “highest selection scores” is used.
Claim(s) 37-39, 43 each recite(s) limitations that raise issues of indefiniteness as set forth above, and their dependent claims are rejected at least based on their direct and/or indirect dependency from the claims listed above. Appropriate explanation and/or amendment is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 28-32, 34-38, 41-42, 46-47 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks) in view of Tsai et al. (Multimodal Transformer for Unaligned Multimodal Language Sequences)
Regarding claim 28
(Note: Hereinafter, if a limitation has bold brackets (i.e. [·]) around claim languages, the bracketed claim languages indicate that they have not been taught yet by the current prior art reference but they will be taught by another prior art reference afterwards.)
Lee teaches
A method performed by one or more computers for using a neural network to generate a network output that characterizes a [multimodal] entity to perform a [multi-modal] processing task, the method comprising:
(Lee [sec(s) 1] “In this paper, we propose a novel set-input deep neural network architecture called the Set Transformer, (cf. Trans former, (Vaswani et al., 2017)). The novelty of the Set Transformer is in three important design choices: … We apply the Set Transformer to several set-input problems and empirically demonstrate the importance and effectiveness of these design choices, and show that we can achieve the state-of-the-art performances for the most of the tasks.” [sec(s) Abs] “We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. … We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating increased performance compared to recent methods for set-structured data.”;)
generating a representation of the [multi-modal] entity as a set of data element embeddings, wherein:
(Lee [sec(s) 3] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions. Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows:”;)
the [multi-modal] entity comprises data of at least two data modalities from a group consisting of: text data, image data, video data, audio data, point cloud data, and protein data; and
(Lee [sec(s) 5] “We split all characters (and corresponding images) into train, validation, and test sets and only train using images from the train character classes. We generate input sets by sampling between 6 and 10 images and we train the model to predict the number of different characters inside the set. We used a Poisson regression model to predict this number, with the rate λ given as the output of a neural network. We maximized the log likelihood of this model using stochastic gradient ascent. We evaluated model performance using sets of images sampled from the test set of characters. Table 2 reports accuracy, measured as the frequency at which the mode of the Poisson distribution chosen by the network is equal to the number of characters inside the input set.” and “Synthetic 2D mixtures of Gaussians: Each dataset contains n ∈ [100,500] points on a 2D plane, each sampled from one of four Gaussians. CIFAR-100: Each dataset contains n ∈ [100,500] images sampled from four random classes in the CIFAR-100 dataset. Each image is represented by a 512-dim vector obtained from a pretrained VGG network (Simonyan & Zisserman, 2014).”;)
generating the set of data element embeddings comprises, for each of a plurality of units included in the [multi-modal] entity, generating a data element embedding for the unit as a combination of: (i) a feature embedding of the unit, [(ii) a positional embedding characterizing a spatial or temporal position of the unit, and (iii) a modality embedding identifying a data modality of the unit];
(Lee [sec(s) 3.1] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions. Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows:” and “A potential problem with using SABs for set-structured data is the quadratic time complexity O(n2), which may be too expensive for large sets (n >> 1). We thus introduce the Induced Set Attention Block (ISAB), which bypasses this problem. Along with the set X ∈ Rn×d, additionally define m d-dimensional vectors I ∈ Rm×d, which we call inducing points. Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network.”;)
obtaining a set of latent embeddings; and
(Lee [sec(s) 3] “A potential problem with using SABs for set-structured data is the quadratic time complexity O(n2), which may be too expensive for large sets (n >> 1). We thus introduce the Induced Set Attention Block (ISAB), which bypasses this problem. Along with the set X ∈ Rn×d, additionally define m d-dimensional vectors I ∈ Rm×d, which we call inducing points. Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network.”;)
processing: (i) the set of data element embeddings, and (ii) the set of latent embeddings, using the neural network to generate the network output characterizing the entity,
(Lee [sec(s) 3] “A potential problem with using SABs for set-structured data is the quadratic time complexity O(n2), which may be too expensive for large sets (n >> 1). We thus introduce the Induced Set Attention Block (ISAB), which bypasses this problem. Along with the set X ∈ Rn×d, additionally define m d-dimensional vectors I ∈ Rm×d, which we call inducing points. Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network. An ISAB with m inducing points I is defined as: ISABm(X) = MAB(X,H) ∈ Rn×d, (9) where H =MAB(I,X) ∈Rm×d. (10) The ISAB first transforms I into H by attending to the input set. The set of transformed inducing points H, which contains information about the input set X, is again attended to by the input set X to finally produce a set of n elements. This is analogous to low-rank projection or autoencoder models, where inputs (X) are first projected onto a low dimensional object (H) and then reconstructed to produce outputs. The difference is that the goal of these methods is reconstruction whereas ISAB aims to obtain good features for the final task. We expect the learned inducing points to encode some global structure which helps explain the inputs X. For example, in the amortized clustering problem on a 2D plane, the inducing points could be appropriately distributed points on the 2D plane so that the encoder can compare elements in the query dataset indirectly through their proximity to these grid points.” [sec(s) 3.3] “After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs.”;)
wherein the neural network comprises a sequence of neural network blocks comprising: (i) one or more cross-attention blocks, (ii) one or more self-attention blocks, and (iii) an output block,
(Lee [sec(s) 4] “One can view the ISAB is the inversion of this idea, where queries I are stored and the input features are used as key-value pairs.” [sec(s) 3.1] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions.” [sec(s) 3] “Similar to other architectures, a Set Transformer consists of an encoder followed by a decoder (cf. Section 2.1), but a distinguishing feature is that each layer in the encoder and decoder attends to their inputs to produce activations.” [sec(s) 3.3] “The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (14) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. … Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15)”;)
wherein each cross-attention block performs operations comprising:
updating each latent embedding in the set of latent embeddings using attention over some or all of the data element embeddings in the set of data element embeddings;
(Lee [sec(s) 4] “One can view the ISAB is the inversion of this idea, where queries I are stored and the input features are used as key-value pairs.” [sec(s) 3.1] “A potential problem with using SABs for set-structured data is the quadratic time complexity O(n2), which may be too expensive for large sets (n >> 1). We thus introduce the Induced Set Attention Block (ISAB), which bypasses this problem. Along with the set X ∈ Rn×d, additionally define m d-dimensional vectors I ∈ Rm×d, which we call inducing points. Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network. An ISAB with m inducing points I is defined as: ISABm(X) = MAB(X,H) ∈ Rn×d, (9) where H =MAB(I,X) ∈Rm×d. (10) The ISAB first transforms I into H by attending to the input set.” [sec(s) 3.1] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions. Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows: MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) rFF is any row-wise feedforward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization (Ba et al., 2016). The MAB is an adaptation of the encoder block of the Trans former (Vaswani et al., 2017) without positional encoding and dropout. Using the MAB, we define the Set Attention Block (SAB) as SAB(X) := MAB(X,X). (8)”;)
wherein each self-attention block performs operations comprising:
updating each latent embedding in the set of latent embeddings using attention over the set of latent embeddings; and
(Lee [sec(s) 3.1] “Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows: MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) rFF is any row-wise feedforward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization (Ba et al., 2016). The MAB is an adaptation of the encoder block of the Trans former (Vaswani et al., 2017) without positional encoding and dropout. Using the MAB, we define the Set Attention Block (SAB) as SAB(X) := MAB(X,X). (8) In other words, an SAB takes a set and performs self attention between the elements in the set, resulting in a set of equal size. Since the output of SAB contains information about pairwise interactions among the elements in the input set X, we can stack multiple SABs to encode higher order interactions.” [sec(s) 3.2] “We use one seed vector (k = 1) in most cases, but for problems such as amortized clustering which requires k correlated outputs, the natural thing to do is to use k seed vectors. To further model the interactions among the k outputs, we apply an SAB afterwards: H =SAB(PMAk(Z)). (12)”;)
wherein the output block performs operations comprising:
after the set of latent embeddings are updated using the one or more cross- attention blocks and the one or more self-attention blocks, processing one or more latent embeddings from the set of latent embeddings to generate the network output characterizing the entity.
(Lee [sec(s) 3.2] “We instead propose to aggregate features by applying multihead attention on a learnable set of k seed vectors S ∈ Rk×d. Let Z ∈ Rn×d be the set of features constructed from an encoder. Pooling by Multihead Attention (PMA) with k seed vectors is defined as PMAk(Z) = MAB(S,rFF(Z)). (11) Note that the output of PMAk is a set of k items. We use one seed vector (k = 1) in most cases, but for problems such as amortized clustering which requires k correlated outputs, the natural thing to do is to use k seed vectors. To further model the interactions among the k outputs, we apply an SAB afterwards: H =SAB(PMAk(Z)). (12)” [sec(s) 3.3] “The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (14) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. … Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)”;)
However, the combination of Lee does not appear to explicitly teach:
A method performed by one or more computers for using a neural network to generate a network output that characterizes a [multimodal] entity to perform a [multi-modal] processing task, the method comprising:
generating a representation of the [multi-modal] entity as a set of data element embeddings, wherein:
the [multi-modal] entity comprises data of at least two data modalities from a group consisting of: text data, image data, video data, audio data, point cloud data, and protein data; and
generating the set of data element embeddings comprises, for each of a plurality of units included in the [multi-modal] entity, generating a data element embedding for the unit as a combination of: (i) a feature embedding of the unit, [(ii) a positional embedding characterizing a spatial or temporal position of the unit, and (iii) a modality embedding identifying a data modality of the unit];
(Note: Hereinafter, if a limitation has one or more bold underlines, the one or more underlined claim languages indicate that they are taught by the current prior art reference, while the one or more non-underlined claim languages indicate that they have been taught already by one or more previous art references.)
Tsai teaches
A method performed by one or more computers for using a neural network to generate a network output that characterizes a multimodal entity to perform a multi-modal processing task, the method comprising:
(Tsai [fig(s) 1] “Example video clip from movie reviews. [Top]: Illustration of word-level alignment where video and audio features are averaged across the time interval of each spoken word. [Bottom] Illustration of cross modal attention weights between text (“spectacle”) and vision/audio.” [sec(s) 1] “For example, the receptors for audio and vision streams may vary with variable receiving frequency, and hence we may not obtain optimal mapping between them. A frowning face may relate to a pessimistically word spoken in the past. That is to say, multimodal language sequences often exhibit “unaligned” nature and require inferring long term dependencies across modalities, which raises a question on performing efficient multimodal fusion. To address the above issues, in this paper we propose the Multimodal Transformer (MulT), an end-to-end model that extends the standard Trans former network (Vaswani et al., 2017) to learn representations directly from unaligned multimodal streams. At the heart of our model is the crossmodal attention module, which attends to the crossmodal interactions at the scale of the entire utterances. This module latently adapts streams from one modality to another (e.g., vision → language) by repeated reinforcing one modality’s features with those from the other modalities, regardless of the need for alignment.”;)
generating a representation of the multi-modal entity as a set of data element embeddings, wherein:
(Tsai [fig(s) 1] “Example video clip from movie reviews. [Top]: Illustration of word-level alignment where video and audio features are averaged across the time interval of each spoken word. [Bottom] Illustration of cross modal attention weights between text (“spectacle”) and vision/audio.” [sec(s) 1] “For example, the receptors for audio and vision streams may vary with variable receiving frequency, and hence we may not obtain optimal mapping between them. A frowning face may relate to a pessimistically word spoken in the past. That is to say, multimodal language sequences often exhibit “unaligned” nature and require inferring long term dependencies across modalities, which raises a question on performing efficient multimodal fusion. To address the above issues, in this paper we propose the Multimodal Transformer (MulT), an end-to-end model that extends the standard Trans former network (Vaswani et al., 2017) to learn representations directly from unaligned multimodal streams. At the heart of our model is the crossmodal attention module, which attends to the crossmodal interactions at the scale of the entire utterances. This module latently adapts streams from one modality to another (e.g., vision → language) by repeated reinforcing one modality’s features with those from the other modalities, regardless of the need for alignment.” [sec(s) 3.2] “we pass the input sequences through a 1D temporal convolutional layer: ˆX{L,V,A} = Conv1D(X{L,V,A},k{L,V,A}) ∈ RT{L,V,A}×d (2) where k{L,V,A} are the sizes of the convolutional kernels for modalities {L,V,A}, and d is a common dimension. … Positional Embedding. To enable the sequences to carry temporal information, following (Vaswani et al., 2017), we augment positional embedding (PE) to ˆX{L,V,A}: Z[0] {L,V,A} = ˆX{L,V,A} + PE(T{L,V,A},d) (3) where PE(T{L,V,A},d) ∈ RT{L,V,A}×d computes the (fixed) embeddings for each position index, and Z[0]{L,V,A} are the resulting low-level position aware features for different modalities.”;)
the multi-modal entity comprises data of at least two data modalities from a group consisting of: text data, image data, video data, audio data, point cloud data, and protein data; and
(Tsai [fig(s) 1] “Example video clip from movie reviews. [Top]: Illustration of word-level alignment where video and audio features are averaged across the time interval of each spoken word. [Bottom] Illustration of cross modal attention weights between text (“spectacle”) and vision/audio.” [sec(s) 1] “For example, the receptors for audio and vision streams may vary with variable receiving frequency, and hence we may not obtain optimal mapping between them. A frowning face may relate to a pessimistically word spoken in the past. That is to say, multimodal language sequences often exhibit “unaligned” nature and require inferring long term dependencies across modalities, which raises a question on performing efficient multimodal fusion. To address the above issues, in this paper we propose the Multimodal Transformer (MulT), an end-to-end model that extends the standard Trans former network (Vaswani et al., 2017) to learn representations directly from unaligned multimodal streams. At the heart of our model is the crossmodal attention module, which attends to the crossmodal interactions at the scale of the entire utterances. This module latently adapts streams from one modality to another (e.g., vision → language) by repeated reinforcing one modality’s features with those from the other modalities, regardless of the need for alignment.”;)
generating the set of data element embeddings comprises, for each of a plurality of units included in the multi-modal entity, generating a data element embedding for the unit as a combination of: (i) a feature embedding of the unit, (ii) a positional embedding characterizing a spatial or temporal position of the unit, and (iii) a modality embedding identifying a data modality of the unit;
(Tsai [fig(s) 1] “Example video clip from movie reviews. [Top]: Illustration of word-level alignment where video and audio features are averaged across the time interval of each spoken word. [Bottom] Illustration of cross modal attention weights between text (“spectacle”) and vision/audio.” [sec(s) 3.2] “Three major modalities are typically involved in multimodal language sequences: language (L), video (V), and audio (A) modalities. We de note with X{L,V,A} ∈ RT{L,V,A}× d{L,V,A} the input feature sequences (and the dimensions thereof) from these 3 modalities. … Temporal Convolutions. To ensure that each element of the input sequences has sufficient awareness of its neighborhood elements, we pass the input sequences through a 1D temporal convolutional layer: ˆX{L,V,A} = Conv1D(X{L,V,A},k{L,V,A}) ∈ RT{L,V,A}×d (2) where k{L,V,A} are the sizes of the convolutional kernels for modalities {L,V,A}, and d is a common dimension. … Positional Embedding. To enable the sequences to carry temporal information, following (Vaswani et al., 2017), we augment positional embedding (PE) to ˆX{L,V,A}: Z[0] {L,V,A} = ˆX{L,V,A} + PE(T{L,V,A},d) (3) where PE(T{L,V,A},d) ∈ RT{L,V,A}×d computes the (fixed) embeddings for each position index, and Z[0]{L,V,A} are the resulting low-level position aware features for different modalities.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lee with the multi-modal entity of Tsai.
One of ordinary skill in the art would have been motived to combine in order to exhibit the best performance when compared to prior methods in prediction.
(Tsai [sec(s) 4] “This observation advocates one of the aforementioned advantage of MulT over conventional alignment (see Section 3.3): crossmodal attention enables MulT to directly capture potentially long-range signals, including those off diagonals on the attention matrix” [sec(s) 5] “Whereas prior approaches focused primarily on the aligned multi modal streams, MulT serves as a strong baseline capable of capturing long-range contingencies, regardless of the alignment assumption. Empirically, we show that MulT exhibits the best performance when compared to prior methods.”)
Regarding claim 29
The combination of Lee, Tsai teaches claim 28.
Lee further teaches
wherein a number of latent embeddings in the set of latent embeddings is less than a number of data element embeddings in the set of data element embeddings, and
(Lee [sec(s) 3.1] “A potential problem with using SABs for set-structured data is the quadratic time complexity O(n2), which may be too expensive for large sets (n >> 1). We thus introduce the Induced Set Attention Block (ISAB), which bypasses this problem. Along with the set X ∈ Rn×d, additionally define m d-dimensional vectors I ∈ Rm×d, which we call inducing points. Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network. … Note that in (9) and (10), attention was computed between a set of size m and a set of size n. Therefore, the time complexity of ISABm(X;λ) is O(nm) where m is a (typically small) hyperparameter — an improvement over the quadratic complexity of the SAB. We also emphasize that both of our set operations (SAB and ISAB) are permutation equivariant (definition in Section 2.1):” [sec(s) 3.3] “We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power.”;)
wherein a number of latent embeddings in the set of latent embeddings is predefined and independent of a number of data element embeddings in the set of data element embeddings.
(Lee [sec(s) 3.1] “A potential problem with using SABs for set-structured data is the quadratic time complexity O(n2), which may be too expensive for large sets (n >> 1). We thus introduce the Induced Set Attention Block (ISAB), which bypasses this problem. Along with the set X ∈ Rn×d, additionally define m d-dimensional vectors I ∈ Rm×d, which we call inducing points. Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network. … Note that in (9) and (10), attention was computed between a set of size m and a set of size n. Therefore, the time complexity of ISABm(X;λ) is O(nm) where m is a (typically small) hyperparameter — an improvement over the quadratic complexity of the SAB. We also emphasize that both of our set operations (SAB and ISAB) are permutation equivariant (definition in Section 2.1):” [sec(s) 3.3] “We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power.”;)
Regarding claim 30
The combination of Lee, Tsai teaches claim 28.
Lee further teaches
wherein the neural network comprises a plurality of cross-attention blocks and a plurality of self-attention blocks, and wherein the plurality of cross-attention blocks and the plurality of self-attention blocks are interleaved.
(Lee [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. Note that PMA with k > 1 seed vectors should be followed by SABs to model the correlation between k outputs. Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 3.1] “Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network. An ISAB with m inducing points I is defined as: ISABm(X) = MAB(X,H) ∈ Rn×d, where H =MAB(I,X) ∈Rm×d. (9) (10) The ISAB first transforms I into H by attending to the input set.”;)
Regarding claim 31
The combination of Lee, Tsai teaches claim 28.
wherein processing, by the output block, one or more latent embeddings from the set of latent embeddings to generate the network output characterizing the entity comprises: (See claim 28)
Lee further teaches
pooling the latent embeddings in the set of latent embeddings to generate a pooled latent embedding; and
(Lee [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. Note that PMA with k > 1 seed vectors should be followed by SABs to model the correlation between k outputs. Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 3.2] “A common aggregation scheme in permutation invariant networks is a dimension-wise average or maximum of the feature vectors (cf. Section 1). We instead propose to aggregate features by applying multihead attention on a learnable set of k seed vectors S ∈ Rk×d. Let Z ∈ Rn×d be the set of features constructed from an encoder. Pooling by Multihead Attention (PMA) with k seed vectors is defined as PMAk(Z) = MAB(S,rFF(Z)). (11)”;)
processing the pooled latent embedding using one or more neural network layers to generate the network output characterizing the entity.
(Lee [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. Note that PMA with k > 1 seed vectors should be followed by SABs to model the correlation between k outputs. Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 3.2] “A common aggregation scheme in permutation invariant networks is a dimension-wise average or maximum of the feature vectors (cf. Section 1). We instead propose to aggregate features by applying multihead attention on a learnable set of k seed vectors S ∈ Rk×d. Let Z ∈ Rn×d be the set of features constructed from an encoder. Pooling by Multihead Attention (PMA) with k seed vectors is defined as PMAk(Z) = MAB(S,rFF(Z)). (11)”;)
Regarding claim 32
The combination of Lee, Tsai teaches claim 31.
Lee further teaches
wherein pooling the latent embeddings in the set of latent embeddings comprises averaging the latent embeddings.
(Lee [sec(s) 3.2] “A common aggregation scheme in permutation invariant networks is a dimension-wise average or maximum of the feature vectors (cf. Section 1). We instead propose to aggregate features by applying multihead attention on a learnable set of k seed vectors S ∈ Rk×d. Let Z ∈ Rn×d be the set of features constructed from an encoder. Pooling by Multihead Attention (PMA) with k seed vectors is defined as PMAk(Z) = MAB(S,rFF(Z)). (11)”;)
Regarding claim 34
The combination of Lee, Tsai teaches claim 28.
wherein for each self-attention block, updating each latent embedding in the set of latent embeddings using attention over the set of latent embeddings comprises: (See claim 28)
Lee further teaches
updating each latent embedding in the set of latent embeddings using query-key-value attention over the set of latent embeddings.
(Lee [sec(s) 2.2] “Assume we have n query vectors (corresponding to a set with n elements) each with dimension dq: Q ∈ Rn×dq. An attention function Att(Q,K,V) is a function that maps queries Q to outputs using nv key-value pairs K ∈ Rnv×dq,V ∈ Rnv×dv. Att(Q,K,V;ω) = ω(QKT)V. (3) The pairwise dot product QKT ∈ Rn×nv measures how similar each pair of query and key vectors is, with weights computed with an activation function ω. The output ω(QKT)V is a weighted sum of V where a value gets more weight if its corresponding key has larger dot product with the query. Multi-head attention, originally introduced in Vaswani et al. (2017), is an extension of the previous attention scheme. Instead of computing a single attention function, this method first projects Q,K,V onto h different dMq, dMq, dMv-dimensional vectors, respectively. An attention function (Att(·;ωj)) is applied to each of these h projections. The output is a linear transformation of the con catenation of all attention outputs: Multihead(Q,K,V;λ,ω) = concat(O1,··· ,Oh)WO, (4) where Oj = Att(QWQj ,KWKj ,VWVj ;ωj) (5)”;)
Regarding claim 35
The combination of Lee, Tsai teaches claim 28.
wherein each self-attention block performs operations comprising: (See claim 28)
Lee further teaches
repeatedly updating each latent embedding in the set of latent embeddings using attention over the set of latent embeddings.
(Lee [sec(s) 3.1] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions.” [sec(s) 3] “Similar to other architectures, a Set Transformer consists of an encoder followed by a decoder (cf. Section 2.1), but a distinguishing feature is that each layer in the encoder and decoder attends to their inputs to produce activations.” [sec(s) 3.3] “The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (14) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. … Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15)”;)
Regarding claim 36
The combination of Lee, Tsai teaches claim 28.
wherein for each cross-attention block, updating each latent embedding in the set of latent embeddings using attention over some or all of the data element embeddings in the set of data element embeddings comprises: (See claim 28)
Lee further teaches
updating each latent embedding in the set of latent embeddings using query-key-value attention over some or all of the data element embeddings in the set of data element embeddings, comprising:
(Lee [sec(s) 2.2] “Assume we have n query vectors (corresponding to a set with n elements) each with dimension dq: Q ∈ Rn×dq. An attention function Att(Q,K,V) is a function that maps queries Q to outputs using nv key-value pairs K ∈ Rnv×dq,V ∈ Rnv×dv. Att(Q,K,V;ω) = ω(QKT)V. (3) The pairwise dot product QKT ∈ Rn×nv measures how similar each pair of query and key vectors is, with weights computed with an activation function ω. The output ω(QKT)V is a weighted sum of V where a value gets more weight if its corresponding key has larger dot product with the query. Multi-head attention, originally introduced in Vaswani et al. (2017), is an extension of the previous attention scheme. Instead of computing a single attention function, this method first projects Q,K,V onto h different dMq, dMq, dMv-dimensional vectors, respectively. An attention function (Att(·;ωj)) is applied to each of these h projections. The output is a linear transformation of the con catenation of all attention outputs: Multihead(Q,K,V;λ,ω) = concat(O1,··· ,Oh)WO, (4) where Oj = Att(QWQj ,KWKj ,VWVj ;ωj) (5)” [sec(s) 3.1] “MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) rFF is any row-wise feedforward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization (Ba et al., 2016).”;)
generating a respective query embedding for each latent embedding in the set of latent embeddings;
(Lee [sec(s) 2.2] “Assume we have n query vectors (corresponding to a set with n elements) each with dimension dq: Q ∈ Rn×dq. An attention function Att(Q,K,V) is a function that maps queries Q to outputs using nv key-value pairs K ∈ Rnv×dq,V ∈ Rnv×dv. Att(Q,K,V;ω) = ω(QKT)V. (3) The pairwise dot product QKT ∈ Rn×nv measures how similar each pair of query and key vectors is, with weights computed with an activation function ω. The output ω(QKT)V is a weighted sum of V where a value gets more weight if its corresponding key has larger dot product with the query. Multi-head attention, originally introduced in Vaswani et al. (2017), is an extension of the previous attention scheme. Instead of computing a single attention function, this method first projects Q,K,V onto h different dMq, dMq, dMv-dimensional vectors, respectively. An attention function (Att(·;ωj)) is applied to each of these h projections. The output is a linear transformation of the con catenation of all attention outputs: Multihead(Q,K,V;λ,ω) = concat(O1,··· ,Oh)WO, (4) where Oj = Att(QWQj ,KWKj ,VWVj ;ωj) (5)” [sec(s) 3.1] “MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) rFF is any row-wise feedforward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization (Ba et al., 2016).”;)
generating a respective key embedding and a respective value embedding for each of a plurality of data element embeddings in the set of data element embeddings; and
(Lee [sec(s) 2.2] “Assume we have n query vectors (corresponding to a set with n elements) each with dimension dq: Q ∈ Rn×dq. An attention function Att(Q,K,V) is a function that maps queries Q to outputs using nv key-value pairs K ∈ Rnv×dq,V ∈ Rnv×dv. Att(Q,K,V;ω) = ω(QKT)V. (3) The pairwise dot product QKT ∈ Rn×nv measures how similar each pair of query and key vectors is, with weights computed with an activation function ω. The output ω(QKT)V is a weighted sum of V where a value gets more weight if its corresponding key has larger dot product with the query. Multi-head attention, originally introduced in Vaswani et al. (2017), is an extension of the previous attention scheme. Instead of computing a single attention function, this method first projects Q,K,V onto h different dMq, dMq, dMv-dimensional vectors, respectively. An attention function (Att(·;ωj)) is applied to each of these h projections. The output is a linear transformation of the con catenation of all attention outputs: Multihead(Q,K,V;λ,ω) = concat(O1,··· ,Oh)WO, (4) where Oj = Att(QWQj ,KWKj ,VWVj ;ωj) (5)” [sec(s) 3.1] “MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) rFF is any row-wise feedforward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization (Ba et al., 2016).”;)
updating each latent embedding in the set of latent embeddings using query-key- value attention over the plurality of data element embeddings in the set of data element embeddings based on: (i) the query embeddings for the latent embeddings, and (ii) the key and value embeddings for the data element embeddings.
(Lee [sec(s) 2.2] “Assume we have n query vectors (corresponding to a set with n elements) each with dimension dq: Q ∈ Rn×dq. An attention function Att(Q,K,V) is a function that maps queries Q to outputs using nv key-value pairs K ∈ Rnv×dq,V ∈ Rnv×dv. Att(Q,K,V;ω) = ω(QKT)V. (3) The pairwise dot product QKT ∈ Rn×nv measures how similar each pair of query and key vectors is, with weights computed with an activation function ω. The output ω(QKT)V is a weighted sum of V where a value gets more weight if its corresponding key has larger dot product with the query. Multi-head attention, originally introduced in Vaswani et al. (2017), is an extension of the previous attention scheme. Instead of computing a single attention function, this method first projects Q,K,V onto h different dMq, dMq, dMv-dimensional vectors, respectively. An attention function (Att(·;ωj)) is applied to each of these h projections. The output is a linear transformation of the con catenation of all attention outputs: Multihead(Q,K,V;λ,ω) = concat(O1,··· ,Oh)WO, (4) where Oj = Att(QWQj ,KWKj ,VWVj ;ωj) (5)” [sec(s) 3.1] “MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) rFF is any row-wise feedforward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization (Ba et al., 2016).”;)
Regarding claim 37
The combination of Lee, Tsai teaches claim 28.
Lee further teaches
wherein the entity comprises a plurality of units arranged in a spatial structure, wherein each unit is associated with positional data that defines a respective position of the unit in the spatial structure, and wherein obtaining the representation of the entity as the set of data element embeddings comprises:
(Lee [sec(s) 5] “We split all characters (and corresponding images) into train, validation, and test sets and only train using images from the train character classes. We generate input sets by sampling between 6 and 10 images and we train the model to predict the number of different characters inside the set.” and “Synthetic 2D mixtures of Gaussians: Each dataset contains n ∈ [100,500] points on a 2D plane, each sampled from one of four Gaussians. CIFAR-100: Each dataset contains n ∈ [100,500] images sampled from four random classes in the CIFAR-100 dataset. Each image is represented by a 512-dim vector obtained from a pretrained VGG network (Simonyan & Zisserman, 2014).” [sec(s) 5.2] “We use the Omniglot (Lake et al., 2015) dataset, which consists of 1,623 different handwritten characters from various alphabets, where each character is represented by 20 different images.” [sec(s) Abs] “Many machine learning tasks such as multiple instance learning, 3D shape recognition and few shot image classification are defined on sets of in stances.” [sec(s) 3] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions. Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows:”;)
Tsai further teaches
generating, for each unit in the entity, a feature embedding of the unit based on features of the unit;
(Tsai [fig(s) 1] [sec(s) 3.2] “Three major modalities are typically involved in multimodal language sequences: language (L), video (V), and audio (A) modalities. We de note with X{L,V,A} ∈ RT{L,V,A}× d{L,V,A} the input feature sequences (and the dimensions thereof) from these 3 modalities. … Temporal Convolutions. To ensure that each element of the input sequences has sufficient awareness of its neighborhood elements, we pass the input sequences through a 1D temporal convolutional layer: ˆX{L,V,A} = Conv1D(X{L,V,A},k{L,V,A}) ∈ RT{L,V,A}×d (2) where k{L,V,A} are the sizes of the convolutional kernels for modalities {L,V,A}, and d is a common dimension. The convolved sequences are expected to contain the local structure of the sequence, which is important since the sequences are collected at different sampling rates.”;)
generating, for each unit in the entity, a positional embedding of the unit based on the position of the unit in the spatial structure; and
(Tsai [fig(s) 1] [sec(s) 3.2] “Three major modalities are typically involved in multimodal language sequences: language (L), video (V), and audio (A) modalities. We de note with X{L,V,A} ∈ RT{L,V,A}× d{L,V,A} the input feature sequences (and the dimensions thereof) from these 3 modalities. … Positional Embedding. To enable the sequences to carry temporal information, following (Vaswani et al., 2017), we augment positional embedding (PE) to ˆX{L,V,A}: Z[0] {L,V,A} = ˆX{L,V,A} + PE(T{L,V,A},d) (3) where PE(T{L,V,A},d) ∈ RT{L,V,A}×d computes the (fixed) embeddings for each position index, and Z[0]{L,V,A} are the resulting low-level position aware features for different modalities.”;)
generating, for each unit in the entity, a data element embedding of the unit based on: (i) the feature embedding of the unit, and (ii) the positional embedding of the unit.
(Tsai [fig(s) 1] [sec(s) 3.2] “Three major modalities are typically involved in multimodal language sequences: language (L), video (V), and audio (A) modalities. We de note with X{L,V,A} ∈ RT{L,V,A}× d{L,V,A} the input feature sequences (and the dimensions thereof) from these 3 modalities. … Positional Embedding. To enable the sequences to carry temporal information, following (Vaswani et al., 2017), we augment positional embedding (PE) to ˆX{L,V,A}: Z[0] {L,V,A} = ˆX{L,V,A} + PE(T{L,V,A},d) (3) where PE(T{L,V,A},d) ∈ RT{L,V,A}×d computes the (fixed) embeddings for each position index, and Z[0]{L,V,A} are the resulting low-level position aware features for different modalities.”;)
The combination of Lee, Tsai is combinable with Tsai for the same rationale as set forth above with respect to claim 1.
Regarding claim 38
The combination of Lee, Tsai teaches claim 37.
wherein for each unit in the entity, generating the data element embedding of the unit based on: (i) the feature embedding of the unit, and (ii) the positional embedding of the unit, comprises: (See claim 37)
Lee further teaches
concatenating the feature embedding of the unit and the positional embedding of the unit.
(Tsai [sec(s) 3.2] “Positional Embedding. To enable the sequences to carry temporal information, following (Vaswani et al., 2017), we augment positional embedding (PE) to ˆX{L,V,A}: Z[0] {L,V,A} = ˆX{L,V,A} + PE(T{L,V,A},d) (3) where PE(T{L,V,A},d) ∈ RT{L,V,A}×d computes the (fixed) embeddings for each position index, and Z[0]{L,V,A} are the resulting low-level position aware features for different modalities. … As a final step, we concatenate the outputs from the crossmodal transformers that share the same target modality to yield Z{L,V,A} … Each of them is then passed through a sequence model to collect temporal information to make predictions. We choose the self-attention transformer (Vaswani et al., 2017). Eventually, the last elements of the sequences models are extracted to pass through fully-connected layers to make predictions.” [sec(s) 3] “In this section, we describe our proposed Multi modal Transformer (MulT) (Figure 2) for modeling unaligned multimodal language sequences. At the high level, MulT merges multimodal time series via a feed-forward fusion process from multiple directional pairwise crossmodal transformers. Specifically, each crossmodal transformer (introduced in Section 3.2) serves to repeatedly reinforce a target modality with the low-level features from another source modality by learning the attention across the two modalities’ features. A MulT architecture hence models all pairs of modalities with such crossmodal transformers, followed by sequence models (e.g., self-attention transformer) that predicts using the fused features.”;)
Regarding claim 41
The combination of Lee, Tsai teaches claim 28.
Lee further teaches
wherein the sequence of neural network blocks of the neural network further comprises one or more selection blocks;
(Lee [fig(s) 1] [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. Note that PMA with k > 1 seed vectors should be followed by SABs to model the correlation between k outputs. Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)”;)
wherein each selection block performs operations comprising:
after the set of latent embeddings are updated using one or more cross-attention blocks, one or more self-attention blocks, or both, processing the set of latent embeddings and the set of data element embeddings to generate a respective selection score for each data element embedding in the set of data element embeddings; and
(Lee [fig(s) 1] [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. Note that PMA with k > 1 seed vectors should be followed by SABs to model the correlation between k outputs. Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 2.2] “An attention function (Att(·;ωj)) is applied to each of these h projections. The output is a linear transformation of the con catenation of all attention outputs: Multihead(Q,K,V;λ,ω) = concat(O1,··· ,Oh)WO, (4) where Oj = Att(QWQj ,KWKj ,VWVj ;ωj) (5)”;)
selecting a proper subset of the set of data element embeddings for use by one or more specified cross-attention blocks based on the selection scores;
(Lee [fig(s) 1] [sec(s) 3.1] “we train along with other parameters of the network” [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. Note that PMA with k > 1 seed vectors should be followed by SABs to model the correlation between k outputs. Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 2.2] “An attention function (Att(·;ωj)) is applied to each of these h projections. The output is a linear transformation of the con catenation of all attention outputs: Multihead(Q,K,V;λ,ω) = concat(O1,··· ,Oh)WO, (4) where Oj = Att(QWQj ,KWKj ,VWVj ;ωj) (5)”;)
wherein each specified cross-attention block updates each latent embedding in the set of latent embeddings using attention over only data element embeddings in the selected proper subset of the set of data element embeddings.
(Lee [sec(s) 3.1] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions. Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows: MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) rFF is any row-wise feedforward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization (Ba et al., 2016). The MAB is an adaptation of the encoder block of the Trans former (Vaswani et al., 2017) without positional encoding and dropout. Using the MAB, we define the Set Attention Block (SAB) as SAB(X) := MAB(X,X). (8)”;)
Regarding claim 42
The combination of Lee, Tsai teaches claim 41.
Lee further teaches
wherein each selection block comprises:
(i) a parameter selection neural network, and (ii) a unit selection neural network, and
(Lee [fig(s) 1] [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. This can result in much lower processing times when using ISAB (as compared to SAB), while still maintaining high representational power. After the encoder transforms data X ∈Rn×dx into features Z ∈ Rn×d, the decoder aggregates them into a single or a set of vectors which is fed into a feed-forward network to get final outputs. Note that PMA with k > 1 seed vectors should be followed by SABs to model the correlation between k outputs. Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 3.1] “Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows: MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) [sec(s) 5.2] “We generate input sets by sampling between 6 and 10 images and we train the model to predict the number of different characters inside the set. … We trained ISABn+PMA on this task while varying the number of inducing points (n).”;)
wherein for each selection block, processing the set of latent embeddings and the set of data element embeddings to generate the respective selection score for each data element embedding in the set of data element embeddings comprises: (See claim 41)
processing the latent embeddings using the parameter selection neural network to generate a network output that defines values of a set of neural network parameters of the unit selection neural network; and
(Lee [fig(s) 1] [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. … Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 3.1] “Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows: MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) … Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network. [sec(s) 5.2] “We generate input sets by sampling between 6 and 10 images and we train the model to predict the number of different characters inside the set. … We trained ISABn+PMA on this task while varying the number of inducing points (n).” [sec(s) 5.3] “Specifically, given a dataset X, we train a neural network to output parameters
PNG
media_image1.png
71
867
media_image1.png
Greyscale
which maximize
PNG
media_image2.png
174
1038
media_image2.png
Greyscale
. (18) We structured f(·;λ) as a set-input neural network and learned its parameters λ using stochastic gradient ascent, where we approximate gradients using minibatches of datasets.”;)
processing each data element embedding in the set of data element embeddings using the unit selection neural network and in accordance with the values of the set of neural network parameters of the unit selection neural network to generate the selection score for the data element embedding.
(Lee [fig(s) 1] [sec(s) 3.3] “Using the ingredients explained above, we describe how we would construct a set transformer consists of an encoder and a decoder. The encoder Encoder : X → Z ∈ Rn×d is a stack of SABs or ISABs, for example: Encoder(X) = SAB(SAB(X)) (13) Encoder(X) = ISABm(ISABm(X)). (13) We point out again that the time complexity for stacks of SABs and ISABs are O(ln2) and O(lnm), respectively. … Decoder(Z;λ) = rFF(SAB(PMAk(Z))) ∈ Rk×d (15) where PMAk(Z) = MAB(S,rFF(Z)) ∈ Rk×d, (16)” [sec(s) 3.1] “Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows: MAB(X,Y) =LayerNorm(H +rFF(H)), (6) where H =LayerNorm(X +Multihead(X,Y,Y;ω)), (7) … Inducing points I are part of the ISAB itself, and they are trainable parameters which we train along with other parameters of the network”;)
Regarding claim 46
The claim is rejected for the reasons set forth in the rejection of Claim 1.
Regarding claim 47
The claim is rejected for the reasons set forth in the rejection of Claim 1.
Claim(s) 33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks) in view of Tsai et al. (Multimodal Transformer for Unaligned Multimodal Language Sequences) in view of Vaswani et al. (Attention Is All You Need)
Regarding claim 33
The combination of Lee, Tsai teaches claim 28.
Tsai further teaches
wherein the network output characterizing the entity comprises a sequence of output elements, and
(Tsai [sec(s) Abs] “How ever, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities.” [sec(s) 3] “In this section, we describe our proposed Multi modal Transformer (MulT) (Figure 2) for modeling unaligned multimodal language sequences. At the high level, MulT merges multimodal time series via a feed-forward fusion process from multiple directional pairwise crossmodal transformers. Specifically, each crossmodal transformer (introduced in Section 3.2) serves to repeatedly reinforce a target modality with the low-level features from another source modality by learning the attention across the two modalities’ features. A MulT architecture hence models all pairs of modalities with such crossmodal transformers, followed by sequence models (e.g., self-attention transformer) that predicts using the fused features.”;)
wherein processing, by the output block, one or more latent embeddings from the set of latent embeddings to generate the network output characterizing the entity comprises (See claim 28), at each of a plurality of time steps:
processing: (i) the one or more latent embeddings from the set of latent embeddings, and [(ii) output elements generated at any preceding time steps], to generate an output element at the time step.
(Tsai [sec(s) 3.2] “As a final step, we concatenate the outputs from the crossmodal transformers that share the same target modality to yield Z{L,V,A} … Each of them is then passed through a sequence model to collect temporal information to make predictions.” [sec(s) 3] “In this section, we describe our proposed Multi modal Transformer (MulT) (Figure 2) for modeling unaligned multimodal language sequences. At the high level, MulT merges multimodal time series via a feed-forward fusion process from multiple directional pairwise crossmodal transformers. Specifically, each crossmodal transformer (introduced in Section 3.2) serves to repeatedly reinforce a target modality with the low-level features from another source modality by learning the attention across the two modalities’ features. A MulT architecture hence models all pairs of modalities with such crossmodal transformers, followed by sequence models (e.g., self-attention transformer) that predicts using the fused features.”;)
However, the combination of Lee, Tsai does not appear to explicitly teach:
processing: (i) the one or more latent embeddings from the set of latent embeddings, and [(ii) output elements generated at any preceding time steps], to generate an output element at the time step.
Vaswani teaches
processing: (i) the one or more latent embeddings from the set of latent embeddings, and (ii) output elements generated at any preceding time steps, to generate an output element at the time step.
(Vaswani [sec(s) 3] “Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 29]. Here, the encoder maps an input sequence of symbol representations (x1,...,xn) to a sequence of continuous representations z = (z1,...,zn). Given z, the decoder then generates an output sequence (y1,...,ym) of symbols one element at a time. At each step the model is auto-regressive [9], consuming the previously generated symbols as additional input when generating the next.” [sec(s) 3.2] “Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position. We need to prevent leftward information flow in the decoder to preserve the auto-regressive property. We implement this inside of scaled dot-product attention by masking out (setting to −∞) all values in the input of the softmax which correspond to illegal connections.” [sec(s) 3.1] “We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lee, Tsai with the preceding time steps of Vaswani.
One of ordinary skill in the art would have been motived to combine in order to achieve a new state of the art in some translation tasks, and outperform even all previously reported ensembles.
(Vaswani [sec(s) 7] “For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles.”)
Claim(s) 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks) in view of Tsai et al. (Multimodal Transformer for Unaligned Multimodal Language Sequences) in view of Tancik et al. (Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains)
Regarding claim 39
The combination of Lee, Tsai teaches claim 37.
Lee further teaches
wherein the spatial structure is a one-dimensional (1D), two-dimensional (2D), or three-dimensional (3D) array of units, and
(Lee [sec(s) 5] “We split all characters (and corresponding images) into train, validation, and test sets and only train using images from the train character classes. We generate input sets by sampling between 6 and 10 images and we train the model to predict the number of different characters inside the set.” and “Synthetic 2D mixtures of Gaussians: Each dataset contains n ∈ [100,500] points on a 2D plane, each sampled from one of four Gaussians. CIFAR-100: Each dataset contains n ∈ [100,500] images sampled from four random classes in the CIFAR-100 dataset. Each image is represented by a 512-dim vector obtained from a pretrained VGG network (Simonyan & Zisserman, 2014).” [sec(s) 5.2] “We use the Omniglot (Lake et al., 2015) dataset, which consists of 1,623 different handwritten characters from various alphabets, where each character is represented by 20 different images.” [sec(s) Abs] “Many machine learning tasks such as multiple instance learning, 3D shape recognition and few shot image classification are defined on sets of in stances.”;)
Tsai further teaches
wherein generating, for each unit in the entity, the positional embedding of the unit based on the position of the unit in the spatial structure comprises:
(Tsai [sec(s) 3.2] “Positional Embedding. To enable the sequences to carry temporal information, following (Vaswani et al., 2017), we augment positional embedding (PE) to ˆX{L,V,A}: Z[0] {L,V,A} = ˆX{L,V,A} + PE(T{L,V,A},d) (3) where PE(T{L,V,A},d) ∈ RT{L,V,A}×d computes the (fixed) embeddings for each position index, and Z[0]{L,V,A} are the resulting low-level position aware features for different modalities. … As a final step, we concatenate the outputs from the crossmodal transformers that share the same target modality to yield Z{L,V,A} … Each of them is then passed through a sequence model to collect temporal information to make predictions. We choose the self-attention transformer (Vaswani et al., 2017). Eventually, the last elements of the sequences models are extracted to pass through fully-connected layers to make predictions.”;)
generating, for each unit in the entity, a [Fourier] feature positional encoding [having frequency bands that are spaced log-linearly over a predefined target frequency range].
(Tsai [sec(s) 3.2] “Positional Embedding. To enable the sequences to carry temporal information, following (Vaswani et al., 2017), we augment positional embedding (PE) to ˆX{L,V,A}: Z[0] {L,V,A} = ˆX{L,V,A} + PE(T{L,V,A},d) (3) where PE(T{L,V,A},d) ∈ RT{L,V,A}×d computes the (fixed) embeddings for each position index, and Z[0]{L,V,A} are the resulting low-level position aware features for different modalities. … As a final step, we concatenate the outputs from the crossmodal transformers that share the same target modality to yield Z{L,V,A} … Each of them is then passed through a sequence model to collect temporal information to make predictions. We choose the self-attention transformer (Vaswani et al., 2017). Eventually, the last elements of the sequences models are extracted to pass through fully-connected layers to make predictions.”;)
The combination of Lee, Tsai is combinable with Tsai for the same rationale as set forth above with respect to claim 1.
However, the combination of Lee, Tsai does not appear to explicitly teach:
generating, for each unit in the entity, a [Fourier] feature positional encoding [having frequency bands that are spaced log-linearly over a predefined target frequency range].
Tancik teaches
generating, for each unit in the entity, a Fourier feature positional encoding having frequency bands that are spaced log-linearly over a predefined target frequency range.
(Tancik [sec(s) 1] “A few recent works [30, 48] have experimentally found that a heuristic sinusoidal mapping of input coordinates (called a “positional encoding”) allows MLPs to represent higher frequency content. We observe that this is a special case of Fourier features [37]: mapping input coordinates v to
PNG
media_image3.png
59
1395
media_image3.png
Greyscale
before passing them into an MLP. We show that this mapping transforms the NTK into a stationary (shift-invariant) kernel and enables tuning the NTK’s spectrum by modifying the frequency vectors bj, thereby controlling the range of frequencies that can be learned by the corresponding MLP. We show that the simple strategy of setting aj = 1 and randomly sampling bj from an isotropic distribution achieves good performance, and that the scale (standard deviation) of this distribution matters much more than its specific shape. We train MLPs with this Fourier feature input mapping across a range of tasks relevant to the computer vision and graphics communities.” [sec(s) 2] “our analysis is intended to clarify experimental results demonstrating that an input mapping of coordinates (which they called a “positional encoding”) using sinusoids with logarithmically-spaced axis-aligned frequencies improves the performance of coordinate-based MLPs on the tasks of novel view synthesis from 2D images [30] and protein structure modeling from cryo-electron microscopy [48]. We analyze this technique to show that it corresponds to a modification of the MLP’s NTK, and we show that other non-axis-aligned frequency distributions can outperform this positional encoding.” [sec(s) 6.1] “Positional encoding:
PNG
media_image4.png
95
1268
media_image4.png
Greyscale
T for j = 0, . . . , m − 1. Uses log-linear spaced frequencies for each dimension, where the scale σ is chosen for each task and dataset by a hyperparameter sweep. This is a generalization of the “positional encoding” used by prior work [30, 43, 48]. Note that this mapping is deterministic and only contains on-axis frequencies, making it naturally biased towards data that has more frequency content along the axes.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lee, Tsai with the Fourier feature of Tancik.
One of ordinary skill in the art would have been motived to combine in order to greatly improve the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.
(Tancik [sec(s) Abs] “To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.”)
Claim(s) 43 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks) in view of Tsai et al. (Multimodal Transformer for Unaligned Multimodal Language Sequences) in view of Goyal et al. (PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination)
Regarding claim 43
The combination of Lee, Tsai teaches claim 41.
wherein selecting a proper subset of the data element embeddings for use by one or more specified cross-attention blocks based on the selection scores comprises: (See claim 41)
Lee further teaches
selecting a predefined number of the data element embeddings having [the highest selection scores] in the set of data element embeddings.
(Lee [sec(s) 3] “We begin by defining our attention-based set operations, which we call SAB and ISAB. While existing pooling methods for sets obtain instance features independently of other instances, we use self-attention to concurrently encode the whole set. This gives the Set Transformer the ability to compute pairwise as well as higher-order interactions among instances during the encoding process. For this purpose, we adapt the multihead attention mechanism used in Transformer. We emphasize that all blocks introduced here are neural network blocks with their own parameters, and not fixed functions. Given matrices X,Y ∈ Rn×d which represent two sets of d-dimensional vectors, we define the Multihead Attention Block (MAB) with parameters ω as follows:”;)
However, the combination of Lee, Tsai does not appear to explicitly teach:
selecting a predefined number of the data element embeddings having the highest selection scores in the set of data element embeddings.
Goyal teaches
selecting a predefined number of the data element embeddings having the highest selection scores in the set of data element embeddings.
(Goyal [sec(s) 3.3] “We next address the task of determining the retention configuration. Analyzing all the possible configurations is untenable due to the exponential search space. Instead, we design a strategy that learns the retention configuration. Intuitively, we wish to retain the word-vectors with the topmost significance scores and the objective is to learn how many to retain. The topmost word-vectors may appear in arbitrary positions across different inputs in the dataset. Therefore, we sort them according to their significance scores. We shall learn the extent to which the sorted positions must be retained. We accomplish the task by introducing soft-extract layers and modifying the loss function” [sec(s) 3.2] “Word-vector Extraction. Given the scoring mechanism, we perform word-vector selection by inserting an extract layer between the self-attention module and the feed forward network. The layer computes the scores and retains the top lj word-vectors. See Figure 4 for an illustration.” [sec(s) 3.1] “BERT derives the final prediction from the word-vector corresponding to the CLS token. We conducted experiments to determine whether it is critical to derive the final prediction from the CLS token during inference.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lee, Tsai with the highest selection scores of Goyal.
One of ordinary skill in the art would have been motived to combine in order to improve inference time gains based on word-vector elimination.
(Goyal [sec(s) 1] “In terms of inference time, removing an encoder can be considered equivalent to eliminating all its output wordvectors. However, encoder elimination is a coarse-grained mechanism that removes the encoders in totality. To achieve considerable gain on inference time, a commensurate number of encoders need to pruned, leading to accuracy loss. In contrast, word-vector elimination is a fine-grained method that keeps the encoders intact and eliminates only a fraction of word-vectors. Consequently, as demonstrated in our experimental study, word-vector elimination leads to improved inference time gains”)
Claim(s) 44, 48 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks) in view of Tsai et al. (Multimodal Transformer for Unaligned Multimodal Language Sequences) in view of Upadhyay et al. (TRANSFORMER BASED REINFORCEMENT LEARNING FOR GAMES)
Regarding claim 44
The combination of Lee, Tsai teaches claim 41.
Lee further teaches
determining a task performance measure based on the network output characterizing the entity;
(Lee [sec(s) 5] “To evaluate the Set Transformer, we apply it to a suite of tasks involving sets of data points. We repeat all experiments five times and report performance metrics evaluated on corresponding test datasets. Along with baselines, we compared various architectures arising from the combination of the choices of having attention in encoders and de coders. Unless specified otherwise, “simple pooling” means average pooling.” [sec(s) Abs] “We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating increased performance compared to recent methods for set-structured data”;)
However, the combination of Lee, Tsai does not appear to explicitly teach:
determining a reward based on the task performance measure; and
training the selection blocks on a reinforcement learning objective function that depends on the reward.
Upadhyay teaches
determining a reward based on the task performance measure; and
(Upadhyay [sec(s) 2] “In our experiments, we set up an environment for the ”Cart pole” game (using OpenAI gym[16]) where the goal is to balance the pole on the cart (i.e., prevent the pole from falling) by moving the cart left or right. Figure 1 shows a frame from the game to visualize the environment. The set of actions A con sists of {left, right} and the environment provides a reward from the set of rewards R consisting of {+1, −1}. For ever time-step where the pole does not fall the environment pro vides a reward of +1 and when the pole falls (i.e., the angle it makes from the cart crosses a certain threshold) the episode is completed and the agent receives a reward of −1. The typical deep reinforcement learning pipeline passes the frames (or sequence of frames) through a deep convolutional neural network to extract the useful features, which are further processed to estimate the value function (V ) or the state-action value function (Q).”;)
training the selection blocks on a reinforcement learning objective function that depends on the reward.
(Upadhyay [sec(s) 2] “Deep Q-Learning (DQN) uses a neural network to approximate the Q-value function. The state is given as the input and the Q-value of all possible actions is generated as the output. In a typical Deep Q learning setup, all the past experiences are first stored in the memory, the next action is then determined by using epsilon greedy policy with respect to current Q values and the final loss is computed using equation 1, where St, at,rt is the state, action taken and reward received at time step t and St+1 is the state at the next time-step.
PNG
media_image5.png
110
1191
media_image5.png
Greyscale
(1)” [sec(s) 4] “In this work, we propose a new transformer based model for reinforcement learning, the inspiration for the same is de rived from the fact that the recent advancements in NLP have been achieved by moving away from RNNs and introducing the new transformer based model. Even though the standard transformer consists of both encoder and decoder modules as it’s designed to perform Seq2Seq task, we decided to use only the encoder module with the multi-head attention mechanism from the transformer to extract important features from the states to learn the Q-values.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lee Tsai with the reward of Upadhyay.
One of ordinary skill in the art would have been motived to combine in order to improve training time as well as ac curacy in NLP tasks based on more effective and explicit attention-mechanism.
(Upadhyay [sec(s) 1] “The transformer was a novel technique introduced to replace this attention-mechanism performed by LSTM by more effective and explicit attention-mechanism. It is also an encoder-decoder model but differs from LSTM by avoiding any usage of recurrent neural networks which is common in LSTM, GRU, etc. This improves training time as well as ac curacy in NLP tasks.”)
Regarding claim 48
The combination of Lee, Tsai teaches claim 28.
However, the combination of Lee, Tsai does not appear to explicitly teach:
wherein the neural network is configured to perform an agent control task of selecting actions to be performed by an agent that is interacting with an environment.
Upadhyay teaches
wherein the neural network is configured to perform an agent control task of selecting actions to be performed by an agent that is interacting with an environment.
(Upadhyay [sec(s) 2] “In our experiments, we set up an environment for the ”Cart pole” game (using OpenAI gym[16]) where the goal is to balance the pole on the cart (i.e., prevent the pole from falling) by moving the cart left or right. Figure 1 shows a frame from the game to visualize the environment. The set of actions A con sists of {left, right} and the environment provides a reward from the set of rewards R consisting of {+1, −1}.” [sec(s) 2] “Deep Q-Learning (DQN) uses a neural network to approximate the Q-value function. The state is given as the input and the Q-value of all possible actions is generated as the output. In a typical Deep Q learning setup, all the past experiences are first stored in the memory, the next action is then determined by using epsilon greedy policy with respect to current Q values and the final loss is computed using equation 1, where St, at,rt is the state, action taken and reward received at time step t and St+1 is the state at the next time-step.
PNG
media_image5.png
110
1191
media_image5.png
Greyscale
(1)” [sec(s) 4] “In this work, we propose a new transformer based model for reinforcement learning, the inspiration for the same is de rived from the fact that the recent advancements in NLP have been achieved by moving away from RNNs and introducing the new transformer based model. Even though the standard transformer consists of both encoder and decoder modules as it’s designed to perform Seq2Seq task, we decided to use only the encoder module with the multi-head attention mechanism from the transformer to extract important features from the states to learn the Q-values.”;)
The combination of Lee, Tsai is combinable with Upadhyay for the same rationale as set forth above with respect to claim 44.
Claim(s) 45 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks) in view of Tsai et al. (Multimodal Transformer for Unaligned Multimodal Language Sequences) in view of Upadhyay et al. (TRANSFORMER BASED REINFORCEMENT LEARNING FOR GAMES) in view of Byravan et al. (Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models)
Regarding claim 45
The combination of Lee, Tsai, Upadhyay teaches claim 44.
However, the combination of Lee, Tsai, Upadhyay does not appear to explicitly teach:
wherein the task performance measure comprises a cross-entropy classification error, and
wherein the reinforcement learning objective function comprises a squared Bellman error.
Byravan teaches
wherein the task performance measure comprises a cross-entropy classification error, and
(Byravan [sec(s) F] “The reconstruction loss is split into two parts (weighted equally), an image reconstruction loss and a proprioception reconstruction loss. We use a squared error term for the proprioception loss and a binary cross entropy loss term for image re construction; in practice we found this to result in better image reconstructions than a squared error term.” [sec(s) 6] “In particular we use CEM: the cross-entropy method [47] using the same model as IVG for transfer (latent rollouts). PG: replacing the value gradients with a likelihood ratio estimator (using 100 imagined rollouts), again using the same model as used for the IVG transfer. Additional details on the baselines are given in the appendix.”;)
wherein the reinforcement learning objective function comprises a squared Bellman error.
(Byravan [sec(s) 4] “Here ζ is a coefficient that weights the two loss terms and the reward loss is
PNG
media_image6.png
64
785
media_image6.png
Greyscale
, (5) where the value-loss is given by the, importance weighted, squared Bellman error
PNG
media_image7.png
129
1712
media_image7.png
Greyscale
, (6)” [sec(s) C] “As described in the main paper value-loss LV involves the calculation of a (squared) Bellman error, which is given by
PNG
media_image8.png
106
1427
media_image8.png
Greyscale
(7)”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lee, Tsai, Upadhyay with the reinforcement learning of Byravan.
One of ordinary skill in the art would have been motived to combine in order to show a significant improvement in learning speed compared to strong off-policy baselines.
(Byravan [sec(s) Abs] “We evaluate the efficacy of our approach in a transfer learning scenario, re-using previously learned models on tasks with different reward structures and visual distractors, and show a significant improvement in learning speed compared to strong off-policy baselines.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEHWAN KIM whose telephone number is (571)270-7409. The examiner can normally be reached Mon - Fri 9:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEHWAN KIM/Examiner, Art Unit 2129
Prosecution Insights