DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in responsive to communication(s): original application filed on 05/08/2024. Claims pending. Claims independent.
Specification
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
The disclosure is objected to because of the following informalities:
in ¶ [0095], "… FPGA circuitry may include an on-board CPU, such as one or more of the example CPU 820 of FIG. 7 …" appears to be "… FPGA circuitry may include an on-board CPU, such as one or more of the example CPU 820 of FIG. 8 …".
Appropriate correction is required.
Claim Objections
Claims 1, 5, 8, 12, 15, and 19 are objected to because of the following informalities:
in Claim 1, lines 1-4, "An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit …" appears to be "An apparatus comprising: interface circuitry; a non-transitory machine readable storage medium storing machine-readable instructions; and at least one processor circuit …" to prevent ambiguous on whether directly claiming non-statutory subject matter "machine-readable instructions";
in Claim 1, lines 8-10, "self attention layer interaction" appears to be "self-attention layer interaction" according to Claims 8 and 15;
in Claim 1, lines 18-19, "… insert masked ones of the first tokens and second tokens to a masked language model …" appears to be "… insert masked ones of the first tokens and the second tokens to a masked language model …";
in Claim 5, lines 1-2; Claim 12, lines 2-3, and Claim 19, lines 1-2, "… wherein … and the object matching neural network is a feed forward network" appears to be "… wherein … and the object matching neural network are a feed forward network" (see also 112 Rejections to Claims 5, 12, and 19)
in Claim 8, lines 1-2, "A non-transitory machine readable storage medium comprising instructions …" appears to be "A non-transitory machine readable storage medium storing instructions …";
in Claim 8, lines 15-16, "… feed masked ones of the first tokens and second tokens to a masked language model …" appears to be "… feed masked ones of the first tokens and the second tokens to a masked language model …";
in Claim 15, lines 14-15, "… feeding masked ones of the first tokens and second tokens to a masked language model …" appears to be "… feeding masked ones of the first tokens and the second tokens to a masked language model …".
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 8, and 15 recites the limitation "the MLM" in lines 23, 20, and 19 respectively. There is insufficient antecedent basis for this limitation in the claim. For examination purpose, "the masked language model" is considered.
Claims 2-7. 9-14, and 16-20 are rejected for fully incorporating the deficiency of their respective base claims.
Claim 5, 12, and 19 recite the limitation "… wherein the group classifier model …" in line 1, 2, and 1 respectively, which rendering these claims indefinite because ".
Allowable Subject Matter
Claims 1-20 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action.
The following is a statement of reasons for the indication of allowable subject matter:
In regard to independent Claims 1, 8, and 15, prior arts of records, either singularly or in combination, do not teach or suggest the combination of claimed elements including "an apparatus comprising: interface circuitry; a non-transitory machine readable storage medium storing machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions to: tokenize a first input and a second input to generate first tokens and second tokens; generate context information based on transformer self-attention layer interaction between the first tokens and the second tokens, the self-attention layer interaction to generate numerical values for respective ones of the first tokens and the second tokens; insert a first average value of the first tokens to a first group classifier model to predict a first group classification; insert a second average value of the second tokens to a second group classifier model to predict a second group classification, the first and second group classifier models trained with supervised data associated with the first and second inputs; insert masked ones of the first tokens and the second tokens to a masked language model; and train a transformer based on an average loss value associated with (a) a first loss value corresponding to the first group classification, (b) a second loss value corresponding to the second group classification, (c) a third loss value corresponding to the masked language model, and (d) a fourth loss value corresponding to an object matching neural network", "a non-transitory machine readable storage medium storing instructions to cause programmable circuitry to at least: tokenize a first input and a second input to generate first tokens and second tokens; generate context information based on transformer self-attention layer interaction between the first tokens and the second tokens, the self-attention layer interaction to generate numerical values for respective ones of the first tokens and the second tokens; feed a first average value of the first tokens to a first group classifier model to predict a first group classification; feed a second average value of the second tokens to a second group classifier model to predict a second group classification, the first and second group classifier models trained with supervised data associated with the first and second inputs; feed masked ones of the first tokens and the second tokens to a masked language model; and train a transformer based on an average loss value associated with (a) a first loss value corresponding to the first group classification, (b) a second loss value corresponding to the second group classification, (c) a third loss value corresponding to the masked language model, and (d) a fourth loss value corresponding to an object matching neural network", or "a method comprising: tokenizing a first input and a second input to generate first tokens and second tokens; generating context information based on transformer self-attention layer interaction between the first tokens and the second tokens, the self-attention layer interaction to generate numerical values for respective ones of the first tokens and the second tokens; feeding a first average value of the first tokens to a first group classifier model to predict a first group classification; feeding a second average value of the second tokens to a second group classifier model to predict a second group classification, the first and second group classifier models trained with supervised data associated with the first and second inputs; feeding masked ones of the first tokens and the second tokens to a masked language model; and training a transformer based on an average loss value associated with (a) a first loss value corresponding to the first group classification, (b) a second loss value corresponding to the second group classification, (c) a third loss value corresponding to the masked language model, and (d) a fourth loss value corresponding to an object matching neural network" when interpreted as a whole.
Mohamed et al. ("Comparison of Transformer-based Architectures for Product Categorization", 2023 International Conference on Computer and Applications (ICCA), Nov. 28-30, 2023, pp. 1-6) discloses in Abstract and Section 1 of Page 1 that (1) In this paper, we aim to evaluate different transformer-based architectures on the Amazon Shopping Queries dataset to determine which architecture is well-suited for the task of product categorization within an e-commerce platform; (2) In particular the attention mechanism has helped enabling the e-commerce search engine to understand the intent behind the user query and properly retrieving & categorizing products available in the catalogue; (3) The different models that achieved state of the art results in language modelling tasks like language translation & text generation were built upon the transformer architecture which basis is the attention mechanism; (4) These models include BERT which stands for Bidirectional Encoder Representations from Transformers, and it utilizes the transformer encoder architecture and GPT which stands for generative pre-trained transformers, and it utilizes the transformer decoder architecture; and (5) Our motivation for this paper is to fine-tune both BERT & GPT-2 models on the amazon dataset to determine which language model is better suited for the task of multi-class product classification. Mohamed further discloses in Section II of Pages 1-3 that (1) The attention mechanism enhanced the encoder-decoder model that was intended for the task of machine translation by receiving an input sequence and then generating the output sequence which is the resultant translated sentence; (2) The attention mechanism works by allocating attention scores to all of the encoded word tokens within an input sequence, these scores are acquired by calculating the similarity score between the output of the decoder with the encoder’s hidden state afterwards these scores are normalized into attention weights using a softmax layer, these weights represent the most important & relevant parts of each word token within an input sequence; (3) The attention weights of each word token in the encoder’s input sequence are then added together along with encoder hidden representations to compute a context vector which is the of the summarization of the overall relevancy of each word in the input sequence, this context vector is utilized iteratively with the state of the decoder to compute each word token in the output sequence; (4) The transformer architecture uses this attention mechanism which represents the important relationships and relevancy between words along with the positional encodings that represents the position of each word within a sentence to maintain the context & dependencies of words with long sequences of data that previous neural networks such as RNN & LSTM failed to achieve; (5) BERT which stands for Bidirectional Encoder Representations from Transformers is a pre-trained language model based on transformers architecture; (6) BERT uses multilayer bi-directional transformer-based encoders to achieve deep understanding of a sentence within its context since language models with fixed vector representations like word2vec and GloVe failed to represent different embeddings for words based on the sentence context; (7) BERT was pretrained on Wikipedia and Bookcorpus datasets using masked language model and next sentence prediction tasks, the next sentence prediction task is used to capture the relationship between a pair of sentences; (8) BERT takes the input as a single sequence, and it utilizes special tokens CLS and SEP, where the SEP token is used to indicate the end of an input and start of another input in the sequence since BERT takes only one input sequence as we previously mentioned and CLS token on the other hand is a classification token at the last hidden state of BERT and it is used in classification tasks; (9) Afterwards BERT uses both the segment and position embeddings for each token, the segment embedding is used to indicate the sentence number which the token belongs to, encoded into a vector and the position embedding is used to indicate the position of the token within that sentence, encoded into a vector; (10) the final embeddings are computed using the sum of token, segment, and position embeddings, then those final embeddings are given to the BERT bidirectional transformer layers to receive output; (11) GPT which stands for Generative Pre-trained Transformer is a language model that is built on the transformer architecture like BERT but the main difference between BERT & GPT is that BERT is built on the encoder part only of the transformer architecture while GPT is built on the decoder part of the transformer architecture without the encoder; (12) The input that GPT receives is a sequence of token and the output is probability distribution of the following tokens in the sequence; (13) The pre-training objective of GPT is Autoregressive task which is the task of text regeneration therefore in the fine-tuning step in the GPT model, any NLP downstream task can be handled as generative in nature; (14) Both GPT & BERT use the same positional encoding technique which is the absolute positional encoding; (15) Tokenization is a procedure of dividing the textual input into smaller segments called tokens, it can be performed on a word, sub-word or even a character to be prepared for processing by any machine learning algorithms; (16) There are several tokenization methods, BERT tokenization method is WordPiece tokenization while GPT uses Byte-Pair Encoding; (17) WordPiece and Byte-Pair Encoding are considered two variants of the same tokenization concept, Sub-word tokenization; (18) The main concept behind sub-word tokenization is adding the words that occur regularly to the vocabulary while less frequent words are split into sub-words; and (19) The difference between BPE & WordPiece is that BPE adds the sub-words with the highest appearance score while WordPiece adds the sub-words that increases the likelihood of the training data. Mohamed also discloses in Section III with FIGS. 1-2 of Pages 3-4 that (1) The text classification task is the process of categorizing sentences into a specific label, and it is considered as a supervised machine learning technique meaning that the sentences used for training had already been labelled with a specific category; (2) We have four relevance judgments associating the complex queries with the products which are Exact, Substitute, Complement and Irrelevant; (3) Exact means that the product is completely relevant to the query and it satisfies all of its requirements, Substitute means that the product satisfies parts of the query in some requirements and some requirements are not, Complement means that the product does not satisfy the query’s purpose but it could be used in conjunction with an exact product and Irrelevant means that the item does not satisfy any aspect of the query; (4) The queries in the dataset are complex user queries that were released by amazon to enhance the performance of different machine learning models in semantic matching tasks and classification; (5) The goal of using the dataset is fine-tuning both BERT & GPT-2 model to accurately classify each product associated with a query into one of the four labels, the dataset has queries and products in three different languages which are English, Spanish, and Japanese but for the purpose of comparing the performance of the two before-mentioned language models, the English language only is going to be covered; (6) We adopted for our classification task the BERT-Base uncased model which consists of 12 transformer encoder layers stacked one above the other, each hidden unit is of size 768, meaning it produces vectors of size 768, it performs the self-attention mechanism with 12 attention heads, and it has 110 million learnable parameters; (7) Before the BERT model is fed the data to predict the label for each query-product pair, the data must be pre-processed first; (8) There are several steps that must be executed in the pre-processing phase; (9) The first step is assigning numerical values to the relevance judgments so that later we can classify each query-product pair and the following step is tokenizing the queries & product titles; (10) The tokenization is the process of transforming the textual sentences of the query and product title into smaller representations called tokens that BERT model can interpret; (19) Therefore, the words in input sentences are divided into smaller sub-words; (20) Afterwards, the concatenation of the query and the product title into a single sequence must be performed since BERT model only receives one input sequence at a time; (21) This is achieved by using the [SEP] token between the query sequence and the product title sequence during the tokenization step; (22) During the tokenization process as well, the classification token [CLS] is appended to the beginning of the input sequence which would be subsequently utilized at the last hidden state to encompass the entire input sequence; (23) Furthermore, the [PAD] token would be added to the input sequence if the size of the input sequence is less than the length of the fixed size that is accepted by the BERT model, alternatively if the input sequence size is greater than the maximum fixed length accepted by BERT, the sequence must be truncated; (24) After the words in the tokenization process are split into multiple smaller sub words or characters, each token is allocated a numerical value conforming to an index from the vocabulary of the BERT’s model; (25) Afterwards, the initial layer of the BERT model which is the embedding layer receives the tokens to be converted to high dimensional vectors called token embeddings that encapsulates the context of the tokens, and the embedding layer also adds the positional & segment embeddings by elementwise addition to the token embeddings to state the position of each token in the input sequence; (26) Since BERT model is transformer-based, meaning that sequences undergo concurrent processing, it is beneficial to have positional embeddings to enable BERT to capture sequential context; (27) Finally, the multiple transformer layers of BERT compute the contextual representations and connections of the input sequences; (28) Afterwards, a classification layer is added on top of the BERT model, which is a fully connected dense layer having a softmax activation function; (28) The classification layer utilizes as input the embedding vector of size 768 output from the BERT model’s first position of the sequence which is value of the [CLS] token; (29) Finally, the classification layer will produce an output vector of size (768,4) that corresponds to the number of class labels for our classification tasks; (30) This is achieved by the usage of the softmax function illustrated as equation (1), which is responsible for transforming the output vectors into normalized probability distribution that indicates which label of the four (ESCI); (31) To diminish the loss and optimize the model’s performance, the weights of the BERT-based classification model are tuned throughout the training process using the cross-entropy loss; (32) This is achieved by calculating the difference between the actual input vector and the predicted probability value computed by the softmax and the goal is to minimize that difference; (33) GPT-2 is a decoder-based model, meaning that the last token of the input sequence would be the token utilized for the ESCI label classification task as it encompasses the contextual embedding of the whole sentence not the first token [CLS] like BERT; (34) The tokenization process of splitting the sequences into sub-words and characters were performed for the GPT-2 model but in different manner to BERT’s like the padding location and tokenization method which was Byte- Pair Encoding; (35) It was challenging to provide both the query & product title embeddings to GPT-2 is not built to handle two separate sentences concurrently as BERT since GPT-2 does not utilize special tokens [CLS], [SEP] and [PAD]; (36) Alternatively, the two sequences which were the query and product title, were concatenated as a single sequence, afterwards this sequence is tokenized to be provided to GPT- 2 model as an input sequence along with the encoded labels; (36) Afterwards, the model receives these tokens to be propagated throughout the 12 decoder layers to obtain context information; (37) Then a linear layer is added to the GPT-2 model to perform the product categorization, the output dimension of that layer must be equal to four corresponding to the number of the labels in our classification task; (38) The last token of the input sequence represents the pooled representation of the whole sequence computed by GPT-2 model; it was then passed through the linear layer to be transformed into a single vector by learning the weight matrix and the bias; (39) That vector encompassed the label predicted for the product relevance classification task; and (40) We then use the cross-entropy function to compute the variability between the predicted label and the actual label for that sequence, the purpose is to adjust the model weights throughout the training to minimize that loss.
Tracz et al. ("BERT-based similarity learning for product matching", Proceedings of the Workshop on Natural Language Processing in E-Commerce (EComNLP), Dec. 12, 2020, pp. 66-75) discloses in Abstract of Page 66 that (1) In this work, we apply BERT-based models in a similarity learning setup to solve the product matching problem; (2) We provide a thorough ablation study, showing the impact of architecture and training objective choices; and (3) Application of transformer-based architectures and proper sampling techniques significantly boosts performance for a range of e-commerce domains, allowing for production deployment. Tracz further discloses in Section 1 of Pages 66-67 that (1)Classical approaches to product matching and its generalization (entity matching) often rely on rule-based methods and hand-crafted features such as string similarity measures; (2) Recent advances of deep learning in natural language processing (NLP) sparked increasing interest in end-to-end deep learning approaches both for entity matching and product matching; (3) we follow this trend and present our experiences with leveraging transformer-based neural language models combined with the similarity learning setup for product matching; (4) While transformer architectures have been very recently applied to entity matching, to the best of our knowledge, we are the first to apply them specifically for the product matching problem in e-commerce; (5) Aside from architectural choices, we show that sampling techniques tailored to the product matching domain prove to be crucial in the progress of similarity learning; (6) The contributions of our paper are as follows: (a) we apply state-of-the-art BERT-based models in the similarity learning setup to solve the product matching task in the e-commerce domain, (b) we compare the usefulness of modern BERT-based architectures such as BERT and DistilBERT (Sanh et al., 2019) for the product matching task, (c) we propose category hard batch construction strategy, which proves to increase the fraction of active training triplets and the performance of the final model; and (d) we adopt and evaluate different batch construction strategies in the similarity learning setup for solving product matching. Tracz also discloses in Section 2 of Pages 67-68 that (1) Product matching aims at identifying offers of the same product across many merchants selling it in an e-commerce portal and integrating the information into a single entry in a product catalog; (2) Production use of a product matching system requires that the system is able to operate correctly for yet unrepresented products; (3) As products are being introduced, the product matching system should be capable to handle new instances; (4) Motivated by this requirement, we formalize the product-matching problem as generalized zero-shot multi-class classification following the definition in (Li et al., 2019); (5) Let O denote the set of offers, and
P
=
P
s
∪
P
u
the set of products consisting of two non-overlapping sets of seen products
P
s
and unseen products
P
u
; (6) Suppose we have three sets of data
D
s
=
D
s
,
D
u
,
D
a
, where
D
s
,
D
u
and
D
u
are training, test and semantic description sets, respectively; (7) The training and test sets consist of matching offer-product pairs, i.e.,
D
s
=
o
,
p
:
o
∈
O
,
p
∈
P
and
D
u
=
o
,
p
:
o
∈
O
,
p
∈
P
; (8) The aim is to learn transferable knowledge from
D
s
to be able to predict matching products for offers present in pairs from
D
u
additionally making use of the semantic descriptions of offers and products
D
a
=
O
,
P
; (9) Importantly, as this is a generalized zero-shot classification problem, not all products are observed in available training data; (10) To be able to classify offers to products not observed during training, not only offers but also product classes need some representation; (11) Since the above definition of product matching is general and representation-agnostic, here we also formalize the specific settings of the product matching problem we solve in this work. We define each offer
o
∈
O
and each product
p
∈
P
as triples
(
t
,
a
,
c
)
, where
t
is the title,
a
is the set of attributes and
c
∈
C
is the category; (12) The set of attributes
a
contains triples
(
a
n
a
m
e
,
a
v
a
l
u
e
s
,
a
u
n
i
t
)
, where
a
n
a
m
e
is the attribute name,
a
v
a
l
u
e
s
is a set of one or more values and
a
u
n
i
t
is an optional unit; (13) Titles of products or offers
t
, attribute names
a
n
a
m
e
, attribute values
a
v
a
l
u
e
s
and attribute units
a
u
n
i
t
are strings possibly containing multiple words; (14) Given an offer
o
=
(
t
o
;
a
o
;
c
o
)
the problem we tackle in this work is to find its matching product p 2 P. In the classic product matching problem
P
=
P
; and (15) Here, we consider a variant of this problem, where
P
=
p
=
t
p
,
a
p
,
c
p
:
p
∈
P
,
c
o
=
c
p
, i.e., we only look for the matching product in a set of products that belong to the same category as the offer. Tracz further teaches in Section 3 of Pages 68-69 that (1) In this work, we interpret the product matching problem as a similarity learning task; (2) This approach allows us to cast product matching problem as a zero-shot learning problem, in result avoiding re-training the model on the introduction of new products or product categories; (3) To solve the product matching problem with triplet loss, we introduce a notion of similarity between offers and products, defined as proximity of their representations in some embedding space; (4) Each training example is defined as a triplet
(
o
,
p
+
,
p
-
)
, denoting an offer (anchor), a matching product (positive) and a non-matching product (negative); (5) Given encoders
E
θ
:
O
→
R
N
and
E
ϕ
:
P
→
R
N
, both transforming respective entities to vector representations in some embedding space, and distance measure d (in our case the cosine distance), we adjust network parameters θ and
ϕ
to minimize the triplet loss objective:
L
o
,
p
+
,
p
-
=
m
a
x
0
,
m
+
d
E
θ
o
,
E
ϕ
p
+
-
d
E
θ
o
,
E
ϕ
p
-
, where margin
m
is a hyperparameter; (6) In this work we tie parameters of
E
θ
and
E
ϕ
, employing a single instance of the encoder for both of them; (6) One shortcoming of the standard triplet loss objective is that majority of possible triplets prove trivial; (7) To ensure that loss values do not vanish after the initial phase of training, a proper batch construction strategy is essential; (8) Properly selected active (non-zero loss) triplets let one effectively evaluate and optimize the specified loss function; (9) The batch construction strategy will be an object of the ablation study; (10) The representation of each offer and product is a concatenated title, attributes values, and units, which is then lower-cased and pass into a fast byte-pair encoder; (11) It is important in the similarity learning setting to properly choose the encoder architecture; (12) Our main focus is the transformer based encoders, which recently gained a lot of attention due to achieving state-of-the-art results on Natural Language Understanding benchmarks; (13) Our BERT usage as an encoder is inspired by (Reimers and Gurevych, 2019); (14) We use a bag of words (BOW) embeddings model based on StarSpace (Wu et al., 2017b) as a simple baselines; (15) The attention mechanism used in the transformer allows the model to learn the context and importance of words in an entity, whereas, in BOW embeddings, all of the words are treated equally and independently; (16) Transformer architectures also allow the model to track the position of the words in a sequence, which we find beneficial in product matching task; (17) To leverage our non-annotated data, we use the masked language model (MLM) pretraining objective described originally in the BERT paper (Devlin et al., 2019), but follow the improved training procedure from (Liu et al., 2019); (18) We also explore scaling down the number of parameters with both the number of layers reduction and the knowledge distillation approach; (19) In our setup the triplets batch construction begins with sampling a batch of matches used as the anchor-positive pairs in triplets; (20) For negative item selection we consider three strategies; (21) The simplest strategy is a modification of the random negative selection which randomly selects a negative from the non-matching products in the category of the anchor, and for the purpose of this work, we name it category random (CR) strategy; (22) The second explored approach is a modification of the batch hard strategy adjusted for our problem setup, which starts with a batch of anchor-positive pairs and then selects the negative as the most similar product from all the non-matching products in the sampled batch of matches, and we refer to this strategy as batch hard (BH) in the experiments; and (23) for the third batch construction method, we propose the category hard (CH) strategy, which is similar to the category random strategy, but instead of selecting the negative at random from the products in a given category, it selects the one that is most similar to the anchor offer.
Myagmar et al. ("Cross-Domain Sentiment Classification With Bidirectional Contextualized Transformer Language Models", IEEE Access, VOLUME 7, Nov. 20, 2019, pp. 163219-163230) discloses in ABSTRACT of Page 163219 that (1) Cross-domain sentiment classification is an important Natural Language Processing (NLP) task that aims at leveraging knowledge obtained from a source domain to train a high-performance learner for sentiment classification on a target domain; (2) Existing transfer learning methods applied on cross-domain sentiment classification mostly focus on inducing a low-dimensional feature representation shared across domains based on pivots and non-pivots, which is still a low-level representation of sequence data; (3) Recently, there have been great progress in the NLP literature in developing high-level representation language models based on Transformer architecture, which are pre-trained on large text corpus and fine-tuned for specific task with an additional layer on top; (4) Among such language models, the bidirectional contextualized Transformer language models of BERT and XLNet have greatly impacted NLP research field; (5) In this paper, we fine-tune BERT and XLNet for the cross-domain sentiment classification; (6) We then explore their transferability in the context of cross-domain sentiment classification through in-depth analysis of two models' performances and update the state-of-the-arts with a significant margin of improvement; and (7) Our results show that such bidirectional contextualized language models outperform the previous state-of-the-arts methods for cross-domain sentiment classification while using up to 120 times less data. Myagmar further discloses in Section I of Pages 163219- 163220 that (1) Sentiment classification, an important task of Natural Language Processing (NLP), aims to identify the emotional tendencies (positive or negative) of given text input; (2) RNN-based deep learning architectures has been the standard for various NLP tasks, including sentiment classification; (3) Transfer learning allows us to deal with this scenario by borrowing information from a relevant source domain with abundant labeled data to help improve the prediction performance in the target domain; (4) Cross-domain sentiment classification (CDSC) aims at leveraging knowledge obtained from a source domain to train a high-performance learner for sentiment classification on a target domain, e.g., book product review, to help classification in the target domain, e.g., electronics product review, with few or no labeled data; (5) Traditional pivot-based CDSC schemes attempt to infer the correlation between pivot words, i.e., the domain-shared sentiment words, and non-pivot words, i.e., the domain-specific sentiment words, by utilizing multiple pivot prediction tasks; (6) However, these schemes share a major limitation that manual selection of pivots is required; (7) substantial work has shown that unsupervised pre-trained language models on large text corpus are beneficial for text classification and other NLP tasks, which can avoid training a new model from scratch; (8) a pre-training technique, Bidirectional Encoder Representations from Transformers (BERT), is proposed and has created state-of-the-art models for a wide variety of NLP tasks, including question answering (SQuAD v1.1), natural language inference, text classification and others; (9) The latest of such pre-trained language models is XLNet, a generalized autoregressive pretraining method that enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order, and overcomes the limitations of BERT thanks to its autoregressive formulation; (10) furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining; (11) In this paper, we fine-tune BERT and XLNet for CDSC and compare them with the current state-of-the-art methods; and (12) our main contributions are summarized as follows: (a) This is the first work to explore the usage of Transformer-based bidirectional contextualized language models for CDSC; (b) Compare and comprehensively analyze the performance of the two highest performing Transformer language models of XLNet and BERT in the context of CDSC; and (c) Achieves new state-of-the-arts results with significant improvements over the previous approaches. Myagmar also discloses in Section III-V with FIG. 1 of Pages 163221- 163224 that (1) Transfer learning comprises of two main concepts: a domain and a task; (2) Given a source domain
D
s
with its task
T
s
and a target domain
c
with its task
T
T
, the objective of cross-domain sentiment classification is to learn the conditional probability distribution PT (YT|XT ) in
D
T
by utilizing the knowledge learned from
D
s
and
T
s
, where
D
S
≠
D
T
and sufficient labeled training data are available in
D
s
; (3) Typically either only few or no labeled data are available in the target domain
D
T
; (4) The task of the cross-domain sentiment classification is to learn a robust classifier PS (YS|XS ) trained on labeled data in the source domain to predict the polarity of unlabeled examples from the target domain using the learned classifier PS (YT|XT ), where YT = YS , i.e., both domains have the same label space; (5) The original Transformer has encoder-decoder structure, with the encoder mapping an input sequence to a sequence of continuous representations, which is used by the decoder to generate an output sequence one element at a time; (6) Each of the encoder and the decoder consists of 6 identical layers, with each containing two sub-layers of 8 parallel self-attention heads and a fully connected feed-forward neural network; (7) The input representation to the first encoder layer is a concatenation of WordPiece embeddings and positional embeddings generated from the input sequence; (8) An attention function can be described as mapping a query and a set of key- value pairs to an output, where the query, keys, values, and output are all vectors; (9) The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key; (10) Specifically, given an embedded vector x for an input sequence, we create a Query, Key, and Value vector for each input embedding token by multiplying the embedding by three learned matrices WQ. WK, WV respectively; (11) For parallel computation, we stack the Query, Key and Value vectors into matrices Q, K, V; (12) Then the self-attention function is given by Eqn. (1); (13) The Transformer performs such self-attention function in parallel with multiple attention heads by projecting the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively; (14) Each Transformer layer consists of two sub-layers, where the first sub-layer is the multi-head attention and its normalized output is fed to the second sub-layer of fully connected feed forward network; (15) Each encoder layer passes its output as an input to the next encoder layer, with the final encoder layer producing the final encoded representation for fine-tuning; (16) BERT, which stands for Bidirectional Encoder Representations from Transformers, is built upon recent works in pre-training contextual representations such as ELMo, and ULMFiT, but these models are either unidirectional or shallowly bidirectional, meaning contextualized representation of a word only considers the words to its left or to its right; (17) BERT, on the other hand, has deeply bidirectional contextualization that combines the representations of both left-context and right-context models; (18) BERT uses two unsupervised prediction tasks: (a) First is next sentence prediction task, where two sentences (A,B) are selected from the text corpus and a classifier is trained to predict whether B actually follows A, where 50% of the time B is the actual next sentence that follows A, and 50% of the time it is a random sentence from the corpus; and (b) The second task is the Masked Language Model task, where they mask some percentage of the input tokens at random, and then predict only those masked tokens; (19) Specifically, given a text sequence x = [x1, …, xT], BERT first constructs a corrupted version
x
^
by randomly setting a 15% of tokens in
x
to a special symbol [MASK]; (20) If denote the masked tokens as
x
-
, then the training objective is to reconstruct
x
-
from
x
^
as shown in Eqn. (4); (21) The biggest advantage of this training objective is it allows the model simultaneous access to the contextual information on both sides of a toke; (22) BERT is the first fine-tuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks, outperforming many systems with task-specific architectures and advances the state-of-the-art for eleven NLP tasks; (23) BERT has achieved strong performances across multiple tasks but it had the following major flaws: (a) The original Transformer architecture can capture con- text within the specified maximum input sequence length, and if a document is longer than the specified length, it would be divided into segments with each of them being processed by the model independently from scratch without any connection between them; (b) BERT is trained to predict tokens replaced with the [MASK] symbol, and however, this [MASK] token never appears in downstream tasks, which creates a discrepancy between pre-training and fine-tuning; and (c) BERT makes predictions for the masked tokens with assumption that there is no dependencies between these masked tokens, which is bit over-simplification and can cause reduced number of dependencies that BERT can learn at once; (24) XLNet solves BERT's first flaw of input length context constraint with the architecture of Transformer-XL, which itself is a modification upon the original Transformer; (25) Transformer-XL introduces Recurrence Mechanism and Relative Positional Encoding to the Transformer architecture to capture long-term dependencies for documents that are longer than the maximum allowed input length; (26) With Recurrence Mechanism, the hidden state sequence computed for the previous segment is fixed and cached to be reused as an extended context when the model processes the next new segment; (27) Although the gradient still remains within a segment, this additional input allows the network to exploit information in the history, leading to an ability of modeling longer-term dependency and avoiding context fragmentation; (28) Relative Positional Encoding encodes position of a context in relative distance from the current token at each attention module, as opposed to encoding position statically only at the beginning like in BERT; (29) This is done so to accommodate the Recurrence Mechanism and avoid having tokens from different segments having the same positional encoding; (30) Despite its ability to capture long-term dependencies, Transformer-XL still only holds unidirectional context, i.e., predicts the current token based on the given sequential context on its left or its right side only; (31) XLNet solves the issue of unidirectional context, without using [MASK] symbol as in BERT, by introducing a language modeling objective called Permutation language modeling that predicts a current token based on the given preceding context just like traditional language model; (32) However, instead of predicting tokens in sequential order, tokens are predicted following a random permutation order; (33) to reduce the optimization difficulty, only the last tokens in a factorization order is chosen for training; (34) XLNet introduces a scheme called Two-Stream Self-Attention that uses two sets of hidden representations: (a) the content stream, and (b) the query stream; (35) The language model is trained to predict each token in the sentence using only the query stream; and (36) We shall ne-tune the pre-trained Transformer models, BERT and XLNet, with a labeled sentiment data from a selected source domain and measure its performance in predicting the sentiment polarity of other domain's sentiment data.
Wang et al. ("Multimodal Semantic Mismatch Detection in Social Media Posts", 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP), Sep 26-28, 2022, pp. 1-6) discloses in Abstract of Page 1 that (1) We develop self-supervised methods to detect semantic mismatch across multiple modalities, namely video, audio and text. We use state-of-the-art language, video and audio models to extract dense features from each modality, and explore transformer architecture together with contrastive learning methods on a dataset of one million Twitter posts from 2021 to 2022; (2) Our best-performing method benefits from the robustness of Noise-Contrastive loss and the context provided by fusing modalities together using a cross-transformer, which outperforms state-of-the-art by over 9% in accuracy; and(3) We further characterize the performance of our system on topic-specific datasets containing COVID-19 and Russia-Ukraine related tweets, and shows that it outperforms state-of-the-art by over 17% in accuracy. Wang further discloses in Section I with FIG. 1 of Page 1 that (1) In this paper, we develop methods to identify social media posts that contain semantic mismatches among their modalities, e.g. video, audio, and text. Semantic mismatches can lead to misinformation, especially those generated at large volumes by automated engines; (2) Many of them are known as cheap fakes, whereby either modality has been crudely manipulated, where an example of a semantic mismatch is shown in Figure 1; (3) The challenges of semantic mismatch detection in social media video posts are two-fold: (a) learning a joint representation of video, audio, and text effectively; and (b) lack of a large labeled dataset for semantic matching; (4) To address the joint representation learning problem, we propose a deep-learning-based method for learning accurate video-language joint distributions; (5) To address the data issues, we introduce a novel training and evaluation method through random-mismatch, which does not require human labeling effort; (6) Specifically, we collect one million social media video posts from Twitter to use as a large self-supervised training corpus and introduce two datasets on specific topics, namely COVID-19 and Russia-Ukraine Crisis related tweets; and (7) For labels used in self-supervision, we consider all collected tweets as semantically matched video and text pairs, and construct mismatched video and text pairs through random mismatch - given a video, we randomly select another tweet’s text to construct a mismatched pair. Wang also discloses in Section III with FIG. 2-4 of Pages 2-4 that (1) Given a video post consisting of a video and a corresponding text description, we first use pre-trained models to extract video and text features; (2) Then, the features are projected to a common representation space, which are learned through contrastive learning. Lastly, we use the projected video and text representations to classify whether the pair of video and text is a match or a mismatch; (3) We first preprocess videos into the input format of our video model, where we convert all videos into 10 frames-per-second and break each video into segments of 32 frames; (4) For video model, we use S3D, pre-trained on activity recognition, to extract one 512-dimensional video feature per video segment; (5) For the text input, we use DeBERTa-v3-Large, pre-trained using Masked Language Modeling, to extract token-level features, where a text feature is generated corresponding to each text token; (6) Given video features of
v
=
v
1
,
…
v
n
,
v
i
∈
R
512
and
t
=
t
1
,
…
t
n
,
t
i
∈
R
1024
, we first use a linear projection to project all features onto the same dimension space
R
1024
,
v
'
=
W
v
v
,
t
'
=
W
t
t
, where
W
v
∈
R
1024
×
512
, and
W
t
∈
R
1024
×
1024
are learned parameters; (6) Since video and text are both sequences of data, it is important to process the temporal information within each modality, rather than naively averaging all features together; (7) As seen in Figure 2, we use transformers to embed features of each modality and to retrieve temporal information; (8) Video features
v
'
=
v
1
'
,
…
v
n
'
and text features
t
'
=
t
1
'
,
…
t
m
'
each are passed through a transformer to generate features of video and text context,
h
=
h
1
,
…
h
n
,
h
i
∈
R
1024
and
k
=
k
1
,
…
k
m
,
k
i
∈
R
1024
, respectively; (9) Next, we apply global mean pooling on features of each modality, h and k, and retrieve aggregated modality features; (10) We use contrastive learning to learn to project features onto a representation space, such that elements of matched semantics are close to each other, while those of mismatched semantics are away from each other; (11) For the architecture shown in Figure 2, we apply two different methods of learning the joint video and text representation using Contrastive Learning: (a) Cosine Embedding Loss and (b) Noise Contrastive Estimation (NCE) loss; (12) We use 2-layer MLPs for projecting
v
e
m
b
e
d
,
t
e
m
b
e
d
to a latent space, but still use the before-projection features as part of our representation; (13) Cosine embedding loss encourages the vector angle between a matching pair of video and text to be smaller, and the angle of a mismatching pair to be larger, which allows the models to learn feature embeddings
v
e
m
b
e
d
and
t
e
m
b
e
d
by contrasting matching pairs with mismatching pairs; (14) One problem with Cosine Embedding Loss is that it only considers one instance of positive or negative sample at a time; (15) The process of constructing negative examples through random mismatch makes the authenticity of the negative sample noisy; (16) Therefore, to reduce noise presented in the negative samples, we sample and learn on multiple negative samples at a time using a variant of NCE loss, similar to NCE loss used in CLIP; (17) Out of all B text features, we learn to classify the text feature matching
v
i
l
a
t
e
n
t
; (18) Therefore, we minimize the cross entropy of each video feature and its matching text feature, versus other text features in the batch; (19) video-to-text loss function is given in Eqn. (2); (20) We also learn the reverse loss in Eqn. (3), namely given
t
i
l
a
t
e
n
t
, we learn to classify which video feature is matching it among all B video features; (21) we optimize using the mean of these two losses; (3) To fully capture the temporal information in each modality, and allow video and text features to interact with each other, we further apply one cross-transformer as shown in Figure 3; (22) It takes in
h
1
,
…
h
n
and
k
1
,
…
k
m
, the last hidden state outputs of both video and text transformers before the global mean pooling, and uses attention mechanism to contextualize video and text features; (23) Cross-transformer outputs new hidden states of video and text,
h
1
'
,
…
h
n
'
and
k
1
'
,
…
k
m
'
, respectively; (24) We then apply global mean pooling on features of each modality to obtain contextualized aggregated modality features; (25) We concatenate both contextualized and learned feature embeddings to obtain fused representation; (26) With representation
R
c
r
o
s
s
, we use a 4-layer MLP over the joint representation
R
c
r
o
s
s
to regress the probability of matching
y
=
σ
(
M
L
P
(
R
c
o
n
c
a
t
)
)
, which we supervise with binary cross-entropy loss LBCE in Eqn. (6); (27) We also explore adding audio as the third input feature to our pipeline, as shown in Figure 4; and (28) To build representation space for audio features, we use a modified NCE loss as in Eqn. (7).
However, closest arts of records, as discussed above, singly or in combination
do not teach or suggest at least following features "tokenize/tokenizing a first input and a second input to generate first tokens and second tokens; generate/generating context information based on transformer self-attention layer interaction between the first tokens and the second tokens, the self-attention layer interaction to generate numerical values for respective ones of the first tokens and the second tokens; insert/feed/feeding a first average value of the first tokens to a first group classifier model to predict a first group classification; insert/feed/feeding a second average value of the second tokens to a second group classifier model to predict a second group classification, the first and second group classifier models trained with supervised data associated with the first and second inputs; insert/feed/feeding masked ones of the first tokens and the second tokens to a masked language model; and train/training a transformer based on an average loss value associated with (a) a first loss value corresponding to the first group classification, (b) a second loss value corresponding to the second group classification, (c) a third loss value corresponding to the masked language model, and (d) a fourth loss value corresponding to an object matching neural network" when combining with all other limitations of these claims as a whole.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142
Prosecution Insights