DETAILED ACTION
This office action is in response to the communication filed on July 07, 2025. Claims 1-20 are currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 07/07/25 have been considered by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 of instant application 19/261,618 are rejected on the ground of nonstatutory double patenting over claims 1, 8, 9, 12, and 17 of U.S. Patent No. 11,327,960, since the claims, if allowed, would improperly extend the “right to exclude” already granted in the patent.
The subject matter claimed in the instant application is fully disclosed in the
patent and is covered by the patent since the patent and the application are claiming
common subject matter, as follows:
US Application 19/261,618
US Patent 11,327,960
10. A method comprising:
receiving a text string including raw data for a transaction;
extracting contextual information from the text string based on at least:
tokenizing the text string, and
applying a masked language model to the tokenized text string;
(see claims 11-14 below)
(see claim 15 below)
identifying entities within the text string based on the contextualized information; and
generating normalized transaction data including the identified entities.
(see claim 17 below)
11. The method of claim 10, wherein tokenizing the text string comprises:
generating a sequence of tokens.
12. The method of claim 11, wherein applying the masked language model comprises:
generating a sequence of vectors.
13. The method of claim 12, wherein individual vectors of the vectors correspond to individual tokens in the sequence of vectors.
14. The method of claim 12, wherein the sequence of vectors is encoded with information regarding one or more surrounding tokens in the sequence of tokens.
15. The method of claim 12, wherein extracting the contextual information is further based on bidirectionally parsing the sequence of vectors to identify tokens indicative of entity information, and wherein identifying entities is based at least in part on the entity information.
17. The method of claim 10, wherein generating the normalized transaction data includes generating at least one of:
location information,
an identified candidate merchant,
generated merchant information,
category information, or
other metadata related to the transaction.
(similarly claims 1-9 and 18-20)
1. A computer-implemented method of parsing raw data associated with one or more transactions, the computer-implemented method comprising, by one or more hardware processors executing program instructions:
receiving a text string including raw data for a transaction;
matching the text string to a plurality of locations within a location corpus to extract location information from the text string;
identifying a candidate entity from the text string based on a similarity score with respect to a plurality of entities within an entity corpus;
in response to the similarity score of the identified candidate entity being less than a threshold score:
tokenizing the text string to create a sequence of tokens;
applying a masked language model to the sequence of tokens to generate a sequence of vectors, each of the vectors corresponding to one of the tokens and being encoded with information regarding one or more of the surrounding tokens in the sequence of tokens;
bidirectionally parsing the sequence of vectors to identify tokens indicative of entity information; and
generating entity information using the tokens indicative of entity information; and
generating normalized transaction data including the extracted location information and one of the identified candidate entity or the generated entity information.
(similarly claims 9, 12, and 17)
16. The method of claim 11, wherein applying the masked language model comprises:
masking a token of the sequence of tokens;
sending the sequence of tokens including the masked token through an embedding layer to generate a matrix;
encoding the matrix with contextual information using a transformer encoder; and
applying a linear transformation to the encoded matrix using a classification layer.
(similarly claim 7)
8. The computer-implemented method of claim 1, wherein the applying of the masked language model comprises,
for each of the tokens in the sequence of tokens:
masking the token,
sending the sequence of tokens including the masked token though an embedding layer to generate a 2D matrix,
encoding the 2D matrix with contextual information using a transformer encoder, and
applying a linear transformation to the encoded 2D matrix using a classification layer,
wherein the sequence of vectors are based on the linearly transformed encoded 2D matrix calculated for each of the tokens.
Claims 1-20 of instant application 19/261,618 are rejected on the ground of nonstatutory double patenting over claims 1, 8, 9, 12, and 17 of U.S. Patent No. 11,327,960 because claims 1, 8, 9, 12, and 17 of U.S. Patent No. 11,327,960 teaches all the limitations of claims 1-20 of the instant application.
Since the limitations of claims 1-20 of the instant application are broader than and/or similar to the limitations in claims 1, 8, 9, 12, and 17 of U.S. Patent No. 11,327,960 (as shown in the table mapping above), the claims are not patentably distinct (see In re Goodman).
Claims 1-20 of instant application 19/261,618 are rejected on the ground of nonstatutory double patenting over claims 1, 5, 9, 13, 15, and 19 of U.S. Patent No. 12,361,216, since the claims, if allowed, would improperly extend the “right to exclude” already granted in the patent.
The subject matter claimed in the instant application is fully disclosed in the
patent and is covered by the patent since the patent and the application are claiming
common subject matter, as follows:
US Application 19/261,618
US Patent 12,361,213
10. A method comprising:
receiving a text string including raw data for a transaction;
extracting contextual information from the text string based on at least:
tokenizing the text string, and
applying a masked language model to the tokenized text string;
(see claims 11-14 below)
(see claim 16 below)
(see claim 15 below)
identifying entities within the text string based on the contextualized information; and
generating normalized transaction data including the identified entities.
(see claim 17 below)
11. The method of claim 10, wherein tokenizing the text string comprises:
generating a sequence of tokens.
12. The method of claim 11, wherein applying the masked language model comprises:
generating a sequence of vectors.
13. The method of claim 12, wherein individual vectors of the vectors correspond to individual tokens in the sequence of vectors.
14. The method of claim 12, wherein the sequence of vectors is encoded with information regarding one or more surrounding tokens in the sequence of tokens.
15. The method of claim 12, wherein extracting the contextual information is further based on bidirectionally parsing the sequence of vectors to identify tokens indicative of entity information, and wherein identifying entities is based at least in part on the entity information.
16. The method of claim 11, wherein applying the masked language model comprises:
masking a token of the sequence of tokens;
sending the sequence of tokens including the masked token through an embedding layer to generate a matrix;
encoding the matrix with contextual information using a transformer encoder; and
applying a linear transformation to the encoded matrix using a classification layer.
17. The method of claim 10, wherein generating the normalized transaction data includes generating at least one of:
location information,
an identified candidate merchant,
generated merchant information,
category information, or
other metadata related to the transaction.
(similarly claims 1-9 and 18-20)
1. A computer-implemented method comprising, by one or more hardware processors executing program instructions:
receiving a text string including raw data for a transaction;
tokenizing the text string to create a sequence of tokens;
applying a masked language model to the sequence of tokens to generate a sequence of vectors, each of the vectors corresponding to one of the tokens and being encoded with information regarding one or more of surrounding tokens in the sequence of tokens,
wherein applying the masked language model comprises:
masking a token of the sequence of tokens,
sending the sequence of tokens including the masked token through an embedding layer to generate a 2D matrix,
encoding the 2D matrix with contextual information using a transformer encoder, and
applying a linear transformation to the encoded 2D matrix using a classification layer,
wherein the sequence of vectors are based on the linearly transformed encoded 2D matrix calculated for each of the tokens;
bidirectionally parsing the sequence of vectors to identify tokens indicative of entity information;
generating entity information using the tokens indicative of entity information; and
generating normalized transaction data including the generated entity information.
(similarly claims 5, 9, 13, 15, and 19)
Claims 1-20 of instant application 19/261,618 are rejected on the ground of nonstatutory double patenting over claims 1, 5, 9, 13, 15, and 19 of U.S. Patent No. 12,361,213 because claims 1, 5, 9, 13, 15, and 19 of U.S. Patent No. 12,361,213 teaches all the limitations of claims 1-18 of the instant application.
Since the limitations of claims 1-20 of the instant application are broader than and/or similar to the limitations in claims 1, 5, 9, 13, 15, and 19 of U.S. Patent No. 12,361,213 (as shown in the table mapping above), the claims are not patentably distinct (see In re Goodman).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
At step 1:
Independent claims 1, 10, and 18 respectively recite a device, a method, and a non-transitory computer-readable medium, which are directed to a statutory category such as a process, machine, or an article of manufacture.
At step 2A, prong one:
Independent claim 1 and similarly independent claims 10 and 18 recites the limitations:
“extract contextual information from the text string based on at least: tokenizing the text string, and applying a masked language model to the tokenized text string”;
A person can mentally or using a pen and paper extract contextual information from a text string based on at least mentally or using a pen and paper tokenizing the text string, and mentally or using a pen and paper applying a masked language model to the tokenized text string.
“identify entities within the text string based on the contextualized information”;
A person can mentally or using a pen and paper identify entities within a text string based on contextualized information.
“generate normalized transaction data including the identified entities”;
A person can mentally or using a pen and paper generate normalized transaction data including identified entities.
The limitations, as recited above, are processes that, under their broadest reasonable interpretation, cover steps that can be performed in the human mind or by a human using a pen and paper, but for recitation of generic computer components.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claims recite an abstract idea.
At step 2A, prong two:
This judicial exception is not integrated into a practical application.
Independent claim 1 and similarly independent claims 10 and 18 recites the limitations:
“receive a text string including raw data for a transaction”, which is a step of receiving data. The step is recited at a high level of generality, and amounts to mere data gathering, which is a form of insignificant extra-solution activity (MPEP 2106.05(g)).
The additional elements “a device, comprising: one or more memories; and one or more processors, coupled to the one or more memories, configured to:” in the steps in claim 1 are recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using generic computer components.
The additional elements “a non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the device to:” in the steps in claim 18 are recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using generic computer components.
Accordingly, the additional elements, individually or in combination, do not
integrate the abstract idea into a practical application, even viewing the claims a whole,
because it does not impose any meaningful limits on practicing the abstract idea.
At step 2B:
Independent claims 1, 10, and 18 recites the same additional elements as
identified in step 2A prong two above. These additional elements are not sufficient to
amount to significantly more than the judicial exception.
Independent claim 1 and similarly independent claims 10 and 18 recites the limitations:
“receive a text string including raw data for a transaction”, which is a step of receiving data, and is recognized as a well understood, routine, and conventional activity within the field of computer functions as an element of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i)).
Accordingly, the additional limitations are not sufficient to amount to significantly more than the judicial exception. Therefore, the claims are directed to an abstract idea and are not patent eligible.
Dependent claim 2 and similarly dependent claims 11 and 19 recites additional limitations, such as:
“…to tokenize the text string, are configured to generate a sequence of tokens”.
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1, 10, and 18, because a person can mentally or using a pen and paper tokenize a text string by mentally or using a pen and paper generating a sequence of tokens, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 3 and similarly dependent claims 12 and 20 recites additional limitations, such as:
“to apply the masked language model, are configured to generate a sequence of vectors”.
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1, 10, and 18, because a person can mentally or using a pen and paper apply a masked language model by mentally or using a pen and paper generating a sequence of vectors, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 4 and similarly dependent claim 13 recites additional limitations, such as:
“wherein a vector of the sequence of vectors corresponds to individual tokens in the sequence of vectors”.
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1 and 10, because a person can mentally or using a pen and paper generate a sequence of vectors where a vector of the sequence of vectors corresponds to individual tokens in the sequence of vectors, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 5 and similarly dependent claim 14 recites additional limitations, such as:
“wherein the sequence of vectors is encoded with information regarding one or more surrounding tokens in the sequence of tokens”.
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1 and 10, because a person can mentally or using a pen and paper encode a sequence of vectors with information regarding one or more surrounding tokens in a sequence of tokens, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 6 and similarly dependent claim 15 recites additional limitations, such as:
“wherein extracting the contextual information is further based on bidirectionally parsing the sequence of vectors to identify tokens indicative of entity information, and wherein identifying entities is based at least in part on the entity information”.
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1 and 10, because a person can mentally or using a pen and paper extract contextual information by bidirectionally parsing a sequence of vectors to identify tokens indicative of entity information, and the person can mentally or using a pen and paper identify entities based at least in part on the entity information, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 7 and similarly dependent claim 16 recites additional limitations, such as:
to apply the masked language model, are configured to:
“mask a token of the sequence of tokens”;
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1 and 10, because a person can mentally or using a pen and paper mask a token of a sequence of tokens, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
“send the sequence of tokens including the masked token through an embedding layer to generate a matrix”, which is a step of sending or transmitting data.
At step 2A prong two, the step is recited at a high level of generality, and amounts to mere data gathering, which is a form of insignificant extra-solution activity.
At step 2B, the step is recognized as a well understood, routine, and conventional activity within the field of computer functions as an element of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i)).
“encode the matrix with contextual information…”;
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1 and 10, because a person can mentally or using a pen and paper encode a matrix with contextual information, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
“apply a linear transformation to the encoded matrix…”.
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1 and 10, because a person can mentally or using a pen and paper apply a linear transformation to an encoded matrix, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
The additional elements “using a transformer encoder” and “using a classification layer” in the steps are recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using generic computer components.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 8 and similarly dependent claim 17 recites additional limitations, such as:
“wherein generating the normalized transaction data includes generating at least one of: location information, an identified candidate merchant, generated merchant information, category information, or other metadata related to the transaction”.
These limitations are directed to the same abstract idea under the mental processes grouping as independent claims 1 and 10, because a person can mentally or using a pen and paper generate normalized transaction data that includes generating at least one of: location information, an identified candidate merchant, generated merchant information, category information, or other metadata related to a transaction, and because the limitations do not recite any additional elements that are sufficient to amount to significantly more.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 9 recites additional limitations, such as:
“wherein the raw data is received from one or more external financial account systems”, which is a step of receiving data.
At step 2A prong two, the step is recited at a high level of generality, and amounts to mere data gathering, which is a form of insignificant extra-solution activity.
At step 2B, the step is recognized as a well understood, routine, and conventional activity within the field of computer functions as an element of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i)).
The additional elements “from one or more external financial account systems” in the steps are recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using generic computer components.
Accordingly, the additional elements, individually or in combination, do not integrate the abstract idea into a practical application, even viewing the claims a whole, because it does not impose any meaningful limits on practicing the abstract idea.
Accordingly, dependent claims 2-9, 11-17, 19 and 20 are also directed to abstract idea without significantly more and are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Torres (US Pub 2021/0209513) in view of Weilert (US Pub 2020/0300656).
With respect to claim 1, Torres discloses a device, comprising:
one or more memories (Torres in [0063] and [0069] discloses computing device including one or more memories and one or more processors); and
one or more processors, coupled to the one or more memories (Torres in [0063] and [0069] discloses computing device including one or more memories and one or more processors), configured to:
receive a text string… (Torres in [0038] discloses receiving input data including input text and multiple NLP task identifiers; here Torres does not explicitly disclose receive a text string including raw data for a transaction, but the Weilert reference discloses the feature, as discussed below);
extract contextual information from the text string (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like) based on at least:
tokenizing the text string (Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors), and
applying a masked language model to the tokenized text string (Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector);
identify entities within the text string based on the contextualized information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc., included in a collection of documents using natural language processing predictions generated by an NLP model that performs named entity recognition tasks, generating a normalized natural NLP prediction; Torres in [0056] and [0061] discloses named entity recognition predictions for input text are displayed to the user, to generate named entity recognition prediction, a task specific output layer having one or more classification layers trained on named entity recognition tasks are included in a fine-tuned model for named entity recognitions); and
generate normalized…data including the identified entities (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc., included in a collection of documents using natural language processing predictions generated by an NLP model that performs named entity recognition tasks, generating a normalized natural NLP prediction; Torres in [0056] and [0061] discloses named entity recognition predictions for input text are displayed to the user, to generate named entity recognition prediction, a task specific output layer having one or more classification layers trained on named entity recognition tasks are included in a fine-tuned model for named entity recognitions; here Torres does not explicitly disclose transaction data, but the Weilert reference discloses the feature, as discussed below).
Torres discloses receiving a text string including information on companies, vendors, products etc., however, Torres does not explicitly disclose:
receive a text string including raw data for a transaction;
The Weilert reference discloses receiving a text string including raw data for a transaction (Weilert in [0026], [0033], and [0062] discloses receiving transaction data and metadata associated with the transaction, metadata including location information; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination; Weilert in [0075] discloses a model can compare string similarity, comparison can be at the tokens level and character level, a machine learning model can use string similarity scores).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having the teachings of Torres and Weilert, to have combined Torres and Weilert. The motivation to combine Torres and Weilert would be to resolve ambiguities utilizing data and metadata collected or inferred from transactions (Weilert: [0002] and [0004]).
With respect to claim 2, Torres in view of Weilert discloses the device of claim 1, wherein the one or more processors, to tokenize the text string, are configured to generate a sequence of tokens (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 3, Torres in view of Weilert discloses the device of claim 2, wherein the one or more processors, to apply the masked language model, are configured to generate a sequence of vectors (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 4, Torres in view of Weilert discloses the device of claim 3, wherein a vector of the sequence of vectors corresponds to individual tokens in the sequence of vectors (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 5, Torres in view of Weilert discloses the device of claim 3, wherein the sequence of vectors is encoded with information regarding one or more surrounding tokens in the sequence of tokens (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 6, Torres in view of Weilert discloses the device of claim 3, wherein extracting the contextual information is further based on bidirectionally parsing the sequence of vectors to identify tokens indicative of entity information, and wherein identifying entities is based at least in part on the entity information (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 7, Torres in view of Weilert discloses the device of claim 2, wherein the one or more processors, to apply the masked language model, are configured to:
mask a token of the sequence of tokens (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens);
send the sequence of tokens including the masked token through an embedding layer to generate a matrix (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens);
encode the matrix with contextual information using a transformer encoder (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens); and
apply a linear transformation to the encoded matrix using a classification layer (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens).
With respect to claim 8, Torres in view of Weilert discloses the device of claim 1, wherein generating the normalized transaction data includes generating at least one of:
location information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination),
an identified candidate merchant (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination),
generated merchant information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination),
category information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination), or
other metadata related to the transaction (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination).
With respect to claim 9, Torres in view of Weilert discloses the device of claim 1, wherein the raw data is received from one or more external financial account systems (Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction, transactional data points are transmitted over a network between terminals and a financial institution; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination).
With respect to claim 10, Torres discloses a method comprising:
receiving a text string… (Torres in [0038] discloses receiving input data including input text and multiple NLP task identifiers; here Torres does not explicitly disclose receive a text string including raw data for a transaction, but the Weilert reference discloses the feature, as discussed below);
extracting contextual information from the text string (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like) based on at least:
tokenizing the text string (Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors), and
applying a masked language model to the tokenized text string (Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector);
identifying entities within the text string based on the contextualized information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc., included in a collection of documents using natural language processing predictions generated by an NLP model that performs named entity recognition tasks, generating a normalized natural NLP prediction; Torres in [0056] and [0061] discloses named entity recognition predictions for input text are displayed to the user, to generate named entity recognition prediction, a task specific output layer having one or more classification layers trained on named entity recognition tasks are included in a fine-tuned model for named entity recognitions); and
generating normalized…data including the identified entities (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc., included in a collection of documents using natural language processing predictions generated by an NLP model that performs named entity recognition tasks, generating a normalized natural NLP prediction; Torres in [0056] and [0061] discloses named entity recognition predictions for input text are displayed to the user, to generate named entity recognition prediction, a task specific output layer having one or more classification layers trained on named entity recognition tasks are included in a fine-tuned model for named entity recognitions; here Torres does not explicitly disclose transaction data, but the Weilert reference discloses the feature, as discussed below).
Torres discloses receiving a text string including information on companies, vendors, products etc., however, Torres does not explicitly disclose:
receive a text string including raw data for a transaction;
The Weilert reference discloses receiving a text string including raw data for a transaction (Weilert in [0026], [0033], and [0062] discloses receiving transaction data and metadata associated with the transaction, metadata including location information; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination; Weilert in [0075] discloses a model can compare string similarity, comparison can be at the tokens level and character level, a machine learning model can use string similarity scores).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having the teachings of Torres and Weilert, to have combined Torres and Weilert. The motivation to combine Torres and Weilert would be to resolve ambiguities utilizing data and metadata collected or inferred from transactions (Weilert: [0002] and [0004]).
With respect to claim 11, Torres in view of Weilert discloses the method of claim 10, wherein tokenizing the text string comprises:
generating a sequence of tokens (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 12, Torres in view of Weilert discloses the method of claim 11, wherein applying the masked language model comprises:
generating a sequence of vectors (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 13, Torres in view of Weilert discloses the method of claim 12, wherein individual vectors of the vectors correspond to individual tokens in the sequence of vectors (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 14, Torres in view of Weilert discloses the method of claim 12, wherein the sequence of vectors is encoded with information regarding one or more surrounding tokens in the sequence of tokens (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 15, Torres in view of Weilert discloses the method of claim 12, wherein extracting the contextual information is further based on bidirectionally parsing the sequence of vectors to identify tokens indicative of entity information, and wherein identifying entities is based at least in part on the entity information (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 16, Torres in view of Weilert discloses the method of claim 11, wherein applying the masked language model comprises:
masking a token of the sequence of tokens (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens);
sending the sequence of tokens including the masked token through an embedding layer to generate a matrix (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens);
encoding the matrix with contextual information using a transformer encoder (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens); and
applying a linear transformation to the encoded matrix using a classification layer (Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0042], [0049], and [0061] discloses base parameters generated by the encoder fed into one or more classification layer to perform an NLP task on input data, output layer includes one or more classification layer than transforms vector data output by the encoded into a form that generates a prediction, output layer having one or more classification layers trained on named entity recognition tasks; Torres in [0045] and [0049] discloses task specific output layer generating a prediction by computing dot product between a final output matrix generated by an encoder with its weight matrix, adding its bias vector, and passing the output to transform the vector values to work values, one dimension of output matrix corresponds to individual input tokens, output layer multiplying output vectors by its weight matrix to transform the encoder output matrix into a matrix with an output size equivalent to that of the input sequence of tokens).
With respect to claim 17, Torres in view of Weilert discloses the method of claim 10, wherein generating the normalized transaction data includes generating at least one of:
location information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination),
an identified candidate merchant (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination),
generated merchant information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination),
category information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination), or
other metadata related to the transaction (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc.; Weilert in [0004] and [0026] discloses transaction metadata including location and category information; Weilert in [0033] and [0062] discloses receiving transaction data and metadata associated with the transaction; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination).
With respect to claim 18, Torres discloses a non-transitory computer-readable medium storing a set of instructions (Torres in [0063] and [0069] discloses computing device including one or more memories and one or more processors, including one or more computer-readable mediums storing instructions executed by processor), the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device (Torres in [0063] and [0069] discloses computing device including one or more memories and one or more processors, including one or more computer-readable mediums storing instructions executed by processor) to:
receive a text string… (Torres in [0038] discloses receiving input data including input text and multiple NLP task identifiers; here Torres does not explicitly disclose receive a text string including raw data for a transaction, but the Weilert reference discloses the feature, as discussed below);
extract contextual information from the text string (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like) based on at least:
tokenizing the text string (Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors), and
applying a masked language model to the tokenized text string (Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector);
identify entities within the text string based on the contextualized information (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc., included in a collection of documents using natural language processing predictions generated by an NLP model that performs named entity recognition tasks, generating a normalized natural NLP prediction; Torres in [0056] and [0061] discloses named entity recognition predictions for input text are displayed to the user, to generate named entity recognition prediction, a task specific output layer having one or more classification layers trained on named entity recognition tasks are included in a fine-tuned model for named entity recognitions); and
generate normalized…data including the identified entities (Torres in [0009] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing; Torres in [0015] and [0032] discloses generating a list of entities, such as companies, vendors, products etc., included in a collection of documents using natural language processing predictions generated by an NLP model that performs named entity recognition tasks, generating a normalized natural NLP prediction; Torres in [0056] and [0061] discloses named entity recognition predictions for input text are displayed to the user, to generate named entity recognition prediction, a task specific output layer having one or more classification layers trained on named entity recognition tasks are included in a fine-tuned model for named entity recognitions; here Torres does not explicitly disclose transaction data, but the Weilert reference discloses the feature, as discussed below).
Torres discloses receiving a text string including information on companies, vendors, products etc., however, Torres does not explicitly disclose:
receive a text string including raw data for a transaction;
The Weilert reference discloses receiving a text string including raw data for a transaction (Weilert in [0026], [0033], and [0062] discloses receiving transaction data and metadata associated with the transaction, metadata including location information; Weilert in [0051] discloses determining similarities/matches between transaction metadata and metadata of at least one point of interest, metadata fields like raw transaction string including merchant name, merchant id, address used to make the determination; Weilert in [0075] discloses a model can compare string similarity, comparison can be at the tokens level and character level, a machine learning model can use string similarity scores).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having the teachings of Torres and Weilert, to have combined Torres and Weilert. The motivation to combine Torres and Weilert would be to resolve ambiguities utilizing data and metadata collected or inferred from transactions (Weilert: [0002] and [0004]).
With respect to claim 19, Torres in view of Weilert discloses the non-transitory computer-readable medium of claim 18, wherein the one or more instructions, that cause the device to tokenize the text string, cause the device to generate a sequence of tokens (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
With respect to claim 20, Torres in view of Weilert discloses the non-transitory computer-readable medium of claim 18, wherein the one or more instructions, that cause the device to apply the masked language model, cause the device to generate a sequence of vectors (Torres in [0009] and [0040] discloses using machine learning models for named entity recognition to automate information extraction from documents and other aspects of document processing, text data for training datasets extracted from a corpus of documents; Torres in [0026] discloses a model can be a language representation model, the model can be a bidirectional encoder representations from transformers model, processing text using a transformer, such as an attention mechanism that learns contextual relations between words and/or sub-words in a text sequence, processing text to aggregate language modeling knowledge, such as insights about the meaning of words, structure of text, context provided by neighboring words, and the like, generating a fine-tuned model from the model, the fine-tuned model specific to a particular category of NLP task, such as sentiment analysis, topic and intent classification, named entity recognition, and the like; Torres in [0027] and [0028] discloses a tokenizer converting text input into contextual embedding vectors, an encoder turning the contextual embedding vectors into a language representation, a task specific output layer producing a prediction for a particular task, input text broken down into a sequence of tokens using a tokenization process, output of the tokenization is a sequence of words or subwords which are mapped to embedding vectors; Torres in [0029] discloses predict the value of a masked word in a sequence based on the context provided by the words around the masked word; Torres in [0044] discloses training a model that predicts original identity of masked words, tokens included in training data are masked, training the model to understand the context of masked words based on the words in the text sequence surrounding the masked words, word context incorporated into training data by adding a positional embedding indication the position of the word within its text to each word token, the positional embedding for each token added to the token’s input vector).
Remarks
The relevant prior art of record that are not used in claim rejections but are pertinent to the claims or disclosure are:
Kota (US Pub 2021/0357784), which discloses a bidirectional transformer encoder that reads a sequence of information to learn context based on all of its surroundings, embedding an inputted sequence of tokens into vectors, replacing vectors with a mask token, predicting original value of masked pieces of information, multiplying vectors by an embedding matrix and transforming them into a dimension, and calculating the probability of each piece of information.
Ganu (US Pub 2020/0394263), which discloses pre-processing texts by replacing terms with tokens denoting a plurality of entities, receiving a document as input and outputting a vector which embeds the document, parsing raw stream of data in the context of organizing patterns into a stream of tokens, and generating vectors corresponding to the tokens.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REZWANUL MAHMOOD whose telephone number is (571)272-5625. The examiner can normally be reached M-F 9-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J. Lo can be reached at 571-272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.M/Examiner, Art Unit 2159 /AMRESH SINGH/Primary Examiner, Art Unit 2159
Prosecution Insights