DETAILED ACTION
This final office action is responsive to application 18/058,242 with applicant’s amendments and request for reconsideration as submitted 25 Mar. 2026.
Claim status is currently pending and under examination for claims 1-20 of which amended claims are 1, 11 and 20 further corresponding to the independent claims.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
Applicant’s responsive remarks filed 03/25/26 are considered together with amendments with regard to the remaining issues which are addressed as follows.
Rejection under 35 U.S.C. 101 as being directed to an abstract idea without significantly more is maintained herein. Applicant’s brief remarks on eligibility are not found persuasive for at least the following reasons. Traversal is noted in light of amendment and that it is part of a system which also has some machine learning that is collectively improved.
However, examiner respectfully disagrees because amendments embellish the abstract idea of mathematical concepts where tensor product formula includes scalar probability values in probability space. Scalar probabilities include real numbers used in performing the tensor product formula. Evidence to support this functionality as math is provided by Manaswi1 demonstrates that scalars in a space of probabilistic values to represent tensors are simply mathematical objects. Further, as set forth per MPEP 2106.05(a)(II) “it is important to keep in mind that an improvement in the abstract idea itself …is not an improvement in technology.” Collective improvement is not sufficiently specific to distill the technical basis for better machine learning. Machine learning was not treated as the abstract idea, this includes generate an encoder… multilayer bidirectional transformer (e.g. BERT as addressed in rejection) which is not part of the amended limitation and such that post-processing may be performed on an already established machine learning model. Doing so does not demonstrate inventive concept as it simply uses that which is known with an abstract idea. Adding to the abstract idea with mental process to include judging relevance or its corollary and labeling data manually by hand, the remarks do not point out how or why these functionalities preclude mental performance in a reasoning stage of the computational pipeline. Therefore, it is not seen that the claims avoid the identified abstract ideas nor that additional elements amount to significantly more or serve to clarify real-world, practical application that is specifically pointed out or traversed. After preponderance of the evidence, balance favors a finding of ineligibility under 35 U.S.C. 101.
Rejection under 35 U.S.C. 103 as being obvious over a combination of prior art is updated to reflect amendments. Remarks point to language of the amended limitation whereby newly identified reference Jiang is applied to meet the scope of claim. Particularly, tensor product representations (TPR) is detailed for the purpose of enriching transformer encoder models. In view of the new finding, an updated rejection follows as detailed below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. In determining whether the claims are subject matter eligible, the examiner applies guidance set forth under MPEP 2106. The response to remarks above are incorporated herein.
Step 1: Is the claim to a process, machine, manufacture, or composition of matter? Yes—all claims fall within one of the four statutory categories: claims 1-10 and 20 are a system/machine, and claims 11-19 are a method/process. Thus, the analysis should proceed per MPEP 2106.03.
Step 2A, prong one: Does the claim recite an abstract idea, law of nature or natural phenomenon? Yes—the claims, under the broadest reasonable interpretation, recites an abstract idea. In this case, claims fall within the enumerated grouping of abstract idea being “Mathematical Concepts” and/or “Mental Processes”, but for the recitation of generic computer components. More particularly, claims recite:
Claims 1 and 11
“generate token sequences based on the input text… set of ground truth token sequences which are labeled irrelevant for irrelevant text data which belongs in an irrelevant classification group, and labeled relevant for relevant text data which belongs in a relevant classification group” (Mental Process, e.g. parse document into paragraphs or topics and label (ir)relevance manually as mental judgment)
“input the encoder output into a relevant linear function and an irrelevant linear function to linearly transform the encoder output, and output a transformed relevant output and a transformed irrelevant output, respectively” (Math function as linear algebra functions)
“compute a relevancy probability and a first irrelevance probability, respectively, by inputting the transformed relevant output and the transformed irrelevant output into a sigmoid function, the relevance probability being a probability that the token sequence belongs in the relevant classification group, the first irrelevance probability being a probability that the token sequence belongs in the irrelevant classification group” (Mathematical Calculation to compute probabilities with sigmoid function)
“compute a second irrelevance probability by inputting the relevance probability and the first irrelevance probability into a tensor product formula operating in probability space, the relevance probability and the first irrelevance probability being scalar probability values in the probability space” (Math Calculation and Formula)
Claim 20- further comprising
“classification group for tax law articles which contain only income tax information or property tax information and no other tax information… classification group for irrelevant tax law articles which contain only administrative tax information and no other tax information” and “classification group for relevant tax law articles which contain tax information other than administrative tax information, income tax information, or property tax information” (Mental Process, observe tax documents for (ir)relevance judgment)
“compute a fourth irrelevance probability by inputting the relevance probability, the first irrelevance probability, and the third irrelevance probability into a product formula” (Mathematical Calculation or Formula)
Focus of the claim concern computing/calculating probabilities with a formula for (ir)relevance. The text of claim 1 is elaborated in claim 20 as tax law articles. When read in light of the instant specification, mathematical formula for probabilistic computations are detailed per [0022-23], [0033-34]. As such, the claims are drawn to mathematical processes which is an abstract idea enumerated under MPEP 2106.04(a)(2).
Step 2A, prong two: Does the claim recite additional elements that integrate the judicial exception into a practical application? No—a practical application is not integrated by the judicial exception because the additional elements are as follows:
“system, comprising: a processor and memory of a computing device, the processor being configured to execute a program using portions of memory” MPEP 2106.05(f) merely uses a computer as a tool to perform an abstract idea, recited at high level of generality.
“receive input text” and “receive input text from the tax law articles” MPEP 2106.05(g) adding insignificant extra-solution activity to the judicial exception, e.g. mere data gathering
“generate an encoder output by inputting the token sequences into a multi-layer bidirectional transformer to transform the token sequences into the encoder output, the multi-layer bidirectional transformer being trained using binary cross-entropy loss”
MPEP 2106.05(h) generally linking the use of the judicial exception to a particular technological environment or field of use
“generate and output an irrelevancy score for the input text based on the second irrelevance probability” MPEP 2106.05(g) adding insignificant extra-solution activity to the judicial exception, e.g. necessary data outputting
Balance of the claim concerns computer implementation, input and output, and an encoder. When read in light of specification [0009], encoder is preferably BERT which is an already established language model that is analogous to ELMo or GPT (which is not abstract, but not inventive concept as of applicant’s effective filing date circa Nov 2022). Applying an off-the-shelf AI encoder model does not meaningfully limit the claim. The encoder having input and output is an ordinary capacity and mere use of a computer is inadequate to cure eligibility. As set forth under MPEP 2106.04(a)(2) “A claim that requires a computer may still recite a mental process” and noting MPEP 2106.05(h) “a competent claim drafter could attach a similar type of limitation to almost any mathematical formula” Accordingly, the claim remains drawn to the judicial exception and the identified additional elements fail to integrate the abstract idea into a practical application.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No—the claims do not include additional elements that amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea in to a practical application, the additional elements are identified with respect to MPEP 2106.05 and do not demonstrate significantly more. Particularly, additional elements comprise:
“system, comprising: a processor and memory of a computing device, the processor being configured to execute a program using portions of memory” MPEP 2106.05(f) merely uses a computer as a tool to perform an abstract idea, recited at high level of generality. Particularly, a general purpose computer does not qualify as a particular machine under MPEP 2106.05(b).
“receive input text” and “receive input text from the tax law articles” MPEP 2106.05(g) adding insignificant extra-solution activity to the judicial exception e.g. mere data gathering. Particularly, said extra-solution activity is a well-understood, routine and conventional (WURC) activity under MPEP 2106.05(d)(II)(i) “The courts have recognized the following computer functions as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity: i. Receiving or transmitting data over a network, e.g., using the Internet to gather data”
“generate an encoder output by inputting the token sequences into a multi-layer bidirectional transformer to transform the token sequences into the encoder output, the multi-layer bidirectional transformer being trained using binary cross-entropy loss”
MPEP 2106.05(h) generally linking the use of the judicial exception to a particular technological environment or field of use. Particularly, encoder such as BERT [0009] was originally authored by Devlin of Google (arXiv: 1810.04805v1, 2018) google scholar shows 150,000+ citations thus being well-known as it is extensively studied in the relevant art. Known variants include LegalBERT e.g. Holzenberger arXiv: 2005.05257v3 Figs 1-2 [Sect5.1] and/or Sabapathy US2023/0419042A1 Fig 5 [0078] relevant use cases as supplemental supporting evidence.
“generate and output an irrelevancy score for the input text based on the second irrelevance probability” MPEP 2106.05(g) adding insignificant extra-solution activity to the judicial exception, e.g. necessary data outputting. Particularly, said extra-solution activity is a well-understood, routine and conventional (WURC) activity under MPEP 2106.05(d)(II) “The courts have recognized the following computer functions as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity: transmitting data over a network, e.g., using the Internet” or “retrieving information in memory… presenting offers”
Significantly more is not established by the additional elements for the reasons noted, especially in consideration of the evidentiary support. If claim language provides only a result-oriented solution, with insufficient detail for how a computer accomplishes it, then claims do contain an inventive concept. Taken alone, their additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Considered as a whole, looking at the limitations as an ordered combination does not elevate the claims to satisfy eligibility. The claims present the skilled artisan with abstract relevancies, math tensor product formula and linear functions that fairly draw the claims to a judicial exception with the addition of using some known transformer model which generally links the judicial exception to machine learning such that the claim as a whole is no more than a drafting effort to monopolize the judicial exception. Their collective functions merely provide conventional computer implementation.
For the above reasons, the claims are not patent eligible. This rejection applies equally to independent claims 1, 11 and 20 as well to dependent claims 2-10 and 12-19. Dependent claims when analyzed as a whole are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitations fail to establish that the claims are not directed to an abstract idea, as they recite further embellishment of the judicial exception, or that they include additional elements which integrate the judicial exception into a practical application or amount to significantly more.
Dependent claims 2 and 12 disclose wherein token sequences are generated so that first token is a predetermined classification token, and token sequences are separated by separation token. This is considered part of the abstract idea to include mental process by evaluating token data. For example, segmenting sentences by beginning first word, phrase word, and last word or end of sentence with commas separating words of a sentence is a tokenization or parsing process which can be performed by hand aided by template or pencil and paper as part of a mental process. A class token may simply designate a topic and the use of ‘or’ in the alternative is not limiting but may be considered as adding words to build a sentence. There are no additional elements.
Dependent claims 3 and 13 disclose wherein the transformer performs dropout in which vectors of tokens are dropped and only output of classification and separation tokens are encoded. This is considered as an additional element which amounts to adding insignificant extra-solution activity to the judicial exception under MPEP 2106.05(g). Particularly, said extra-solution activity is a well-understood, routine and conventional activity as evidenced by Hou arXiv: 2203.13240v1 at [P.1 ¶4] extensive studies have been performed on dropout in the field of BERT encoder transformers. In view of the foregoing, the additional elements are not an inventive concept and fail to integrate the judicial exception into a practical application or amount to significantly more.
Dependent claims 4 and 14 disclose wherein a number of limitations are performed, limitations which are considered part of the abstract idea are as follows:
“the irrelevant classification group is a first irrelevant classification group; the irrelevant linear function is a first irrelevant linear function; the transformed irrelevant output is a transformed first irrelevant output; the encoder output is inputted into the first irrelevant linear function, and a second irrelevant linear function to linearly transform the encoder output, and output the transformed first irrelevant output and a transformed second irrelevant output, respectively; a third irrelevance probability is computed by inputting the transformed second irrelevant output into the sigmoid function, the third irrelevance probability being a probability that the token sequence belongs in the second irrelevant classification group; a fourth irrelevance probability is computed by inputting the relevance probability, the first irrelevance probability, and the third irrelevance probability into a product formula; the product formula is SIP + (RP * TIP), where SIP stands for the second irrelevance probability, RP stands for the relevance probability, and TIP stands for the third irrelevance probability;”
These limitations embellish mathematical calculations and formula as noted above and which is an enumerated grouping of abstract idea under MPEP 2106.04(a)(2). Additional elements comprise:
“the multi-layer bidirectional transformer is trained using binary cross entropy loss on the set of ground truth token sequences which are labeled first irrelevant for irrelevant text data which belongs in the first irrelevant classification group, and labeled second irrelevant for irrelevant text data which belongs in a second irrelevant classification group; and the irrelevancy score for the input text is generated and outputted based on the fourth irrelevance probability.”
These additional elements regard transformer training on binary cross entropy and outputting the generated probability. As was similarly noted in rejection of claim 1, these additional elements fall under MPEP 2105.05(h)(g) as generally linking the use of the judicial exception to a particular technological environment or field of use and adding insignificant extra-solution activity. Particularly, the necessary data output is well-understood, routine and conventional activity under MPEP 2106.05(d) and transformer is widely cited as evidenced by Devlin and/or Holzenberger. Accordingly, these additional elements do not integrate the judicial exception into a practical application or amount to significantly more.
Dependent claims 5 and 15 disclose wherein the tensor product formula is as specified. This is considered part of the abstract idea as a mathematical formula. There are no additional elements.
Dependent claim 6 and 16 discloses wherein the binary cross entropy loss is calculated for the first irrelevancy probability and the second irrelevance probability. This is considered part of the abstract idea being mathematical calculations. There are no additional elements.
Dependent claim 7 and 17 disclose wherein the sigmoid function is a logistic sigmoid function to determine the relevance probability, the first irrelevance probability, and the second irrelevance probability. This is considered part of the abstract idea being mathematical calculations to further specify the sigmoid function for determining probabilities.
Dependent claims 8 and 18 disclose wherein the irrelevancy score includes a classification of token sequence as relevant or irrelevant based on second irrelevance probability. This is considered part of the abstract idea being math and/or mental process. For example, classifying by scoring probability can be simple ranking by ascending/descending order, and/or calculated as a softmax function as described per specification [0017]. There are no additional elements.
Dependent claims 9 and 19 disclose wherein token sequence is categorized as irrelevant if the second irrelevance probability is greater than threshold. This is considered part of the abstract idea being math and/or mental process. The threshold probability can be a cutoff or max/min operand for evaluating or judging categories to sort or rank information. There are no additional elements.
Dependent claim 10 discloses wherein the transformer is a BERT encoder. This is considered as an additional element. The additional element BERT encoder was invented by Devlin of Google in 2018 as was already noted, and therefore is demonstrably not an inventive concept. The additional element amounts to generally linking the use of the judicial exception to a particular technological environment or field of use under MPEP 2106.05(h) and which does not meaningfully limit the claim under MPEP 2106.05(e) as it merely applies an already established model to the judicial exception. Therefore, the additional element does not integrate the judicial exception into a practical application or amount to significantly more.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 6, 8-12, 16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over: Gu et al., “Domain-Specific Language Model Pre-Training for Korean Tax Law Classification” hereinafter Gu, in view of Chen et al., “Extreme Multi-Label Classification with Label Masking for Product Attribute Value Extraction” hereinafter Chen, as evidenced by Sabapathy et al., US PG Pub No 2023/0419042A1 hereinafter Sabapathy, and further in view of Jiang et al., “Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization” hereafter Jiang (arXiv: 2106.01317v1).
With respect to claim 1, Gu teaches:
A computing system, comprising: a processor and memory of a computing device, the processor being configured to execute a program using portions of memory to: {Gu [P.46348 Sect IV.A] discloses system with CPU/GPU processors and Pytorch program for requisite memory of database shown Fig 1 to convey computer implementation of KTL-BERT (Korean tax law BERT bidirectional encoder repr. transformer)}
receive input text {Gu [P.46345 Sect.III] discloses collecting and preprocessing of tax law dataset with vocabulary for input, e.g. input sentences [P.46346 Last¶]};
generate token sequences based on the input text {Gu Figs 1 and 7 show tokenizer introduced [P.46343 Sect.I Last3¶] to build vocabulary, and vectorization thereof [P.46346 Rt.Col], similar [P.46351 ¶5] “tokenized by our proposed KTL-BERT”};
generate an encoder output by inputting the token sequences into a multi-layer bidirectional transformer to transform the token sequences into the encoder output {Gu [P.46342 ¶3] “Bidirectional Encoder Representations from Transformers (BERT)” shown Fig 1 the arrows indicate input and output, BERT is multilayer known to have 12 layers, example layers shown Fig 2 which may include performing operations of attention and feed forward implemented by Eqs. 1-3},
However, Gu does not expressly disclose binary cross-entropy (BCE) loss or sigmoid function which is disclosed by Chen:
the multi-layer bidirectional transformer being trained using binary cross entropy loss on a set of ground truth token sequences which are labeled irrelevant for irrelevant text data which belongs in an irrelevant classification group, and labeled relevant for relevant text data which belongs in a relevant classification group {Chen Fig 1 shows BERT transformer and BCE (binary cross entropy) loss detailed [P.136 Sect4.2] comprising “irrelevant labels… relevant labels” respectively negative & positive for a multi-label classification task to train XMC (extreme multi-label classification) models introduced [P.135 Sect.4] where {t1, t2, …, tn} is sequence of tokenized titles/text Fig2 in tuple form <c, t, d> = x input and a ground truth may be yj in the equation. Additionally, both irrelevant labels and relevant labels can be used for training as is evidenced by Sabapathy at [0093, 98-99], [0078]};
compute a relevance probability and a first irrelevance probability, respectively, by inputting the transformed relevant output and the transformed irrelevant output into a sigmoid function, the relevance probability being a probability that the token sequence belongs in the relevant classification group, the first irrelevance probability being a probability that the token sequence belongs in the irrelevant classification group {Chen [P.136 ¶2-8] “probability through a sigmoid layer… probability returned from the model exceeds 0.5 among labels relevant” i.e. >0.5 is relevant thus ≤0.5 is irrelevant, Fig 1 shows the sigmoid layer which is commonly denoted σ in the statistical BCE, “predicts score in range of [0.0, 1.0] to all labels by giving x. σ tends to be close to 1.0 to the j-th label when yj = 1” meaning [0,1] range assigned where 1 is max relevance and 0 is most irrelevant, applied over ‘all’ labels in the multi-class classification. Additionally, a percentage can be used for relevance score as is further evidenced by Sabapathy at [0074]};
Chen is directed to BERT encoder transformer for text thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to employ the BCE and sigmoid functions of Chen in combination for a motivation of alleviating an imbalance between positive and negative labels as it “reduces bias in the distribution between positive and negative labels” (Chen [P.136 Sect4.2], [P.135 ¶1]). Further, Sabapathy in the same field of endeavor as BERT-based models, supports training with both irrelevant and relevant labels as motivated by determining the training label based on the relevance score (Sabapathy [0093, 98-99]).
However, the combination Gu, Chen and Sabapathy does not teach tensor product formula with scalar probability values and plurality of linear functions which is taught by Jiang:
input the encoder output into a relevant linear function and an irrelevant linear function to linearly transform the encoder output, and output a transformed relevant output and a transformed irrelevant output, respectively {Jiang Fig 2 shows plurality of linear functions to linearly transform by [P.4 ¶2,4] “linear projection that computes the attention scores… second linear projection” the encoder is described [P.3-4 Sect 2.2.1-2.3] “encoder applied to a sequence of tokens… encoder’s output” Eq.9 tokenized text are inputs, outputs are calculated by multi-head attention Eqs.3,8, whereby attention determines degree of relevance and irrelevance such that multiple heads for multiple linear functions are individually calculated Eq.2, example tasks include semantic role labeling [P.7 Sect4.1] See also Fig 3, [P.13 App.C] “We use a BERT-base model”};
compute a second irrelevance probability by inputting the relevance probability and the first irrelevance probability into a tensor product formula operating in probability space, the relevance probability and the first irrelevance probability being scalar probability values in the probability space {Jiang Fig 2 TPR tensor product representation introduced [P.3 Sect2.1] and detailed [P.4 Sect2.2.2] Eq.6 “We use a Hadamard product to approximate the full Tensor product… Hadamard product allows learning an optimal [sic] lower-rank approximation” where low-rank and “∈ R” i.e. being of real number set membership convey scalar probability values in the probability space. The terms R and F of Eq.6 are shown Fig 2 multi-headed attention transformer such that attention determines degree of relevance and irrelevance which is calculated Eqs. 3,8 with each attention head reducing to Eq.2, example tasks include semantic role labeling [P.7 Sect4.1]. See also Fig 3, [App.C] “We use a BERT-base model”}; and
generate and output an irrelevancy score for the input text based on the second irrelevance probability {Jiang [P.2 ¶4] “generate Tensor Product representation” including [P.3-4 Sect. 2.2.1-2.3] “attention output… attention scores” Eqs. 3,8 scores output by attention of generative model which inputs tokenized text Fig 1 or Tbl. 1, irrelevance exemplified [P.6 Last¶] “summaries that mention facts not included in the source article” and/or [P.10 ¶2] “unreliable text… model hallucinations”}.
Jiang is directed to transformer encoder model training thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to employ the tensor product representations, linear functions and teachings of Jiang in combination to arrive at the invention as claimed for a motivation being [P.9 Sect.6] “we enrich the Transformer model with structured Tensor Product Representation for abstractive summarization… our TP-Transformer with discrete roles outperforms Transformer and TP-Transformer with continuous roles on several abstractive summarization datasets, in both metrics scores and human evaluation” and/or [P.8 Last2¶] “improves interpretability of the representations… we show that having the compositional TPR results in more interpretable final representations at every layer …thus lead to summaries of better quality.”
Considered as a whole, Gu sets up a problem of BERT-based tax law classification while the reference Chen details BCE with sigmoid, relevance and irrelevance with support from Sabapathy, and Jiang teaches tensor products and linear functions for multi-head attention to enrich transformers. Taken together, the teachings provide a skilled artisan adept at model tuning with the requisite tools to arrive at claimed invention with a reasonable level of experimentation. Thus, it is respectfully submitted that the combination of prior arts support a finding of obviousness under 35 U.S.C. 103.
With respect to claim 2, the combination of Gu, Chen, Sabapathy and Jiang teaches the computing system of claim 1, wherein
the token sequences are generated so that a first token of every token sequence is configured to be a predetermined classification token; and the token sequences are separated by a predetermined separation token, or a learned embedding corresponding to each token sequence is added to each token of the token sequence {Chen [P.135 Last¶] “CLS and SEP are special tokens to represent a classifier token and a separator, respectively” consistent with instant specification [0016]}. Employing [CLS] and [SEP] tokens for classification and separation is in the standard BERT vocabulary.
With respect to claim 6, the combination of Gu, Chen, Sabapathy and Jiang teaches the computing system of claim 1, wherein
the binary cross entropy loss is calculated for the first irrelevance probability and the second irrelevance probability {Chen [P.136 Sect.4] BCE binary cross entropy equation statistically computes loss over irrelevant labels for a probability in the range [0.1, 1.0]. Doing so for a second probability regards the second token in a token sequence [P.135 Sect.4]}.
With respect to claim 8, the combination of Gu, Chen, Sabapathy and Jiang teaches the computing system of claim 1, wherein
the irrelevancy score includes a classification of the token sequence as relevant or irrelevant based on the second irrelevance probability {Chen [P.135 Sect.4] “classifier token” describes tokens for XMC extreme multi-label classification, class labels being relevant and irrelevant with statistical computation via BCE [P.136 Sect4.2] Additionally see Sabapathy Fig 5 binary classifier for relevant or irrelevant, relevance scores shown Fig 9:920}.
With respect to claim 9, the combination of Gu, Chen, Sabapathy and Jiang teaches the computing system of claim 8, wherein
the token sequence is categorized as irrelevant if the second irrelevance probability is greater than a predetermined threshold {Chen [P.136 ¶3] “probability… exceeds 0.5 among labels relevant” where exceeding conveys threshold predetermined at 0.5 to distinguish irrelevant from relevant. Additionally see Sabapathy [0098] “score threshold” described for irrelevant training classifier Fig 5}.
With respect to claim 10, the combination of Gu, Chen, Sabapathy and Jiang teaches the computing system of claim 1, wherein
the multi-layer bidirectional transformer is a BERT (Bidirectional Encoder Representations from Transformers) encoder {Gu discloses [P.46342 ¶3] “Bidirectional Encoder Representations from Transformers (BERT)” shown Fig 1}.
With respect to claim 11, the rejection of claim 1 is incorporated. The difference in scope being a method to perform the steps of system claim 1. Gu discloses [P.46345 Sect.III] “proposed method” shown Fig 1. The remainder of this claim is rejected for the same rationale as claim 1.
With respect to claim 12, the combination of Gu, Chen, Sabapathy and Jiang teaches the method of claim 11, and further teaches the limitation of claim 2. Therefore, the rejection of claim 2 is applied to claim 12.
With respect to claim 16, the combination of Gu, Chen, Sabapathy and Jiang teaches the method of claim 11, and further teaches the limitation of claim 6. Therefore, the rejection of claim 6 is applied to claim 16.
With respect to claim 18, the combination of Gu, Chen, Sabapathy and Jiang teaches the method of claim 11, and further teaches the limitation of claim 8. Therefore, the rejection of claim 8 is applied to claim 18.
With respect to claim 19, the combination of Gu, Chen, Sabapathy and Jiang teaches the method of claim 18, and further teaches the limitation of claim 9. Therefore, the rejection of claim 9 is applied to claim 19.
With respect to claim 20, the rejection of claim 1 is incorporated. The difference in scope regards tax law articles as the received input to be classified with encoder where the use of ‘or’ provides for income tax information or property tax information, and articles which contain only administrative tax information. Gu at Fig 1 bottom right illustrates multiple categories of tax information including “Global income tax” as well as tax categories for capital gains, VAT, withholding, etc. and describes “Asset revaluation tax” [P.46346 ¶2] where asset conveys property tax, and VAT tax is administered by governing body and/or corporate tax administered by a corporate entity such as payroll withholding. Moreover, Gu discloses [P.46342 ¶2] “select the category of tax… automatically classify the category of tax” teaches that the exact category of tax law is selectable and points to solution for automatic classification thereof. Additionally, computed is a third and fourth irrelevance probability. A third, fourth, fifth or Nth value would be obvious because the data is sequence. Chen’s tokenized sequence is introduced [P.135 Last2¶] and shown as words Fig 2, such that irrelevance is probabilistically computed over the plurality of words of a description that is categorized for multi-class classification. The remainder of the claim is rejected for the same rationale as claim 1.
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over: Gu, Chen, Sabapathy and Jiang in view of Hou et al., “Token Dropping for Efficient BERT Pretraining” hereinafter Hou (arXiv: 2203.13240v1).
With respect to claim 3, the combination of Gu, Chen, Sabapathy and Jiang teaches the computing system of claim 2. Hou teaches wherein in the multi-layer bidirectional transformer {Hou Fig 1 BERT},
a dropout is configured to be performed in which output vectors corresponding to all tokens other than the predetermined separation token and/or the predetermined classification token are dropped, and only the output corresponding to the predetermined classification token and the predetermined separation token are encoded {Hou [P.3 Sect.3] Token-Dropping “Figure 2 gives an illustration of where the unimportant tokens are dropped in a BERT model” Eq.1 details vectorization and “we never drop special tokens including [MSK], [CLS], and [SEP]. In other words, we always treat these tokens as important tokens” [P.4 Rt.Col] where [CLS] and [SEP] are classification and separation tokens, respectively, and encoder is BERT}.
Hou is directed to BERT-based models with importance scores for text thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to perform dropout per Hou in combination to arrive at the invention as claimed for a motivation “our token dropping strategy can save 25% of pretraining time while achieving similar performance” and “BERT models can be pretrained with only a subset of the layers focusing on important tokens. Even though the model is trained on sub-sequences of important tokens only, it generalizes well to full sequences during fine-tuning on downstream tasks… with minimal computational overhead and without modifying the model architecture” (Hou [P.2 ¶3]).
With respect to claim 13, the combination of Gu, Chen, Sabapathy and Jiang teaches the method of claims 12, and further combination with Hou teaches the limitation of claim 3. Therefore, the rejection of claim 3 with equal motivation is applied to claim 13.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over: Gu, Chen, Sabapathy and Jiang in view of Dana et al., US PG Pub No 2024/0004907A1 hereinafter Dana.
With respect to claim 7, the combination of Gu, Chen, Sabapathy and Jiang teaches the computing system of claim 1. Dan teaches wherein
the sigmoid function computes a logistic sigmoid function of elements of the encoder output to determine the relevance probability, the first irrelevance probability, and the second irrelevance probability {Dana [0040] “logits followed by a sigmoid layer” logits are logistic then fed into sigmoid subsequent to binary cross-entropy lgs as described for BERT encoder and softmax probabilities Fig 2, and relevancies are conveyed by cosine similarity [0026]}.
Dana is directed to BERT-based encoder transformers thus being analogous. A person having ordinary skill in the art would have considered it obvious prior to the effective filing date to employ logit/logistic sigmoid per Dana for the (ir)relevance probabilities of Chen’s sigmoid layer Fig 1 in combination to arrive at the invention as claimed as applying known techniques to known methods ready for improvement to yield predictable results where a plurality of layer operations are stacked in a modular neural architecture (Dana [0003]).
With respect to claim 17, the combination of Gu, Chen, Sabapathy and Jiang teaches the method of claim 11, and further combination with Dana teaches the limitation of claim 7. Therefore, the rejection of claim 7 with equal motivation is applied to claim 17.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Moradshah et al., “HUBERT Untangles BERT to Improve Transfer Across NLP Tasks” arXiv: 1910.12647v2 see Fig 1 TPR stacked on BERT
Chang et al., US PG Pub No 2024/0185833A1 Google discloses [0035] “tensor fusion”
Yao et al., “ReprBERT: Distilling BERT to an Efficient Representation-Based Relevance Model for E-Commerce” see [P.4364 ¶1] irrelevancy and BCE Eqs.6-7, Fig 1
Zhang et al., “Multimodal Sentiment Analysis Based on Attention Mechanism and Tensor Fusion Network” see Fig 1, Eqs. 5-6
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Chase P Hinckley whose telephone number is (571)272-7935. The examiner can normally be reached M-F 9:00 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda M. Huang can be reached at 571-270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHASE P. HINCKLEY/Examiner, Art Unit 2124
1 Manaswi, N.K., “Basics of TensorFlow” [P.2 ¶2] “A tensor is a mathematical object and a generalization of scalars, vectors, and matrices. A tensor can be represented as a multidimensional array. A tensor of zero rank (order) is nothing but a scalar”
Prosecution Insights