DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on October 23rd, 2025 has been entered and Claims 1-20 is/are pending. Applicant’s amendments to the Claims overcome the 35 U.S.C 101 rejection(s) previously set forth in the Non-Final Action mailed on June 23rd, 2025.
Response to Arguments
Applicant’s arguments filed October 23rd, 2025 have been fully considered but they are not persuasive.
Applicant’s arguments with respect to claim(s) 1, 5, and 17-18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
The reliance on an earlier publicly available version of Polo et al., is necessitated by Applicant’s amendment and does not constitute a new ground of rejection.
Applicant’s arguments with respect to the matter of publication dates have been considered. With respect to Crichton et al., the attached copy explicitly identifies a date of 08/15/2017 on page 13. With respect to Elnaggar et al., the attached copy explicitly identifies a date of September of 2018. With respect to Dong et al., the attached copy explicitly identifies a date of 10/15/2019.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Elnaggar et al., Non-Patent Literature (“Multi-Task Deep Learning for Legal Document Translation, Summarization and Multi-Label Classification”) in view of Crichton et al., Non-Patent Literature (“A Neural Network Multi-Task Learning Approach to Biomedical Named Entity Recognition”) and Polo et al. (“Interpretable Approach in the Classification of Sequences of Legal Texts.”)
Regarding claim 1:
Elnaggar teaches:
training, by a computer device, a machine learning model (Section 5.1, lines 7-10, “In case of our Multi-Task model at each training step we trained the model for the same batch size of each problem sequentially. To speed up the process, we trained the algorithms on four machines.”)
with a legal centric multi-label dataset comprising a plurality of documents each annotated with at least one label selected from a plurality of predefined labels for a given domain, (Section 3, paragraph 2, lines 6-8, “The JRC-Acquis is a collection of legislative documents [plurality of documents], retrieved from the European Union (EU) law, stating EU laws and policies for 22 languages [legal centric], which have to be implemented by each member state.” Paragraph 6, “The JRC-Acquis dataset was used for Multi-Labeling [multi-label], where each document is assigned to EuroVoc thesaurus annotations [annotated with at least one label]. These EuroVoc thesaurus has a hierarchical structure with over than 6000 classes [plurality of predefined labels].”)
wherein the machine learning model is trained using a multi-task learning process (Introduction, paragraph 3, “One way to overcome this problem is by using multi-task deep learning. In this approach, we train multiple tasks using only one model to provide better results of these problems.”)
Elnaggar does not explicitly teach:
splitting the plurality of documents in the legal centric multi-label dataset into at least three splits including a test split, a train split and a development (dev) split wherein splitting involves populating the train split with more than half of the plurality of documents;
a first task for training the machine learning model with respect to all of the labels used for the plurality of documents in the dataset, and
a second task for training the machine learning model with respect to a subset of all of the labels used for the plurality of documents in the dataset; and
predicting, by the computer device, using the trained machine learning model, at least one label from the plurality of labels for at least one other document.
Crichton teaches:
a first task for training the machine learning model with respect to all of the labels used for the plurality of documents in the dataset, and (Page 3, Col 2, paragraph 2, lines 4-5, “They first trained a model [a first task for training the machine learning model] on supervised classification task with fully-labeled examples [all of the labels]”)
a second task for training the machine learning model with respect to a subset of all of the labels used for the plurality of documents in the dataset; and (Page 3, Col 2, paragraph 2, lines 5-7, “then shared some layers of the model with a semi-supervised model which is trained on only partially-labeled examples [subset of all of the labels]”)
Crichton and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Crichton it would have been obvious for a person of ordinary skill in the art to apply the teachings of Crichton to Elnaggar before the effective filing date of the claimed invention in order to improve the accuracy and efficiency of processing documents (Crichton, Introduction, paragraph 3,“The question of how to improve the performance of NER, especially in the very common situation where only limited annotations are available, is still an open area of research. One potentially promising solution is to use multiple annotated datasets together to train a model for improved performance on a single dataset. This can help since datasets may contain complementary information that can help to solve individual tasks more accurately when trained jointly.”).
Polo teaches:
splitting the plurality of documents in the legal centric multi-label dataset into at least three splits including a test split, a train split and a development (dev) split wherein splitting involves populating the train split with more than half of the plurality of documents; (Section II, paragraph 3, “The three labels of interest (Archived, Active, Suspended) reflect the most practical classification of the status of a proceeding. Although in Procedural Law there may be some other subtle categories of analog status for proceedings (such as extinction and dismissal), the status of being archived, active or suspended is related to the activities of all personnel involved with these proceedings (i.e., wherein under the broadest reasonable interpretation the proceedings are interpreted as documents in legal centric multi label). For example, the suspension of a proceeding means that, even if not extinct (and therefore subject to reactivation of the same lawsuit), and from a practical view these proceedings are out of the judiciary routine of certain portfolios from courts, law firms, civil associations or legal aid organizations”…Section IV, part G, “we splitted at random our labeled dataset in three parts: training set (70%) [a train split] (i.e., wherein splitting involves populating the train split with more than half, hence ‘70%’ is more than half), validation set (10%) [development (dev) split] (i.e., wherein under the broadest reasonable interpretation (BRI) development split is interpreted as ‘validation set’) and test set (20%) [test split].We used the training set to fit the model, the validation set to choose the best hyperparameters and the test set just to check the performance of the final model.”)
predicting, by the computer device, using the trained machine learning model, at least one label from the plurality of labels for at least one other document. ((Section 2, “The objective of this paper is to develop an interpretable model [trained machine learning model] for the classification of Brazilian legal proceedings in three possible classes [label from the plurality of labels] of status: (i) archived proceedings, (ii) active proceedings, and (iii) suspended proceedings.”)
Polo and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Polo it would have been obvious for a person of ordinary skill in the art to apply the teachings of Polo to Elnaggar before the effective filing date of the claimed invention in order to improve efficiency in processing documents (Polo, Section II, paragraph 2, “These objectives and criteria have been chosen because they are key features to any task related to legal proceedings in Brazil an possibly a good amount of countries. Although there are 90 different Courts in Brazil (State, Labour, Federal and others)– plus the Supreme Court–, all legal proceedings in Brazil must be included in one of the three presented classes (Archived, Active, Suspended). In spite of the status of a proceeding being an objective information, sometimes it can be hard for public or private organizations with large portfolios to track it because the information: (i) is non-structured and non-standardized, (ii) can be spread in hundreds of separate individual Courts’ web pages and (iii) it can be imprecise, incorrect or outdated.”)
Regarding claim 2:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1, wherein Polo further teaches:
wherein the dataset comprises a plurality of legal documents, (Section 4, lines 2-3, “a dataset containing 6449 legal proceedings, [plurality of legal documents] each with an individual and a variable number of motions, but which have been labeled by lawyers”)
the plurality of labels comprises a plurality of procedural postures, and the at least one annotated document is labeled with at least one procedural posture. (Section 2, paragraph 1., lines 1-4, “The objective of this paper is to develop predictive models [trained machine learning model] for the classification of Brazilian legal proceedings in three possible classes [label from the plurality of labels] of status: (i) archived proceedings, (ii) active proceedings, and (iii) suspended proceedings”) and (Section 4, paragraph 1, “Among the labeled data, 47.14% is classified as archived (class 1), 45.23% is classified as active (class 2), and 7.63% is classified as suspended (class 3) [at least one annotated document is labeled with at least one procedural posture].”)
The motivation for claim 2 is the same as the motivation for claim 1.
Claim(s) 3, 6-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Elnaggar, as modified by Crichton and Polo in view of Chalkidis et al. Non-Patent Literature ("Large-Scale Multi-Label Text Classification on EU Legislation.")
Regarding claim 3:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 2.
Elnaggar, as modified by Crichton does not explicitly teach:
wherein the dataset comprises a plurality of documents labeled with at least one of a first set of procedural postures in no more than 0.1% of the documents in the dataset and
wherein the machine learning model is trained to label the at least one other document with the first set of procedural postures.
Polo further teaches:
wherein the machine learning model is trained to label the at least one other document with the first set of procedural postures (Section 2, “The objective of this paper is to develop predictive models [trained machine learning model] for the classification of Brazilian legal proceedings in three possible classes [label] of status: (i) archived proceedings, (ii) active proceedings, and (iii) suspended proceedings”) and (Section 4, “Among the labeled data, 47.14% is classified as archived (class 1), 45.23% is classified as active (class 2), and 7.63% is classified as suspended (class 3) [procedural postures].”)
Polo does not explicitly teach:
wherein the dataset comprises a plurality of documents labeled with at least one of a first set of procedural postures in no more than 0.1% of the documents in the dataset and
Chalkidis teaches:
wherein the dataset comprises a plurality of documents labeled with at least one of a first set of procedural postures in no more than 0.1% of the documents in the dataset and (Section 3, paragraph 3, “All the documents of the dataset have been annotated by the Publications Office of EU4 with multiple concepts from EUROVOC. While EUROVOC includes approx. 7k concepts (labels), only 4,271 (59.31%) are present in EURLEX57K, from which only 2,049 (47.97%) have been assigned to more than 10 documents. We split EURLEX57K into training (45k documents), development (6k), and test subsets (6k). We also divide the 4,271 labels into frequent (746 labels), few-shot (3,362), and zeroshot (163), depending on whether they were assigned to more than 50, fewer than 50 but at least one, or no training documents, respectively [no more than 0.1% of the documents].”)
Chalkidis and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Chalkidis it would have been obvious for a person of ordinary skill in the art to apply the teachings of Chalkidis to Elnaggar before the effective filing date of the claimed invention in order to improve the efficiency of processing documents (Chalkidis, Introduction, “Large-scale multi-label text classification (LMTC) is the task of assigning to each document all the relevant labels from a large set, typically containing thousands of labels (classes). Applications include building web directories (Partalas et al., 2015), labeling scientific publications with concepts from ontologies (Tsatsaronis et al., 2015), assigning diagnostic and procedure labels to medical records (Mullenbach et al., 2018; Rios and Kavuluru, 2018). We focus on legal text processing, an emerging NLP field with many applications (e.g., legal judgment (Nallapati and Manning, 2008; Aletras et al., 2016), contract element extraction (Chalkidis et al., 2017), obligation extraction (Chalkidis et al., 2018)), but limited publicly available resources.”)
Regarding claim 6:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton and Polo does not explicitly teach:
wherein the machine learning model is a machine learning model pretrained with general-domain corpora
and the computer implemented process comprises continued training of the general-domain corpora pretrained machine learning model with the legal centric multi-label dataset.
Chalkidis further teaches:
wherein the machine learning model is a machine learning model pretrained with general-domain corpora (Section 4, paragraph 6, “BERT (Devlin et al., 2018) is a language model based on Transformers (Vaswani et al., 2017) pretrained on large corpora [general-domain corpora].”)
and the computer implemented process comprises continued training of the general-domain corpora pretrained machine learning model with the legal centric multi-label dataset (Abstract, “We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EUR-LEX, annotated with 4.3k EUROVOC labels [legal centric multi-label]”…Section 4, paragraph 6, “For a new target task, a task-specific layer is added on top of BERT. The extra layer is trained jointly with BERT by fine-tuning on task-specific data.”)
The motivation for claim 6 is the same as the motivation for claim 3.
Regarding claim 7:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton and Polo does not explicitly teach:
wherein the plurality of documents in the dataset are labeled following a Zipfian distribution.
Chalkidis further teaches:
wherein the plurality of documents in the dataset are labeled following a Zipfian distribution (Appendix, “Figure 3 shows the distribution of labels across EURLEX57K documents. From the 7k labels fewer than 50% appear in more than 10 documents. Such an aggressive Zipfian distribution has also been noted in medical code predictions (Rios and Kavuluru, 2018), where such thesauri are used to classify documents, demonstrating the practical importance of few-shot and zero-shot learning.”)
The motivation for claim 7 is the same as the motivation for claim 3.
Regarding claim 8:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton and Polo does not explicitly teach:
wherein the subset of labels includes classes determined based on how many documents in the plurality of documents in the dataset are tagged a given label.
Chalkidis further teaches:
wherein the subset of labels includes classes determined based on how many documents in the plurality of documents in the dataset are tagged a given label (Section 3, paragraph 3, “We split EURLEX57K into training (45k documents), development (6k), and test subsets (6k). We also divide the 4,271 labels into frequent (746 labels), few-shot (3,362), and zeroshot (163), depending on whether they were assigned to more than 50, fewer than 50 but at least one, or no training documents, respectively.”)
The motivation for claim 8 is the same as the motivation for claim 3.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Elnaggar, as modified by Crichton and Polo, further in view of Xiao et al. Non-Patent Literature ("Label-Specific Document Representation for Multi-Label Text Classification.")
Regarding claim 4:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton and Polo does not explicitly teach:
wherein the dataset comprises a document labeled with a first procedural posture in only one of the documents in the dataset and
wherein the machine learning model is trained to label at least one other document with the first procedural posture
Xiao teaches:
wherein the dataset comprises a document labeled with a first procedural posture in only one of the documents in the dataset and (Section 3.3, “Figure 2 shows the distribution of label frequency on EUR-Lex, F is the frequency of label. Among it, nearly 55% of labels occur between 1 and 5 times to form the first label group (Group1).”)
wherein the machine learning model is trained to label at least one other document with the first procedural posture (Section 3.3, paragraph 2, “Figure 3 shows the prediction results in terms of P@1, P@3 and P@5 obtained by AttentionXML, EXAM and LSAN. Three methods become better and better from Group1 to Group3, which is reasonable because more and more documents are included in each label from Group1 to Group3. LSAN significantly improves the prediction performance on Group1. Especially, LSAN obtains an average of more than 83.82%, 182.55%, 244.62% gain on three metrices for group 1 to AttentionXML, and 3.85%, 27.19%, 58.27% gain to EXAM. This result demonstrates the superiority of the proposed model on multi-label text data with tail labels.”)
Xiao and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Xiao it would have been obvious for a person of ordinary skill in the art to apply the teachings of Xiao to Elnaggar before the effective filing date of the claimed invention in order to improve the efficiency of processing documents (Xiao, Introduction, paragraph 1, “In this paper, we focus on multi-label text classification (MLTC) because it has become one of the core tasks in natural language processing and has been widely applied in topic recognition (Yang et al., 2016), question answering (Kumar et al., 2016), sentimental analysis (Cambria et al., 2014) and so on. With the boom of big data, MLTC becomes significantly challenging because it has to handle the massive documents, words and labels simultaneously. Therefore, it is an emergency to develop effective multi-label text classifier for various practical applications.”)
Claim(s) 9-11, 13 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Elnaggar, as modified by Crichton, Polo, and Chalkidis, further in view of Peng et al. Non-Patent Literature ("Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification.")
Regarding claim 9:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton, Polo and Chalkidis does not explicitly teach:
representing each of the labels for the first task as a vector,
computing a cosine distance between the labels for the first task and an output from the machine learning model,
determining a weight matrix for the first task based on the computed cosine distances,
classifying a sample of the output from the machine learning model therewith providing a dense output for the first task, and
multiplying the dense output for the first task with the weight matrix for the first task therewith providing a final classification output.
Peng teaches:
representing each of the labels for the first task as a vector, (Section 1, paragraph 7, lines 5-10, “Therefore, the hierarchical taxonomy relation among the labels can be encoded in a continuous vector space with the skip-gram [27] on the sequences. In this way, the distance between two labels can be measured by calculating the cosine similarity of their label vectors.”)
computing a cosine distance between the labels for the first task and an output from the machine learning model, (Section 4, paragraph 4, “Thus, given a tag label/class l ∈ V, we can approximate a semantic distance between any other label li , i ∈ [1, L] by calculating their discrete cosine distance between their embedding vectors.”)
determining a weight matrix for the first task based on the computed cosine distances, (Section 1, paragraph 7, lines 10-13, “By taking the distance between labels into consideration, we design a new weighted margin loss to guide the training of proposed models in multi-label text classification.”)
classifying a sample of the output from the machine learning model therewith providing a dense output for the first task, and (Section 3.2, paragraph 2, “The capsules contain groups of locally invariant neurons that learn to recognize the presence of features and encode their properties into vector outputs, with the vector length representing the presence of the features. The primary capsule layer is a convolution capsule layer with M channels of capsules.”)
multiplying the dense output for the first task with the weight matrix for the first task therewith providing a final classification output (Section 3.2, paragraph 2, “For all but the first layer of capsules, the total input to a capsule j is a weighted sum over all the prediction vectors uˆj|i from the capsules in the layer below, and is calculated by multiplying the output ui of a capsule in the layer below by a weight matrix Wij.”)
Peng and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Peng it would have been obvious for a person of ordinary skill in the art to apply the teachings of Peng to Elnaggar before the effective filing date of the claimed invention to apply multi-task machine learning process [i.e., first task] in order to improve the efficiency of processing documents (Peng, Introduction, paragraph 8, “We conduct extensive evaluations on our proposed framework by comparing it with state-of-the-art methods on three benchmark datasets, comparing with traditional shallow models and recent deep learning models. The results show that our approach outperforms them by a large margin in both efficiency and effectiveness on large-scale multi-label text classification.”)
Regarding claim 10:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton and Polo does not explicitly teach:
representing each of the labels for the second task as a vector,
computing a cosine distance between the labels for the second task and an output from the machine learning model,
determining a weight matrix for the second task based on the computed cosine distances,
classifying a sample of the output from the machine learning model therewith providing a dense output for the second task,
wherein labels for the second task include classes determined based on how many documents in the plurality of documents in the dataset are tagged a given label, and
multiplying the dense output for the second task with the weight matrix for the second task therewith providing a small classification output.
Chalkidis further teaches:
wherein labels for the second task include classes determined based on how many documents in the plurality of documents in the dataset are tagged a given label, (Section 3, paragraph 3, “We split EURLEX57K into training (45k documents), development (6k), and test subsets (6k). We also divide the 4,271 labels into frequent (746 labels), few-shot (3,362), and zeroshot (163), depending on whether they were assigned to more than 50, fewer than 50 but at least one, or no training documents, respectively.”)
Peng further teaches:
representing each of the labels for the second task as a vector, (Section 1, paragraph 7, lines 5-10, “Therefore, the hierarchical taxonomy relation among the labels can be encoded in a continuous vector space with the skip-gram [27] on the sequences. In this way, the distance between two labels can be measured by calculating the cosine similarity of their label vectors.”)
computing a cosine distance between the labels for the second task and an output from the machine learning model, (Section 4, paragraph 4, “Thus, given a tag label/class l ∈ V, we can approximate a semantic distance between any other label li , i ∈ [1, L] by calculating their discrete cosine distance between their embedding vectors.”)
determining a weight matrix for the second task based on the computed cosine distances, (Section 1, paragraph 7, “By taking the distance between labels into consideration, we design a new weighted margin loss to guide the training of proposed models in multi-label text classification.”)
classifying a sample of the output from the machine learning model therewith providing a dense output for the second task, (Section 3.2, paragraph 2, “The capsules contain groups of locally invariant neurons that learn to recognize the presence of features and encode their properties into vector outputs, with the vector length representing the presence of the features. The primary capsule layer is a convolution capsule layer with M channels of capsules.”)
multiplying the dense output for the second task with the weight matrix for the second task therewith providing a small classification output (Section 3.2, paragraph 2, “For all but the first layer of capsules, the total input to a capsule j is a weighted sum over all the prediction vectors uˆj|i from the capsules in the layer below, and is calculated by multiplying the output ui of a capsule in the layer below by a weight matrix Wij.”)
Peng and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Peng it would have been obvious for a person of ordinary skill in the art to apply the teachings of Peng to Elnaggar before the effective filing date of the claimed invention to apply multi-task machine learning process [i.e., second task] in order to improve the efficiency of processing documents (Peng, Introduction, paragraph 8, “We conduct extensive evaluations on our proposed framework by comparing it with state-of-the-art methods on three benchmark datasets, comparing with traditional shallow models and recent deep learning models. The results show that our approach outperforms them by a large margin in both efficiency and effectiveness on large-scale multi-label text classification.”)
Regarding claim 11:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton, Polo and Chalkidis does not explicitly teach:
comprising pretraining the machine learning model using a portion of at least one document in the dataset (Section 2.3, lines 6-10, “In this way, we have a 3-D tensor representation for each document, where the padded vectors are zero vectors with the same dimension. Then the convolution, recurrent and capsule networks introduced in the next section will be operated over the unified representations of the documents.”)
Peng and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Peng it would have been obvious for a person of ordinary skill in the art to apply the teachings of Peng to Elnaggar before the effective filing date of the claimed invention in order to improve the efficiency of processing documents (Peng, Introduction, paragraph 8, “We conduct extensive evaluations on our proposed framework by comparing it with state-of-the-art methods on three benchmark datasets, comparing with traditional shallow models and recent deep learning models. The results show that our approach outperforms them by a large margin in both efficiency and effectiveness on large-scale multi-label text classification.”)
Regarding claim 13 and analogous claim 19:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton, Polo and Chalkidis does not explicitly teach:
comprising pretraining the machine learning model using documents processed with N-Gram topic modeling.
Peng further teaches:
comprising pretraining the machine learning model using documents processed with N-Gram topic modeling (Section 5.2, paragraph 2, “N-gram, sequence-of-words or graph-of-words based models. These methods extract N-gram features, sequence of-words or graph-of-words from the document as the input of classification models.”)
The motivation for claim 13 is the same as the motivation for claim 11.
Claim(s) 12, 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Elnaggar, as modified by Crichton and Polo, further in view of Singh et al. Non-Patent Literature ("An Ensemble Approach for Extractive Text Summarization.")
Regarding claim 12:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton and Polo does not explicitly teach:
comprising pretraining the machine learning model using at least one document with noisy text filtered therefrom.
Singh teaches:
comprising pretraining the machine learning model using at least one document with noisy text filtered therefrom (Page 2, Section III(A), “The input text is preprocessed by tokenization, removing stopwords [noisy text filtered] and then performing stemming.”)
Singh and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Singh it would have been obvious for a person of ordinary skill in the art to apply the teachings of Singh to Elnaggar before the effective filing date of the claimed invention in order to improve the efficiency of processing documents (Singh, Introduction, paragraph 1, “With the increase in the number of digital media and publishing documents, extracting information from a huge amount of data is a prolonged task. Thus, there is a need to hire an experienced person to create human-generated summaries. Moreover, editing so many documents consisting of a large amount of text is a time-consuming task and the expected outcome is not guaranteed. A solution to this is a system that can automatically generate summaries from the given text.”)
Regarding claim 14:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton and Polo does not explicitly teach:
comprising pretraining the machine learning model using sentence reranking.
Singh further teaches:
comprising pretraining the machine learning model using sentence reranking (Page 2, Section III(A), “The input text is preprocessed by tokenization, removing stopwords and then performing stemming. After that, the following mentioned features were extracted from the text. Two approaches were used for scoring the sentences: (Page 3-4, 2) Sentence Scoring) Sentence Scoring: In this, the sentence features are analyzed. The approaches following this idea are as follows: 1) Cue-phrases: Sentences starting with ‘in conclusion’, ’example’, ’therefore’, ‘our investigation’, ‘the article describes’, ‘the best, ‘the most important’, ‘if particular’, ‘according to’, ‘significantly’ are of great importance in a text document. The sentence consisting of cue phrase or words are assigned high score. To normalize the results of cuescore, each sentence is divided by the maximum value of cuescore. 2) Numerical Data: Sentence containing numerical data like event dates, transaction having numerical data, year, age, etc. are important sentences and must be included if the summary.”)
The motivation for claim 14 is the same as the motivation for claim 12.
Claim(s) 5, 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Elnaggar, as modified by Crichton, Polo, Chalkidis, and Singh, further in view of Dong et al. Non-Patent Literature ("Unified Language Model Pre-training for Natural Language Understanding and Generation.")
Regarding claim 5:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 2.
Elnaggar, as modified by Crichton and Polo, Singh does not explicitly teach:
wherein the machine learning model is a neural language model
and pretraining the machine learning model by establishing an embedding layer with at least three distinct embedding sublayers including a token key embedding sublayer, a position embedding sublayer and a token sequence embedding sublayer
Chalkidis further teaches:
wherein the machine learning model is a neural language model (Section 4, paragraph 6, “BERT (Devlin et al., 2018) is a language model based on Transformers pretrained on large corpora. For a new target task, a task-specific layer is added on top of BERT. The extra layer is trained jointly with BERT by fine-tuning on task-specific data” [neural language model].)
Dong teaches:
and pretraining the machine learning model by establishing an embedding layer with at least three distinct embedding sublayers including a token key embedding sublayer, a position embedding sublayer and a token sequence embedding sublayer (Introduction, paragraph 3, “In this work we propose a new UNIfied pre-trained Language Model (UNILM) that can be applied to both natural language understanding (NLU) and natural language generation (NLG) tasks [pretraining the machine learning model]”…Section 2.1, “For each input token, its vector representation is computed by summing the corresponding token embedding [token key embedding sublayer], position embedding [a position embedding sublayer], and segment embedding [a token sequence embedding sublayer] (i.e., wherein sequence embedding under the broadest reasonable interpretation (BRI) is interpreted as ‘segment embedding’). Since UNI LM is trained using multiple LM tasks, segment embeddings also play a role of LM identifier in that we use different segment embeddings for different LM objectives.”)
Dong and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Dong it would have been obvious for a person of ordinary skill in the art to apply the teachings of Dong to Elnaggar before the effective filing date of the claimed invention in order to improve the efficiency of processing documents (Dong, Introduction, paragraph 1, “Language model (LM) pre-training has substantially advanced the state of the art across a variety of natural language processing tasks [8, 29, 19, 31, 9, 1]. Pre-trained LMs learn contextualized text representations by predicting words based on their context using large amounts of text data, and can be fine-tuned to adapt to downstream tasks.”)
Regarding claim 15:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton, Polo, Chalkidis and Singh does not explicitly teach:
comprising pretraining the machine learning model using masked language modeling.
Dong teaches:
comprising pretraining the machine learning model using masked language modeling (Section 2.3, “We pretrain UNILM using four cloze tasks designed for different language modeling objectives.”)
Dong and Elnaggar are both related to the same field of endeavor (i.e., natural language processing). In view of the teachings of Dong it would have been obvious for a person of ordinary skill in the art to apply the teachings of Dong to Elnaggar before the effective filing date of the claimed invention in order to improve the efficiency of processing documents (Dong, Introduction, paragraph 1, “Language model (LM) pre-training has substantially advanced the state of the art across a variety of natural language processing tasks [8, 29, 19, 31, 9, 1]. Pre-trained LMs learn contextualized text representations by predicting words based on their context using large amounts of text data, and can be fine-tuned to adapt to downstream tasks.”)
Regarding claim 16:
Elnaggar, as modified by Crichton and Polo, teaches the method of claim 1.
Elnaggar, as modified by Crichton, Polo, Chalkidis and Singh does not explicitly teach:
wherein masked language modeling comprises randomly selecting tokens from an original document and replacing a portion of the selected tokens with a mask token, and wherein the machine learning model predicts token values based on tokens surrounding the mask token.
Dong further teaches:
wherein masked language modeling comprises randomly selecting tokens from an original document and replacing a portion of the selected tokens with a mask token, and wherein the machine learning model predicts token values based on tokens surrounding the mask token (Section 2.3, paragraph 1, “We pretrain UNILM using four cloze tasks designed for different language modeling objectives. In a cloze task, we randomly choose some WordPiece tokens in the input, and replace them with special token [MASK]. Then, we feed their corresponding output vectors computed by the Transformer network into a softmax classifier to predict the masked token. The parameters of UNILM are learned to minimize the cross-entropy loss computed using the predicted tokens and the original tokens.”)
The motivation for claim 16 is the same as the motivation for claim 15.
Claim(s) 17-18, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Elnaggar, as modified by Crichton and Polo, further in view of Chalkidis, Peng, Xiao, Singh and Dong.
Regarding claim 17:
Chalkidis teaches:
training, by a computer device, a general-domain corpora pretrained machine learning model (Section 4, paragraph 6, “BERT (Devlin et al., 2018) is a language model based on Transformers (Vaswani et al., 2017) pretrained on large corpora [general-domain corpora]”) with a legal centric multi-label dataset comprising a plurality of documents (Abstract, “We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EUR-LEX, annotated with 4.3k EUROVOC labels [legal centric multi-label]”…Section 4, paragraph 6, “For a new target task, a task-specific layer is added on top of BERT. The extra layer is trained jointly with BERT by fine-tuning on task-specific data.”) each annotated with at one label selected from a plurality of predefined labels (Section 3, paragraph 3, “All the documents of the dataset have been annotated by the Publications Office of EU4 with multiple concepts from EUROVOC. While EUROVOC includes approx. 7k concepts (labels), only 4,271 (59.31%) are present in EURLEX57K, from which only 2,049 (47.97%) have been assigned to more than 10 documents. We split EURLEX57K into training (45k documents), development (6k), and test subsets (6k) for a given domain following a Zipfian distribution, (Appendix, “Figure 3 shows the distribution of labels across EURLEX57K documents. From the 7k labels fewer than 50% appear in more than 10 documents. Such an aggressive Zipfian distribution has also been noted in medical code predictions (Rios and Kavuluru, 2018), where such thesauri are used to classify documents, demonstrating the practical importance of few-shot and zero-shot learning.”) which subset includes classes determined based on class frequency, wherein under BRI, it is interpreted that class frequency is shown through the Zipfian distribution ( [0047] Finally, in the POSTURESOK dataset, the labels follow a Zipfian distribution in terms of their count, and many posture categories occur in 50 cases or less (below). Few-shot and zero-shot learning may be some of the promising candidate techniques to addressing the task we are proposing with this dataset.)
Elnaggar teaches:
wherein the machine learning model is trained using a multi-task learning process comprising: (Introduction, paragraph 3, “One way to overcome this problem is by using multi-task deep learning. In this approach, we train multiple tasks using only one model to provide better results of these problems.”)
Crichton teaches:
a first task for training the machine learning model with respect to all of the labels used for the plurality of documents in the dataset, and “They first trained a model [a first task for training the machine learning model] on supervised classification task with fully-labeled examples [all of the labels]”)
a second task for training the machine learning model with respect to a subset of all of the labels used for the plurality of documents in the dataset, (Page 3, Col 2, paragraph 2, lines 5-7, “then shared some layers of the model with a semi-supervised model which is trained on only partially-labeled examples [subset of all of the labels].”)
wherein at least one of the first and the second task for training the machine learning model comprises: “They first trained a model [a first task for training the machine learning model] on supervised classification task with fully-labeled examples [all of the labels]”) (Page 3, Col 2, paragraph 2, lines 5-7, “then shared some layers of the model with a semi-supervised model which is trained on only partially-labeled examples [subset of all of the labels],” In combination, shows a first task and second task.)
Peng teaches:
representing each of the labels for the at least one of the first and the second task as a vector, (Section 1, paragraph 7, lines 5-10, “Therefore, the hierarchical taxonomy relation among the labels can be encoded in a continuous vector space with the skip-gram [27] on the sequences. In this way, the distance between two labels can be measured by calculating the cosine similarity of their label vectors.”)
computing a cosine distance between the labels for the at least one of the first and the second task and an output from the machine learning model, (Section 4, paragraph 4, “Thus, given a tag label/class l ∈ V, we can approximate a semantic distance between any other label li , i ∈ [1, L] by calculating their discrete cosine distance between their embedding vectors.”)
determining a weight matrix for the at least one of the first and the second task based on the computed cosine distances (Section 1, paragraph 7, “By taking the distance between labels into consideration, we design a new weighted margin loss to guide the training of proposed models in multi-label text classification.”)
classifying a sample of the output from the machine learning model therewith providing a dense output for the at least one of the first and the second task, and (Section 3.2, paragraph 2, “The capsules contain groups of locally invariant neurons that learn to recognize the presence of features and encode their properties into vector outputs, with the vector length representing the presence of the features. The primary capsule layer is a convolution capsule layer with M channels of capsules.”)
multiplying the dense output for the at least one of the first and the second task with the weight matrix for the first task therewith providing a classification output; and (Section 3.2, paragraph 2, “For all but the first layer of capsules, the total input to a capsule j is a weighted sum over all the prediction vectors uˆj|i from the capsules in the layer below, and is calculated by multiplying the output ui of a capsule in the layer below by a weight matrix Wij.”)
Polo teaches:
splitting the plurality of documents in the legal centric multi-label dataset into at least three splits including a test split, a train split and a development (dev) split wherein splitting involves populating the train split with more than half of the plurality of documents; (Section II, paragraph 3, “The three labels of interest (Archived, Active, Suspended) reflect the most practical classification of the status of a proceeding. Although in Procedural Law there may be some other subtle categories of analog status for proceedings (such as extinction and dismissal), the status of being archived, active or suspended is related to the activities of all personnel involved with these proceedings (i.e., wherein under the broadest reasonable interpretation the proceedings are interpreted as documents in legal centric multi label). For example, the suspension of a proceeding means that, even if not extinct (and therefore subject to reactivation of the same lawsuit), and from a practical view these proceedings are out of the judiciary routine of certain portfolios from courts, law firms, civil associations or legal aid organizations”…Section IV, part G, “we splitted at random our labeled dataset in three parts: training set (70%) [a train split] (i.e., wherein splitting involves populating the train split with more than half, hence ‘70%’ is more than half), validation set (10%) [development (dev) split] (i.e., wherein under the broadest reasonable interpretation (BRI) development split is interpreted as ‘validation set’) and test set (20%) [test split].We used the training set to fit the model, the validation set to choose the best hyperparameters and the test set just to check the performance of the final model.”)
predicting, by the computer device, using the trained machine learning model, at least one label from the plurality of labels for at least one other document (Section 2, “The objective of this paper is to develop predictive models [trained machine learning model] for the classification of Brazilian legal proceedings in three possible classes [label from the plurality of labels] of status: (i) archived proceedings, (ii) active proceedings, and (iii) suspended proceedings”)
In combination for the same reasons shared above, all references are related to the same field of endeavor (i.e., natural language processing). It would have been obvious for a person of ordinary skill in the art to apply the teachings in combination to the teachings of Elnaggar before the effective filing date of the claimed invention in order to improve the accuracy and efficiency of processing documents of legal documents.
Regarding claim 18:
Elnaggar, as modified by Crichton, Chalkidis, Peng and Polo, teaches claim 17.
Elnaggar, as modified by Crichton and Polo, Chalkidis, Peng, and Xiao does not explicitly teach:
comprising pretraining the machine learning model using at least one document processed using at least one of sentence reranking and filtering noisy text therefrom.
Singh further teaches:
comprising pretraining the machine learning model using at least one document processed using at least one of sentence reranking (Page 2, Section III(A), “The input text is preprocessed by tokenization, removing stopwords and then performing stemming. After that, the following mentioned features were extracted from the text. Two approaches were used for scoring the sentences: (Page 3-4, 2) Sentence Scoring) Sentence Scoring: In this, the sentence features are analyzed. The approaches following this idea are as follows: 1) Cue-phrases: Sentences starting with ‘in conclusion’, ’example’, ’therefore’, ‘our investigation’, ‘the article describes’, ‘the best, ‘the most important’, ‘if particular’, ‘according to’, ‘significantly’ are of great importance in a text document. The sentence consisting of cue phrase or words are assigned high score. To normalize the results of cuescore, each sentence is divided by the maximum value of cuescore. 2) Numerical Data: Sentence containing numerical data like event dates, transaction having numerical data, year, age, etc. are important sentences and must be included if the summary.”) and filtering noisy text therefrom Page 2, Section III(A), “The input text is preprocessed by tokenization, removing stopwords [noisy text filtered] and then performing stemming.”)
Dong teaches:
and pretraining the machine learning model by establishing an embedding layer with at least three distinct embedding sublayers including a token key embedding sublayer, a position embedding sublayer and a token sequence embedding sublayer (Introduction, paragraph 3, “In this work we propose a new UNIfied pre-trained Language Model (UNILM) that can be applied to both natural language understanding (NLU) and natural language generation (NLG) tasks [pretraining the machine learning model]”…Section 2.1, “For each input token, its vector representation is computed by summing the corresponding token embedding [token key embedding sublayer], position embedding [a position embedding sublayer], and segment embedding [a token sequence embedding sublayer] (i.e., wherein sequence embedding under the broadest reasonable interpretation (BRI) is interpreted as ‘segment embedding’). Since UNI LM is trained using multiple LM tasks, segment embeddings also play a role of LM identifier in that we use different segment embeddings for different LM objectives.”)
The motivation for claim 18 is the same as the motivation for claim 17.
Regarding claim 20:
Elnaggar, as modified by Crichton, Chalkidis, Peng and Polo, teaches claim 17.
Elnaggar, as modified by Crichton, Polo, Chalkidis, Singh, Peng, and Xiao does not explicitly teach:
comprising pretraining the machine learning model using masked language modeling.
Dong further teaches:
comprising pretraining the machine learning model using masked language modeling (Section 2.3, paragraph 1, “We pretrain UNILM using four cloze tasks designed for different language modeling objectives)
The motivation for claim 20 is the same as the motivation for claim 17.
Conclusion
Applicant’s amendment necessitated the new ground(s) of rejection presented in this Office action. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMINA BENOURAIDA whose telephone number is (571)272-4340. The examiner can normally be reached Monday-Friday 8:30am-5pm ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J. Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMINA MORENO BENOURAIDA/Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129