DETAILED ACTION
This communication is in response to the application filed 9/29/23 in which claims 1-20 were presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 9/29/23 and 2/5/25 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Allowable Subject Matter
Claims 6 and 10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1-5, 7, 11-16, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Linhan, et al. "Weighted sampling for masked language modeling." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023 (“Zhang”) in view of Sage, Clément, et al. "Data-efficient information extraction from documents with pre-trained language models." International Conference on Document Analysis and Recognition. Cham: Springer International Publishing, 2021 (“Sage”) and Morariu (US 2023/0376687 A1; published Nov. 23, 2023).
Regarding claim 1, Zhang discloses [a] computer-implemented method comprising:
obtaining first training data based on a plurality of unlabeled documents for use in training a first model for document information extraction; (Zhang Section 3.2 (“BERTCP degrades GLUE performance compared to BERT, probably because WikiText (373.28M data size) used for continual pre-training is much smaller and less diverse than the standard BERT pre-training dataset (Wikipedia and Bookscorpus, 16GB data size), which could hurt generalizability of PLMs.”))
pretraining the first model [according to a dynamic window adjustable to a word token count] for each document of the plurality of unlabeled documents, wherein the pretraining comprises evaluating word tokens in each of the plurality of unlabeled documents where masking is applied according to individual masking rates determined for the word tokens, and wherein the individual masking rates are indicative of respective informative relevance of the word tokens; and (Zhang Introduction (“Our work focuses on alleviating the frequency bias issue in MLM and improving quality of PLMs. We propose two Weighted Sampling methods for masking tokens based on token frequency or training loss. The latter one can dynamically adjust sampling weights and achieve a good balance between masking probabilities of common tokens and rare tokens based on the learning status of PLMs. Our Weighted Sampling methods can be applied to any MLM-pretrained PLMs. In this work, we focus on investigating the effectiveness of applying Weighted Sampling to BERT as the backbone. We initialize from BERT and continue pre-training with Weighted Sampling.”),
PNG
media_image1.png
454
584
media_image1.png
Greyscale
.
Zhang does not expressly disclose providing the pretrained first model for initializing a second document information extraction model to be trained based on labeled documents as second training data (but see Sage Abstract (“Like for many text understanding and generation tasks, pretrained languages models have emerged as a powerful approach for extracting information from business documents. However, their performance has not been properly studied in data-constrained settings which are often encountered in industrial applications. In this paper, we show that LayoutLM, a pre-trained model recently proposed for encoding 2D documents, reveals a high sample-efficiency when fine-tuned on public and real-world Information Extraction (IE) datasets.”), Section 4.1/4.2 (training on SROIE and PO-51K annotated datasets)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang to incorporate the teachings of Sage to fine-tune the pre-trained model for extracting information from business documents, at least because doing so would enable high sample-efficiency.
Zhang does not expressly disclose [a] computer-implemented method (but see Morariu ¶ 71 (“Computing device 1100 includes bus 1110 that directly or indirectly couples the following devices: memory 1112, one or more processors 1114, one or more presentation components 1116, input/output (I/O) ports 1118, input/output components 1120, and illustrative power supply 1122. Bus 1110 represents what may be one or more busses (such as an address bus, data bus, or combination thereof). Although the various blocks of FIG. 11 are shown with lines for the sake of clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would more accurately be gray and fuzzy. For example, one may consider a presentation component such as a display device to be an I/O component. Also, processors have memory.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang and Sage to incorporate the teachings of Morariu to store the claimed process as instructions executable by a processor, at least because doing so would enable a processing device to perform any of the operations required by the document information extraction algorithm.
Claim 11 is a CRM claim corresponding to claim 1 and, therefore, is similarly rejected. Zhang and Sage do not expressly disclose [a] non-transitory, computer-readable medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors (but see Morariu ¶ 73 (“Memory 1112 includes computer storage media in the form of volatile and/or nonvolatile memory. As depicted, memory 1112 includes instructions 1124. Instructions 1124, when executed by processor(s) 1114 are configured to cause the computing device to perform any of the operations described herein, in reference to the above discussed figures, or to implement any program modules described herein. The memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical-disc drives, etc. Computing device 1100 includes one or more processors that read data from various entities such as memory 1112 or I/O components 1120. Presentation component(s) 1116 present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang and Sage to incorporate the teachings of Morariu to store the claimed process as instructions in a computer storage medium, at least because doing so would enable a processing device to perform any of the operations required by the document information extraction algorithm.
Claim 16 is a system claim corresponding to claim 1 and, therefore, is similarly rejected. Zhang and Sage do not expressly disclose [a] computer-implemented system comprising: one or more processors; and one or more computer-readable memories coupled to the one or more processors and having instructions stored thereon that are executable by the one or more processors (but see Morariu ¶ 71 (“Computing device 1100 includes bus 1110 that directly or indirectly couples the following devices: memory 1112, one or more processors 1114, one or more presentation components 1116, input/output (I/O) ports 1118, input/output components 1120, and illustrative power supply 1122. Bus 1110 represents what may be one or more busses (such as an address bus, data bus, or combination thereof). Although the various blocks of FIG. 11 are shown with lines for the sake of clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would more accurately be gray and fuzzy. For example, one may consider a presentation component such as a display device to be an I/O component. Also, processors have memory.”), ¶ 72 (“Computing device 1100 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 1100 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang and Sage to incorporate the teachings of Morariu to store the claimed process as instructions executable by a processor, at least because doing so would enable a processing device to perform any of the operations required by the document information extraction algorithm.
Regarding claim 2, Zhang, in view of Sage and Morariu, discloses the invention of claim 1 as discussed above. Zhang and Sage do not expressly disclose wherein obtaining the training data includes performing an optical character recognition to determine words as part of at least some of the plurality of unlabeled document (but see Morariu (US 2023/0376687 A1; published Nov. 23, 2023) ¶ 46 (“In an embodiment, an OCR model, CNN, and/or other machine learning model generates a set of input feature vectors based at least in part on the document 320, the set of input feature vectors are processed by the multi-modal multi-granular model and then provided, as set of output feature vectors (e.g., the result of the multi-modal multi-granular model processing the set of input feature vectors) to a document classification model to perform the document classification task.”), ¶ 50 (“In one example, an OCR application extracts characters, words, and/or sub-words from the document and provides candidate regions and/or bounding boxes. In various embodiments, the textual content of a particular granularity is provided to the sentence encoder and a vector is obtained.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang to incorporate the teachings of Morariu to use an OCR application to extract characters, words, and/or sub words from the document, at least because doing so would enable extract text with various granularities from the document and process it by a sentence encoder. See Morariu ¶ 50.
Claim 12 is a CRM claim corresponding to claim 2 and, therefore, is similarly rejected.
Claim 17 is a system claim corresponding to claim 2 and, therefore, is similarly rejected.
Regarding claim 3, Zhang, in view of Sage and Morariu, discloses the invention of claim 1 as discussed above. Zhang further discloses wherein pretraining the first model comprises:
tokenizing each document of the plurality of unlabeled documents into a respective sequence of word tokens and spatial embeddings; (Zhang Section 2.1 Masked Language Modeling (“For a sentence S = {t1,t2,…,tn}, where n is the number of tokens and ti is a token, the standard masking strategy as in [1] randomly chooses 15% of tokens to mask.”))
calculating the individual masking rates for the word tokens that identify the informative relevance of respective word tokens in the plurality of unlabeled documents; and (Zhang Section 2.2 Weighted Sampling (“In order to tackle the frequency bias problem, we propose two weighted sampling strategies, namely, Frequency Weighted Sampling and Dynamic Weighted Sampling, to compute the masking probability for each token, based on statistical signals and model-based signals, respectively.”), Section 2.2.1 Frequency Weighted Sampling (“A natural statistical signal characterizing the informativeness of a token w is its frequency freq(w) in the pre-training corpus.”), Section 2.2.2 Dynamic Weighted Sampling (“Frequency Weighted Sampling produces constant sampling probabilities for tokens and does not consider the learning status of the backbone masked language model that it applies for. We hypothesize that the signal of informativeness of a token w may also be derived from how poorly a masked language model predicts it. Therefore, we propose the Dynamic Weighted Sampling strategy shown in Figure 2. We use a weight dictionary in memory to store the sampling weights of each token after each batch in each iteration instead of updating the sampling weights after processing all batches in an iteration as the sampling only happens once in the latter case.”))
learning a representation of the first model as a document foundation model, wherein the first model includes weights as parameters of the model defining strength of connection across different layers in the model (Zhang Section 2.2.2. Dynamic Weighted Sampling (“Firstly, we set an initial sampling weight wt(ti) = 1 for each token ti ∈ T in the weight dictionary, where T denotes all tokens in the pre-training dataset. Then, we compute the sampling probabilities using sampling weights in the weight dictionary based on Eqn. 3 and train a masked language model. During each mini-batch, the masked tokens are predicted by the current model and we compute the total cross-entropy loss for token ti as:
PNG
media_image2.png
78
672
media_image2.png
Greyscale
where x denotes the input masked sequence and θ denotes the parameters of the current masked language model. Then, we use Lti to compute the sampling weight wt(ti) based on Eqn. 5.
PNG
media_image3.png
70
684
media_image3.png
Greyscale
where τ is a temperature parameter with default as 0.2. Finally, for the next mini-batch, we compute the sampling probability p(ti) for token ti by normalizing wt(ti) following Eqn. 3.”)).
Claim 13 is a CRM claim corresponding to claim 3 and, therefore, is similarly rejected.
Claim 18 is a system claim corresponding to claim 3 and, therefore, is similarly rejected.
Regarding claim 4, Zhang, in view of Sage and Morariu, discloses the invention of claim 1 as discussed above. Zhang further discloses wherein the word tokens include at least one of single words or subwords (Zhang Section 2.1 Masked Language Modeling (“For a sentence S = {t1,t2,…,tn}, where n is the number of tokens and ti is a token, the standard masking strategy as in [1] randomly chooses 15% of tokens to mask.”)).
Claim 14 is a CRM claim corresponding to claim 4 and, therefore, is similarly rejected.
Claim 19 is a system claim corresponding to claim 4 and, therefore, is similarly rejected.
Regarding claim 5, Zhang, in view of Sage and Morariu, discloses the invention of claim 1 as discussed above. Zhang further discloses wherein the pretraining comprises:
defining sequences of word tokens for each of the plurality of unlabeled documents, wherein the sequences have different lengths corresponding to a count of the word tokens determined for each of the plurality of unlabeled documents after performing text recognition; and (Zhang Section 2.1 Masked Language Modeling (“For a sentence S = {t1,t2,…,tn}, where n is the number of tokens and ti is a token, the standard masking strategy as in [1] randomly chooses 15% of tokens to mask.”))
obtaining the individual masking rates for each word token in the sequences of word tokens (Section 2.2.2 Dynamic Weighted Sampling (“Frequency Weighted Sampling produces constant sampling probabilities for tokens and does not consider the learning status of the backbone masked language model that it applies for. We hypothesize that the signal of informativeness of a token w may also be derived from how poorly a masked language model predicts it. Therefore, we propose the Dynamic Weighted Sampling strategy shown in Figure 2. We use a weight dictionary in memory to store the sampling weights of each token after each batch in each iteration instead of updating the sampling weights after processing all batches in an iteration as the sampling only happens once in the latter case.”)).
Claim 15 is a CRM claim corresponding to claim 5 and, therefore, is similarly rejected.
Regarding claim 7, Zhang, in view of Sage and Morariu, discloses the invention of claim 1 as discussed above. Zhang does not expressly disclose training the second document information extraction model for document information extraction based on labeled documents, wherein the training comprises: initializing the second document information extraction model based on input provided from pretrained first model (but see Sage, Clément, et al. "Data-efficient information extraction from documents with pre-trained language models." International Conference on Document Analysis and Recognition. Cham: Springer International Publishing, 2021 Abstract (“Like for many text understanding and generation tasks, pretrained languages models have emerged as a powerful approach for extracting information from business documents. However, their performance has not been properly studied in data-constrained settings which are often encountered in industrial applications. In this paper, we show that LayoutLM, a pre-trained model recently proposed for encoding 2D documents, reveals a high sample-efficiency when fine-tuned on public and real-world Information Extraction (IE) datasets.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang to incorporate the teachings of Sage to fine-tune the pre-trained model for extracting information from business documents, at least because doing so would enable high sample-efficiency.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Sage, and Morariu as applied to claim 1 above, and further in view of Wu, Yi, David Bamman, and Stuart Russell. "Adversarial training for relation extraction." Proceedings of the 2017 conference on empirical methods in natural language processing. 2017 (“Wu”).
Regarding claim 8, Zhang, in view of Sage and Morariu, discloses the invention of claim 1 as discussed above. Zhang and Sage do not expressly disclose executing adversarial training over the second document information extraction model that is initialized based on the pretrained first model (but see Wu Abstract (“Adversarial training is a mean of regularizing classification algorithms by generating adversarial noise to the training data. We apply adversarial training in relation extraction within the multi-instance multi-label learning framework. We evaluate various neural network architectures on two different datasets. Experimental results demonstrate that adversarial training is generally effective for both CNN and RNN models and significantly improves the precision of predicted relations.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang to incorporate the teachings of Wu to use adversarial training to regularize the document information extraction model, at least because doing would significantly improve the precision of predicted tokens.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Sage, and Morariu as applied to claim 1 above, and further in view of Xia (US 2021/0374603 A1; published Dec. 2, 2021).
Regarding claim 9, Zhang, in view of Sage and Morariu, discloses the invention of claim 1 as discussed above. Zhang and Sage do not expressly disclose:
initializing the second model based on weights provided from the first model as pretrained; (but see Xia ¶ 85 (“Specifically, both the encoder and the decoder of the CLANG model 130 use six trans-former layers. Pre-trained weights from BERT-base are used to initialize the embeddings and the transformer layers. The weights from the first six layers in BERT-base are used to initialize the trans-former layers in the encoder and the later six layers are used to initialize the decoder.”))
performing fine-tuning of the pretrained first model based on labeled business documents as second training data, wherein the labeled business documents are labeled and evaluated according to a virtual adversarial training (VAT); and based on the performed fine-tuning, generating a classifier on top of the second document information extraction model (but see Xia ¶ 81 (“Utterances for few-shot intents can be generated by sampling two latent variables, z.sub.d and z.sub.a, separately from multivariate standard Gaussian distributions. Beam search may be applied to do the generation. To improve the diversity of the generated utterances, the latent variables may be sampled for s times and the top k results are stored for each time. The sampled latent variables are then sent to the decoder 403 in FIG. 4 to generate an utterance. These generated utterances are then added to the original training dataset to alleviate the scarce annotation problem. A language model, such as BERT, may then be fine-tuned with the augmented dataset to solve the generalized few-shot intent detection task.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Zhang to incorporate the teachings of Xia to fine tune the pre-trained model with an augmented data set, at least because doing so would enable solving the generalized few-shot intent detection task.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Song et al. (US 2022/0138572 A1; published May 5, 2022)
Tang et al. (US 12,494,077 B1; published Dec. 9, 2025).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAHID KHAN whose telephone number is (571)270-0419. The examiner can normally be reached M-F, 9-5 est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571)272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAHID K KHAN/Primary Examiner, Art Unit 2146
Prosecution Insights