Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
With regards to claim 1-12 and 14-20, their prior 35 USC § 101 rejections are withdrawn in view of applicant’s amendments.
With regards to claim 13, the 35 USC 112(b) rejection is withdrawn in view of applicant’s amendments.
The following rejections are withdrawn in view of applicant’s amendments:
Claim(s) 1-12, 14-16, 19 and 20 rejected under 35 U.S.C. 102(a)(2) as being anticipated by Meng et al (US Application: US 20230022845, published: Jan 26, 2023, filed: Jul. 13, 2021).
Claim(s) 13 rejected under 35 U.S.C. 103 as being unpatentable over Meng et al (US Application: US 20230022845, published: Jan 26, 2023, filed: Jul. 13, 2021) in view of Tretiakoff et al (US Application: US 2003/0134256, published: Jul. 17, 2003, filed: Jan. 15, 2003).
Claim(s) 17 rejected under 35 U.S.C. 103 as being unpatentable over Meng et al (US Application: US 20230022845, published: Jan 26, 2023, filed: Jul. 13, 2021) in view of Panakkal (US Application: US 2021/0042516, published: Feb. 11, 2021, filed: Dec. 20, 2019).
Claim(s) 18 rejected under 35 U.S.C. 103 as being unpatentable over Meng et al (US Application: US 20230022845, published: Jan 26, 2023, filed: Jul. 13, 2021) in view of Xu et al (US Application: US 20240095460, published: Mar. 21, 2024, filed: Sep. 19, 2022).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-11, 14-16, 19 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Meng et al (US Application: US 20230022845, published: Jan 26, 2023, filed: Jul. 13, 2021) in view of Tretiakoff et al (US Application: US 2003/0134256, published: Jul. 17, 2003, filed: Jan. 15, 2003) in view of Panakkal (US Application: US 2021/0042516, published: Feb. 11, 2021, filed: Dec. 20, 2019).
With regards to claim 1, Meng et al teaches a process for querying one or more documents (fig. 15: a computer implemented processing having a processor and memory is implemented), comprising:
receiving a document query including a natural language query and information identifying one or more document images (Fig. 7, paragraph 0101, claim 9 of Meng et al, a user can provide a question that identifies an invoice document such as “what is the total amount of the invoice?”);
processing the document query in a machine learning model, the machine learning model being trained using language features and vision features for joint learning (paragraph 0077: the document query can be processed through a BERT (LLM type model)); and
generating an answer based on processing of the document query by the machine learning model, the answer including text and a bounding box indicating a location of the source of the answer (Fig. 7: an answer is provided to include such as ‘Amount’ in an answer summary area and attention is rendered in the form of an arrow pointing to a bounding box of a location of the source of the answer (ref 704) );
wherein processing comprises partitioning the one or more document images into different regions (paragraph 0070: the document is partitioned into block regions).
However Meng et al does not expressly teach … and resampling the one or more document images associated with the different regions, and wherein the machine learning model is pretrained by instructing the model to predict the line above, below, to the left and to the right of a given item of text.
Yet Tretiakoff et al teaches … and resampling the one or more document images associated with the different regions (paragraph 0032: different regions can be identified for resampling).
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to have modified Meng et al’s ability to process /analyze images having different regions, such that specific regions could have been selected for resampling as taught by Tretiakoff et al. The combination would have allowed an ability to assessed regions of the image(s) in optimal resolution/quality to yield a better/more-positive assessment/analysis result.
However the combination does not expressly teach … wherein the machine learning model is pretrained by instructing the model to predict the line above, below, to the left and to the right of a given item of text.
Yet Panakkal teaches … wherein the machine learning model is pretrained by instructing the model to predict the line above, below, to the left and to the right of a given item of text (paragraphs 0024, 0027, 0030, 0036, 0037, 0045, 0052, and Fig. 3: values with respect to text horizontally or vertical to an item of text are recognized through trained/learned data which includes vertically and horizontally aligned data (as known in the art as data in any direction vertically (up and/or down) and data in any direction horizontally (left and right). More specifically the model is trained to identify regions with particular characteristics and these characteristics include for example a label on a line is left of a line-value (which also means the line value is to the right of the label on the line), or a line-label above a line-value (which also means the value of the line is below the line-label). Thus the model recognizes/predictions the regions having these characteristics of above, below, left or right of a given item of text. Since the claim language leaves open to interpretation what is ‘a given item of text’, since it does not explain how a given item of text is identified/selected, the examiner emphasizes that ‘a given item of text’ could be either one of the label(s) on a line or one of the value(s) in a line mentioned above).
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to have modified Meng et al and Tretiakoff et al’s ability to pretrain a model to help predict and correlate data with corresponding block areas for processing document image(s), such that the pretraining data would have further included recognizing regions/areas based upon spatial context that includes spatial relationships of text content such as horizontal (left, right) and/or vertical (above below) for text content items of label and value, as taught by Panakkal et al. The combination would have allowed Meng et al to have implemented efficient extraction of specific information from scanned documents (Panakkal et al, paragraph 0002).
With regards to claim 2. The process of claim 1, Meng et al teaches wherein the language features are associated with a large language model (LLM) (paragraph 0077: a BERT model is a type of LLM model).
With regards to claim 3. The process of claim 1, Meng et al teaches wherein the vision features are associated with a large vision model (paragraph 0125: a vision machine learning model is implemented)
With regards to claim 4. The process of claim 1, Meng et al teaches wherein the machine learning model is based on a language- image model (Fig. 11, paragraph 0111-0116 and 0125: the system integrates a vision CNN model in combination with a BERT language model).
With regards to claim 5. The process of claim 4, Meng et al teaches wherein the model is pretrained on image-text tasks (Fig. 5, paragraph 0030: pre training and fine tuning is implemented to integrate text natural language tasks to detected blocks in a document image).
With regards to claim 6. The process of claim 5, Meng et al teaches wherein the model is tuned using one or more datasets and one or more tasks (paragraph 0063: the model is tuned with invoice document data and tasks relating to question/answer).
With regards to claim 7. The process of claim 6, Meng et al teaches wherein the one or more datasets include key-value pair data, specific entity data or generic entity data (paragraph 0062 and 0063: pairs of context, question and/or answer(s) are included with respect to entity data such as Bill entity or named entity).
With regards to claim 8. The process of claim 4, Meng et al teaches wherein the model is fine-tuned to predict text in a bounding box specified by one or more polygon vertices (paragraph 0065, 00125, Fig. 5 and Fig 7: the model is tuned for question/answer with respect to coordinates/location that are blocked and as shown in Fig. 7, the blocks are specified for display having vertices/corners where lines meet).
With regards to claim 9. The process of claim 8, Meng et al teaches wherein the model uses machine learning inferences to make predictions on new data (paragraph 0064: inference makes predictions on data based on pre training and fine tuning).
With regards to claim 10. The process of claim 9, Meng et al teaches wherein the model uses GPUs or TPUs for inferencing (paragraphs 0064, 0116: a GPU is implemented to implement the model (having inferencing).
With regards to claim 11. The process of claim 1, Meng et al teaches wherein processing comprises applying optical character recognition (OCR) to the one or more document images (paragraph 0039: OCR is applied to a document image).
With regards to claim 14. The process of claim 1, Meng et al teaches comprising verifying the answer against optical character recognition (OCR) generated text and a location parameter (Fig 7: the answer is provided for user visual verification via answer text in right pane and reference location of a bounding box/block for the answer, which is rendered).
With regards to claim 15. The process of claim 14, Meng et al teaches wherein the location parameters define the bounding box (as similarly explained in the rejection of claim 14, location data is referenced to position/render a box at location in document image), and is rejected under similar rationale.
With regards to claim 16. The process of claim 1, Meng et al teaches wherein the machine learning model is pretrained by masking spans of optical character recognition (OCR) serialized text and requesting the machine learning model to predict the spans of masked OCR serialized text (paragraph 0093: the text obtained via OCR is then masked and predictions for the masked spans are implemented).
With regards to claim 19, Meng et al, Tretiakoff et al and Panakkal teaches a system for querying one or more documents, comprising: one or more processing devices; a non-transitory computer-readable medium storing instructions and coupled to the one or more processing devices, the instruction causing the one or more processing devices to: receive a document query including a natural language query and information identifying one or more document images; process the document query in a machine learning model, the machine learning model being trained using language features and vision features for joint learning, wherein process the query comprises partition the one or more document images into different regions and resampling the one or more document images associated with the different regions; and generate an answer based on processing of the document query by the machine learning model, the answer including text and a bounding box indicating a location of the source of the answer, wherein the machine learning model is pretrained by instructing the model to predict the line above, below, to the left and to the right of a given item of text, as similarly explained in the rejection of claim 1, and is rejected under similar rationale.
With regards to claim 20. The system of claim 19, Meng et al teaches wherein the instructions cause the one or more processing devices to process the document query by: applying optical character recognition (OCR) to the one or more document images to produce OCR generated text (as similarly explained in the rejection of claim 11, and is rejected under similar rationale); and verifying the answer against the OCR generated text and a location parameter (Fig. 2, paragraph 0070: the document is partitioned into block regions).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Meng et al (US Application: US 20230022845, published: Jan 26, 2023, filed: Jul. 13, 2021) in view of Tretiakoff et al (US Application: US 2003/0134256, published: Jul. 17, 2003, filed: Jan. 15, 2003) in view of Panakkal (US Application: US 2021/0042516, published: Feb. 11, 2021, filed: Dec. 20, 2019) in view of Xu et al (US Application: US 20240095460, published: Mar. 21, 2024, filed: Sep. 19, 2022).
With regards to claim 18. The process of claim 1, Meng et al teaches wherein the natural language query, as similarly explained in the rejection of claim 1, and is rejected under similar rationale.
However Meng et al, Tretiakoff et al and Panakkal does not expressly teach the query comprises an audible question.
Yet Xu et al teaches the query comprises an audible question (Abstract, paragraph 0003: a query/question provided in an audible/speech based manner is implemented , such that the question is then processed and an answer is provided via a model with an answer ).
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to have modified Meng et al, Tretiakoff et al and Panakkal’s ability to use a model to process a user’s natural language query/prompt to provide an answer, such that the well-known technique to process a prompt/question as an audio prompt is also supported when providing the prompt to the model, as taught by Xu et al. The combination would have allowed Meng et al, Tretiakoff et al and Panakkal to have included an additional modality (speech) to provide a user’s question to a large language model that is able to flexibly interpreted questions having different forms (Xu et al, paragraph 0004).
Response to Arguments
Applicant's arguments filed 12/30/2025 have been fully considered but they are not persuasive.
With regards to claim 17, the applicant argues Panakkal does not teach “to predict” what’s ‘above, below, to the left’ or to the right of the given item of text’ and the applicant further argues with respect to paragraphs 0036, 0037 and 0045 of Panakkal that ‘Panakkal merely discloses training a model to identify regions of interest’ and there is no teaching in the prior art relevant to predicting anything relative to a given item of text.
However this argument is not persuasive as the claim requires predicting the line above, below to the left and to the right of a given item of text, and Panakkal’s ability to recognize regions of interest is based on this concept. More specifically, as indicated in paragraphs 0024, 0027, 0036, 0037, 0045 and 0052: Panakkal predicts regions of interest through spatial prediction/recognition of line data. In other words, the model is trained to identify regions with prediction/recognition of particular spatial characteristics and these spatial characteristics include for example a text label on a line being left of a line-value (which also means the line value is spatially to the right of the label on the line), or a line-label above a line-value (which also means the value of the line is spatially below the line-label). Thus the model recognizes/predicts the regions by further predicting/recognizing these characteristics of above, below, left or right of a given item of text. Since the claim language leaves open to interpretation what is ‘a given item of text’, since it does not explain how a given item of text is identified/selected, the examiner emphasizes that ‘a given item of text’ could be either one of the label(s) on a line or one of the value(s) in a line mentioned above. The examiner points out a few things in the interest for applicant’s consideration: 1) Potential clarification of training data that is masked out, and spans of masked data being predicted in combination with the claimed lined prediction, 2) The current claim language includes interpretation of predicting a ‘line’ in general (which can encompass predicting spatial aspects of types of line data), rather than predicting values for an answer, and thus potential consideration for what ‘predicting the line’ in the claim encompasses, and 3) the claimed limitation that references ‘answer’ and ‘including text’ and ‘bounding box indication a location ‘ does not reference the ‘predict the line’ being above, below, left, right’ result of the ‘pretrained step and thus the generated answer does not use any spatial (above, below, left, right) line prediction and does not use or is based on ‘a given item of text’.
The applicant argues the remaining independent claim(s) is/are allowable in view of the arguments presented by the applicant for claim 1 above, however this argument is not persuasive since claim 1 has been shown/explained to be rejected above.
The applicant argues the claims that depend upon the independent claims are allowable due to applicant’s arguments that the independent claims are allowable. However this argument is not persuasive since the independent claims have been explained/shown to be rejected above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILSON W TSUI whose telephone number is (571)272-7596. The examiner can normally be reached Monday - Friday 9 am -6 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Queler can be reached at (571) 272-4140. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILSON W TSUI/Primary Examiner, Art Unit 2172
Prosecution Insights