DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/19/24 and 08/20/25 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8 and 15-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al US 2020/0273078 in view of Saraswat et al US 2019/0205636 further in view of Costello US 2020/0065384 further in view of King et al US 7593605.
Regarding claim 1, Xu et al teaches A computer-implemented method for extracting data from an image of a document, the computer-implemented method comprising:
retrieving object data pertaining to an object that has been detected in the image of the document, the object data denoting at least a portion of the document having the object (identifying key-value pairs in a document (fig 3 and paragraph 0070), the object data denoting at least a portion of the document having the object (fig 2B and paragraph 0067);
acquiring text pertaining to the portion of the document having the object, the text having been recognized from the image of the document (the parsing system 100 uses OCR techniques to identify the text inside each bounding box, and thereafter identifies valid key-value pairs enclosed by the bounding boxes (fig 2B and paragraph 0067);
determining a key type (date, time, invoice # paragraph 0057) for the object based on the text and a machine learned model (each bounding box 114 is predicted to enclose textual data that defines: (i) a key, and (ii) a value corresponding to the key. For example, a bounding box may enclose the textual data “Name: John Smith” which defines the key “Name” and the corresponding value “John Smith”. The detection model 106 may be configured to generate bounding boxes 114 which each enclose a single key-value pair, detection model to implicitly recognized visual signals to enable to identified key value pair (paragraph 0046 and 0051),
determining a value for the object based on the determined key type for the object (each bounding box 114 is predicted to enclose textual data that defines: (i) a key, and (ii) a value corresponding to the key. For example, a bounding box may enclose the textual data “Name: John Smith” which defines the key “Name” and the corresponding value “John Smith”. The detection model 106 may be configured to generate bounding boxes 114 which each enclose a single key-value pair (paragraph 0046 and 0051); and
wherein the determining of the key type primarily attempts to determine the key type using the character level neural network, wherein the key type is determined using the character level neural network model if the primary attempt was successful (detection model 106 may be a neural network object detection model (e.g., including one or more convolutional neural networks), where the “objects” correspond to key-value pairs in the document (paragraph 0053-0054) Note: the object is detected by detection model 106 thereby making the attempt successful,
Xu et al fails to teach storing the determined key type and the determined value for the object,
Saraswat et al teaches storing the determined key type and the determined value for the object (Once identified, one or more key-value pairs data objects and or free text data objects may be stored in the data repository 140 and associated with the corresponding electronic document (paragraph 0041),
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al with storing the determined key type and the determined value for the object.
The reason of doing so would be to allow a device to identify text or words more accurately
Xu et al in view of Saraswat et al fails to teach the machine learned model pertaining to at least a character level neural network model and/or a pattern matching model;
Costello teaches the machine learned model pertaining to at least a character level neural network model and/or a pattern matching model (the least one convolutional neural networks-based model can be a character-level convolutional neural network model (short as CharCNN hereafter) (paragraph 0068). The plurality of models include at least one recurrent neural networks-based model (paragraph 0069). As such, each of the at least one recurrent neural networks-based model can be a bidirectional GRU cell-based recurrent neural network model (short as BiGRU hereafter) (paragraph 0070)
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al with machine learned model pertaining to at least a character level neural network model and/or a pattern matching model.
The reason of doing so would be to allow a device to identify and predict text or words more accurately
Xu et al in view of Saraswat et al further in view of Costello fails to teach wherein the determining of the key type secondarily attempts to determine the key type using the pattern matching model if the primary attempt was unsuccessful.
King et al teaches wherein the determining of the key type secondarily attempts to determine the key type using the pattern matching model if the primary attempt was unsuccessful (a low resolution mobile phone camera is used to take a picture of a rendered document but it cannot resolve the image to the character level. The document can still be identified by searching for a matching sequential pattern of long and short words (column 92, lines 20-30).
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al further in view of Costello with wherein the determining of the key type secondarily attempts to determine the key type using the pattern matching model if the primary attempt was unsuccessful.
The reason of doing so would be to allow to accurately identify and predict text or words
Regarding claim 2, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the text is recognized through Optical Character Resolution (OCR) of at least a portion of the document (Xu et al: system 100 may use OCR techniques to identify the text in the document 102 (paragraph 0055).
Regarding claim 3, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the object is an object block, and the object data is provided within the object block (Xu et al: bounding boxes 114, the system 100 processes the portion of the image 112 enclosed by the bounding box using the OCR engine 108 to identify the textual data (i.e., the text 116) enclosed by the bounding box (paragraph 0055).
Regarding claim 4, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the object is a key-value block (Xu et al: “objects” correspond to key-value pairs in the document (paragraph 0053).
.
Regarding claim 5, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the key type comprises a textual descriptor (Xu et al: text enclosed by the bounding box includes a valid key, the filtering engine 110 identifies a “type” (e.g., alphabetical, numerical, temporal) of the portion of text enclosed by the bounding box (paragraph 0059).
Regarding claim 6, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the machine learned model is a Natural Language Processing (NLP) model (Saraswat et al: smart data engine computing system 130 has been programmed with a generic framework running on a machine learning and/or natural language processing (NLP) backbone built on top computing language (e.g., Python, Java, C, C++, and the like) libraries (paragraph 0036).
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al further in view of Costello further in view of King et al with wherein the machine learned model is a Natural Language Processing (NLP) model.
The reason of doing so would be to allow a device to identify text or words and to process the text or words so that a user can easily identify them
Regarding claim 7, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein a user input is provided to the character level neural network model, and wherein the character level neural network model determines the key type for the object based at least in part on the user input (Xu et al: Before using the detection model 106, the system 100 may identify a “class” of the document 102 (e.g., invoice, pay stub, or purchase receipt). For example, a user of the system 100 may identify (user input) the class of the document 102 upon providing the document to the system 100 (paragraph 0055).
Regarding claim 8, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the document comprises a business transaction document (Xu et al: a document 102 (e.g., an invoice, pay stub, or purchase receipt) (paragraph 0043), and wherein the image of the document is provided in a graphical file format (Xu et al: the document 102 can be represented in any appropriate unstructured data format, for example, as a Portable Document Format (PDF) document or as an image document (e.g., a Portable Network Graphics (PNG) or Joint Photographic Experts Group (JPEG) document) (paragraph 0044).
Regarding claim 15, Xu et al teaches A non-transitory computer readable medium including at least computer program code for extracting data from an image of a document (one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus (paragraph 0084), the computer readable medium comprising:
computer program code for retrieving object data pertaining to an object that has been detected in the image of the document, the object data denoting at least a portion of the document having the object (identifying key-value pairs in a document (fig 3 and paragraph 0070), the object data denoting at least a portion of the document having the object (fig 2B and paragraph 0067);
computer program code for acquiring text pertaining to the portion of the document having the object, the text having been recognized from the image of the document (the parsing system 100 uses OCR techniques to identify the text inside each bounding box, and thereafter identifies valid key-value pairs enclosed by the bounding boxes (fig 2B and paragraph 0067);
computer program code for determining a key type for the object based on at least the text (each bounding box 114 is predicted to enclose textual data that defines: (i) a key, and (ii) a value corresponding to the key. For example, a bounding box may enclose the textual data “Name: John Smith” which defines the key “Name” and the corresponding value “John Smith”. The detection model 106 may be configured to generate bounding boxes 114 which each enclose a single key-value pair (paragraph 0046 and 0051);
computer program code for determining a value for the object based on the determined key type for the object (each bounding box 114 is predicted to enclose textual data that defines: (i) a key, and (ii) a value corresponding to the key. For example, a bounding box may enclose the textual data “Name: John Smith” which defines the key “Name” and the corresponding value “John Smith”. The detection model 106 may be configured to generate bounding boxes 114 which each enclose a single key-value pair (paragraph 0046 and 0051); and
wherein the determining of the key type primarily attempts to determine the key type using the character level neural network, wherein the key type is determined using the character level neural network model if the primary attempt was successful (detection model 106 may be a neural network object detection model (e.g., including one or more convolutional neural networks), where the “objects” correspond to key-value pairs in the document (paragraph 0053-0054) Note: the object is detected by detection model 106 thereby making the attempt successful,
Xu et al fails to teach computer program code for storing the determined key type and the determined value for the object,
Saraswat et al teaches computer program code for storing the determined key type and the determined value for the object (Once identified, one or more key-value pairs data objects and or free text data objects may be stored in the data repository 140 and associated with the corresponding electronic document (paragraph 0041)
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al with storing the determined key type and the determined value for the object.
The reason of doing so would be to allow a device to identify text or words more accurately
Xu et al in view of Saraswat et al fails to teach a machine learned model, the machine learned model pertaining to at least a character level neural network model and/or a pattern matching model
Costello teaches a machine learned model, the machine learned model pertaining to at least a character level neural network model and/or a pattern matching model (the least one convolutional neural networks-based model can be a character-level convolutional neural network model (short as CharCNN hereafter) (paragraph 0068). The plurality of models include at least one recurrent neural networks-based model (paragraph 0069). As such, each of the at least one recurrent neural networks-based model can be a bidirectional GRU cell-based recurrent neural network model (short as BiGRU hereafter) (paragraph 0070)
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al with machine learned model pertaining to at least a character level neural network model and/or a pattern matching model.
The reason of doing so would be to allow a device to identify and predict text or words more accurately
Xu et al in view of Saraswat et al further in view of Costello fails to teach wherein the determining of the key type secondarily attempts to determine the key type using the pattern matching model if the primary attempt was unsuccessful.
King et al teaches wherein the determining of the key type secondarily attempts to determine the key type using the pattern matching model if the primary attempt was unsuccessful (a low resolution mobile phone camera is used to take a picture of a rendered document but it cannot resolve the image to the character level. The document can still be identified by searching for a matching sequential pattern of long and short words (column 92, lines 20-30).
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al further in view of Costello with wherein the determining of the key type secondarily attempts to determine the key type using the pattern matching model if the primary attempt was unsuccessful.
The reason of doing so would be to allow to accurately identify and predict text or words
Regarding claim 16, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the text is recognized through Optical Character Resolution (OCR) of at least a portion of the document (Xu et al: system 100 may use OCR techniques to identify the text in the document 102 (paragraph 0055).
.
Regarding claim 17, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the object is an object block, and the object data is provided within the object block (Xu et al: bounding boxes 114, the system 100 processes the portion of the image 112 enclosed by the bounding box using the OCR engine 108 to identify the textual data (i.e., the text 116) enclosed by the bounding box (paragraph 0055).
Regarding claim 18, Xu et al in view of Saraswat et al further in view of Costello further in view of King et al teaches wherein the document comprises a business transaction document (Xu et al: a document 102 (e.g., an invoice, pay stub, or purchase receipt) (paragraph 0043), and wherein the image of the document is provided in a graphical file format (Xu et al: the document 102 can be represented in any appropriate unstructured data format, for example, as a Portable Document Format (PDF) document or as an image document (e.g., a Portable Network Graphics (PNG) or Joint Photographic Experts Group (JPEG) document) (paragraph 0044).
Claim(s) 9-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al US 2020/0273078 in view of Saraswat et al US 2019/0205636 further in view of Costello US 2020/0065384
Regarding claim 9, Xu et al teaches A data extraction system for extracting data from an image of a document, the data extraction system comprising:
a neural network model that receives an object block and recognized text within at least a portion of the document as recognized from the image of the document, the character level neural network model predicting a key type and a value for the object block (detection model 106 may be a neural network object detection model (e.g., including one or more convolutional neural networks), where the “objects” correspond to key-value pairs in the document (paragraph 0053-0054); and
a pattern matching model that receives an object block and recognized text within at least a portion of the document as recognized from the image of the document, (the filtering engine 110 may determine whether the text enclosed by the bounding box includes a valid key from a predetermined set of valid keys. For example, the set of valid keys may include: “Date”, “Time”, “Invoice #”, “Amount Due”, and the like. In comparing different portions of text to determine whether the text enclosed by the bounding box includes a valid key, the filtering engine 110 may determine that two portions of text are “matching” (paragraph 0056-0057),
wherein key type and value predicted using the character level neural network model are accepted and recorded if such prediction is successful (detection model 106 may be a neural network object detection model (e.g., including one or more convolutional neural networks), where the “objects” correspond to key-value pairs in the document (paragraph 0053-0054) Note: the object is detected by detection model 106 thereby making the attempt successful,
Xu et al fails to teach the pattern matching model predicting a key type and a value for the object block
Saraswat et al teaches the pattern matching model predicting a key type and a value for the object block (machine-learning engine may use mass spectra of the historical data together with the knowledge base of procedures, contracts, electronic transaction information, industry and/or business sector information, to predict or identify a set of key value pairs and/or free text summaries (paragraph 0039). Once identified, one or more key-value pairs data objects and or free text data objects may be stored in the data repository 140 and associated with the corresponding electronic document (paragraph 0041)
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al with the pattern matching model predicting a key type and a value for the object block.
The reason of doing so would be to allow a device to identify text or words more accurately
Xu et al in view of Saraswat et al fails to teach a character level neural network model
Costello teaches a character level neural network model (the least one convolutional neural networks-based model can be a character-level convolutional neural network model (short as CharCNN hereafter) (paragraph 0068). The plurality of models include at least one recurrent neural networks-based model (paragraph 0069). As such, each of the at least one recurrent neural networks-based model can be a bidirectional GRU cell-based recurrent neural network model (short as BiGRU hereafter) (paragraph 0070)
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al with a character level neural network model.
The reason of doing so would be to allow a device to identify and predict text or words more accurately
Regarding claim 10, Xu et al in view of Saraswat et al further in view of Costello teaches wherein the object block is a key-value block, wherein key type comprises a textual descriptor (Xu et al: text enclosed by the bounding box includes a valid key, the filtering engine 110 identifies a “type” (e.g., alphabetical, numerical, temporal) of the portion of text enclosed by the bounding box (paragraph 0059), and wherein the image of the document is provided in a graphical file format (Xu et al: document 102 can be represented in any appropriate unstructured data format, for example, as a Portable Document Format (PDF) document or as an image document (e.g., a Portable Network Graphics (PNG) or Joint Photographic Experts Group (JPEG) document) (paragraph 0044)
.
Regarding claim 11, Xu et al in view of Saraswat et al further in view of Costello teaches wherein the image of the document is provided in a graphical file format (Xu et al: document 102 can be represented in any appropriate unstructured data format, for example, as a Portable Document Format (PDF) document or as an image document (e.g., a Portable Network Graphics (PNG) or Joint Photographic Experts Group (JPEG) document) (paragraph 0044)
.
Regarding claim 12, Xu et al in view of Saraswat et al further in view of Costello teaches wherein the object block is a key-value block (Xu et al: “objects” correspond to key-value pairs in the document (paragraph 0053).
Claim(s) 13 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al US 2020/0273078 in view of Saraswat et al US 2019/0205636 further in view of Costello US 2020/0065384 further in view of Nuolf et al US 2020/0125635.
Regarding claim 13, Xu et al in view of Saraswat et al further in view of Costello teaches all of the limitations of claim 9
Xu et al in view of Saraswat et al further in view of Costello fails to teach wherein a user input is provided to the character level neural network model, and wherein the character level neural network model predicts a key type and a value for the object block based at least in part on the user input.
Nuolf et al teaches wherein a user input is provided to the character level neural network model, and wherein the character level neural network model predicts a key type and a value for the object block based at least in part on the user input (user can enter a description of the item into that data field. When the user tabs to the next data field, the input received from the user can be automatically transmitted to the machine learning and artificial intelligence model. The model can evaluate the input (e.g., the text describing the item) and predict which four code segments corresponds to the input (paragraph 0008)
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al further in view of Costello with wherein a user input is provided to the character level neural network model, and wherein the character level neural network model predicts a key type and a value for the object block based at least in part on the user input.
The reason of doing so would be to allow a device to identify text or words more accurately
Regarding claim 14, Xu et al in view of Saraswat et al further in view of Costello teaches all of the limitations of claim 9
Xu et al in view of Saraswat et al further in view of Costello fails to teach wherein a user input is provided to the pattern matching model, and wherein the pattern matching model predicts a key type and a value for the object block based at least in part on the user input.
Nuolf et al teaches wherein a user input is provided to the pattern matching model, and wherein the pattern matching model predicts a key type and a value for the object block based at least in part on the user input (the user can select the data field corresponding to the description of the item associated with the line item. Then, the user can enter a description of the item into that data field. When the user tabs to the next data field, the input received from the user can be automatically transmitted to the machine learning and artificial intelligence model. The model can evaluate the input (e.g., the text describing the item) (pattern matching by matching the description input of the user with the data fields)
Therefore, it would have been obvious to a person with ordinary skill in the art to have modified Xu et al in view of Saraswat et al further in view of Costello with wherein a user input is provided to the pattern matching model, and wherein the pattern matching model predicts a key type and a value for the object block based at least in part on the user input.
The reason of doing so would be to allow a device to identify text or words more accurately
Conclusion
Any inquiry concerning this communication should be directed to Michael Burleson whose telephone number is (571) 272-7460 and fax number is (571) 273-7460. The examiner can normally be reached Monday thru Friday from 8:00 a.m. – 4:30p.m. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi Sarpong can be reached at (571) 270- 3438.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Michael Burleson
Patent Examiner
Art Unit 2681
Michael Burleson
September 30, 2026
/MICHAEL BURLESON/
/AKWASI M SARPONG/SPE, Art Unit 2681 10/02/2026
