DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/21/2026 has been entered.
Response to Amendment
Claims 1-2, 6-16, and 18-24 were previously pending and subject to final action filed on 04/06/2026. In the response filed 05/21/2026, claims 1, 7, 15, 16 and 24 were amended, claim 12, 18 and 23 were canceled. Claims 25-27 were newly added claims. Therefore, claims 1-2, 6-11, 13-16, and 19-27 are currently pending and subject to the final action below.
Response to Arguments
Applicant’s arguments, see page 11, filed 05/21/2026, with respect to the Objection to the Drawing have been fully considered and are persuasive. The amendment to the specification filed on 05/21/2026 was amended to recite element 480. The objection of the Drawing has been withdrawn.
Applicant’s arguments, see pages 11-16, filed on 05/21/2026 with respect to claims 1-2, 6-16, and 18-24 under the prior rejections have been considered but are moot because the arguments do not apply to the new combination of references being used in the current rejection.
Examiner Notes:
The term “full cell” is a cell that is not empty and contain data within the cell. The term “empty cell” is a cell consisting of “no character/numerical data”, “no data” or “white spaces without no character/numerical data”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6-11, 13-16, 19-22, and 24-25 are rejected under 35 U.S.C. 103 as being unpatentable over Shraga US 20220043794 A1, Pub Date: Feb. 10, 2022) in view of DUTA (US 20190340240 A1, Pub Date: Nov. 7, 2019) in view of VENKATESWARAN (US 20200097711 A1, Pub Date: Mar. 26, 2020) and further in view of Ahmed Nassar (TableFormer: Table Structure Understanding with Transformers, Pub Date: Mar. 11, 2022, hereinafter “Nassar”).
Regarding independent claim 1, Shraga teaches: A computer implemented method of determining a table structure of a table, wherein the method comprises:
receiving an image of the table; (Shraga − [0002] receiving a query and searching for pertinent information in a corpus of electronic data—be it…, images; Information retrieval systems, which are used for conducting searches in the organization's internal collection (corpus) of electronic documents and other data. [0003] Electronic documents, which typically contain unstructured data in the form of free text, sometimes also include tables—which are a form of structured data.) Receiving images of an electronic document that includes tables.
and receiving the table structure of the table in response to inputting the image of the table into a table recognition neural network, (Shraga − [0010] In some embodiments, the separate machine learning encoders comprise a Recurrent Convolutional Neural Network (RCNN) that encodes the description of the table. a first Three-Dimensional Convolutional Neural Network (3D-CNN) that encodes the rows of the table; and a second 3D-CNN that encodes the columns of the table. [0032] The instructions of multimodal table encoding module 108 may cause system 100 to receive an electronic document which includes a table 110, process the electronic document, and output a fused encoding of the table 112.) process the electronic document (receiving) using a RCNN for encoding the table and output a fused encoding of the table 112.
wherein the table structure comprises a row of tokens for each row of the table. (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
Shraga continue to teach a single token for end of rows/columns but does not explicitly teach: a single token assigned to each cell of the table
However, DUTA teaches: wherein the table structure comprises a single token assigned to each cell of the table, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
wherein the single token assigned to each cell of the table is selected from a finite number of tokens, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
wherein the finite number of tokens includes at least one cell identifier token comprised of a full cell identifier and an empty cell identifier, (DUTA − [0014] FIG. 3 provides an example of splitting and merging column grids of a table candidate generated by the Table Extractor, as described herein. Fig. 3 shows at least one empty cell and full cell, each with token identifier.)
identifying an error in a token of the table structure (DUTA – [0098] In other words, the tokenization process may have errors. For example, what should be one token may be incorrectly broken into multiple adjacent tokens.)
and wherein the table structure comprises a row of tokens for each row of the table; (DUTA – [0102] The Table Extractor inserts horizontal grid lines between consecutive rows of tokens. Rows of tokens for each row of the table identified by the Table Extractor)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga and DUTA to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Shraga does not explicitly teach: identifying an error in a token of the table structure by sequentially comparing a single token assigned to each cell to a predefined logic
VENKATESWARAN teaches: identifying an error in a token of the table structure by sequentially comparing a single token assigned to each cell to a predefined logic; (VENKATESWARAN − [0011] FIG. 5C illustrates a screen for a converted and validated structured table using a spreadsheet export application, according to an example; [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring.)
and correcting the error in the token using a predefined correction algorithm. (VENKATESWARAN − [0039] The system memory 204 may include image processing 206, assisted previews 208, cell detection 210, or other various hardware, software, or modules. Image processing 206 may be hardware or software that, when coordinated or executed by the processing unit 202, provides perspective correction for misaligned source images may be provided.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Shraga does not explicitly teach: which is implemented using an image encoder decoder network or an image encoder dual decoder network, wherein the predefined logic to identify the error comprises checking for row length consistency and checking if the table structure is rectangular;
However, Nassar teaches: which is implemented using an image encoder decoder network or an image encoder dual decoder network, (Nassar − [pages 2, 4-5] Fig. 1, image to text architecture include image encoder/dual decoder networks; TableFormer includes a CNN Backbone Network that encodes the table image and provides the encoded image to a Structure Decoder and Cell BBox Decoder.)
PNG
media_image1.png
368
338
media_image1.png
Greyscale
wherein the predefined logic to identify the error comprises checking for row length consistency and checking if the table structure is rectangular; (Nassar − [page 11 1.1. Data preparation] As such, we prefer to have “strict” table, i.e., tables where every row and has exactly the same length. A strict HTML structure looks always rectangular. [page 12] describe post-processing of the predicted table structure using it’s row and column grid dimensions)
PNG
media_image2.png
628
354
media_image2.png
Greyscale
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 2, depends on claim 1, Shraga teaches: wherein the finite number of tokens further comprises: a horizontal group member cell token, (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
a vertical group member cell token, (Shraga − wherein: [0007] (i) the schema of the table is encoded together with end-of-column tokens)
a two-dimensional cell member token, (Shraga − wherein: [0007] end-of-row tokens is a two-dimension cell member token for region of the table)
Shraga does not explicitly teach: and a new line token
However, DUTA teaches: and a new line token. (DUTA − [0103] a first token in a column (of the vertical column grid) may indicate new cell (i.e., a new row) first token for new row is a new line token)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 6, depends on claim 2, Shraga does not explicitly teach: wherein the predefined logic to identify the error comprises any one of the following: a check for row length consistency; a check if the table structure is rectangular; a check that there are no horizontal group member cell tokens in a first column of the table structure; a check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure; a check that the top-left corner of a two-dimensional cell is either a two-dimensional cell token or a cell identifier token; a check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it; a check that a horizontal group member cell only has horizontal group member cells tokens or cell identifier tokens to the left of it; and combinations thereof.
However, VENKATESWARAN teaches: wherein the predefined logic to identify the error comprises any one of the following: a check for row length consistency; a check if the table structure is rectangular; a check that there are no horizontal group member cell tokens in a first column of the table structure; a check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure; a check that the top-left corner of a two-dimensional cell is either a two-dimensional cell token or a cell identifier token; a check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it; a check that a horizontal group member cell only has horizontal group member cells tokens or cell identifier tokens to the left of it; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring. the claim recites at least one of, VNEKATESWARAN teaches checking cell and characters misrecognized for comparison and validation of errors )
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 7, depends on claim 6, Shraga does not explicitly teach: wherein the method further comprises correcting the error of the token by performing any one of the following: padding rows shorter that a maximum row length and that end with a cell identifier token or a horizontal group member token with horizontal group member tokens if the check for row length consistency or the check if the table structure is rectangular failed, wherein other rows shorter than the maximum row length are padded with cell identifier tokens if the check for row length consistency or the check if the table structure is rectangular failed; if the check that there are no horizontal group member cell tokens in a first column of the table structure is failed then replace the token with the horizontal group member cell token; if the check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure fails then replace the token with the cell identifier token; if the check that the top-left corner of a two-dimensional cell is either a two-dimensional cell member token or a cell identifier token fails then replace the top-left corner with the cell identifier token or the two-dimensional cell token; if the check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it fails then replace the token with the token with the group member cell token or the cell identifier token; if the check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it fails then replace the token with the horizontal group member token or the cell identifier token; and combinations thereof.
However, VENKATESWARAN teaches: wherein the method further comprises correcting the error of the token by performing any one of the following: padding rows shorter that a maximum row length and that end with a cell identifier token or a horizontal group member token with horizontal group member tokens if the check for row length consistency or the check if the table structure is rectangular failed, wherein other rows shorter than the maximum row length are padded with cell identifier tokens if the check for row length consistency or the check if the table structure is rectangular failed; if the check that there are no horizontal group member cell tokens in a first column of the table structure is failed then replace the token with the horizontal group member cell token; if the check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure fails then replace the token with the cell identifier token; if the check that the top-left corner of a two-dimensional cell is either a two-dimensional cell member token or a cell identifier token fails then replace the top-left corner with the cell identifier token or the two-dimensional cell token; if the check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it fails then replace the token with the token with the group member cell token or the cell identifier token; if the check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it fails then replace the token with the horizontal group member token or the cell identifier token; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0058] an examination of edges of words and how they are aligned (to the left, right) relative to these large spacing between table columns, the claim recites at least one of, VNEKATESWARAN teaches white spaces (padding))
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 8, depends on claim 1, Shraga does not explicitly teach: wherein the method further comprises converting the table structure into HTML, XML, LaTeX, or MD by iteratively converting rows identified by the new line token into tags and/or elements.
However, VENKATESWARAN teaches: wherein the method further comprises converting the table structure into HTML, XML, LaTeX, or MD by iteratively converting rows identified by the new line token into tags and/or elements.( VENKATESWARAN − [0064] Once the table structure (e.g., borders, cells, spanning formation, etc.) and table content is sorted in 315 and 316, a structured table may be created 317. output into any file format that supports tables. These may include XLSX format 318, but also include others, such as HTML, DOCX, and PDF, to name a few.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 9, depends on claim 1, Shraga does not explicitly teach: wherein the method further comprises: correlating the table structure with locations in the image of the table; extracting data corresponding to the location of cell identifier locations from the image of the table; and inserting the data into the table structure.
However, DUTA teaches: wherein the method further comprises: correlating the table structure with locations in the image of the table; (DUTA − [0077] Following tokenization of the characters on a page, the Table Extractor computes vertical token alignments for each of those tokens. For example, in various implementations, for each x-coordinate on the page, the Table Extractor applies a vertical accumulator sampling window extending the length of the page (in the y-direction) and traversing the page along the x-axis (e.g., horizontal direction))
extracting data corresponding to the location of cell identifier locations from the image of the table; (DUTA − [0077] Following tokenization of the characters on a page, the Table Extractor computes vertical token alignments for each of those tokens. For example, in various implementations, for each x-coordinate on the page, the Table Extractor applies a vertical accumulator sampling window extending the length of the page (in the y-direction) and traversing the page along the x-axis (e.g., horizontal direction))
and inserting the data into the table structure. (DUTA – [0102] The Table Extractor inserts horizontal grid lines between consecutive rows of tokens in the vertical column grid whenever the interrow distance is larger than the most frequent distance, or optionally, when the starting token begins with a capitalized character.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 10, depends on claim 9, Shraga does not explicitly teach: optical character recognition
However, DUTA teaches: optical character recognition and extracting the data from a source document of the image of the table. (DUTA – [0057] image-based documents (e.g., a document photograph or scanned document) on which optical character recognition (OCR) operations; the Table Extractor processes the document using any combination of OCR, PDF-to-Text extraction, or any other image and text-based techniques to generate machine-readable characters (e.g., letters, numbers, special characters, emojis, etc.) in combination with at least the absolute and/or relative positions of those characters on particular pages within the arbitrary input document.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 11, depends on claim 1, Shraga teaches: wherein the table recognition neural network is implemented using a [ResNet] neural network that feeds a feature vector to a transformer encoder neural network with multiple encoder layers, (Shraga − [0010-0011] In some embodiments, the separate machine learning encoders comprise a Recurrent Convolutional Neural Network (RCNN) that encodes the description of the table. [0054-0056] For example, this may be an RCNN, which has shown to be highly-effective for text classification and similar NLP tasks. RCNN, as known in the art, is a bidirectional Long Short-Term Memory (bi-LSTM) followed by a max-pooling layer. The pooling layer allows to obtain a unified vector representation over all the tokens, capturing the most important latent factors in this textual modality.)
wherein the [ResNet] neural network is configured for receiving the image of the table, (Shraga − [0002] receiving a query and searching for pertinent information in a corpus of electronic data—be it…, images; Information retrieval systems, which are used for conducting searches in the organization's internal collection (corpus) of electronic documents and other data. [0026] A Recurrent Convolutional Neural Network (RCNN) may be used for encoding the description of the table.)
Shraga does not explicitly teach: decoder within the transformer
However, Nassar teaches: wherein the table recognition neural network is further implemented such that the output of the transformer encoder neural network is passed to a structure decoder neural network, (Nassar – [page 5] the Structure Decoder is modeled using a transformer encoder and transformer decoder; the transformer decoder receives as input and the output features are produced by the transformer encoder.)
wherein the structure decoder neural network is implemented as a transformer encoder with multiple decoder layers, (Nassar – [page 5] the Structure Decoder is modeled as a transformer encoder with two encoder layers and a transformer decoder made from a stack of four decoder layers.)
wherein the structure decoder neural network is configured for outputting the table structure. (Nassar – [page 4] the encoder image feature vector is passed to the Structure Decoder to produce a sequence of HTML tags that represent the structure of the table; Fig. 4, the Structure Decoder predicts the table structure tags.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 13, depends on claim 1, Shraga does not explicitly teach: decoder network
However, Nassar teaches: wherein the table recognition neural network is implemented using an image encoder text decoder neural network architecture. (SMOCK − [0040] [0083] At operation 1330, the table objects are transformed into a structured representation of the table. At operation 1340, data is extracted into the structured table representation. Extraction may be performed by recognizing text from each table object and associating the text with cells represented by the structured representation of the table.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 14, depends on claim 1, Shraga continue to teach a single token for end of rows/columns but does not explicitly teach: the single token assigned to each cell of the table
However, DUTA teaches: wherein the single token assigned to each cell of the table is associated with a token specific area of the image of the table or a bounding box identifying the token specific area of the image of the table. (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens; [0006] the Table Extractor begins operation on an arbitrary document by identifying each character in that document and a positional bounding box for each of those characters.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding independent claim 15, is directed to a computer system comprising a processor and a memory storing machine executable instructions (Shraga [0008]). Claim 15 have similar/same technical features/limitations as claim 1. Claim 15 is rejected under the same rationale.
Regarding independent claim 16, Shraga teaches: A method of training a table recognition neural network, wherein the method comprises:
receiving training data, (Shraga − [0076] The performance evaluation included comparing the tabular information retrieved by the present technique to the ground truth defined by Zhang (2018),)
wherein the table structure comprises a row of tokens for each row of the table; (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
Shraga does not explicitly teach: wherein the method further comprises identifying an error in a token of the table structure by sequentially comparing single token assigned to each cell to a predefined logic.
However, DUTA teaches: wherein the table structure comprises a single token assigned to each cell of the table, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
wherein the single token assigned to each cell of the table is selected from a finite number of tokens, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, and DUTA to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Shraga does not explicitly teach: wherein the markup language or the structure representations is at least one of HTML, XML, LaTex, or markdown,
However, VENKATESWARAN teaches: wherein the markup language or structure representation is any one of the following: into HTML, XML, LaTeX, or markdown. ( VENKATESWARAN − [0064] Once the table structure (e.g., borders, cells, spanning formation, etc.) and table content is sorted in 315 and 316, a structured table may be created 317. output into any file format that supports tables. These may include XLSX format 318, but also include others, such as HTML, DOCX, and PDF, to name a few.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Shraga does not explicitly teach: receiving training data, with ground truth data in a markup language or structure representation, wherein the training data comprise pairs of training images containing a table and ground truth data descriptive of a table structure of the table,
However, Nassar teaches: receiving training data, with ground truth data in a markup language or structure representation, wherein the markup language or the structure representations is at least one of HTML, XML, LaTex, or markdown, wherein the training data comprise pairs of training images containing a table and ground truth data descriptive of a table structure of the table, (Nassar − [page 5] During training, the transformer decoder receives as input the output feature produced by the transformer encoder, and the tokenized input of the HTML ground-truth tags. Figure 3: TableFormer takes in an image of the PDF and creates bounding box and HTML structure predictions that are synchronized. The bounding boxes grabs the content from the PDF and inserts it in the structure.)
PNG
media_image3.png
906
730
media_image3.png
Greyscale
converting the ground truth data from the markup language or the structure representation to the table structure with the single token assigned to each cell of the table before training; (Nassar – [page 3] PubTabNet annotations comprise table structure represented in HTML format; [page 5, Fig. 4] during training, the Structure Decoder receives “tokenized tags” of the HTML code that represent table structure; the transformer decoder receives tokenized input of HTML ground-truth tags”;)
and training the table recognition neural network using the training data. (Nassar – [page 6] the Cross-Entropy loss is used to trained to train the Structure Decoder which predicts the structure tokens; TableFormer is trained using Adam optimizers for the CNN Backbone Network, Structure Decoder, and Cell BBox Decoder.)
PNG
media_image4.png
274
734
media_image4.png
Greyscale
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 19, depends on claim 16, Shraga does not explicitly teach: wherein the method further comprises identifying an error in a token of the table structure by sequentially comparing single token assigned to each cell to a predefined logic.
However, VENKATESWARAN teaches: wherein the method further comprises identifying an error in a token of the table structure by sequentially comparing single token assigned to each cell to a predefined logic, (VENKATESWARAN − [0011] FIG. 5C illustrates a screen for a converted and validated structured table using a spreadsheet export application, according to an example; [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring.)
wherein the predefined logic to identify the error comprises any one of the following: a check for row length consistency; a check if the table structure is rectangular; a check that there are no horizontal group member cell tokens in a first column of the table structure; a check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure; a check that the top-left corner of a two-dimensional cell is either a two-dimensional cell token or a cell identifier token; a check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it; a check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring. the claim recites at least one of, VNEKATESWARAN teaches checking cell and characters misrecognized for comparison and validation of errors )
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 20, depends on claim 19, Shraga does not explicitly teach: wherein the method further comprises correcting the error of the token using a predefined correction algorithm, (VENKATESWARAN − [0039] The system memory 204 may include image processing 206, assisted previews 208, cell detection 210, or other various hardware, software, or modules. Image processing 206 may be hardware or software that, when coordinated or executed by the processing unit 202, provides perspective correction for misaligned source images may be provided.)
wherein the method further comprises correcting the error of the token by performing any one of the following: padding rows shorter that a maximum row length and that end with a cell identifier token or a horizontal group member token with horizontal group member tokens if the check for row length consistency or the check if the table structure is rectangular failed, wherein other rows shorter than the maximum row length are padded with cell identifier tokens if the check for row length consistency or the check if the table structure is rectangular failed; if the check that there are no horizontal group member cell tokens in a first column of the table structure is failed then replace the token with the horizontal group member cell token; if the check that check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure fails then replace the token with the cell identifier token; if the check that check that the top-left corner of a two-dimensional cell is either a two-dimensional cell member token or a cell identifier token fails then replace the top-left corner with the cell identifier token or the two-dimensional cell token; if the check that check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it fails then replace the token with the token with the group member cell toke or the cell identifier token; if the check that check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it fails then replace the token with the horizontal group member token or the cell identifier token; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0058] an examination of edges of words and how they are aligned (to the left, right) relative to these large spacing between table columns, the claim recites at least one of, VNEKATESWARAN teaches white spaces (padding))
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 21, depends on claim 15, Shraga teaches: wherein the finite number of tokens further comprises: a horizontal group member cell token, (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
a vertical group member cell token, (Shraga − wherein: [0007] (i) the schema of the table is encoded together with end-of-column tokens)
a two-dimensional cell member token, (Shraga − wherein: [0007] end-of-row tokens is a two-dimension cell member token for region of the table)
Shraga does not explicitly teach: and a new line token
However, DUTA teaches: and a new line token. (DUTA − [0103] a first token in a column (of the vertical column grid) may indicate new cell (i.e., a new row) first token for new row is a new line token)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 22, depends on claim 1, Shraga does not explicitly teach: checking an output of the table recognition neural network to determine a validity
DUTA teaches: teaches: wherein the sequentially comparing further comprises: checking an output of the table recognition neural network to determine a validity of the table structure by comparing adjacent tokens sequentially. (DUTA [0097] Once the upper, lower, left and right extents of the table candidate have been determined as described above, the Table Extractor then further processes the table candidate to split and merge consecutive vertical column grids, if necessary, and to determine whether or not the table candidate is a valid table)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 24, depends on claim 1, Shraga does not explicitly teach: padding rows shorter than a maximum row length and that end with the at least one cell identifier token, wherein other rows shorter than the maximum row lengths are padded with cell identifier tokens if the checking for the row length consistency or the checking for the table structure rectangularity failed.
However, Nassar teaches: padding rows shorter than a maximum row length and that end with the at least one cell identifier token, wherein other rows shorter than the maximum row lengths are padded with cell identifier tokens if the checking for the row length consistency or the checking for the table structure rectangularity failed. (Nassar − [page 11 1.1. Data preparation] As such, we prefer to have “strict” table, i.e., tables where every row and has exactly the same length. A strict HTML structure looks always rectangular. [page 12] describe post-processing of the predicted table structure using it’s row and column grid dimensions)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN and Nassar to provide a system for extracting table data from images, with a reasonable expectation of success since the references are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process by generating a structured table representation in a markup format and training the table recognition with ground truth table structure data, thereby improving the generation and recognition of table structure from images.
Regarding dependent claim 25, depends on claim 1, Shraga teaches further comprising: training the table recognition neural network. (Shraga − [0010] In some embodiments, the separate machine learning encoders comprise a Recurrent Convolutional Neural Network (RCNN) that encodes the description of the table. a first Three-Dimensional Convolutional Neural Network (3D-CNN) that encodes the rows of the table; and a second 3D-CNN that encodes the columns of the table. [0032] The instructions of multimodal table encoding module 108 may cause system 100 to receive an electronic document which includes a table 110, process the electronic document, and output a fused encoding of the table 112.)
Allowable Subject Matter
Claim 26 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Claim 27, which depends from claim 26.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Xu Zhong - Image-based table recognition: data, model, and evaluation; discusses image-based table recognition using an attention based encoder dual decoder architecture for converting table images into structure HTML representations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARL E BARNES JR whose telephone number is (571)270-3395. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at (571) 272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CARL E BARNES JR/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
