DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-20 were previously pending and subject to non-final action filed on 10/01/2025. In the response filed 01/02/2026, claims 1-2, 6, 15-16, 20 were amended, claims 3-5, 17 were canceled, and 21-24 were newly added claims. Therefore, claims 1-2, 6-16, and 18-24 are currently pending and subject to the final action below.
Response to Arguments
Applicant's arguments, see page 11, filed 01/02/2026, with respect to the objection to the Drawing have been fully considered but they are not persuasive.
Applicant’s argument: The objection to the drawings and the specification were discussed and Examiner agreed the amendments to the specification included herewith would overcome both objections.
Examiner Response: Applicant’s filed amendments to the specification on 01/02/2026. Amendments was made to specification paragraph 0087, 0103, 0110, to correct the elements missing elements not found in the specification that was recited in Fig. 4 element 480, Fig. 16 element 1608, and Fig. 23 element 2308.
Correction to the specification satisfied Fig. 16 and Fig. 23., however Fig. 4 element 480 is not shown in Fig. 4.
The amendments to the specification filed on 01/02/2026 was amended to recite element 408. However, element 480 is recited in Fig. 4, not element 408. Therefore, the objection to the Drawing is maintained.
Applicant’s arguments, see page 11, filed on 01/02/2026, with respect to the objection to the specification have been fully considered and are persuasive. The objection of specification has been withdrawn.
Applicant’s arguments, see page 11, filed on 01/02/2026, with respect to claim 17 under 35 U.S.C. 112 (b) have been fully considered and are persuasive. The 112 (b) rejection of claim 17 has been withdrawn.
Applicant’s arguments, see pages 12-15, filed on 01/04/2026 with respect to claim(s) 1-20 under 35 U.S.C. 103 have been considered but are moot because the arguments do not apply to the new combination of references being used in the current rejection.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: See below:
Fig. 4 element 480 is recited in the Drawing, however is not mention in the specification filed on 05/12/2023.
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: See below:
Specification filed on 01/02/2026 recites in paragraph [0087] “It can be seen that there is a region with a horizontal span “408”. However, the element 408 is not recited in Fig. 4. It is unclear if the element 408 should be 408 or the specification 408 should be 480.
Examiner Notes:
The term “full cell” is a cell that is not empty and contain data within the cell. The term “empty cell” is a cell consisting of “no character/numerical data”, “no data” or “white spaces without no character/numerical data”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6-10, 14-15, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Shraga US 20220043794 A1, Pub Date: Feb. 10, 2022) in view of DUTA (US 20190340240 A1, Pub Date: Nov. 7, 2019) and further in view of VENKATESWARAN (US 20200097711 A1, Pub Date: Mar. 26, 2020).
Regarding independent claim 1, Shraga teaches: A computer implemented method of determining a table structure of a table, wherein the method comprises:
receiving an image of the table; (Shraga − [0002] receiving a query and searching for pertinent information in a corpus of electronic data—be it…, images; Information retrieval systems, which are used for conducting searches in the organization's internal collection (corpus) of electronic documents and other data. [0003] Electronic documents, which typically contain unstructured data in the form of free text, sometimes also include tables—which are a form of structured data.) Receiving images of an electronic document that includes tables.
and receiving the table structure of the table in response to inputting the image of the table into a table recognition neural network, (Shraga − [0010] In some embodiments, the separate machine learning encoders comprise a Recurrent Convolutional Neural Network (RCNN) that encodes the description of the table. a first Three-Dimensional Convolutional Neural Network (3D-CNN) that encodes the rows of the table; and a second 3D-CNN that encodes the columns of the table. [0032] The instructions of multimodal table encoding module 108 may cause system 100 to receive an electronic document which includes a table 110, process the electronic document, and output a fused encoding of the table 112.) process the electronic document (receiving) using a RCNN for encoding the table and output a fused encoding of the table 112.
wherein the table structure comprises a row of tokens for each row of the table. (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
Shraga continue to teach a single token for end of rows/columns but does not explicitly teach: a single token assigned to each cell of the table
However, DUTA teaches: wherein the table structure comprises a single token assigned to each cell of the table, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
wherein the single token assigned to each cell of the table is selected from a finite number of tokens, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
wherein the finite number of tokens includes at least one cell identifier token comprised of a full cell identifier and an empty cell identifier, (DUTA − [0014] FIG. 3 provides an example of splitting and merging column grids of a table candidate generated by the Table Extractor, as described herein. Fig. 3 shows at least one empty cell and full cell, each with token identifier.)
identifying an error in a token of the table structure (DUTA – [0098] In other words, the tokenization process may have errors. For example, what should be one token may be incorrectly broken into multiple adjacent tokens.)
and wherein the table structure comprises a row of tokens for each row of the table; (DUTA – [0102] The Table Extractor inserts horizontal grid lines between consecutive rows of tokens. Rows of tokens for each row of the table identified by the Table Extractor)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga and DUTA to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Shraga does not explicitly teach: identifying an error in a token of the table structure by sequentially comparing a single token assigned to each cell to a predefined logic
VENKATESWARAN teaches: identifying an error in a token of the table structure by sequentially comparing a single token assigned to each cell to a predefined logic; (VENKATESWARAN − [0011] FIG. 5C illustrates a screen for a converted and validated structured table using a spreadsheet export application, according to an example; [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring.)
and correcting the error in the token using a predefined correction algorithm. (VENKATESWARAN − [0039] The system memory 204 may include image processing 206, assisted previews 208, cell detection 210, or other various hardware, software, or modules. Image processing 206 may be hardware or software that, when coordinated or executed by the processing unit 202, provides perspective correction for misaligned source images may be provided.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 2, depends on claim 1, Shraga teaches: wherein the finite number of tokens further comprises: a horizontal group member cell token, (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
a vertical group member cell token, (Shraga − wherein: [0007] (i) the schema of the table is encoded together with end-of-column tokens)
a two-dimensional cell member token, (Shraga − wherein: [0007] end-of-row tokens is a two-dimension cell member token for region of the table)
Shraga does not explicitly teach: and a new line token
However, DUTA teaches: and a new line token. (DUTA − [0103] a first token in a column (of the vertical column grid) may indicate new cell (i.e., a new row) first token for new row is a new line token)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 6, depends on claim 2, Shraga does not explicitly teach: wherein the predefined logic to identify the error comprises any one of the following: a check for row length consistency; a check if the table structure is rectangular; a check that there are no horizontal group member cell tokens in a first column of the table structure; a check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure; a check that the top-left corner of a two-dimensional cell is either a two-dimensional cell token or a cell identifier token; a check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it; a check that a horizontal group member cell only has horizontal group member cells tokens or cell identifier tokens to the left of it; and combinations thereof.
However, VENKATESWARAN teaches: wherein the predefined logic to identify the error comprises any one of the following: a check for row length consistency; a check if the table structure is rectangular; a check that there are no horizontal group member cell tokens in a first column of the table structure; a check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure; a check that the top-left corner of a two-dimensional cell is either a two-dimensional cell token or a cell identifier token; a check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it; a check that a horizontal group member cell only has horizontal group member cells tokens or cell identifier tokens to the left of it; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring. the claim recites at least one of, VNEKATESWARAN teaches checking cell and characters misrecognized for comparison and validation of errors )
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to validating table extraction data.
Regarding dependent claim 7, depends on claim 6, Shraga does not explicitly teach: wherein the method further comprises correcting the error of the token by performing any one of the following: padding rows shorter that a maximum row length and that end with a cell identifier token or a horizontal group member token with horizontal group member tokens if the check for row length consistency or the check if the table structure is rectangular failed, wherein other rows shorter than the maximum row length are padded with cell identifier tokens if the check for row length consistency or the check if the table structure is rectangular failed; if the check that there are no horizontal group member cell tokens in a first column of the table structure is failed then replace the token with the horizontal group member cell token; if the check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure fails then replace the token with the cell identifier token; if the check that the top-left corner of a two-dimensional cell is either a two-dimensional cell member token or a cell identifier token fails then replace the top-left corner with the cell identifier token or the two-dimensional cell token; if the check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it fails then replace the token with the token with the group member cell token or the cell identifier token; if the check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it fails then replace the token with the horizontal group member token or the cell identifier token; and combinations thereof.
However, VENKATESWARAN teaches: wherein the method further comprises correcting the error of the token by performing any one of the following: padding rows shorter that a maximum row length and that end with a cell identifier token or a horizontal group member token with horizontal group member tokens if the check for row length consistency or the check if the table structure is rectangular failed, wherein other rows shorter than the maximum row length are padded with cell identifier tokens if the check for row length consistency or the check if the table structure is rectangular failed; if the check that there are no horizontal group member cell tokens in a first column of the table structure is failed then replace the token with the horizontal group member cell token; if the check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure fails then replace the token with the cell identifier token; if the check that the top-left corner of a two-dimensional cell is either a two-dimensional cell member token or a cell identifier token fails then replace the top-left corner with the cell identifier token or the two-dimensional cell token; if the check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it fails then replace the token with the token with the group member cell token or the cell identifier token; if the check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it fails then replace the token with the horizontal group member token or the cell identifier token; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0058] an examination of edges of words and how they are aligned (to the left, right) relative to these large spacing between table columns, the claim recites at least one of, VNEKATESWARAN teaches white spaces (padding))
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to validating table extraction data.
Regarding dependent claim 8, depends on claim 1, Shraga does not explicitly teach: wherein the method further comprises converting the table structure into HTML, XML, LaTeX, or MD by iteratively converting rows identified by the new line token into tags and/or elements.
However, VENKATESWARAN teaches: wherein the method further comprises converting the table structure into HTML, XML, LaTeX, or MD by iteratively converting rows identified by the new line token into tags and/or elements.( VENKATESWARAN − [0064] Once the table structure (e.g., borders, cells, spanning formation, etc.) and table content is sorted in 315 and 316, a structured table may be created 317. output into any file format that supports tables. These may include XLSX format 318, but also include others, such as HTML, DOCX, and PDF, to name a few.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to validating table extraction data.
Regarding dependent claim 9, depends on claim 1, Shraga does not explicitly teach: wherein the method further comprises: correlating the table structure with locations in the image of the table; extracting data corresponding to the location of cell identifier locations from the image of the table; and inserting the data into the table structure.
However, DUTA teaches: wherein the method further comprises: correlating the table structure with locations in the image of the table; (DUTA − [0077] Following tokenization of the characters on a page, the Table Extractor computes vertical token alignments for each of those tokens. For example, in various implementations, for each x-coordinate on the page, the Table Extractor applies a vertical accumulator sampling window extending the length of the page (in the y-direction) and traversing the page along the x-axis (e.g., horizontal direction))
extracting data corresponding to the location of cell identifier locations from the image of the table; (DUTA − [0077] Following tokenization of the characters on a page, the Table Extractor computes vertical token alignments for each of those tokens. For example, in various implementations, for each x-coordinate on the page, the Table Extractor applies a vertical accumulator sampling window extending the length of the page (in the y-direction) and traversing the page along the x-axis (e.g., horizontal direction))
and inserting the data into the table structure. (DUTA – [0102] The Table Extractor inserts horizontal grid lines between consecutive rows of tokens in the vertical column grid whenever the interrow distance is larger than the most frequent distance, or optionally, when the starting token begins with a capitalized character.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 10, depends on claim 9, Shraga does not explicitly teach: optical character recognition
However, DUTA teaches: optical character recognition and extracting the data from a source document of the image of the table. (DUTA – [0057] image-based documents (e.g., a document photograph or scanned document) on which optical character recognition (OCR) operations; the Table Extractor processes the document using any combination of OCR, PDF-to-Text extraction, or any other image and text-based techniques to generate machine-readable characters (e.g., letters, numbers, special characters, emojis, etc.) in combination with at least the absolute and/or relative positions of those characters on particular pages within the arbitrary input document.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 14, depends on claim 1, Shraga continue to teach a single token for end of rows/columns but does not explicitly teach: the single token assigned to each cell of the table
However, DUTA teaches: wherein the single token assigned to each cell of the table is associated with a token specific area of the image of the table or a bounding box identifying the token specific area of the image of the table. (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens; [0006] the Table Extractor begins operation on an arbitrary document by identifying each character in that document and a positional bounding box for each of those characters.)
Regarding independent claim 15, is directed to an computer system comprising a processor and a memory storing machine executable instructions (Shraga [0008]). Claim 15 have similar/same technical features/limitations as claim 1. Claim 15 is rejected under the same rationale.
Regarding dependent claim 21, depends on claim 15, Shraga teaches: wherein the finite number of tokens further comprises: a horizontal group member cell token, (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
a vertical group member cell token, (Shraga − wherein: [0007] (i) the schema of the table is encoded together with end-of-column tokens)
a two-dimensional cell member token, (Shraga − wherein: [0007] end-of-row tokens is a two-dimension cell member token for region of the table)
Shraga does not explicitly teach: and a new line token
However, DUTA teaches: and a new line token. (DUTA − [0103] a first token in a column (of the vertical column grid) may indicate new cell (i.e., a new row) first token for new row is a new line token)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 22, depends on claim 1, Shraga does not explicitly teach: checking an output of the table recognition neural network to determine a validity
DUTA teaches: teaches: wherein the sequentially comparing further comprises: checking an output of the table recognition neural network to determine a validity of the table structure by comparing adjacent tokens sequentially. (DUTA [0097] Once the upper, lower, left and right extents of the table candidate have been determined as described above, the Table Extractor then further processes the table candidate to split and merge consecutive vertical column grids, if necessary, and to determine whether or not the table candidate is a valid table)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Claim(s) 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Shraga, DUTA and VENKATESWARAN as applied to claim 1 above, and further in view of SMOCK (US 20220335240 A1, Pub Date: Oct. 20, 2022).
Regarding dependent claim 11, depends on claim 1, Shraga teaches: wherein the table recognition neural network is implemented using a [ResNet] neural network that feeds a feature vector to a transformer encoder neural network with multiple encoder layers, (Shraga − [0010-0011] In some embodiments, the separate machine learning encoders comprise a Recurrent Convolutional Neural Network (RCNN) that encodes the description of the table. [0054-0056] For example, this may be an RCNN, which has shown to be highly-effective for text classification and similar NLP tasks. RCNN, as known in the art, is a bidirectional Long Short-Term Memory (bi-LSTM) followed by a max-pooling layer. The pooling layer allows to obtain a unified vector representation over all the tokens, capturing the most important latent factors in this textual modality.)
wherein the [ResNet] neural network is configured for receiving the image of the table, (Shraga − [0002] receiving a query and searching for pertinent information in a corpus of electronic data—be it…, images; Information retrieval systems, which are used for conducting searches in the organization's internal collection (corpus) of electronic documents and other data. [0026] A Recurrent Convolutional Neural Network (RCNN) may be used for encoding the description of the table.)
Shraga does not explicitly teach: decoder within the transformer
However, SMOCK teaches: wherein the table recognition neural network is further implemented such that the output of the transformer encoder neural network is passed to a structure decoder neural network, (SMOCK − [0040] followed by a transformer encoder and then a transformer decoder. The CNN backbone is a ResNet-18 model.)
wherein the structure decoder neural network is implemented as a transformer encoder with multiple decoder layers, (SMOCK − [0040] decoder each have six layers of self-attention)
wherein the structure decoder neural network is configured for outputting the table structure. (SMOCK − [0035] [0040] the transformer network architecture can implement object detection with a generality sufficient to output the complete set of table objects)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA, VENKATESWARAN and Smock to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 12, depends on claim 1, Shraga does not explicitly teach: decoder network
However, SMOCK teaches: wherein the table recognition neural network is implemented using an image encoder decoder network or an image encoder dual decoder network. (SMOCK − [0040] The table detection model 115 in one embodiment is a DETR (detection transformer) neural network model, composed of three submodules in sequence: a convolutional neural network (CNN) backbone, followed by a transformer encoder and then a transformer decoder. The CNN backbone is a ResNet-18 model.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA, VENKATESWARAN and Smock to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 13, depends on claim 1, Shraga does not explicitly teach: decoder network
However, SMOCK teaches: wherein the table recognition neural network is implemented using an image encoder text decoder neural network architecture. (SMOCK − [0040] [0083] At operation 1330, the table objects are transformed into a structured representation of the table. At operation 1340, data is extracted into the structured table representation. Extraction may be performed by recognizing text from each table object and associating the text with cells represented by the structured representation of the table.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA, VENKATESWARAN and Smock to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Claim(s) 23-24 are rejected under 35 U.S.C. 103 as being unpatentable over Shraga, DUTA and VENKATESWARAN as applied to claim 1 above, and further in view of Thompson (US 20220318224 A1, Pub Date: Oct. 6, 2022).
Regarding dependent claim 23, depends on claim 1, Shraga does not explicitly teach: logic to identify the error comprises checking for row length consistency and checking if the table structure is rectangular
However, Thompson teaches: wherein the error is in the table structure, and wherein the predefined logic to identify the error comprises checking for row length consistency and checking if the table structure is rectangular. (Thompson − [0158] Detecting Table Boundaries, detection of table boundaries.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA, VENKATESWARAN and Thompson to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 24, depends on claim 1, Shraga does not explicitly teach: padding rows shorter than a maximum row length and that end with the at least one cell identifier token, wherein other rows shorter than the maximum row lengths are padded with cell identifier tokens if the checking for the row length consistency or the checking for the table structure rectangularity failed.
However, Thompson teaches: wherein correcting the error further comprises: padding rows shorter than a maximum row length and that end with the at least one cell identifier token, wherein other rows shorter than the maximum row lengths are padded with cell identifier tokens if the checking for the row length consistency or the checking for the table structure rectangularity failed. (Thompson − [0214] Additionally, a custom padding process is employed to build horizontal rows from pixels distributed across neighboring rows (growing the row boundary to include such distributed pixels, and optionally pixels therebetween)) Examiner Notes: the claim limitation appear to recite a contingent Limitations within a process claim, if rectangularity does not fail.
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA, VENKATESWARAN and Thompson to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Claim(s) 16 is rejected under 35 U.S.C. 103 as being unpatentable over Shraga US 20220043794 A1, Pub Date: Feb. 10, 2022) in view of DUTA (US 20190340240 A1, Pub Date: Nov. 7, 2019) in view of Chan (US 20210406266 A1, Pub Date: Dec. 20, 2021) and further in view of Benincasa (US 11768884 B2, Filed Date: Jun. 24, 2022).
Regarding independent claim 16, Shraga teaches: A method of training a table recognition neural network, wherein the method comprises:
receiving training data, (Shraga − [0076] The performance evaluation included comparing the tabular information retrieved by the present technique to the ground truth defined by Zhang (2018),)
wherein the table structure comprises a row of tokens for each row of the table; (Shraga − [0007] wherein: (i) the schema of the table is encoded together with end-of-column tokens or end-of-row tokens,)
Shraga does not explicitly teach: wherein the method further comprises identifying an error in a token of the table structure by sequentially comparing single token assigned to each cell to a predefined logic.
However, DUTA teaches: wherein the table structure comprises a single token assigned to each cell of the table, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
wherein the single token assigned to each cell of the table is selected from a finite number of tokens, (DUTA − [0007] In various implementations, the Table Extractor then selects one of the table candidates having a highest number of tokens. This selected table candidate is then segmented into a plurality of rows and a plurality of columns that jointly delimit a plurality of table cells, with each of those cells encompassing one or more corresponding tokens)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, and DUTA to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
However, Chan teaches: receiving training data, with ground truth data in a markup language or structure representation, (Chan − [0044] In some embodiments, one or more machine learning models can be used and trained to generate tighter bounding boxes for each object.)
wherein the training data comprise pairs of training images containing a table and ground truth data descriptive of a table structure of the table, (Chan − [0045] Some embodiments use machine learning models, such as CNNs to detect tables. For example, some embodiments use TABLESENSE algorithms, where an active training/learning approach is taken to label data in iterations. The purpose of PBR is to minimize the absolute deviations between predicted boundary boxes and their ground truth values.)
training the table recognition neural network using the training data. (Chan − [0044] In some embodiments, one or more machine learning models can be used and trained to generate tighter bounding boxes for each object. In this way, bounding boxes can change in shape and confidence levels for classification/prediction can be increased based on increased training sessions.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and Chan to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Shraga does not explicitly teach: converting the ground truth data from the markup language or the structure representation to the table structure with the single token assigned to each cell of the table before training;
However, Benincasa teaches: converting the ground truth data from the markup language or the structure representation to the table structure with the single token assigned to each cell of the table before training; (Benincasa − [Col. 7 ll 19-25] The data extraction model may have been trained using ground truth labels assigned to tokenized training documents, and the ground truth labels may have also been used to select a subset of section tokens of the tokenized training document, and learn at least one property of the subset of section tokens, wherein a feature value is extracted for each section token based on the learned property.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA, Chan and Benincasa to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Claim(s) 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Shraga, DUTA, Benincasa and Chan as applied to claim 16 above, and further in view of VENKATESWARAN (US 20200097711 A1, Pub Date: Mar. 26, 2020).
Regarding dependent claim 18, depends on claim 16, Shraga does not explicitly teach: wherein the markup language or structure representation is any one of the following: into HTML, XML, LaTeX, or markdown.
However, VENKATESWARAN teaches: wherein the markup language or structure representation is any one of the following: into HTML, XML, LaTeX, or markdown. ( VENKATESWARAN − [0064] Once the table structure (e.g., borders, cells, spanning formation, etc.) and table content is sorted in 315 and 316, a structured table may be created 317. output into any file format that supports tables. These may include XLSX format 318, but also include others, such as HTML, DOCX, and PDF, to name a few.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and Chan, Benincasa and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 19, depends on claim 16, Shraga does not explicitly teach: wherein the method further comprises identifying an error in a token of the table structure by sequentially comparing single token assigned to each cell to a predefined logic.
However, VENKATESWARAN teaches: wherein the method further comprises identifying an error in a token of the table structure by sequentially comparing single token assigned to each cell to a predefined logic, (VENKATESWARAN − [0011] FIG. 5C illustrates a screen for a converted and validated structured table using a spreadsheet export application, according to an example; [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring.)
wherein the predefined logic to identify the error comprises any one of the following: a check for row length consistency; a check if the table structure is rectangular; a check that there are no horizontal group member cell tokens in a first column of the table structure; a check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure; a check that the top-left corner of a two-dimensional cell is either a two-dimensional cell token or a cell identifier token; a check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it; a check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0020] In these assisted previews, additional mapping and detection of low confidence content in cells or tables may be provided. Cell mapping and detection/highlighting of low confidence results may be provided for ease of comparison and validation. For example, this may include detecting cells that might have some potential errors in them (e.g., misrecognized characters, etc.), which may have been identified during table border identification and table restructuring. the claim recites at least one of, VNEKATESWARAN teaches checking cell and characters misrecognized for comparison and validation of errors )
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and Chan, Benincasa and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Regarding dependent claim 20, depends on claim 19, Shraga does not explicitly teach: wherein the method further comprises correcting the error of the token using a predefined correction algorithm, (VENKATESWARAN − [0039] The system memory 204 may include image processing 206, assisted previews 208, cell detection 210, or other various hardware, software, or modules. Image processing 206 may be hardware or software that, when coordinated or executed by the processing unit 202, provides perspective correction for misaligned source images may be provided.)
wherein the method further comprises correcting the error of the token by performing any one of the following: padding rows shorter that a maximum row length and that end with a cell identifier token or a horizontal group member token with horizontal group member tokens if the check for row length consistency or the check if the table structure is rectangular failed, wherein other rows shorter than the maximum row length are padded with cell identifier tokens if the check for row length consistency or the check if the table structure is rectangular failed; if the check that there are no horizontal group member cell tokens in a first column of the table structure is failed then replace the token with the horizontal group member cell token; if the check that check for two-dimensional cell member tokens to determine if they have only two-dimensional cell member tokens above and to the left in the table structure fails then replace the token with the cell identifier token; if the check that check that the top-left corner of a two-dimensional cell is either a two-dimensional cell member token or a cell identifier token fails then replace the top-left corner with the cell identifier token or the two-dimensional cell token; if the check that check that a vertical group member cell only has vertical group member cell tokens or cell identifier tokens above it fails then replace the token with the token with the group member cell toke or the cell identifier token; if the check that check that a horizontal group member cells only has horizontal group member cells tokens or cell identifier tokens to the left of it fails then replace the token with the horizontal group member token or the cell identifier token; and combinations thereof. (VENKATESWARAN − [0020] [0041] [0058] an examination of edges of words and how they are aligned (to the left, right) relative to these large spacing between table columns, the claim recites at least one of, VNEKATESWARAN teaches white spaces (padding))
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have combine Shraga, DUTA and Chan, Benincasa and VENKATESWARAN to provide a system for extracting table data from images, with a reasonable expectation of success since both are in the same field of endeavor of generating table structure for extracting table data. The motivation to combines provides the improvement to table extraction process of identifying tables within an image.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARL E BARNES JR whose telephone number is (571)270-3395. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at (571) 272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CARL E BARNES JR/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
Prosecution Insights