Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The United States Patent & Trademark Office appreciates the application that is submitted by the inventor/assignee. The United States Patent & Trademark Office reviewed the following application and has made the following comments below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/10/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/24/2026 has been entered.
Overview
Claims 1-5, 7-15, and 17-20 are pending in this application and have been considered below.
Claims 6 and 16 are cancelled.
Claims 1-5, 7-15, and 17-20 are rejected.
Applicant Arguments
In regards to Argument 1, see remarks (page 6), filed 08/24/2026, Applicant/s state/s “The Examiner states that Streltsov satisfies element a) of claims 1 and 11. Applicant respectfully disagrees. This element recites (emphasis added): "responsive to receipt of an input image of a form, placing first bounding boxes around text in the form;" Streltsov's input document already contains bounding boxes, as the Examiner acknowledges (see, e.g., Office Action, paragraph bridging pages 7 and 8). Consequently, Streltsov's bounding boxes are not placed in response to receipt of an input image of a form.”
In regards to Argument 2, see remarks (pages 6-7), filed 08/24/2026, Applicant/s state/s “The Examiner states that Streltsov satisfies element f)3) of claims 1 and 11. Again, Applicant respectfully disagrees. Element f)l) recites, "randomly selecting a second entity". Element f)3) recites, "adding the second entity to a dictionary". Element f)3) relies on element f)1) for antecedent basis, meaning that element f)3) comes after element f)l). According to the Examiner's reading, Streltsov's alleged second entity comes from a dictionary (see, e.g., Office Action, paragraph bridging pages 11 and 12), meaning that it already is stored there. Consequently, in Streltsov, storage of the alleged second entity in the dictionary comes before the entity is randomly selected. This is the opposite of the order that claims 1 and 11 recite.”
In regards to Argument 3, see remarks (page 7), filed 08/24/2026, Applicant/s state/s “To distinguish independent claims 1 and 11 further, Applicant has amended these to recite that the second entities are different from the first entities (element f) more generally), and identifying the second bounding boxes as new bounding boxes (element g)). This last change, which comes from paragraph [0055] of the specification, addresses the Examiner's characterization of Streltsov's modified bounding boxes as second bounding boxes (see, e.g., Office Action, p. 13). Modified bounding boxes are not new bounding boxes.”
In regards to Argument 4, see remarks (page 7), filed 08/24/2026, Applicant/s state/s “The Examiner acknowledges that Streltsov lacks elements b) and c) in claims 1 and 11, and relies on Ast for these. Before proceeding to discuss Ast, however, Applicant notes that if Streltsov lacks element c) as the Examiner acknowledges, then it does not identify regions on the form, and so cannot satisfy elements d) or e), both of which rely on the identified regions from element c).”
In regards to Argument 5, see remarks (page 7), filed 08/24/2026, Applicant/s state/s “Also, Applicant notes that element b), reciting inputting semantic information for text in the first bounding boxes, cannot happen in Streltsov because Streltsov already has bounding boxes around text ab initio in the images as they are input, whereas in claims 1 and 11 in the present application, the bounding boxes come after the image is input.”
In regards to Argument 6, see remarks (page 7), filed 08/24/2026, Applicant/s state/s “The Examiner relies on Ast for disclosure of identifying regions on a form, the regions to contain one or more of the first bounding boxes. Assuming for the sake of argument that Ast does disclose this, that does not explain how or why Streltsov would use such regions for purposes of elements d) or e) in claims 1 and 11. Consequently, Applicant submits that no reasonable combination of Streltsov and Ast can render claims 1 and 11 obvious.”
In regards to Argument 7, see remarks (page 7), filed 08/24/2026, Applicant/s state/s “Gohari and Buban do not remedy the above-identified deficiencies of the Streltsov/Ast combination. Consequently, pursuant to the foregoing discussion, Applicant submits that all of claims 1-5, 7-15, and 17-20 in the subject application are patentable.”
Examiner’s Responses
In response to Argument 1, see remarks (page 6), filed 08/24/2026, with respect to element a) of claims 1 and 11 and Streltsov’s teaching of the limitation: "responsive to receipt of an input image of a form, placing first bounding boxes around text in the form,” has been fully considered and is persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made under 35 U.S.C. 103 in view of Streltsov et al. (U.S. Patent Pub. No. 20230334309 A1, hereafter referred to as Streltsov) in view of Ast (U.S. Patent Pub No. 20220139099 A1, hereafter referred to as Ast) in further view of Yebes-Torres et al. (U.S. Patent Pub. No. 20250148443 A1, hereafter referred to as Yebes-Torres). The 35 U.S.C. 103 rejection does not rely on Streltsov for the “responsive to receipt of an input image of a form” portion of the limitation. Instead, the Examiner relies on Ast (U.S. Patent Pub. No. 20200242348 A1) to teach that limitation. Specifically, Ast discloses an input image can be provided to OCR module. OCR module identifies characters depicted within input image and produces an output that includes the identified characters and corresponding geometries. The position information can be represented in output as bounding boxes. Output can be a text document that contains the text of the input image and corresponding position information (Paragraph [0030], Fig. 1).
In response to Argument 2, see remarks (pages 6-7), filed 08/24/2026, with respect to element f)3) of claims 1 and 11 have been fully considered but they are not persuasive.
Specifically regarding Applicant’s argument that “Element f)l) recites, "randomly selecting a second entity". Element f)3) recites, "adding the second entity to a dictionary". Element f)3) relies on element f)1) for antecedent basis, meaning that element f)3) comes after element f)l). … This is the opposite of the order that claims 1 and 11 recite”, in light of paragraph [0054] of Applicant’s specification, which states “at 661, in a semantic region … a particular piece of semantic information (for example, a date, a name of an industrial part, a telephone number, a street address, or the like) may be identified. (This piece of information also may be referred to as an entity.) A mask may be placed over that entity… At 663, a dictionary may be consulted, and a matching, but different entity selected. At 665, the masked entity may be replaced with the selected entity, and the selected entity added to a dictionary as a new entity. With this addition, the selected entity may be used at some point later on as content in form generation.” Therefore, in light of paragraph [0054] of Applicant’s specification, the Examiner asserts that the randomly selected “different entity” to replace the “identified first entities” can be selected from the dictionary prior to the “different entity” being added to the dictionary. Also Applicant’s Drawing, Fig. 6, which shows “663” where “a dictionary may be consulted, and a matching, but different entity selected”, followed by “665” “the masked entity may be replaced with the selected entity” and finally, at “667”, the new entity is added to a dictionary, as written in paragraph [0054] of Applicant’s specification and as depicted in Fig. 6. The relevant portion of Applicant’s Fig. 6 is included directly below. Therefore, the Examiner asserts the randomly selected “second entity” can be selected from a dictionary.
PNG
media_image1.png
405
200
media_image1.png
Greyscale
In response to Argument 3, see remarks (page 7), filed 08/24/2026, with respect to the Applicant’s assertion that Streltsov’s modified bounding boxes are not “new” bounding boxes has been fully considered but they are not persuasive. Streltsov teaches the bounding boxes (110) may be modified to account for the new text. For example, if the semantic augmentation adds two new lines of text lines, the size of bounding box (110) may be increased accordingly (Paragraph [0062]). The Examiner asserts that under Broadest Reasonable Interpretation (BRI), the bounding boxes modified to account for the new text are “new bounding boxes”. Specifically, a bounding box is well known in the field of computer vision to describe the spatial location of an object. Bounding boxes are defined by the x and y coordinates of the upper-left corner and the lower-right corner of the box/rectangle. Thus, by changing the coordinates (and the size) of the bounding box to account for new text, the bounding box is “new” i.e., it has different location coordinates and also, a different size. In addition, the claim does not specify what a “new” bounding box is or may include.
In response to Argument 4, see remarks (page 7), filed 08/24/2026, with respect to the rejection(s) of claims 1 and 11 specifically, regarding elements b) and c), have been fully considered but they are not persuasive. The Examiner asserts that Streltsov does teach identifying regions on the form. Specifically, Streltsov teaches sections of the form such as sections (102), (104), and (106) may automatically be determined based on rules defined by the user. See the defined sections in Fig. 1A, dashed and solid lines define the section’s boundaries (Paragraph [0038], Fig. 1A). However, Streltsov does not explicitly disclose using “a deep learning system” to identify the regions on the form. Ast is relied upon to teach this limitation. Specifically, Ast teaches using a region-based convolutional neural network (R-CNN) trained to extract regions from the semantic image (Paragraphs [0011], [0049]). Therefore, the Examiner asserts that since Streltsov does identify regions on the form, and Ast fulfils the deep learning system aspect of the claim limitation, elements d) and e) can be satisfied by Streltsov. See below for details of the rejection and motivation to combine Streltsov and Ast references.
In response to Argument 5, see remarks (page 7), filed 08/24/2026, with respect to Applicant’s argument regarding element b), which states, “inputting semantic information for text in the first bounding boxes, cannot happen in Streltsov because Streltsov already has bounding boxes around text ab initio in the images as they are input, whereas in claims 1 and 11 in the present application, the bounding boxes come after the image is input” has been fully considered but is not persuasive. Streltsov teaches semantic augmentations may be applied to annotated data fields (i.e., replacing text/semantic information with different text/semantic information). The Examiner interprets “inputting semantic information for text in the first bounding boxes” could simply mean replacing the text in the first bounding boxes with new text, since semantic information is text (a date, a name of an industrial part, a telephone number, a street address, etc.), in light of paragraph [0054] of Applicant’s specification, which states, “a particular piece of semantic information (for example, a date, a name of an industrial part, a telephone number, a street address, or the like) may be identified. (This piece of information also may be referred to as an entity.)” Therefore, replacing the text in the bounding boxes in the original form with new text (semantic augmentations) is “inputting semantic information for text”. Further, the secondary reference, Ast, also teaches the limitation “b) inputting semantic information for text in the first bounding boxes.” Specifically, Ast teaches converting text bounded by the geometries (bounding boxes) to semantic information by a semantics converter module (Paragraphs [0007-8]).
In response to Argument 6, see remarks (page 7), filed 08/24/2026, with respect to the teachings of Streltsov and Ast have been fully considered but are not persuasive. Streltsov does teach identifying regions in a form. Specifically, Streltsov teaches sections of the form such as sections (102), (104), and (106) may automatically be determined based on rules defined by the user. See the defined sections in Fig. 1A, dashed and solid lines define the section’s boundaries (Paragraph [0038], Fig. 1A). However, Streltsov does not explicitly disclose using “a deep learning system” to identify the regions on the form. Ast is relied upon to teach this limitation. Specifically, Ast teaches using a region-based convolutional neural network (R-CNN) trained to extract regions from the semantic image (Paragraphs [0011], [0049]). Therefore, the Examiner asserts that since Streltsov does identify regions on the form, and Ast fulfils the deep learning system aspect of the claim limitation, elements d) and e) can be satisfied by Streltsov. See below for details of the rejection and motivation to combine Streltsov and Ast references.
In response to Argument 7, see remarks (page 7), filed 08/24/2026, with respect to the rejection(s) of claims 1-5, 7-15, and 17-20 have been fully considered but are not persuasive. See below for the details of the rejection.
Claims 1-2, 4-5, 7-8, 11-12, 14-15, and 17 are rejected under 35 U.S.C. 103(a) as being unpatentable Streltsov et al. (U.S. Patent Pub. No. 20230334309 A1, hereafter referred to as Streltsov) in view of Ast (U.S. Patent Pub No. 20220139099 A1, hereafter referred to as Ast) in further view of Yebes-Torres et al. (U.S. Patent Pub. No. 20250148443 A1, hereafter referred to as Yebes-Torres).
Claims 3, 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Streltsov et al. (U.S. Patent Pub. No. 2023/0334309, hereafter referred to as Streltsov) in view of Ast (U.S. Patent Pub No. 20220139099 A1, hereafter referred to as Ast) in further view of Yebes-Torres et al. (U.S. Patent Pub. No. 20250148443 A1, hereafter referred to as Yebes-Torres) in further view of Gohari (U.S. Patent Pub. No. 2022/0318492, hereafter referred to as Gohari).
Claims 9, 10, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Streltsov et al. (U.S. Patent Pub. No. 2023/0334309, hereafter referred to as Streltsov) in view of Ast (U.S. Patent Pub No. 20220139099 A1, hereafter referred to as Ast) in further view of Yebes-Torres et al. (U.S. Patent Pub. No. 20250148443 A1, hereafter referred to as Yebes-Torres) in further view of Buban et al. (U.S. Patent No. 12,094,231, hereafter referred to as Buban).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(B) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-5, 7-15, and 17-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention. The Examiner strongly suggested that appropriate corrections be made to clarify the claim scope.
With respect to Claim 1, the claim recites the following, each of which renders the claim indefinite:
“second, different entities” on line 10 (unclear antecedent basis);
“a/the second entity” on lines 12-14 (unclear antecedent basis); it is unclear as to whether the “second, different entities” recited on line 10 are the same or different from “a/the second entity” on lines 12-14.
“placing new bounding boxes around text in the second entities” on line 17 (unclear to what this refers); it is unclear what “around text in the second entities” means since “entities”, as defined in Applicant’s specification (paragraph [0054]), are pieces of semantic information such as a date, a name of an industrial part, a telephone number, a street address or the like, which are “text”. Therefore, it is unclear how bounding boxes are placed around text in the second entities since the second entities are text (semantic information such as date, name, phone number, etc.). It is unclear how text is in the second entities if the second entities are textual.
Claims 1-5 and 7-10 are also rejected under 35 U.S.C. 112(b) by virtue of their dependence on rejected claim 1.
With respect to Claim 11, the claim recites the following, each of which renders the claim indefinite:
“second, different entities” on line 13 (unclear antecedent basis);
“a second entity” on lines 15-17 (unclear antecedent basis); it is unclear as to whether the “second, different entities” recited on line 13 are the same or different from “a second entity” on lines 15-17.
“placing new bounding boxes around text in the second entities” on line 20 (unclear to what this refers); it is unclear what “around text in the second entities” means since “entities”, as defined in Applicant’s specification (paragraph [0054]), are pieces of semantic information such as a date, a name of an industrial part, a telephone number, a street address or the like, which are examples of “text”. Therefore, it is unclear how bounding boxes are placed around text in the second entities since the second entities are text (semantic information such as date, name, phone number, etc.). It is unclear how text is in the second entities if the second entities are textual.
Claims 12-14 and 17-20 are also rejected under 35 U.S.C. 112(b) by virtue of their dependence on rejected claim 11.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-5, 7-8, 11-12, 14-15, and 17 are rejected under 35 U.S.C. 103(a) as being unpatentable Streltsov et al. (U.S. Patent Pub. No. 20230334309 A1, hereafter referred to as Streltsov) in view of Ast (U.S. Patent Pub No. 20220139099 A1, hereafter referred to as Ast) in further view of Yebes Torres et al. (U.S. Patent Pub. No. 20250148443 A1, Yebes Torres).
Regarding Claim 1, Streltsov teaches the following:
A method (Abstract, Streltsov discloses a method for generating a synthetic training data set from an original unstructured electronic document.) comprising:
[AltContent: arrow]a) (Paragraphs [0038], [0056], Fig. 1A, Streltsov teaches obtaining an original electronic document (100) which may be an electronic document such as an invoice, payment advice, a paycheck, a purchase order, a receipt, or any other electronic document.),
PNG
media_image2.png
290
621
media_image2.png
Greyscale
(Paragraphs [0039-40], Fig. 1A, reference character 110, Streltsov teaches the original electronic document (100) which may comprise a plurality of annotated data fields (108). Each section may comprise annotated data fields (108). Each annotated data field may comprise a bounding box (110) and an associated label. Each bounding box (110) may have coordinate data stored therefore (e.g., pixel positions of each corner of the bounding box (110). Bounding boxes may be automatically determined for original electronic document (100).);
PNG
media_image3.png
789
563
media_image3.png
Greyscale
b) inputting semantic information for text in the first bounding boxes (Paragraph [0048], Figs. 1A & 1B, Streltsov teaches semantic augmentations may be applied to annotated data fields (108) to change the text thereof. The semantic augmentations are retrieved from a dictionary or other string data store. In light of paragraphs [0034], [0052], and [0054], Applicant’s specification, the Examiner interprets semantic information to be, for example, company address, customer address, company name, zip code, telephone number, and fax numbers.);
c) (Paragraph [0038], Fig. 1A, Streltsov teaches sections (102), (104), (106) may be automatically determined based on rules defined by the user. See regions 102, 104, and 106 in Fig. 1A above. Each region is defined, see dashed and solid lines defining each section’s boundaries.), the regions to contain one or more of the first bounding boxes (Paragraph [0039], Fig. 1A, Streltsov teaches original electronic document (100) may comprise a plurality of annotated data fields (108). Each section (102), (104), (106) may comprise annotated data fields (108). Each annotated data field (108) may comprise a bounding box (110) and an associated label. See Fig. 1A above, each section (102, 104, and 106) contains one or more bounding boxes.);
d) one of randomly scaling or randomly translating each of the one or more of the first bounding boxes (Paragraphs [0061], [0063], Streltsov teaches performing “micro-operations” such as applying a shift operation to all labels in the original electronic document (100). The shift operations comprise a substantially small percent shift. For example, a label may be shifted by 2% downwards by modifying coordinates of the bounding box. Micro operations may comprise applying a shift operation to all annotated data fields, each of which comprise a label and a bounding box. Micro operations, such as shift operations may be randomly generated to reduce the likelihood that identical synthetic electronic documents are created.) within its own region (Paragraphs [0043], [0061], Streltsov teaches defining a distance limit for the shift operations such that the shifted annotated data fields may not be shifted outside of a specified region of the synthetic electronic document. The distance limit may vary for each annotated data field. The distance limit may be a pixel limit or a percentage limit based on the coordinates of the bounding box. The Examiner interprets a defined distance limit applied for shifts as a specified region in the document. The Examiner interprets “region” broadly as an area or location since the claim is silent to the specifications of the “region.”);
e) identifying, in a region, first entities containing semantic information (Paragraph [0039], Streltsov teaches each section (102, 104, 106) may comprise annotated data fields (108). Each annotated data field (108) may comprise a bounding box (110) and an associated label. For example, an annotated data field (108) for an address field in header section (102) may comprise a label indicating that data within the respective bounding box (110) comprises address data. Similarly, a label may indicate data within a bounding box (110) is total invoice data. Each bounding box (110) may have coordinate data stored therefor (e.g., pixel positions of each corner of the bounding box (110).);
f) replacing the identified first entities with second, different entities containing semantic information (Paragraph [0004], Figs. 1A & 1B, Streltsov teaches “semantic augmentations,” which may comprise changing a text string in the data field. For example, an address field in the original electronic document may be changed to a random address in the synthetic electronic document. The Examiner interprets “changing” and “replacing” to be synonymous in the context of changing one text string to another, different text string. Additionally, the second entities show new/different text that has been replaced, as shown below in the comparison between Fig. 1A and Fig. 1B. For example, the company name in the original electronic document (100) (Fig. 1A) reads “Company Ltd” and the company name in the synthetic electronic document (150) (Fig. 1B) reads “Sala Jurado.” The entities (text strings) depicted in Fig. 1A have been replaced with different entities (text strings) as shown in Fig. 1B. The Examiner interprets “semantic information” to include a text string including, for example, a date, a name of an industrial part, a telephone number, a street address, or the like in light of paragraph [0054] of Applicant’s specification, which provides several examples of pieces of semantic information, such as those specifically listed above.)
[AltContent: arrow][AltContent: arrow]
PNG
media_image4.png
785
1123
media_image4.png
Greyscale
wherein the replacing comprises:
1) randomly selecting a second entity (Paragraph [0052], Streltsov teaches replacing the address with a random address retrieved from the dictionary.);
2) replacing one of the first entities with the second entity (Paragraph [0004], Streltsov teaches “semantic augmentations,” which may comprise changing a text string in the data field. For example, an address field in the original electronic document may be changed to a random address in the synthetic electronic document. The Examiner interprets “changing” and “replacing” to be synonymous in this context. Additionally, the second entities show new/different text that has been replaced, as shown below in Fig. 1B. See Fig. 1A and Fig. 2A above.);
3)
4) repeating said randomly selecting (Paragraph [0052], Figs. 1A & 1B, Streltsov teaches replacing the address with a random address retrieved from the dictionary. This rule may then be repeatedly applied.), said replacing (Paragraphs [0062], [0004], Figs. 1A and 1B, Streltsov teaches semantically augmenting each annotated data field in an original synthetic document (100). The semantic augmentations may comprise changing a string in the data field, requiring each of the steps mentioned above, including selecting the first entity, replacing the first entity with the second entity, and adding the second entity to a dictionary. The Figures (1A and 1B) shown side-by-side below depict all of the first entities (original text) have been replaced with second entities (new/different text). See dashed arrows below for examples.) (Paragraph [0062], Streltsov teaches micro operations (314) may comprise semantic augmentations to text in labels. Each annotated data field (108) may be semantically augmented.);
[AltContent: arrow]
PNG
media_image4.png
785
1123
media_image4.png
Greyscale
g) placing new bounding boxes around text in the second entities (Fig. 1B, reference character 108, Paragraph [0062], Streltsov teaches modifying bounding boxes to account for the new text. For example, if semantic augmentation adds two new lines of text, the size of the bounding box may be increased accordingly. The Examiner interprets “placing second bounding boxes around the text in second entities” to include updating the position and/or size of the bounding box to account for the newly inserted text in the second entity, because the claim is silent as to how the “new” bounding boxes differ or what constitutes a “new” bounding box. See Fig. 1B below, the sizes of the bounding boxes have been changed to account for the replaced text.);
PNG
media_image5.png
785
1123
media_image5.png
Greyscale
h) forming text images to generate new training data for the deep learning system (Abstract, Paragraphs [0001], [0028], [0036], Fig. 1B, Streltsov teaches generating synthetic electronic documents forming a training data set to train a deep learning model. Fig. 1B (below) depicts an example synthetic electronic document generated from the example original electronic document. The Examiner interprets the synthetic electronic document to be a “text image” since the document contains text and the claim is silent to the specifications of “text image.” Additionally, the Examiner interprets the synthetic electronic documents to be “new” training data since they are generated by performing geometric and/or semantic data augmentations to the original electronic document to add structural and/or syntactic variance from the original electronic documents.); and
PNG
media_image6.png
729
483
media_image6.png
Greyscale
i) training the deep learning system using the new training data (Abstract, Paragraphs [0069], [0005], Fig. 4, Fig. 1B, reference character 412, Streltsov teaches providing the training data set comprising a plurality of synthetic electronic documents, such as the one shown in Fig. 1B, to the learning model for training thereon.).
PNG
media_image7.png
591
554
media_image7.png
Greyscale
Streltsov does not explicitly disclose the following:
a) responsive to receipt of an input image of a form, placing first bounding boxes around text in the form
c) using a deep learning system, identifying regions on the form
d) using the deep learning system, performing one of randomly scaling or randomly translating each of the one or more of the first bounding boxes
wherein the replacing comprises:
3) adding the second entity to a dictionary; and
4) repeating said randomly selecting,
Ast is in the same field of art of generating training data from input images of documents to improve the training of a machine learning model. Further, Ast teaches:
[AltContent: arrow][AltContent: arrow]a) responsive to receipt of an input image of a form (Paragraph [0030], Fig. 1, Ast teaches an input image (101) can be provided to or received by OCR module.), placing first bounding boxes around text in the form (Paragraph [0030], Ast teaches OCR module (110) is configured for identifying characters that are depicted within input image (101) can producing an output (115) that includes identified characters and corresponding geometries. “Geometries” refers to the position information associated with the characters. The position information identifies where particular characters are found in the image. The position information can be represented in output (115) as bounding boxes. Output (115) can be a text document that contains the text of the input image and corresponding position information (i.e., bounding boxes).)
[AltContent: arrow]
PNG
media_image8.png
226
785
media_image8.png
Greyscale
b) inputting semantic information for text in the first bounding boxes (Paragraphs [0007-8], Ast teaches converting the text bounded by the geometries (e.g., bounding boxes) to semantic information. A semantics converter module converts the text to semantic information.)
c) using a deep learning system (Abstract, Paragraph [0011], Ast teaches a region-based convolutional neural network (R-CNN).), identifying regions on the form (Abstract, Paragraph [0049], Ast teaches a region-based convolutional neural network (R-CNN) can be trained to extract regions from the semantic image.)
d) using the deep learning system (Abstract, Paragraph [0011], Ast teaches a region-based convolutional neural network (R-CNN).),
wherein the replacing comprises:
3) adding the second entity to a dictionary (Paragraphs [0034-35], Fig. 2, Ast teaches the entries in a language knowledge base for domain-specific semantic types may be supplemented by manually adding these entries to the language knowledge base, or through training and using the system. Over time, language knowledge base (230) can be augmented with additional entries, for instance, automatically as the system is trained and used.); and
PNG
media_image9.png
431
494
media_image9.png
Greyscale
4) repeating (Paragraphs [0034-35], Ast teaches language knowledge base (230) can be augmented with additional entries, for instance, automatically as the system is trained and used.)
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov by localizing text in the input form image using bounding boxes, applying a deep learning model to identify form regions and automate geometric augmentations, and continuously expanding the system’s dictionary that is taught by Ast to make the invention that enables the deep learning system to locate and extract text from regions more accurately and automates the generation of training data with diverse geometric layouts and text/semantic information; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the rule-based or manual process of altering forms by employing a deep learning system to carry out the semantic and geometric augmentations to the annotated data fields and bounding boxes, thereby reducing or eliminating the programming previously required to implement algorithmic rules used in training ML engines (Ast, Col. 2, lines 20-32). In addition, by augmenting the dictionary over time with additional entries, it may help prevent label imbalance such that the model is biased towards labels seen more often during training (Paragraph [0002], Streltsov). Thus, by training the model on a diverse corpus of forms containing varied words, phrases, and text, the accuracy may be improved for handling real-world documents.
Streltsov in view of Ast does not explicitly disclose the following:
wherein the replacing comprises:
4) repeating for all of the first entities;
Yebes-Torres is in the same field of art of training a deep learning model to extract information (text, regions of interest, structure of document, etc.) from an image of a document, such as a receipt using bounding boxes, OCR, and transcribing text.
Further, Yebes-Torres teaches 4) repeating all of the first entities (Paragraphs [0207-208], Yebes-Torres teaches updating internal dictionary (124) to leverage historic data for recognized items in previous processed receipts. A new entry will be added per recognized receipt description.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov in view of Ast by adding the entity or “entry” per recognized item on the receipt (document) to the internal dictionary that is taught by Yebes-Torres, to make the invention that updates the dictionary to be able to recognize and/or generate documents having a wider range of text, words, phrases, or “entities”; thus, one of ordinary skilled in the art would be motivated to combine the references since creating a training data set (documents/forms) having large semantic variation may help prevent label imbalance such that the model is biased towards labels seen more often during training. Thus, the model may be better equipped to identify labels that are seen less often during the training phase (Streltsov, Paragraph [0002]). By updating the dictionary and adding additional entities to the dictionary, the training data generated with a wide variety of entries selected from the dictionary may be better for training the learning model to accurately extract information from documents/forms.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 2, Streltsov in view of Ast in further view of Yebes-Torres teaches the method of claim 1, wherein the input image of the form comprises at least one table (Paragraphs [0037-38], Fig. 1A, reference character 104, Streltsov teaches the electronic document may comprise a tabular layout, such as often seen on an electronic invoice with line-item information.), the method further comprising randomly moving one or more columns and/or one or more rows within the table (Paragraph [0052], Streltsov teaches cloning a random number of line items in a table section (104) to create a plurality of synthetic electronic documents (150) with various sized tables for training the model. Semantic and geometric augmentations such as shifts to the cloned line items may also be performed. The Examiner interprets that by cloning a random number of line items (rows) in a table, as well as applying shifts to the line items, this moves the rows within the table. The claim is silent as to how the rows are moved.).
In regards to Claim 4, Streltsov in view of Ast in further view of Yebes-Torres teaches the method of claim 1, further comprising performing text spotting in the input image of the form (Paragraph [0056], Streltsov teaches automatically generating annotated data fields (108).), and optical character recognition (OCR) on the input image of the form (Paragraphs [0038], [0056], Fig. 3, reference character 306, Streltsov teaches an optical character recognition (OCR) file (306) generated by an OCR model for the original electronic document.).
In regards to Claim 5, Streltsov in view of Ast in further view of Yebes-Torres teaches the method of claim 1, wherein there are first bounding boxes for all of the first entities (Paragraph [0039], Streltsov teaches each annotated data field (108) in the original electronic document comprises a bounding box.).
In regards to Claim 7, Streltsov in view of Ast in further view of Yebes-Torres teaches the method of claim 1, wherein the randomly translating the one or more of the first bounding boxes within one or more of the regions (Paragraphs [0061], [0052], Streltsov teaches applying a “micro-operation,” to the original electronic document (100) such as applying a shift operation to all annotated data fields (108) in the original electronic document. The amount of shift applied to the field may be randomly applied.) comprises, for a bounding box in a region, translating the bounding box only within said region (Paragraph [0061], Streltsov teaches performing a substantially small percent shift, such as downwards by 2% by modifying the coordinates of the bounding box (110). The randomness of the geometric augmentations may also be controlled by the user by setting upper and lower limits on the shift distance. “Region” is being interpreted broadly as any different area or spot on the form since the claim does not specify the constraints of the “region” of the form.).
In regards to Claim 8, Streltsov in view of Ast in further view of Yebes-Torres teaches the method of claim 1, wherein the randomly translating the one or more of the first bounding boxes within one or more of the regions comprises, for a bounding box in a region, translating the bounding box from said region to another of the one or more of the regions (Paragraphs [0010], [0041], Streltsov teaches identifying the header (102) and footer section (106) in the original electronic document (100), and responsive to identifying, deleting the header section and the footer section, and shifting the table section in an arbitrary direction in the sub-template. The position of sections (102, 104, 106) may change due to segmenting annotated data fields (108). The Examiner interprets “shifting the table section in an arbitrary direction” to be a random translation. Additionally, the Examiner interprets shifting the table section, which contains bounding boxes to an arbitrary region since the claim is silent to how the regions differ. “Region” is being interpreted broadly as any different area or spot on the form since the claim is silent to the definition of “region.”).
In regards to Claim 11, Streltsov teaches:
An apparatus (Paragraph [0004], Streltsov teaches a system for generating synthetic electronic documents from an original electronic document for training learning models.) comprising:
a deep learning system (Paragraph [0005], Streltsov teaches a deep learning model.) comprising at least one processor (Paragraph [0019], Streltsov teaches a system including at least one processor.) and a non-transitory memory that contains instructions that, when executed, enable the deep learning system to perform a method (Paragraph [0019], Streltsov teaches one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the at least one processor perform a method for generating the synthetic training data set for training the deep learning model.) comprising:
a) (Paragraphs [0038], [0056], Fig. 1A, Streltsov teaches obtaining an original electronic document (100) which may be an electronic document such as an invoice, payment advice, a paycheck, a purchase order, a receipt, or any other electronic document.), (Paragraphs [0039-40], Fig. 1A, reference character 110, Streltsov teaches the original electronic document (100) which may comprise a plurality of annotated data fields (108). Each section may comprise annotated data fields (108). Each annotated data field may comprise a bounding box (110) and an associated label. Each bounding box (110) may have coordinate data stored therefore (e.g., pixel positions of each corner of the bounding box (110). Bounding boxes may be automatically determined for original electronic document (100).);
b) inputting semantic information for text in the first bounding boxes (Paragraph [0048], Figs. 1A & 1B, Streltsov teaches semantic augmentations may be applied to annotated data fields (108) to change the text thereof. The semantic augmentations are retrieved from a dictionary or other string data store. In light of paragraphs [0034], [0052], and [0054], Applicant’s specification, the Examiner interprets semantic information to be, for example, company address, customer address, company name, zip code, telephone number, and fax numbers.);
c) (Paragraph [0038], Fig. 1A, Streltsov teaches sections (102), (104), (106) may be automatically determined based on rules defined by the user. See regions 102, 104, and 106 in Fig. 1A above. Each region is defined, see dashed and solid lines defining each section’s boundaries.), the regions to contain one or more of the first bounding boxes (Paragraph [0039], Fig. 1A, Streltsov teaches original electronic document (100) may comprise a plurality of annotated data fields (108). Each section (102), (104), (106) may comprise annotated data fields (108). Each annotated data field (108) may comprise a bounding box (110) and an associated label. See Fig. 1A above, each section (102, 104, and 106) contains one or more bounding boxes.);
d) one of randomly scaling or randomly translating each of the one or more of the first bounding boxes (Paragraphs [0061], [0063], Streltsov teaches performing “micro-operations” such as applying a shift operation to all labels in the original electronic document (100). The shift operations comprise a substantially small percent shift. For example, a label may be shifted by 2% downwards by modifying coordinates of the bounding box. Micro operations may comprise applying a shift operation to all annotated data fields, each of which comprise a label and a bounding box. Micro operations, such as shift operations may be randomly generated to reduce the likelihood that identical synthetic electronic documents are created.) within its own region (Paragraphs [0043], [0061], Streltsov teaches defining a distance limit for the shift operations such that the shifted annotated data fields may not be shifted outside of a specified region of the synthetic electronic document. The distance limit may vary for each annotated data field. The distance limit may be a pixel limit or a percentage limit based on the coordinates of the bounding box. The Examiner interprets a defined distance limit applied for shifts as a specified region in the document. The Examiner interprets “region” broadly as an area or location since the claim is silent to the specifications of the “region.”);
e) identifying, in a region, first entities containing semantic information (Paragraph [0039], Streltsov teaches each section (102, 104, 106) may comprise annotated data fields (108). Each annotated data field (108) may comprise a bounding box (110) and an associated label. For example, an annotated data field (108) for an address field in header section (102) may comprise a label indicating that data within the respective bounding box (110) comprises address data. Similarly, a label may indicate data within a bounding box (110) is total invoice data. Each bounding box (110) may have coordinate data stored therefor (e.g., pixel positions of each corner of the bounding box (110).);
f) replacing the identified first entities with second, different entities containing semantic information (Paragraph [0004], Figs. 1A & 1B, Streltsov teaches “semantic augmentations,” which may comprise changing a text string in the data field. For example, an address field in the original electronic document may be changed to a random address in the synthetic electronic document. The Examiner interprets “changing” and “replacing” to be synonymous in the context of changing one text string to another, different text string. Additionally, the second entities show new/different text that has been replaced, as shown below in the comparison between Fig. 1A and Fig. 1B. For example, the company name in the original electronic document (100) (Fig. 1A) reads “Company Ltd” and the company name in the synthetic electronic document (150) (Fig. 1B) reads “Sala Jurado.” The entities (text strings) depicted in Fig. 1A have been replaced with different entities (text strings) as shown in Fig. 1B. The Examiner interprets “semantic information” to include a text string including, for example, a date, a name of an industrial part, a telephone number, a street address, or the like in light of paragraph [0054] of Applicant’s specification, which provides several examples of pieces of semantic information, such as those specifically listed above.)
wherein the replacing comprises:
1) randomly selecting a second entity (Paragraph [0052], Streltsov teaches replacing the address with a random address retrieved from the dictionary.);
2) replacing one of the first entities with the second entity (Paragraph [0004], Streltsov teaches “semantic augmentations,” which may comprise changing a text string in the data field. For example, an address field in the original electronic document may be changed to a random address in the synthetic electronic document. The Examiner interprets “changing” and “replacing” to be synonymous in this context. Additionally, the second entities show new/different text that has been replaced, as shown below in Fig. 1B. See Fig. 1A and Fig. 2A above.);
3)
4) repeating said randomly selecting (Paragraph [0052], Figs. 1A & 1B, Streltsov teaches replacing the address with a random address retrieved from the dictionary. This rule may then be repeatedly applied.), said replacing (Paragraphs [0062], [0004], Figs. 1A and 1B, Streltsov teaches semantically augmenting each annotated data field in an original synthetic document (100). The semantic augmentations may comprise changing a string in the data field, requiring each of the steps mentioned above, including selecting the first entity, replacing the first entity with the second entity, and adding the second entity to a dictionary. The Figures (1A and 1B) shown side-by-side below depict all of the first entities (original text) have been replaced with second entities (new/different text). See dashed arrows below for examples.) (Paragraph [0062], Streltsov teaches micro operations (314) may comprise semantic augmentations to text in labels. Each annotated data field (108) may be semantically augmented.);
g) placing new bounding boxes around text in the second entities (Fig. 1B, reference character 108, Paragraph [0062], Streltsov teaches modifying bounding boxes to account for the new text. For example, if semantic augmentation adds two new lines of text, the size of the bounding box may be increased accordingly. The Examiner interprets “placing second bounding boxes around the text in second entities” to include updating the position and/or size of the bounding box to account for the newly inserted text in the second entity, because the claim is silent as to how the “new” bounding boxes differ. See Fig. 1B below, the sizes of the bounding boxes have been adjusted to account for the newly replaced text.);
h) forming text images to generate new training data for the deep learning system (Abstract, Paragraphs [0001], [0028], [0036], Fig. 1B, Streltsov teaches generating synthetic electronic documents forming a training data set to train a deep learning model. Fig. 1B (below) depicts an example synthetic electronic document generated from the example original electronic document. The Examiner interprets the synthetic electronic document to be a “text image” since the document contains text and the claim is silent to the specifications of “text image.” Additionally, the Examiner interprets the synthetic electronic documents to be “new” training data since they are generated by performing geometric and/or semantic data augmentations to the original electronic document to add structural and/or syntactic variance from the original electronic documents.); and
i) training the deep learning system using the new training data (Abstract, Paragraphs [0069], [0005], Fig. 4, Fig. 1B, reference character 412, Streltsov teaches providing the training data set comprising a plurality of synthetic electronic documents, such as the one shown in Fig. 1B, to the learning model for training thereon.).
Streltsov does not explicitly disclose the following:
a) responsive to receipt of an input image of a form, placing first bounding boxes around text in the form
c) using a deep learning system, identifying regions on the form
d) using the deep learning system, performing one of randomly scaling or randomly translating each of the one or more of the first bounding boxes
wherein the replacing comprises:
3) adding the second entity to a dictionary; and
4) repeating said randomly selecting,
Ast is in the same field of art of generating training data from input images of documents to improve the training of a machine learning model. Further, Ast teaches:
a) responsive to receipt of an input image of a form (Paragraph [0030], Fig. 1, Ast teaches an input image (101) can be provided to or received by OCR module.), placing first bounding boxes around text in the form (Paragraph [0030], Ast teaches OCR module (110) is configured for identifying characters that are depicted within input image (101) can producing an output (115) that includes identified characters and corresponding geometries. “Geometries” refers to the position information associated with the characters. The position information identifies where particular characters are found in the image. The position information can be represented in output (115) as bounding boxes. Output (115) can be a text document that contains the text of the input image and corresponding position information (i.e., bounding boxes).)
b) inputting semantic information for text in the first bounding boxes (Paragraphs [0007-8], Ast teaches converting the text bounded by the geometries (e.g., bounding boxes) to semantic information. A semantics converter module converts the text to semantic information.)
c) using a deep learning system (Abstract, Paragraph [0011], Ast teaches a region-based convolutional neural network (R-CNN).), identifying regions on the form (Abstract, Paragraph [0049], Ast teaches a region-based convolutional neural network (R-CNN) can be trained to extract regions from the semantic image.)
d) using the deep learning system (Abstract, Paragraph [0011], Ast teaches a region-based convolutional neural network (R-CNN).),
wherein the replacing comprises:
3) adding the second entity to a dictionary (Paragraphs [0034-35], Fig. 2, Ast teaches the entries in a language knowledge base for domain-specific semantic types may be supplemented by manually adding these entries to the language knowledge base, or through training and using the system. Over time, language knowledge base (230) can be augmented with additional entries, for instance, automatically as the system is trained and used.); and
4) repeating (Paragraphs [0034-35], Ast teaches language knowledge base (230) can be augmented with additional entries, for instance, automatically as the system is trained and used.)
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov by localizing text in the input form image using bounding boxes, inputting semantic information for the text in the bounding boxes, applying a deep learning model to identify form regions and automate geometric augmentations, and continuously expand the system’s dictionary that is taught by Ast to make the invention that enables the deep learning system to locate and extract text from regions more accurately and automates the generation of training data with diverse geometric layouts and text/semantic information; thus, one of ordinary skilled in the art would be motivated to combine the references to automate the rule-based or manual process of altering forms by employing a deep learning system to carry out the semantic and geometric augmentations to the annotated data fields and bounding boxes, thereby reducing or eliminating the programming previously required to implement algorithmic rules used in training ML engines (Ast, Col. 2, lines 20-32). In addition, by augmenting the dictionary over time with additional entries, it may help prevent label imbalance such that the model is biased towards labels seen more often during training (Paragraph [0002], Streltsov). Thus, by training the model on a diverse corpus of forms containing varied words, phrases, and text, the accuracy may be improved for handling real-world documents.
Streltsov in view of Ast does not explicitly disclose the following:
wherein the replacing comprises:
4) repeating for all of the first entities;
Yebes-Torres is in the same field of art of training a deep learning model to extract information (text, regions of interest, structure of document, etc.) from an image of a document, such as a receipt using bounding boxes, OCR, and transcribing text.
Further, Yebes-Torres teaches 4) repeating all of the first entities (Paragraphs [0207-208], Yebes-Torres teaches updating internal dictionary (124) to leverage historic data for recognized items in previous processed receipts. A new entry will be added per recognized receipt description.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov in view of Ast by adding the entity or “entry” per recognized item on the receipt (document) to the internal dictionary that is taught by Yebes-Torres, to make the invention that updates the dictionary to be able to recognize and/or generate documents having a wider range of text, words, phrases, or “entities”; thus, one of ordinary skilled in the art would be motivated to combine the references since creating a training data set (documents/forms) having large semantic variation may help prevent label imbalance such that the model is biased towards labels seen more often during training. Thus, the model may be better equipped to identify labels that are seen less often during the training phase (Streltsov, Paragraph [0002]). By updating the dictionary and adding additional entities to the dictionary, the training data generated with a wide variety of entries selected from the dictionary may be better for training the learning model to accurately extract information from documents/forms.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 12, Streltsov in view of Ast in further view of Yebes-Torres teaches the apparatus of claim 11, wherein the input image of the form comprises at least one table (Paragraphs [0037-38], Fig. 1A, reference character 104, Streltsov teaches the electronic document may comprise a tabular layout, such as often seen on an electronic invoice with line-item information.), the method further comprising randomly moving one or more columns and/or one or more rows within the table (Paragraph [0052], Streltsov teaches cloning a random number of line items in a table section (104) to create a plurality of synthetic electronic documents (150) with various sized tables for training the model. Semantic and geometric augmentations such as shifts to the cloned line items may also be performed. The Examiner interprets that by cloning a random number of line items (rows) in a table, as well as applying shifts to the line items, this moves the rows within the table.).
In regards to Claim 14, Streltsov in view of Ast in further view of Yebes-Torres teaches the apparatus of claim 11, further comprising performing text spotting in the input image of the form (Paragraph [0056], Abstract, Streltsov teaches automatically generating annotated data fields (108) for the original electronic document (100).) Each annotated data field may comprise a label.), and optical character recognition (OCR) on the input image of the form (Paragraphs [0038], [0056], Fig. 3, reference character 306, Streltsov teaches an optical character recognition (OCR) file (306) generated by an OCR model.).
In regards to Claim 15, Streltsov in view of Ast in further view of Yebes-Torres teaches the apparatus of claim 11, wherein there are first bounding boxes for all of the first entities (Paragraph [0039], Streltsov teaches each annotated data field (108) in the original electronic document comprises a bounding box.).
In regards to Claim 17, Streltsov in view of Ast in further view of Yebes-Torres teaches the apparatus of claim 15, wherein the randomly translating the one or more of the first bounding boxes within one or more of the regions (Paragraphs [0061], [0052], Streltsov teaches applying a “micro-operation,” to the original electronic document (100) such as applying a shift operation to all annotated data fields (108) in the original electronic document. The amount of shift applied to the field may be randomly applied.) comprises, for a bounding box in a region, translating the bounding box only within said region (Paragraph [0061], Streltsov teaches performing a substantially small percent shift, such as downwards by 2% by modifying the coordinates of the bounding box (110). The randomness of the geometric augmentations may also be controlled by the user by setting upper and lower limits on the shift distance. “Region” is being interpreted broadly as any different area or spot on the form since the claim does not specify the constraints of the region on the form.).
Claims 3, 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Streltsov et al. (U.S. Patent Pub. No. 2023/0334309, hereafter referred to as Streltsov) in view of Ast (U.S. Patent Pub No. 20220139099 A1, hereafter referred to as Ast) in further view of Yebes-Torres et al. (U.S. Patent Pub. No. 20250148443 A1, hereafter referred to as Yebes-Torres) in further view of Gohari (U.S. Patent Pub. No. 2022/0318492, hereafter referred to as Gohari).
Regarding Claim 3, Streltsov in view of Ast in further view of Yebes-Torres discloses the method of Claim 1.
Streltsov in view of Ast in further view of Yebes-Torres does not explicitly disclose updating weights of nodes in the deep learning system being trained, responsive to one or more of said randomly scaling, said randomly translating, said replacing, or said forming.
Gohari is in the same field of art of form generation and data extraction using a deep learning system. Further, Gohari teaches updating weights of nodes in the deep learning system being trained (Paragraphs [0016-20], [0033], Fig. 7, Gohari teaches updating weights of nodes in the deep learning model responsive to updating one or more of the identifying text differences and the identifying graphics differences.),
PNG
media_image10.png
543
568
media_image10.png
Greyscale
responsive to one or more of said randomly scaling, said randomly translating, said replacing, or said forming (Paragraphs [0033], [0036], [0059], Figs. 1A and 1B, Fig. 4, Gohari teaches inputting sets of synthetically generated blank forms to the neural network, and comparing the forms to identify differences between them. The synthetically generated documents may be similar, but may have minor changes from form to form, to enable weights of nodes to be altered properly. For example, the header in the form containing the word “Part may be replaced with the word “Widget.” If there are differences in text between two forms, those differences are identified and categorized. Weights for nodes in the neural network are updated and flow is returned to the next pair of forms for training.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov in view of Ast in further view of Yebes-Torres by updating weights of nodes based on the differences between sets of synthetically generated forms/documents that is taught by Gohari, to make the invention that effectively trains the deep learning system to recognize relevant differences between otherwise similar forms, including the types and locations of keywords and potential locations of values corresponding to keywords; thus, one of ordinary skilled in the art would be motivated to combine the references since automating the manual template creation process would save both time and labor associated with manual form creation for training the deep learning model to extract data (Gohari, Abstract, Paragraph [0004]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 13, Streltsov in view of Ast in further view of Yebes-Torres discloses the apparatus of claim 11.
Streltsov in view of Ast in further view of Yebes-Torres does not explicitly disclose updating weights of nodes in the deep learning system being trained, responsive to one or more of said randomly scaling, said randomly translating, said replacing, or said forming.
Gohari is in the same field of art of form generation and data extraction using a deep learning system. Further, Gohari teaches updating weights of nodes in the deep learning system being trained (Paragraphs [0016-20], [0033], Fig. 7, Gohari teaches updating weights of nodes in the deep learning model responsive to updating one or more of the identifying text differences and the identifying graphics differences.), responsive to one or more of said randomly scaling, said randomly translating, said replacing, or said forming (Paragraphs [0033], [0036], [0059], Figs. 1A and 1B, Fig. 4, Gohari teaches inputting sets of synthetically generated blank forms to the neural network, and comparing the forms to identify differences between them. The synthetically generated documents may be similar, but may have minor changes from form to form, to enable weights of nodes to be altered properly. For example, the header in the form containing the word “Part may be replaced with the word “Widget.” If there are differences in text between two forms, those differences are identified and categorized. Weights for nodes in the neural network are updated and flow is returned to the next pair of forms for training.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov in view of Ast in further view of Yebes-Torres by updating weights of nodes based on the differences between sets of synthetically generated forms/documents that is taught by Gohari, to make the invention that effectively trains the deep learning system to recognize relevant differences between otherwise similar forms, including the types and locations of keywords and potential locations of values corresponding to keywords; thus, one of ordinary skilled in the art would be motivated to combine the references since automating the manual template creation process would save both time and labor associated with manual form creation for training the deep learning model to extract data (Gohari, Abstract, Paragraph [0004]).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 18, Streltsov in view of Ast and Yebes-Torres in further view of Gohari disclose the apparatus of claim 13, wherein the randomly translating the one or more of the bounding boxes within one or more of the regions comprises, for a bounding box in a region, translating the bounding box from said region to another of the one or more of the regions (Paragraphs [0010], [0041], Streltsov teaches identifying the header (102) and footer section (106) in the original electronic document (100), responsive to identifying, deleting the header section and the footer section, and shifting the table section in an arbitrary direction in the sub-template. The position of sections (102, 104, 106) may change due to segmenting annotated data fields (108). The Examiner interprets “shifting the table section in an arbitrary direction” to be a random translation. Additionally, the Examiner interprets shifting the table section, which contains bounding boxes to an arbitrary region since the claim is silent to how the regions differ. “Region” is being interpreted broadly as any different area or spot on the form.).
Claims 9, 10, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Streltsov et al. (U.S. Patent Pub. No. 2023/0334309, hereafter referred to as Streltsov) in view of Ast (U.S. Patent Pub No. 20220139099 A1, hereafter referred to as Ast) in further view of Yebes-Torres et al. (U.S. Patent Pub. No. 20250148443 A1, hereafter referred to as Yebes-Torres) in further view of Buban et al. (U.S. Patent No. 12,094,231, hereafter referred to as Buban).
Regarding Claim 9, Ast in view of Streltsov in further view of Yebes-Torres discloses the method of claim 1.
Ast in view of Streltsov in further view of Yebes-Torres does not explicitly disclose wherein the randomly scaling comprises, for a bounding box in a region, enlarging or shrinking the bounding box.
Buban is in the same field of art of form generation for training a machine learning model. Further, Buban discloses wherein the randomly scaling comprises, for a bounding box in a region, enlarging or shrinking the bounding box (Col. 5, lines 28-43, Buban teaches augmenting documents by scaling them. The bounding boxes on the document are transformed according to the augmentation applied to the document. The Examiner interprets the term “scaling” by its well-known definition in the art, in which scaling is defined as a linear transformation that either enlarges or shrinks an object while maintaining its shape.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov in view of Ast in further view of Yebes-Torres by, randomly scaling the bounding boxes in the augmented document by enlarging or shrinking the bounding box that is taught by Buban, to make the invention that augments a smaller set of human-labeled documents to be used to generate a larger set of documents that introduces variance to the training data set (Buban, Col. 5, lines 28-43); thus, one of ordinary skill in the art would be motivated to combine the references to improve the accuracy of identifying and classifying form fields by training the deep learning system on a set of forms with sufficient variance (Buban, Col. 5, lines 65-67 and Col. 12, lines 4-7).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 10, Ast in view of Streltsov and Yebes-Torres in further view of Buban discloses the method of claim 9, further comprising enlarging or shrinking all of the bounding boxes in the region (Col. 5, lines 37-40, Buban teaches transforming the labeled bounding boxes according to the augmentation applied to the entire document. For example, if a document is scaled, the bounding box coordinates of the augmented version of the document are all adjusted accordingly.).
In regards to Claim 19, Ast in view of Streltsov in further view of Yebes-Torres discloses the apparatus of claim 11.
Ast in view of Streltsov in further view of Yebes-Torres does not explicitly disclose wherein the randomly scaling comprises, for a bounding box in a region, enlarging or shrinking the bounding box.
Buban is in the same field of art of form generation for training a machine learning model. Further, Buban discloses wherein the randomly scaling comprises, for a bounding box in a region, enlarging or shrinking the bounding box (Col. 5, lines 28-43, Buban teaches augmenting documents by scaling them. The bounding boxes on the document are transformed according to the augmentation applied to the document. The Examiner interprets the term “scaling” by its well-known definition in the art, in which scaling is defined as a linear transformation that either enlarges or shrinks an object while maintaining its shape.).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Streltsov in view of Ast in further view of Yebes-Torres by, randomly scaling the bounding boxes in the augmented document by enlarging or shrinking the bounding box that is taught by Buban, to make the invention that augments a smaller set of human-labeled documents to be used to generate a larger set of documents that introduces variance to the training data set (Buban, Col. 5, lines 28-43); thus, one of ordinary skill in the art would be motivated to combine the references to improve the accuracy of identifying and classifying form fields by training the deep learning system on a set of forms with sufficient variance (Buban, Col. 5, lines 65-67 and Col. 12, lines 4-7).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
In regards to Claim 20, Ast in view of Streltsov and Yebes-Torres in further view of Buban discloses the apparatus of claim 19, wherein the method further comprises enlarging or shrinking all of the bounding boxes in the region (Col. 5, lines 37-40, Buban teaches transforming the labeled bounding boxes according to the augmentation applied to the entire document. For example, if a document is scaled, the bounding box coordinates of the augmented version of the document are all adjusted accordingly.).
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Panda et al. (U.S. Patent Pub. No. 20250014374 A1) teaches techniques that enable a machine learning (“ML”) model to accurately distinguish in-distribution (“in-D”) elements and out-of-distribution (“OO-D”) elements within an unstructured document. This is achieved using novel training techniques that enable ML models to be trained such that the trained ML models can accurately classify elements extracted from an unstructured document as either belonging to one of a set of in-D classes or to an OO-D class.
Thompson et al. (U.S. Patent Pub. No. 20220405265 A1) teaches computerized analysis of documents to detect tables therein, including single and multi-page tables, as well as tables having diverse layouts, formatting, content and/or terminology; extraction of such tables and/or content depicted therein from documents; and/or classification of tables, and/or portions thereof (especially columns) according to relevancy for downstream applications.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7651. The examiner can normally be reached 8:30am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SYDNEY L BLACKSTEN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
