DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed 08/12/2025.
Claims 1-20 are pending.
Response to Amendment
This Office Action is in response to Applicant’s communication filed 08/12/2025 in response to office action mailed 06/30/2025. The Applicant’s remarks and any amendments to the claims or specification have been considered with the results that follow.
Response to Arguments
Regarding 35 U.S.C. 112(b)
In Remarks page 7, Argument 1
(Examiner summarizes Applicant’s arguments) Applicant argues that Claim 20 as amended clarifies the limitations and thus the 112(b) should be withdrawn.
In response to Argument 1,
Applicant’s amendments are convincing and thus the rejections under 35 U.S.C. 112(b) are overcome and withdrawn.
In Remarks pages 8-10, Argument 2
(Examiner summarizes Applicant’s arguments) Applicant argues that Kukla, Aggarwal, and the instant application are not analogous because machine learning is a very broad field of endeavor and that Kukla and Aggarwal do not address the same problem.
In response to Argument 2,
Examiner disagrees. MPEP 2141.01(a) recites:
“Note that "same field of endeavor" and "reasonably pertinent" are two separate tests for establishing analogous art; it is not necessary for a reference to fulfill both tests in order to qualify as analogous art.”
“These circumstances are to be weighed "from the vantage point of the common sense likely to be exerted by one of ordinary skill in the art in assessing the scope of the endeavor." Airbus, 41 F.3d at 1380. See also Donner Technology, LLC v. Pro Stage Gear, LLC, 979 F.3d 1353, 2020 USPQ2d 11335 (Fed. Cir. 2020); Sanofi-Aventis, 66 F.4th at 1378; and Netflix, Inc. v. DivX, LLC, 80 F.4th 1352, 1358-59, 2023 USPQ2d 1057 (Fed. Cir. 2023) ("The field of endeavor is ‘not limited to the specific point of novelty, the narrowest possible conception of the field, or the particular focus within a given field.’") (quoting Unwired Planet, LLC v. Google Inc., 841 F.3d 995, 1001, 120 USPQ2d 1593, 1597 (Fed. Cir. 2016)).”
Examiner’s position is that, although Aggarwal does not solve the problem of using synthetic documents for machine learning training, it is still analogous to the instant application because they are in the same field of endeavor. Moreover, Kukla is directed to the problem of using synthetic documents for machine learning training. Since the field of endeavor need not be limited to the “narrowest conception of the field”, a person having ordinary skill in the art would recognize that Kukla and Aggarwal generally teach machine learning techniques, similar to the machine learning techniques of the instant application and are therefore analogous even if the type of machine learning is not narrowly the same as the machine learning of the instant application. Moreover, Kukla and Aggarwal are both directed to automatic generation of synthetic documents using machine learning, as is the instant application. Kukla, Aggarwal, and the instant application have much in common and a person having ordinary skill in the art would recognize that they are in the same field of endeavor.
In Remarks pages 10-11, Argument 3
(Examiner summarizes Applicant’s arguments) Applicant argues that (1) the office action does not set forth any sufficient reason to combine Kukla and Aggarwal in the manner of the claims and therefore Kukla nor Agarwal nor their combination teaches the independent claim (2) Kukla does not teach the modified documents of Aggarwal, and Aggarwal does not teach using the modified document for generating synthetic documents or machine learning training (3) If the GANs are removed by using Aggarwal instead, Kukla no longer is able to annotate the documents.
In response to Argument 3,
Examiner disagrees.
Regarding (1), the office action does provide a reason to combine Kukla with Aggarwal. See office action page 10 (Aggarwal paragraph [0018]) “Indeed, the section reflow system can intelligently reposition individual data objects to fill empty space in a modified digital document, split document objects between pages, apply scaling factors or other transformations to document objects, and generate a modified digital document that accurately and efficiently flow between selected sections. In some implementations, the section reflow system can further transmit the modified digital document to a printing device for printing the modified digital document.” In other words, the techniques of Aggarwal improve the formatting of digital documents and eliminate unnecessary whitespace, among other things. A person having ordinary skill in the art would have recognized that the improved document formatting procedures of Aggarwal would provide substantial benefits to the synthetic document generating of Kukla.
Regarding (2), in response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Regarding (3), the office action nor Kukla nor Aggarwal suggest removing the GAN of Kukla to replace it with the techniques of Aggarwal. In fact, office action page 10 suggests modifying “the document layout generation of Kukla with the bounding boxes and shift/delete taught by Aggarwal”. This does not require removing the GAN of Kukla and in fact these operations could be performed in tandem or one after the other. A person having ordinary skill in the art before the effective filing date would be capable of doing so. Therefore, the rejection under 35 U.S.C. 103 is maintained.
In Remarks page 11, Argument 4
(Examiner summarizes Applicant’s arguments) Applicant argues that, by virtue of claim 1, independent claims 8 and 15 as well as all dependent claims are non-obvious for the same reasons.
In response to Argument 4,
Examiner disagrees. For the reasons provided in responses to arguments above, independent claim 1 is not allowable under 35 U.S.C. 103. Therefore the dependent and analogous claims are not allowable for the same reasons.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 8-11, and 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over Kukla et al. (PGPUB No. US 20220229969 A1) in view of Aggarwal et al. (PGPUB no. US 20210368064 A1).
Regarding Claim 1
Kukla teaches:
One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by at least one processor, perform a method comprising:
(paragraph [0006]) “Embodiments also include computer implemented techniques including individual subsystems that are responsible for completing different tasks while a combination of those small parts create the entire system.”
receiving an original electronic document comprising a plurality of annotated data fields,
(paragraph [0034]) “FIG. 2 shows an example of an input 24 to the system, such as may be received by the intake subsystem 12. In this example, the document includes the following elements which may be extracted by the system: Table 30, Paragraph 28, Title 26, and Footer 32”; [*Examiner notes: FIG 2 below shows an electronic document with annotations according to paragraph 34.]
PNG
media_image1.png
607
639
media_image1.png
Greyscale
generating, based on the original electronic document, a plurality of sub-templates, wherein each sub-template of the plurality of sub-templates comprises a distinct layout
(paragraph [0038]) “First, the layout 38 provides a page “template” that describes the arrangement of elements on the document page[*Examiner notes: based on original electronic document].”; (paragraph [0048]) “. In some embodiments, the generated documents may include documents that have identical layouts to the original document(s), and/or pages that have layouts that are similar but not identical to the layouts of the original document(s). Generally it may be preferred for the generated documents to have layouts that vary in specific arrangement and content to the original documents[*Examiner notes: generating a plurality of sub-templates comprising a distinct layout], even where the original corpus includes many documents, to further improve the usefulness of the generated documents in training machine learning and other subsequent systems. That is, it may be preferred to generate documents with a degree of likely diversity, i.e., having pages that aren't already available but that are very plausible to exist within a particular domain of pages of the same type.”
and wherein the generating of at least one sub-template of the plurality of sub-templates comprises: identifying at least three sections of the original electronic document, each section comprising one or more of the plurality of annotated data fields
(paragraph [0034]) “FIG. 2 shows an example of an input 24 to the system, such as may be received by the intake subsystem 12. In this example, the document includes the following elements which may be extracted by the system[*Examiner notes: identifying the sections]: Table 30, Paragraph 28, Title 26, and Footer 32”
PNG
media_image1.png
607
639
media_image1.png
Greyscale
generating a plurality of synthetic electronic documents by applying a plurality of data augmentations to the plurality of sub-templates and the original electronic document;
(paragraph [0048]) “The processes previously disclosed herein may be repeated for each page in the document, so that multiple layouts may be used, each of which may include some or all of the same elements or types of elements. When creating new synthetic documents, pages based upon each layout may be generated, with the types of elements incorporated into the pages of the synthetic documents being determined based upon the layout and annotations created for each page of the original document.”; (paragraph [0047]) “The outputs of the GANs 20 may be combined with the color-coded layouts[*Examiner notes: plurality of augmentations applied to plurality of sub-templates] 18 as previously disclosed to generate a stream of artificial documents 22 based on the original documents using the various element types, as well as the annotations derived from the original documents.”
and training a deep learning model using the plurality of synthetic electronic documents
(paragraph [0022]) “Embodiments disclosed herein may be used to generate a corpus of training documents based on one or more original documents, which then may be used to create, modify, or train other document processing systems.”
Kukla does not explicitly teach:
each annotated data field comprising an annotation that includes a bounding box and a label that identifies data stored within the bounding box
deleting at least two of the identified at least three sections of the original electronic document
and shifting at least one remaining section within the original electronic document by an arbitrary value and in an arbitrary direction of the sub-template;
However, Aggarwal teaches:
each annotated data field comprising an annotation that includes a bounding box and a label that identifies data stored within the bounding box
(paragraph [0057) “In some embodiments, the section reflow system 106 can train the object-detection machine-learning model 304 to predict identify and predict boundaries of document objects based on ground truth data (e.g., labeled document objects with labeled boundaries).”; Figure 5C
deleting at least two of the identified at least three sections of the original electronic document; and shifting at least one remaining section within the original electronic document by an arbitrary value and in an arbitrary direction of the sub-template;
(paragraph [0002]) “For example, the disclosed systems can intelligently remove unselected document objects and reposition the selected document objects to fill in blank space and flow between pages of the modified digital document. Indeed, based on document-object types within a digital document (e.g., text, tables, images, etc.), the disclosed systems can modify positions, scales, and breaking points to generate a modified digital document that smoothly and accurately flows between the document sections selected for printing.”; (paragraph [0115]) “Based on removal of the document sections 708 and 712 […]”; [*Examiner notes: The broadest reasonable interpretation of shifting “by an arbitrary value and in an arbitrary direction” includes shifting the section as necessitated by any given circumstances (e.g. filling blank space).]
Kukla, Aggarwal, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla with the bounding boxes and shift/delete as taught by Aggarwal because (Aggarwal paragraph [0018]) “Indeed, the section reflow system can intelligently reposition individual digital objects to fill empty space in a modified digital document, split document objects between pages, apply scaling factors or other transformations to document objects, and generate a modified digital document that accurately and efficiently flows between selected document sections. In some implementations, the section reflow system can further transmit the modified digital document to a printing device for printing the modified digital document.”
Regarding Claim 3:
Kukla in view of Aggarwal teaches:
The non-transit computer-readable media of claim 1
(see rejection of claim 1)
And Aggarwal further teaches
wherein the method further comprises receiving, from a user, a rule for applying the plurality of data augmentations to the plurality of sub- templates, or the original electronic document.
[*Examiner notes: The broadest reasonable interpretation of “a rule for applying the plurality of data augmentations” includes any input that influences the transformation of the original electronic document to obtain the synthetic electronic document]; (paragraph [0067]) “In response to a user selection of one or more of these selectable elements, the disclosed system can generate a modified digital document”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Kukla with Aggarwal for the same reasons given in claim 1 above.
Regarding Claim 5
Kukla in view of Aggarwal teaches:
The non-transitory computer-readable media of claim 1
(see rejection of claim 1)
And Aggarwal further teaches:
wherein: the identifying of the at least three sections of the original electronic document comprise identifying a header section, a table section, and a footer section of the original electronic document.
(paragraph [0023]) “One approach to automatic processing, analysis, and understanding of documents is to intelligently segment documents to smaller parts such as heading, paragraph, table, footer, header, pictures”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Kukla and Aggarwal for the same reasons given in claim 1 above.
Regarding Claim 8
Kukla teaches:
A method comprising: receiving an original electronic document comprising a plurality of annotated data fields,
(paragraph [0034]) “FIG. 2 shows an example of an input 24 to the system, such as may be received by the intake subsystem 12. In this example, the document includes the following elements which may be extracted by the system: Table 30, Paragraph 28, Title 26, and Footer 32”
PNG
media_image1.png
607
639
media_image1.png
Greyscale
wherein the creating of the synthetic electronic document comprises: identifying at least three sections of the original electronic document, each section comprising one or more of the plurality of annotated data fields;
(paragraph [0034]) “FIG. 2 shows an example of an input 24 to the system, such as may be received by the intake subsystem 12. In this example, the document includes the following elements which may be extracted by the system: Table 30, Paragraph 28, Title 26, and Footer 32”
PNG
media_image1.png
607
639
media_image1.png
Greyscale
and training a deep learning model using the synthetic electronic document
(paragraph [0022]) “Embodiments disclosed herein may be used to generate a corpus of training documents based on one or more original documents, which then may be used to create, modify, or train other document processing systems.”
Kukla does not explicitly teach:
each annotated data field comprising an annotation that includes a bounding box and a label that identifies data stored within the bounding box;
receiving, from a user, at least one data augmentation to apply to at least one annotated data field of the plurality of annotated data fields;
and responsive to receiving the at least one data augmentation, applying the at least one data augmentation to the original electronic document to create a synthetic electronic document,
deleting at least two of the identified at least three sections of the original electronic document
and shifting at least one remaining section within the original electronic document by an arbitrary value and in an arbitrary direction
However, Aggarwal teaches:
each annotated data field comprising an annotation that includes a bounding box and a label that identifies data stored within the bounding box;
(paragraph [0057) “In some embodiments, the section reflow system 106 can train the object-detection machine-learning model 304 to predict identify and predict boundaries of document objects based on ground truth data (e.g., labeled document objects with labeled boundaries).”; Figure 5C
receiving, from a user, at least one data augmentation to apply to at least one annotated data field of the plurality of annotated data fields
and responsive to receiving the at least one data augmentation, applying the at least one data augmentation to the original electronic document to create a synthetic electronic document
(paragraph [0002]) “In response to a user selection of one or more of these selectable elements, the disclosed system can generate a modified digital document for printing by reflowing the identified document objects in accordance with the user selection.”; (paragraph [0133]) “For example, the modified digital document generator 1006 can remove unselected document sections and reposition selected document sections. To do so, the modified digital document generator 1006 may crop portions of the document that correspond to the identified metes and bounds of unselected document sections (e.g., utilizing the location data of document objects provided by the object identification engine 1002). In addition, the modified digital document generator 1006 can move one or more of the selected document sections to fill whitespace resulting from the removal of the unselected document sections.”
deleting at least two of the identified at least three sections of the original electronic document; and shifting at least one remaining section within the original electronic document by an arbitrary value and in an arbitrary direction
(paragraph [0002]) “For example, the disclosed systems can intelligently remove unselected document objects and reposition the selected document objects to fill in blank space and flow between pages of the modified digital document. Indeed, based on document-object types within a digital document (e.g., text, tables, images, etc.), the disclosed systems can modify positions, scales, and breaking points to generate a modified digital document that smoothly and accurately flows between the document sections selected for printing.”; (paragraph [0115]) “Based on removal of the document sections 708 and 712 […]”; [*Examiner notes: The broadest reasonable interpretation of shifting “by an arbitrary value and in an arbitrary direction” includes shifting the section as necessitated by any given circumstances (e.g. filling blank space).]
Kukla, Aggarwal, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla with the bounding boxes and shifting/deleting as taught by Aggarwal because (Aggarwal paragraph [0018]) “Indeed, the section reflow system can intelligently reposition individual digital objects to fill empty space in a modified digital document, split document objects between pages, apply scaling factors or other transformations to document objects, and generate a modified digital document that accurately and efficiently flows between selected document sections. In some implementations, the section reflow system can further transmit the modified digital document to a printing device for printing the modified digital document.”
Regarding Claim 9
Kukla in view of Aggarwal teaches:
The method of claim 8
(see rejection of claim 8)
Aggarwal further teaches:
wherein the method further comprises: receiving, from the user, a first selection of a first bounding box of a first portion of the original electronic document; and receiving, from the user, a second selection of a second bounding box of a second portion of the original electronic document
(paragraph [0022]) “For example, the section reflow system can remove unselected document sections and/or reposition selected document sections to generate a modified document.”; (paragraph [0115]) “Based on removal of the document sections 708 and 712[*Examiner notes: selection of first bounding box and second bounding box], the section reflow system 106 can determine whether the document section 710 fits in whitespace that results from removal of the document section 708.”
wherein the at least one data augmentation is applied between the first bounding box and the second bounding box.
(paragraph [0002]) “For example, the disclosed systems can intelligently remove unselected document objects and reposition the selected document objects to fill in blank space and flow between pages of the modified digital document. Indeed, based on document-object types within a digital document (e.g., text, tables, images, etc.), the disclosed systems can modify positions, scales, and breaking points[*Examiner notes: at least one data augmentation is applied] to generate a modified digital document that smoothly and accurately flows between the document sections[*Examiner notes: between the first bounding box and the second bounding box] selected for printing.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Kukla and Aggarwal for the same reasons given in claim 8 above.
Regarding Claim 10
Kukla in view of Aggarwal teaches:
The method of claim 9
(see rejection of claim 9)
And Aggarwal further teaches:
wherein the at least one data augmentation comprises at least one of a swap, a copy, or a move augmentation
(paragraph [0002]) “For example, the disclosed systems can intelligently remove unselected document objects and reposition the selected document objects to fill in blank space and flow between pages of the modified digital document. Indeed, based on document-object types within a digital document (e.g., text, tables, images, etc.), the disclosed systems can modify positions[*Examiner notes: move augmentation], scales, and breaking points to generate a modified digital document that smoothly and accurately flows between the document sections selected for printing.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Kukla and Aggarwal for the same reasons given in claim 8 above.
Regarding Claim 11
Kukla in view of Aggarwal teaches:
The method of claim 9
(see rejection of claim 9)
And Aggarwal further teaches:
wherein at least one of the first bounding box or the second bounding box comprises at least a subset of the plurality of annotated data fields
“In some embodiments, the section reflow system 106 can train the object-detection machine-learning model 304 to predict identify and predict boundaries of document objects based on ground truth data (e.g., labeled document objects with labeled boundaries[*Examiner notes: annotated data fields])”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Kukla and Aggarwal for the same reasons given in claim 8 above.
Regarding Claim 13
Kukla in view of Aggarwal teaches:
The method of claim 8
(see rejection of claim 8)
And Aggarwal further teaches:
wherein the method further comprises: receiving, from the user, a selection of a bounding box in the original electronic document, the bounding box comprising at least a subset of the plurality of annotated data fields; wherein the at least one data augmentation is applied to the subset of the plurality of annotated data fields
(paragraph [0081]) “After removing the one or more unselected document sections at the act 340, the section reflow system 106 can perform the act 344 by moving one or more document objects[*Examiner notes: bounding box] corresponding to the selected document sections[*Examiner notes: augmentation applied to annotated data fields] (e.g., document sections 2.2.2 and 2.2.4). As described in greater detail in FIGS. 5A-5C, 7A-7B, and 8A-8B, the section reflow system 106 can shift the selected document sections corresponding to the user selection 332 upwards in the digital document.”; (paragraph [0019]) “For example, the section reflow system can identify a variety of objects from digital documents, including text blocks, images, tables, sections, and headings[*Examiner notes: annotated fields].”; Figure 5C
PNG
media_image2.png
538
760
media_image2.png
Greyscale
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Kukla and Aggarwal for the same reasons given in claim 8 above.
Regarding Claim 14
Kukla in view of Aggarwal teaches:
The method of claim 13
(see rejection of claim 13)
And Aggarwal further teaches:
wherein the at least one data augmentation comprises at least one of a shift, a clone, a delete, or a copy augmentation
(paragraph [0002]) “For example, the disclosed systems can intelligently remove unselected document objects and reposition the selected document objects[*Examiner notes: shift augmentation] to fill in blank space and flow between pages of the modified digital document.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Kukla and Aggarwal for the same reasons given in claim 8 above.
Regarding Claim 15
Kukla teaches:
A system comprising: at least one processor; a datastore; and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the at least one processor, perform a method comprising:
(paragraph [0006]) “Embodiments also include computer implemented techniques including individual subsystems that are responsible for completing different tasks while a combination of those small parts create the entire system.”
receiving an original electronic document comprising a plurality of annotated data fields,
(paragraph [0034]) “FIG. 2 shows an example of an input 24 to the system, such as may be received by the intake subsystem 12. In this example, the document includes the following elements which may be extracted by the system: Table 30, Paragraph 28, Title 26, and Footer 32”
PNG
media_image1.png
607
639
media_image1.png
Greyscale
generating, based on the original electronic document, a first sub- template and a second sub-template
(paragraph [0038]) “First, the layout 38 provides a page “template” that describes the arrangement of elements on the document page[*Examiner notes: based on original electronic document].”; (paragraph [0048]) “. In some embodiments, the generated documents may include documents that have identical layouts to the original document(s), and/or pages that have layouts that are similar but not identical to the layouts of the original document(s). Generally it may be preferred for the generated documents to have layouts that vary in specific arrangement and content to the original documents[*Examiner notes: generating a first sub-template and a second sub-template], even where the original corpus includes many documents, to further improve the usefulness of the generated documents in training machine learning and other subsequent systems. That is, it may be preferred to generate documents with a degree of likely diversity, i.e., having pages that aren't already available but that are very plausible to exist within a particular domain of pages of the same type.”
wherein the generating of the first sub-template comprises: identifying at least three sections of the original electronic document, each section comprising one or more of the plurality of annotated data fields
(paragraph [0034]) “FIG. 2 shows an example of an input 24 to the system, such as may be received by the intake subsystem 12. In this example, the document includes the following elements which may be extracted by the system: Table 30, Paragraph 28, Title 26, and Footer 32”
PNG
media_image1.png
607
639
media_image1.png
Greyscale
and generating a plurality of synthetic electronic documents by applying a plurality of data augmentations to the first sub-template, the second sub-template, and the at least one original electronic document,
(paragraph [0048]) “The processes previously disclosed herein may be repeated for each page in the document, so that multiple layouts may be used, each of which may include some or all of the same elements or types of elements. When creating new synthetic documents, pages based upon each layout may be generated, with the types of elements incorporated into the pages of the synthetic documents being determined based upon the layout and annotations created for each page of the original document.”; (paragraph [0047]) “The outputs of the GANs 20 may be combined with the color-coded layouts[*Examiner notes: plurality of augmentations applied to plurality of sub-templates] 18 as previously disclosed to generate a stream of artificial documents 22 based on the original documents using the various element types, as well as the annotations derived from the original documents.”
and training a deep learning model using the plurality of synthetic electronic documents
(paragraph [0022]) “Embodiments disclosed herein may be used to generate a corpus of training documents based on one or more original documents, which then may be used to create, modify, or train other document processing systems.”
Kukla does not explicitly teach:
each annotated data field comprising an annotation that includes a bounding box and a label that identifies data stored within the bounding box
deleting at least two of the identified at least three sections of the original electronic document;
and shifting at least one remaining section within the original electronic document by an arbitrary value and in an arbitrary direction of the first sub- template
However, Aggarwal teaches:
each annotated data field comprising an annotation that includes a bounding box and a label that identifies data stored within the bounding box
(paragraph [0057) “In some embodiments, the section reflow system 106 can train the object-detection machine-learning model 304 to predict identify and predict boundaries of document objects based on ground truth data (e.g., labeled document objects with labeled boundaries).”; Figure 5C
deleting at least two of the identified at least three sections of the original electronic document;
and shifting at least one remaining section within the original electronic document by an arbitrary value and in an arbitrary direction of the first sub- template
(paragraph [0002]) “For example, the disclosed systems can intelligently remove unselected document objects and reposition the selected document objects to fill in blank space and flow between pages of the modified digital document. Indeed, based on document-object types within a digital document (e.g., text, tables, images, etc.), the disclosed systems can modify positions, scales, and breaking points to generate a modified digital document that smoothly and accurately flows between the document sections selected for printing.”; (paragraph [0115]) “Based on removal of the document sections 708 and 712 […]”; [*Examiner notes: The broadest reasonable interpretation of shifting “by an arbitrary value and in an arbitrary direction” includes shifting the section as necessitated by any given circumstances (e.g. filling blank space).]
Kukla, Aggarwal, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla with the bounding boxes and shifting/deleting as taught by Aggarwal because (Aggarwal paragraph [0018]) “Indeed, the section reflow system can intelligently reposition individual digital objects to fill empty space in a modified digital document, split document objects between pages, apply scaling factors or other transformations to document objects, and generate a modified digital document that accurately and efficiently flows between selected document sections. In some implementations, the section reflow system can further transmit the modified digital document to a printing device for printing the modified digital document.”
Regarding Claim 16
Kukla in view of Aggarwal teaches:
The system of claim 15
(see rejection of claim 15)
And Kukla further teaches:
wherein the first sub-template comprises a table page layout, and wherein the second sub-template comprises a footer page layout
(paragraph [0023]) “One approach to automatic processing, analysis, and understanding of documents is to intelligently segment documents to smaller parts such as heading, paragraph, table, footer, header, pictures”; [*Examiner notes: The broadest reasonable interpretation of the terms “table page layout” and “footer page layout” includes any document layout that has a table and a footer in it, respectively. The documents of Kukla include footers and tables and thus are table page layouts and footer page layouts.]
Claims 2 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kukla in view of Aggarwal, and further in view of Kaynig-Fittkau et al. (PGPUB no. US 20210158093 A1), herein referred to as Kaynig-Fittkau.
Regarding Claim 2
Kukla in view of Aggarwal teaches:
The non-transitory computer-readable media of claim 1
(see rejection of claim 1)
Kukla in view of Aggarwal does not explicitly teach:
wherein a data augmentation of the plurality of data augmentations comprises at least one of a shift, a clone, a swap, a delete, or a crop.
However, Kaynig-Fittkau teaches:
wherein a data augmentation of the plurality of data augmentations comprises at least one of a shift, a clone, a swap, a delete, or a crop.
(paragraph [0079]) “As mentioned, the synthetic document generation system 106 can use an algorithmic approach to swap out page elements within real image layouts with desired page elements to create synthetic image layouts.”
Kukla, Aggarwal, Kaynig-Fittkau, and the instant application are analogous because they are all directed to machine learning.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla and Aggarwal with the swap operation of Kaynig-Fittkau because (Kaynig-Fittkau paragraph [0078]) “For instance, the synthetic document generation system 106 may access an insufficient number of image layouts for documents including asides (i.e., a particular page element) to train the image layout prediction neural network 402 to generate image layouts with asides. To account for cases where synthetic image layouts (e.g., the real image layouts 416) are scarce, the synthetic document generation system 106 can utilize algorithmic approaches to transform real image layouts into synthetic image layouts that include the desired page elements.”
Regarding Claim 20
Kukla in view of Aggarwal teaches:
The system of claim 15
(see rejection of claim 15)
Kukla in view of Aggarwal does not explicitly teach:
wherein the method further comprises: receiving, from a user, a creation of a new label; and responsive to receiving, remapping the label to the new label, wherein the plurality of data augmentations is applied to the new label
However, Kaynig-Fittkau teaches:
wherein the method further comprises: receiving, from a user, a creation of a new label; and responsive to receiving, remapping the label to the new label, wherein the plurality of data augmentations is applied to the new label
(paragraph [0057]) “In at least one embodiment, the synthetic document generation system 106 may utilize a different set of labels for the page elements 202-214 based on user input. For example, based on user input to generate synthetic documents that qualify as technical paper, the synthetic document generation system 106 generates synthetic documents[*Examiner notes: plurality of data augmentations applied] including page elements consistent with technical papers[*Examiner notes: remapping label to new label] including introductions, tables of contents, abstracts, etc. The synthetic document generation system 106 labels the page elements accordingly.”
Kukla, Aggarwal, Kaynig-Fittkau, and the instant application are analogous because they are all directed to machine learning and/or data processing.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the Kukla in view of Aggarwal with the remapping of Kaynig-Fittkau (paragraph [0031]) “Furthermore, the synthetic document generation system can make improvements to flexibility relative to conventional systems. […] Because the synthetic document generation system applies styling parameters when inserting synthetic content into documents generated based on the image layouts, the synthetic document generation system can adjust for spacing and other variations across different languages when generating labeled synthetic documents.”
Claims 4 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Kukla, Aggarwal, and further in view Dakin et al. (PGPUB no. US 20160217119 A1).
Regarding Claim 4:
Kukla in view of Aggarwal teaches:
The non-transitory computer-readable media of claim 1
(see rejection of claim 1)
And Kukla further teaches:
wherein a data augmentation of the plurality of data augmentations comprises a semantic data augmentation
(paragraph [0041]) “Elements extracted from documents may be fed to an embedding space subsystem 16. […] In cases where the model was trained on different data than the document being processed by the subsystem, some low-level learned features, such as edge detectors and the like that have shown value in transfer learning systems, may be used to convert the images to semantically meaningful vectors.”
Kukla in view of Aggarwal does not explicitly teach:
and wherein the method further comprises retrieving, from an electronic dictionary associated with an annotated data field of the plurality of annotated data fields, a string for the semantic data augmentation
However, Dakin teaches:
and wherein the method further comprises retrieving, from an electronic dictionary associated with an annotated data field of the plurality of annotated data fields, a string for the semantic data augmentation
[*Examiner notes: The broadest reasonable interpretation of the term “dictionary” includes any storage which is used to store and retrieve textual information associated with annotated data fields.]; (paragraph [0002]) “A form is a document with fields or spaces into which a person can enter data or select a pre-defined data value from a list. […] For example, a form with fields for first and last name may be automatically filled[*Examiner notes: semantic data augmentation] with the user's first and last names[*Examiner notes: mapped to string], which are stored in a database[*Examiner notes: mapped to electronic dictionary associated with annotated data field] from previous user interactions with forms having similar fields.”
Kukla, Aggarwal, Dakin, and the instant application are analogous because they are all directed to document processing and/or artificial intelligence.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla in view of Aggarwal with the electronic dictionary taught by Dakin because (Dakin paragraph [0002]) “Automatic form filling reduces the burden on the user to manually enter the same data repeatedly on different forms” and (Dakin paragraph [0016]) “In this manner, minimal user intervention can be achieved when filling in flat forms, which improves productivity for all users, as well as accessibility for visually impaired users.”
Regarding Claim 17
Kukla in view of Aggarwal teaches:
The system of claim 16
(see rejection of claim 16)
Kukla in view of Aggarwal does not explicitly teach
wherein the plurality of data augmentations comprises a clone operation applied to each label in the table page layout
However, Dakin teaches:
wherein the plurality of data augmentations comprises a clone operation applied to each label in the table page layout
[*Examiner notes: The broadest reasonable interpretation of the term “clone” can include operations such as copying/pasting information from a database as taught by Dakin]; (paragraph [0002]) “A form is a document with fields or spaces into which a person can enter data or select a pre-defined data value from a list. […] For example, a form with fields for first and last name may be automatically filled[*Examiner notes: mapped to clone operation] with the user's first and last names, which are stored in a database from previous user interactions with forms having similar fields.”
Kukla, Aggarwal, Dakin, and the instant application are analogous because they are all directed to document processing and/or artificial intelligence.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla in view of Aggarwal with the clone operation taught by Dakin because (Dakin paragraph [0002]) “Automatic form filling reduces the burden on the user to manually enter the same data repeatedly on different forms” and (Dakin paragraph [0016]) “In this manner, minimal user intervention can be achieved when filling in flat forms, which improves productivity for all users, as well as accessibility for visually impaired users.”
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Kukla, Aggarwal, Ryall and further in view of NPL Liu et al. “IDENTIFICATION OF HEADERS AND FOOTERS IN NOISY DOCUMENTS” herein referred to as Liu, and Ryall et al. (US 20140237350 A1) herein referred to as Ryall.
Regarding Claim 6
Kukla in view of Aggarwal teaches:
The non-transitory computer-readable media of claim 5
(see rejection of claim 5)
Kukla in view of Aggarwal does not explicitly teach:
wherein: the deleting of the at least two of the identified sections comprises deleting the header section and the footer section
and the shifting of the at least one remaining section comprises shifting the table section
However, Liu teaches:
wherein: the deleting of the at least two of the at least three identified sections comprises deleting the header section and the footer section
(page 22 section 3.2.3) “After the second step, a grade for each candidate header/footer line is obtained. The next and also the last step is to evaluate candidate lines and extract headers/footers. Theoretically the lines with “high enough” score are considered to be headers/footers and thus removed automatically by the algorithm.”
Kukla, Aggarwal, Liu, and the instant application are analogous because they are all directed to document processing.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla in view of Aggarwal with the deleting of the header and footer section taught by Liu because (Liu page iii abstract) “Optical Recognition Technology is typically used to convert hard copy printed material into its electronic form. Many presentational artifacts such as end-of-line hyphenations, running headers and footers are literally converted. These artifacts can possibly hinder proximity and exact match searching.” Examiner notes that it would be further obvious to merely select the header and footer sections identified (identification taught by Aggarwal) to be deleted because it amounts to choosing from a finite number of identified solutions (obvious to try).
And Ryall teaches:
and the shifting of the at least one remaining section comprises shifting the table section
(paragraph [0070]) “In an embodiment, layout definition logic 34 is configured to allow a user to move content such as text, tables, images, and links among columns within a section, while concurrently respecting editing restrictions with regard to other portions of the page. Embodiments provide this functionality with no requirement for the user to know the HTML or other coding underlying the electronic document.”
Kukla, Aggarwal, Liu, Ryall, and the instant application are analogous because they are all directed to document processing and/or artificial intelligence.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla, Aggarwal, and Liu with the shifting of the table section taught by Ryall because (Ryall paragraph [0070]) “In an embodiment, layout definition logic 34 is configured to allow a user to move content such as text, tables, images, and links among columns within a section, while concurrently respecting editing restrictions with regard to other portions of the page. Embodiments provide this functionality with no requirement for the user to know the HTML or other coding underlying the electronic document”. Examiner notes that it would be further obvious to merely select the table section identified (identification taught by Aggarwal) to be shifted because it amounts to choosing from a finite number of identified solutions (obvious to try).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Kukla, Aggarwal, Ryall, and further in view of NPL reference Tang et al. “Email Data Cleaning” herein referred to as Tang.
Regarding Claim 7
Kukla in view of Aggarwal teaches:
The non-transitory computer-readable media of claim 5,
(see rejection of claim 5)
Kukla in view of Aggarwal does not explicitly teach:
wherein: the deleting of the at least two of the identified at least three sections comprises deleting the header section and the table section
and the shifting of the at least one remaining section comprises shifting the footer section .
However, Tang teaches:
wherein: the deleting of the at least two of the identified sections comprises deleting the header section and the table section
(page 491 column 2 paragraph 5) “Header, signature, quotation (in forwarded message or replied message), program code, and table are usually irrelevant for mining, and thus should be identified and removed”
Kukla, Aggarwal, Tang, and the instant application are analogous because they are all directed to document processing and/or artificial intelligence.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the document layout generation of Kukla in view of Aggarwal with the deletion of the header section and the table section of Tang because (Tang page 497 column 2 “conclusion”) “Experimental results show that our approach can significantly outperform baseline methods for email data cleaning. When applying it to term extraction from emails, we observe a significant improvement on extraction acc