Detailed Action
This communication is in response to the Request for Continued Examination filed on 11/14/2025.
Claims 1-12 are pending and have been examined.
Claims 1 is an independent system claim.
This Application is still unpublished.
Apparent priority: 10/20/2022.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments and Arguments
Applicant has amended the independent claim to include “a non-transitory memory communicatively coupled to the one or more processors, wherein the one or more processors are execute instructions stored in the non-transitory memory to detect and extract a checkbox symbol and a checkbox option from the digital document comprising one or more checkbox questions; and a linking module configured to establish a relationship between the checkbox option and a corresponding checkbox question, wherein the one or more processors are configured to: detect, using a first machine learning model, the location of the checkbox symbol and the location of the corresponding checkbox option; detect, using a computer vision model, visual shapes of the checkbox symbol and assign a unique visual token; extract textual information from the checkbox option, and group with the corresponding checkbox symbol based on the assigned unique visual token wherein the unique visual token is an anchor to group the textual information with the corresponding checkbox symbol in the digital document; and link, using the linking module, the checkbox option with its corresponding checkbox question based on a context of”
Regarding the Claim Objection of Claim 2. The applicant has amended the dependent claim and as a result the examiner withdraws the objection.
Regarding the 35 USC § 101 rejection applicant notes the 2019 Revised Patent Subject Matter Eligibility Guidance that the claims recite a system, which falls within the "machine" category of patent-eligible subject matter.
Examiner notes the claims are directed to eligible subject matter. The claims recite “A system for detecting and extracting…”
Applicant notes The amended claim 1 underscores a specific and technical implementation that leverages computer vision and multiple trained machine learning models to process digital documents in a manner far beyond human capability. The technical aspects and advantages include:
* Integration of Specialized Machine Learning Models:
" Data Grouping via Unique Visual Tokens:
" Detect Checkbox Symbols While Eliminating False Positives:
Applicant notes the human brain is incapable of processing the vast volumes of pixel-level visual data, textual sequences, and contextual relationships required by the current application with the speed, scale, and precision attained through the integrated computer vision and machine learning models.
Examiner notes that the Specialized Machine Learning Models, the first machine learning model and the computer vision model represent generic learning models. The claims do not include language sufficient to show how they are trained or how they differ from other generic learning models. The claims do not show technological improvement. Examiner further notes a human is naturally capable of Data Grouping via Unique Visual Tokens using logic and resoning in the human mind, and the visual system is capable of Detecting Checkbox Symbols and using logic and reasoning a human can Eliminate False Positives.
Applicant notes that the amended claim 1 is not directed to an abstract idea but to a specific improvement in computer functionality. In Enfish v. Microsoft (Fed. Cir. 2016), the Federal Circuit held that claims reciting a self-referential table were patent-eligible because they improved the way computers stored and retrieved data, cautioning that claims should not be described at too high a level of abstraction.
Examiner notes the additional element of “processor” and “ memory” these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional element of using a processor, hardware, memory computer instructions, firmware is noted as a general computer. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Further, the additional limitation in the claims noted above are directed towards insignificant solution activity. The claims are not patent eligible.
Applicant notes Similarly, in McRO v. Bandai (Fed. Cir. 2016), the Federal Circuit upheld claims directed to animation because they used specific computational rules that improved a technological process and were not merely mental steps.
Applicant notes the current application is similar to these cases because they recite a concrete technological solution, integrating multiple machines learning models, applying computer vision, and assigning unique visual tokens as computational anchors to group textual and non-textual information, linking checkbox options to questions, and eliminate false positives.
Examiner notes the machine learning models are generic machine models and do not amount to more than the abstract idea.
Applicant notes the claims are evaluated to assess whether the claim recites additional elements that integrate the exception into a practical application of the exception.
Applicant notes According to revised step 2A examiners should ensure that they give weight to all additional elements, whether or not they are conventional or described with a high-level of generality, when evaluating whether a judicial exception has been integrated into a practical application. While the initial argument outlined the limitations of the human brain, it is important to outline the technological advancements that contribute to the effectiveness of the claimed device of the current application.
1. Technological Advancement:
* Detecting checkbox symbols and checkbox options in structured and unstructured
digital documents. (Examiner notes a human can detect symbols using the visual system.)
* Detecting the locations of the checkbox symbols and checkbox options using a
first machine learning model. (Examiner notes a human can detect locations using the visual system.)
* Visually processing the checkbox symbols to detect their visual shapes. (Examiner notes a human can detect shapes using the visual system and the human mind.)
* Assign a unique visual token to each of the checkbox symbols. (Examiner notes a human can assign a token using pen and paper or in the human mind.)
* Extracting textual information of the checkbox option and grouping with the corresponding checkbox symbols based on the assigned unique visual tokens. (Examiner notes a human can extract information and group symbols using pen and paper or in the human mind.)
* Employing the unique visual tokens as anchors to group textual information with the corresponding checkbox symbols in the digital document. (Examiner notes a human can assign a token using pen and paper or in the human mind.)
* Establishing links between checkbox options and their corresponding checkbox questions. (Examiner notes a human can establish links between options in the human mind using logic and reasoning.)
2. Practical Application
Applicant notes The current application finds practical application in the field of processing structured and unstructured documents. The amended claim 1 recites non-conventional elements that address core technical challenges, including the use of unique visual tokens as computational anchors to group textual information (checkbox option and question) and non-textual information (checkbox symbol). This approach enables accurate association of checkbox options with their corresponding checkbox symbols, even when document layouts are irregular or non-standardized, thereby overcoming limitations of conventional template-based methods.
Examiner notes a human can establish visual tokens and group textual information and non-textual information in the human mind or using pen and paper.
3. Integration into a Technical Field
Applicant notes The application addresses the technical field of documenting structured and unstructured documents. The current application discloses the system that interacts with components such as a computer vision model, an optical character recognition module and a plurality of machine learning module. The current application's integration with these computational components demonstrates a tangible application of technology rather than the abstract manipulation of information.
Examiner notes a human can interact with structured and unstructured documents. Examiner further notes that computer vision model and the optical character recognition module represent generic models that are not specified how they are trained in order to show how they differ from generic computer vision and optical character recognition.
Applicant notes Further, we rely on McRO, Inc. v. Bandai Namco Games America Inc. (Fed. Cir. 2016). An excerpt from McRO is presented below, wherein reasoning is provided for patent eligibility based on improvement over existing technology.
"As the specification confirms, the claimed improvement here is allowing computers to produce "accurate and realistic lip synchronization and facial expressions in animated characters"
that previously could only be produced by human animators. '576 patent col. 2 ll. 49-50. As the district court correctly recognized, this computer automation is realized by improving the prior art through "the use of rules, rather than artists, to set the morph
weights and transitions between phonemes."
Examiner notes the claims are capable of being performed in the human mind. and examiner further notes that the claims do not integration of the abstract idea into a practical application due to their generic computer components.
The applicant’s arguments and amendments do not overcome the § 35 USC 101 rejection.
Regarding the rejection under 35 USC § 103 Applicant notes Williamson discloses a system for converting structured documents into interactive forms, where mapping of fields is performed based on existing document structure. In contrast, the current application does not rely on pre-existing structured field definitions but instead detects the visual shape of checkbox symbols and assigns a unique visual token to operate as an anchor that groups the extracted textual information of the checkbox options with their corresponding checkbox symbols.
Examiner notes Williamson “translates a structured document into a dynamic form” and “determines input fillable fields” and “analyses the structure and field relationships of the input” [0038] Examiner Further notes there is no mention of structure in the claims.
Applicant notes Williamson refers to "fields" that represent structural metadata embedded within or parsed from a PDF document. These "fields" do not correspond to the visual shapes of checkbox symbols as recited in claim 1 of the current application.
Examiner notes Williamson teaches computer vision that “analyses the structure and field relationships of the input.”
Applicant notes Williamson's "map of possible field names, data types, and relationships" merely assigns textual labels to predefined structured fields within a document and does not perform extraction of textual checkbox options anchored to the visually detected checkbox symbol, as disclosed in claim 1.
Examiner notes Williamson teaches using computer vision technology and machine learning to “match common form patterns and extract contents of each field.” [0020]
Applicant notes Furthermore, Williamson's "best matches" represent a probabilistic classification of field types generated by a machine learning algorithm, and do not involve the deterministic assignment of a unique visual token that anchors textual information to a corresponding checkbox symbol detected through visual processing, as recited in claim 1.
Examiner notes Williamson teaches this limitation for field assignment [0039] “This process uses a number of methods to detect fields in the form, including the ability to parse the PDF fields from fillable forms as well as using computer vision to visually detect fields, and to use system learned best matches to question text, intent and type of information being captured.” [0040] “Match PDF Structure to User Experience 25. The raw PDF structure of fields and their position are sent to the machine learning service (3 of FIG. 1) to draw on historical data for like fields and usage. The service determines how to name the field and possible user interface data type.”
Applicant notes Examiner relied on Hu for "identification of the location of at least one checkbox symbol and its corresponding location relative to the checkbox option."
Applicant notes Hu discloses a line-based approach for predicting field locations on a form image by identifying visible line segments and creating a data structure that maps adjacent image portions to data elements. Field boundaries in Hu are derived from boxes or table lines, and the system predicts a field location when a user moves a cursor over the form, optionally using OCR to assign a label. Thus, Hu is limited to detecting structural lies and boxes for interactive field digitization, not automated detection or contextual linking of visual checkbox symbols as recited in claim 1.
Examiner notes Hu teaches “the computer system automatically displays a predicted location of the field, including a bounding box that represents a boundary of the field.” See the Abstract.
Furthermore, the current application performs fully automated, machine-learning-based detection without relying on predefined templates, manual input, or line-based boundary prediction as in Hu.
Examiner notes Hu teaches this limitation in Figure 1A. Hu creates a structure, to represent a form without a template.
By using trained machine-learning and computer-vision models, the system detects visual checkbox shapes and semantically associates them with text. The unique visual token provides a deterministic, context-aware anchor between visual and textual elements, while the linking module refines these associations to generate a structured, semantic digital representation of the document. Hu neither teaches nor suggests any of these machine- learning, visual-token, or contextual-linking features.
Examiner notes Hu teaches this limitation (2:42-53) “(15) For example, (X,Y) coordinates of the “name” field box, or the “R1C1” field box, can be used to locate an area on the completed form where a person's name or the row one column one table data is expected to appear. Optical character recognition (OCR) can be run on an area of the form that is based on the field boundary, and the person's name or the row one column one table data can be automatically extracted based on the OCR results. For example, the boundary of the field can be oversized by a predetermined amount, and OCR can be run on the area of the form that overlaps with the oversized boundary box to determine text that appears within the oversized boundary box.”
The applicant’s arguments and amendments do not overcome the previously applied prior art.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The independent Claims are directed to statutory categories:
Claim 1 is a system claim and directed to the machine or manufacture category of patentable subject matter.
Independent claim 1 recites,
“1. A system for detecting and extracting checkbox symbol from a digital document, the system comprises one or more processors a non-transitory memory communicatively coupled to the one or more processors, wherein the one or more processors are configured to execute instructions stored in the non-transitory memory to detect and extract a checkbox symbol and a checkbox option from the digital document comprising one or more checkbox questions; and a linking module configured to establish a relationship between the checkbox option and a corresponding checkbox question, wherein the one or more processors are configured to: detect, using a first machine learning model, the location of the corresponding checkbox symbol the location of with respect to the checkbox option; [A human can detect and extract a checkbox symbol in the human mind and using pen and paper. A human can detect a location of a checkbox symbol using the visual system.] [A human can determine context of textual information using natural language understanding.]
Detect, using a computer vision model, visual shapes of the checkbox symbol and assign a unique visual token; [A human naturally detect pictorial representations using the visual system and logic in the human mind.]
extract the textual information from the checkbox option and group with the corresponding checkbox symbol based on the assigned unique visual token [A human can extract and group information using natural logic and understanding. I human can perform visual processing naturally.]
wherein the unique visual token as an anchor to group the textual information with the corresponding checkbox symbol in the digital document; [A human can group textual information with non-textual information using logic and language understanding.]
and link, using the linking module, the checkbox option with its corresponding checkbox question based on a context of the textual information corresponding to the checkbox option.” [A human can identify a link between option and question using logic and reasoning.]
The Dependent Claims do not include additional limitations that could incorporate the abstract idea into a practical application or cause the Claim as a whole to amount to significantly more than the underlying abstract idea.
This judicial exception is not integrated into a practical application. In particular, claim 1 recites the additional element of “processor” and “memory”. For example, in [000109] of the as filed specification, there is description of using a processor, implemented as appropriate in hardware, computer-executable instructions, firmware, or combinations thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using a processor, memory, hardware, computer instructions, firmware is noted as a general computer. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Further, the additional limitation in the claims noted above are directed towards insignificant solution activity. The claims are not patent eligible.
Dependent claim 2 recites,
“2. The system according to claim 1, wherein the one or more processors are configured to: detect location of the checkbox symbol selected by a user using a computer vision model, [This relates to a human observing a location of a checkbox symbol.] wherein the computer vision model is configured to eliminate false checkboxes in the digital document; [This relates to a human ignoring a false checkbox.] and utilize the first machine learning model to detect and validate the checkbox symbol in the digital document.” [This relates to a human validating if a checkbox is real.]
Dependent claim 3 recites,
“3. The system according to claim 2, wherein the one or more processors are configured to train the first machine learning model by: receiving annotation indicating locations of the checkbox symbols in a training corpus, [This relates to a human receiving an annotated version.] wherein in the training corpus, the checkbox symbols comprise of both user-selected and user-unselected checkbox symbols. [This relates to a human observing data of selected and unselected checkbox symbols.]
Dependent claim 4 recites,
“4. The system according to claim 3, wherein the first machine learning model is trained to: validate status of the checkbox symbols and allow selected checkbox symbols to be detected; [This relates to a human validating a status of a checkbox symbol.] and identify a corresponding location of the checkbox option with respect to the detected checkbox symbol.” [This relates to a human identifying a location of a checkbox in relation to a symbol.]
Dependent claim 5 recites,
“5. The system according to claim 1, wherein the one or more processors are configured to: determine the context of the textual information corresponding to the checkbox option in the digital document using the textual processing, [This relates to a human determining context using natural language understanding.] wherein the context of the textual information is determined by pre-processing the textual information in the digital document using an optical character recognition module by: converting the document into an image; [This relates to a human converting a document into an image using pen and paper.] arranging words in the textual information into a two-dimensional sequence; [This relates to a human arranging words using pen and paper.] obtaining a sorted information for each of the words present in the checkbox option; [This relates to a human receiving sorted information.] and feeding the sorted information to train a second machine learning model. [This relates to a human submitting the sorted information to a training system.]
Dependent claim 7 recites,
“6. The system according to claim 5, wherein the second machine learning model is trained by: creating a sequence of words from the sorted information of words present in the textual information; [This relates to a human training by a sequence of words.] tagging words corresponding to the checkbox option corresponding to the checkbox symbol with a start point and an end point, [This relates to a human tagging words using pen and paper.] wherein the checkbox symbol is detected using the first machine learning model; [This relates to a human visually detecting a checkbox symbol.] tagging the checkbox question corresponding to the option in the textual information; [This relates to a human tagging a question using pen and paper.] and tagging the checkbox symbol.; [This relates to a human tagging a symbol using pen and paper.]
Dependent claim 7 recites,
“7. The system according to claim 6, wherein the first machine learning model is trained to detect the checkbox symbol by identifying a plurality of pictorial representations of the checkbox symbol.; [This relates to a human trained to detect a symbol by identifying a representation of a symbol.]
Dependent claim 8 recites,
8. The system according to claim 6, wherein the second machine learning model is further trained by: grouping the tagged information pertaining to the checkbox option, the checkbox question and the checkbox symbol by the unique visual token, [This relates to a human grouping information using natural language understanding and logic and reasoning.] [This relates to a human assigning a token to information in a document using pen and paper.] wherein words corresponding to the checkbox option are grouped with the checkbox symbol using the unique visual token; [This relates to a human grouping a checkbox with a token.] and classify the checkbox option using the textual processing and the unique visual token.; [This relates to a human classifying a checkbox with a token.]
Dependent claim 9 recites,
“9. The system of in claim 8, wherein the unique visual token corresponds to the non-textual information in the digital document which is utilized to classify the textual information in the digital document. [This relates to a human determining using logic and reasoning that a token corresponds to non-textual information in a document.]
Dependent claim 10 recites,
“10. The system according to claim 8, wherein a plurality of unique visual tokens are assigned based on corresponding plurality of pictorial representations of corresponding checkbox symbols across the digital documents. [This relates to a human assigning tokens based on pictorial representations using pen and paper.]
Dependent claim 11 recites,
“11. The system according to claim 6, wherein the tagged information corresponding to the checkbox option and the checkbox question is extracted to be fed to a third machine learning model, [This relates to a human extracting information and submitting to a model.] wherein the third machine learning model is trained to: link the checkbox option to the checkbox question using a linking module, [This relates to a human linking the checkbox to a question using pen and paper.] wherein the third machine learning model is trained to identify a relationship between the checkbox option and the checkbox question based on the tagged information pertaining to the checkbox option and the checkbox question. [This relates to a human trained to identify a relationship between a checkbox option and a queston using logic and reasoning.]
Dependent claim 12 recites,
“12. The system according to claim 11, wherein the third machine learning model is trained by: masking checkbox questions and checkbox options randomly in a training corpus using a masking module; [This relates to a human masking questions and check boxes by hiding them.] generating a plurality of links in the training corpus, [This relates to a human generating links using pen and paper.] wherein the plurality of links between the checkbox options and the checkbox questions are randomly masked; [This relates to a human randomly hiding options and questions using pen and paper.] enabling the third machine learning model to predict a masked information in the training corpus, [This relates to a human predicting information from training information using logic and reasoning in the human mind.] wherein the masked information pertains to a linking of the checkbox question to the checkbox option, [This relates to a human linking a question to an option in the human mind or using pen and paper.] wherein, the third machine learning model with the help of the second machine learning model is trained to: analyze nearby words and context using position and sequence information of word vectors in word tokens between the checkbox options and the checkbox questions; [This relates to a human analyzing words and context using natural language understanding.] predict at least a correct link between the checkbox question and the checkbox option; [This relates to a human predicting a correct link using logic and reasoning.] and extract the checkbox question with the checkbox option. [This relates to a human extracting a question and an option using pen and paper.]
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 are rejected under 35 U.S.C. 103 as being unpatentable over WILLIAMSON (U.S. Patent Number US 20210012060 A1), in view of Hu (U.S. Patent Number US 9910842 B2).
Regarding Claim 1, WILLIAMSON teaches “1. A system for detecting and extracting checkbox symbol from a digital document, the system comprises one or more processors a non-transitory memory communicatively coupled to the one or more processors, wherein the one or more processors are configured to execute instructions stored in the non-transitory memory to detect and extract a checkbox symbol and a checkbox option from the digital document comprising one or more checkbox questions; (see WILLIAMSON [0020] “In one embodiment, it is designed to handle conversion of: 1) Fillable PDF forms, by reading the pre-existing fillable field definitions within the PDF format. 2) Non-fillable PDF forms, which are characterized by needing to be printed to fill in the spaces provided, by using computer vision technology and machine learning to match common form patterns and extract contents of each field.”) (see WILLIAMSON [0024] Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) and a linking module configured to establish a relationship between the checkbox option and a corresponding checkbox question, wherein the one or more processors are configured to: (see WILLIAMSON [0039] This process uses a number of methods to detect fields in the form, including the ability to parse the PDF fields from fillable forms as well as using computer vision to visually detect fields, and to use system learned best matches to question text, intent and type of information being captured.”) (see WILLIAMSON [0057] “…8. A method as claimed in claim 1 wherein, when said structured document includes non fillable forms, said step (b) includes rendering the structured document into a corresponding image, utilizing optical character recognition to determine corresponding textual information, and applying machine learning techniques to said textual information to determine corresponding input fillable fields in said PDF structured document.”) Detect, using a computer vision model, visual shapes of the checkbox symbol and assign a unique visual token; (see WILLIAMSON [0013] “In some embodiments, when the structured document includes non fillable forms, the step (b) preferably can include rendering the structured document into a corresponding image, utilizing optical character recognition to determine corresponding textual information, and applying machine learning techniques to the textual information to determine corresponding input fillable fields in the PDF structured document.”) extract the textual information from the checkbox option and group with the corresponding checkbox symbol based on the assigned unique visual token (see WILLIAMSON [0013] “In some embodiments, when the structured document includes non fillable forms, the step (b) preferably can include rendering the structured document into a corresponding image, utilizing optical character recognition to determine corresponding textual information, and applying machine learning techniques to the textual information to determine corresponding input fillable fields in the PDF structured document.”) (see WILLIAMSON [0023] “For non-fillable PDF forms, the systems converts each page into its corresponding image, and leverages existing computer vision technology to find potential form fields visually. This process uses machine learning based on example tagged PDF forms to learn visual patterns to break up fields successfully.”) wherein the unique visual token is an anchor to group the textual information with the corresponding checkbox symbol in the digital document; (see WILLIAMSON [0039] This process uses a number of methods to detect fields in the form, including the ability to parse the PDF fields from fillable forms as well as using computer vision to visually detect fields, and to use system learned best matches to question text, intent and type (examiner interprets visual token as “best matches to…”) of information being captured.”) (see WILLIAMSON [0022-0024] “The embodiments process, read and interpret any existing PDF fillable form fields from the underlying PDF structure. Identifying and interpreting field names, types and relationships between fields. [0023] For non-fillable PDF forms, the systems converts each page into its corresponding image, and leverages existing computer vision technology to find potential form fields visually. This process uses machine learning based on example tagged PDF forms to learn visual patterns to break up fields successfully. [0024] Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) and link, using the linking module, the checkbox option with its corresponding checkbox question based on a context of the textual information corresponding to the checkbox option. (see WILLIAMSON [0024] “Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) (Examiner notes here in [0024] that Williamson states that the fillable fields can be checkboxes)
WILLIAMSON does not specifically teach detect, using a first machine learning model, the location of the corresponding checkbox symbol the location of with respect to the checkbox option; However Hu does teach this limitation (see Hu (15:35-50) “… determining to digitize the field based on an indication by a user to create the field; and digitizing the field by: writing coordinates that define a boundary of the field to a database that is associated with the image; and determining a label for the field by: executing an optical character recognition (OCR) algorithm on a portion of the image that is determined based on the coordinates of the rectangular box that represents the field; based on results of said executing the OCR algorithm, displaying a suggested label for the field; and based on an indication to use the suggested label, writing the suggested label to the database.”)
WILLIAMSON and Hu are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of WILLIAMSON to incorporate identify location of the checkbox symbol and its corresponding location with respect to the checkbox option of Hu. This allows for an accurate determination of the checkbox location which results in improved a user experience with the document as recognized by Hu (6:28).
As to Claim 2, WILLIAMSON in view of Hu teaches: 2. The system according to claim 1, Furthermore, WILLIAMSON teaches wherein the computer vision model is configured to eliminate false checkboxes in the digital document; (See WILLIAMSON [0020] “In one embodiment, it is designed to handle conversion of: 1) Fillable PDF forms, by reading the pre-existing fillable field definitions within the PDF format. 2) Non-fillable PDF forms, which are characterized by needing to be printed to fill in the spaces provided, by using computer vision technology and machine learning to match common form patterns and extract contents of each field.”) (See WILLIAMSON [0023] For non-fillable PDF forms, the systems converts each page into its corresponding image, and leverages existing computer vision technology to find potential form fields visually. This process uses machine learning based on example tagged PDF forms to learn visual patterns to break up fields successfully.”) (See WILLIAMSON [0012] “In some embodiments the method can further include the step of: providing an interactive user interface for a user to review the determination of input fillable fields. The step (b) further preferably can include: utilizing machine learning on a series of historical document examples to determine probabilistically if a document has input fillable fields. In some embodiments, upon completion of the creation of the second structured document, the second structured document can be added to the series of historical document examples.”) and utilize the first machine learning model to detect and validate the checkbox symbol in the digital document. (See WILLIAMSON [0013] “In some embodiments, when the structured document includes non fillable forms, the step (b) preferably can include rendering the structured document into a corresponding image, utilizing optical character recognition to determine corresponding textual information, and applying machine learning techniques to the textual information to determine corresponding input fillable fields in the PDF structured document.”)
WILLIAMSON does not specifically teach wherein the one or more processors are configured to: detect location of checkbox symbol selected by a user using a computer vision model, However, Hu does teach this limitation (See Hu, (3:17-31) “(18) The user now wishes to digitize a table of the form. The user depresses a click/select indicator of a mouse outside one corner of the table and moves the cursor to outside the opposite corner of the table, which causes a table indicator box to be drawn such that the box encloses the table. The computer system predicts the locations of fields of the table, as well as field names/labels for the fields. The user indicates that he wants to digitize the fields of the table. In some embodiments, the user can interactively modify the size of the bounding boxes that represent the extents of the fields of the table, and can change the name/label of the fields of the table. Once finalized, the user can cause the field information (e.g., the bounding box coordinates, the bounding box locations, the name/label of the fields, etc.) for fields of the table to be written to a database.”)
WILLIAMSON and Hu are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of WILLIAMSON to incorporate the one or more processors are configured to: detect location of checkbox symbol selected by a user using a computer vision model of Hu. This allows for an accurate determination of the checkbox location which results in improved a user experience with the document as recognized by Hu (6:28).
As to Claim 3, WILLIAMSON in view of Hu teaches: 3. The system according to claim 2 Furthermore, WILLIAMSON teaches, wherein the one or more processors are configured to train the first machine learning model by: receiving annotation indicating locations of the checkbox symbols in a training corpus, (See WILLIAMSON [0040] Match PDF Structure to User Experience 25. The raw PDF structure of fields and their position are sent to the machine learning service (3 of FIG. 1) to draw on historical data for like fields and usage. The service determines how to name the field and possible user interface data type. For example, Date of Birth would be detected with high confidence of being a Date and is matched to the Date user interface data type. Further relationships are made with surrounding fields on the form which may result in the algorithm returning a single datatype for many fields. For example, the detection of the text field named Address 1 in close proximity on the PDF to Address 2, City or State may return a high confidence that all of those fields can be represented by a single field named Address which is a compound user interface data type of Address. This intelligent resolution capability simplifies the review process and allows the generation in stage 27 to be a highly dynamic user experience.”) wherein in the training corpus, the checkbox symbols comprise of both user-selected and user-unselected checkbox symbols. (See WILLIAMSON [0012] “In some embodiments the method can further include the step of: providing an interactive user interface for a user to review the determination of input fillable fields. The step (b) further preferably can include: utilizing machine learning on a series of historical document examples to determine probabilistically if a document has input fillable fields. In some embodiments, upon completion of the creation of the second structured document, the second structured document can be added to the series of historical document examples.”) [0042] User confirms or modifies matches 27: The machine learning algorithm ranks it's results and provides the user with the best guesses based on the history contained in the database (4 of FIG. 1). The user is presented with a preview window showing the PDF file as an image marked-up with the various discovered fields.”)
As to Claim 4, WILLIAMSON in view of Hu teaches: 4. The system according to claim 3, Furthermore, WILLIAMSON teaches, wherein the first machine learning model is trained to: validate status of the checkbox symbols and allow selected checkbox symbols to be detected; (See WILLIAMSON [0037] “PDF file structure analysis 23. This step is performed by the PDF structure analyser (5 of FIG. 1). The conversion subsystem parses the PDF file to detect fields/questions, as well as form input structures and data types.”) (See WILLIAMSON [0038] “Render Page by Page Preview image 24. To assist the user experience and as input to the computer vision subsystem (9 of FIG. 1) the system component renders an image format representation of each page of the input. If the PDF structure analysis is not complete, the images will be processed by the computer vision subsystem (9 of FIG. 1) to further analyse the structure and field relationships of the input.”) (See WILLIAMSON [0039] “This process uses a number of methods to detect fields in the form, including the ability to parse the PDF fields from fillable forms as well as using computer vision to visually detect fields, and to use system learned best matches to question text, intent and type of information being captured.”) and identify a corresponding location of the checkbox option with respect to the detected checkbox symbol. (See WILLIAMSON [0024] “Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) (See WILLIAMSON [0042] User confirms or modifies matches 27: The machine learning algorithm ranks it's results and provides the user with the best guesses based on the history contained in the database (4 of FIG. 1). The user is presented with a preview window showing the PDF file as an image marked-up with the various discovered fields.”)
Claims 5-11 are rejected under 35 U.S.C. 103 as being unpatentable over WILLIAMSON (U.S. Patent Number US 20210012060 A1), in view of Hu (U.S. Patent Number US 9910842 B2), and further in view of Khandekar (U.S. Patent Number US 20210042518 A1).
As to Claim 5, WILLIAMSON in view of Hu teaches: 5. The system according to claim 1, Furthermore, WILLIAMSON teaches wherein the one or more processors are configured to: determine the context of the textual information corresponding to the checkbox option in the digital document using the textual processing, (See WILLIAMSON [0024] “Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) (See WILLIAMSON [0042] “User confirms or modifies matches 27: The machine learning algorithm ranks it's results and provides the user with the best guesses based on the history contained in the database (4 of FIG. 1). The user is presented with a preview window showing the PDF file as an image marked-up with the various discovered fields.”) wherein the context of the textual information is determined by pre-processing the textual information in the digital document using an optical character recognition module by: converting the document into an image; (See WILLIAMSON [0013] “In some embodiments, when the structured document includes non fillable forms, the step (b) preferably can include rendering the structured document into a corresponding image, utilizing optical character recognition to determine corresponding textual information, and applying machine learning techniques to the textual information to determine corresponding input fillable fields in the PDF structured document.”) and feeding the sorted information to train a second machine learning model. (See WILLIAMSON [0025] The user is presented with this map and an opportunity to modify what the system has determined programmatically. The resulting map is simultaneously sent back to the machine learning algorithm as additional inputs for future system cycles as well as to a generation engine that processes the map into an adaptive user experience.
WILLIAMSON does not specifically teach arranging words in the textual information into a two-dimensional sequence; However, Khandekar does teach this limitation. (See Khandekar “[0170] The system then scans the word coordinates to find the sequence of words that match all user-provided labels defining information-of-interest. For example, the system finds the sequence of words in word coordinates that match “Fiscal Year 2018” and “Capital Assets,” or their user-provided synonyms. These are the column-header and line identifying labels used by a person's eye to detect where the “Capital Assets” for “Fiscal Year 2018” amount is on the page. The system scans word coordinates multiple times to allow for detection of in-line or wrapped labels.”) obtaining a sorted information for each of the words present in the checkbox option; (see Khandekar [0228] “Minimum number of dark pixels in checked checkbox or radio button—Once the system finds the unknown location of the labels describing a checkbox or a radio button in a source document, it may use the actual location of the labels to find the checkbox or radio button itself and save its cropped image to a local folder. Then the system counts the dark pixels in that saved image using a well-known, readily available API called MICROSOFT System.Drawing. If the number of dark pixels in the locally saved image is greater than this user-provided control value, the system writes the value “True” in the output XML for this information-of-interest; otherwise it writes “False” in the output XML. This is just like a person's eyes noticing the dark area in a checked check box or a clicked radio button. For example, see FIG. 26 for an example of multiple checkboxes that are visually described by column headers and line identifiers. For example, the user will provide the visual relationship “Below Declarations//Borrower//Yes and RightOf Are_You_a_Party_To_a_Lawsuit as Image” to tell the system find the line/column visual intersection, crop that area's image, count the dark pixels and decide if the checkbox has a dark mark in it or not. A good cutoff value for this parameter is “40,” though it may be controlled by the user. In this example, the number of dark pixels in the cropped intersection image is 32, i.e., less than 40; hence, the system will correctly mark “False” in the output XML.”)
WILLIAMSON in view of Hu and Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of WILLIAMSON and Hu to incorporate arranging words in the textual information into a two-dimensional sequence; obtaining a sorted information for each of the words present in the checkbox option of Khandekar. This allows for the system to prevent wrong data extraction from data sources that do not genuinely contain information-of-interest, as recognized by Khandekar [0167].
As to Claim 6, WILLIAMSON in view of Hu and further in view of Khandekar teaches: 6. The system according to claim 5,
Furthermore, Williamson teaches wherein the ch