Detailed Action
This communication is in response to the Arguments and Amendments filed on 4/16/2026.
Claims 1-14 are pending and have been examined.
Claims 13 and 14 are new
Claims 1 is an independent system claim.
This Application is still unpublished.
Apparent priority: 10/20/2022.
Any previous objection/rejection not mentioned in this Office Action has been withdrawn by the Examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments and Arguments
Applicant has amended the independent claim to include “1. (Currently amended) A system for detecting and extracting at least one checkbox symbol from a digital document, wherein the digital document comprises structured documents and unstructured documents, the system comprises: one or more processors; a non-transitory memory communicatively coupled to the one or more processors, wherein the one or more processors are configured to execute instructions stored in the non-transitory memory to detect and extract a checkbox symbol and a checkbox option from the digital document comprising one or more checkbox questions; and a linking module configured to establish a relationship between the checkbox option and a corresponding checkbox question, wherein the one or more processors are configured to: detect, using a first machine learning model, [[the ]]a location of the checkbox symbol and [[the ]]a location of the corresponding checkbox option; erform, using a computer vision model, visual processing on pixels of the digital document to detect a pictorial representation of non-textual information corresponding to by detecting closed contours in the digital document filter the detected closed contours using at least one threshold value to distinguish true checkbox symbols from false checkbox characters; assign a visual token to each detected true checkbox symbol, wherein each distinct symbolic representation of the detected checkbox symbol is assigned with the visual token that is unique to that symbolic representation, wherein the visual token assigned to said symbolic representation is replicated for other checkbox symbols having the same symbolic representation across the digital document; extract textual information from the checkbox option, and group with the corresponding checkbox symbol based on the assigned non-textual information corresponding checkbox symbol in the digital document; and link, using the linking module, the checkbox option with its corresponding checkbox question based on a context of the textual information corresponding to the checkbox option.”
Regarding the 35 USC § 101 rejection, the applicant’s arguments and amendments do not overcome the § 35 USC 101 rejection.
Regarding the rejection under 35 USC § 103
Applicant’s arguments with respect to claim(s) 1-14 have been considered but are moot because the new ground of rejection does not rely on the primary reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Hence, new grounds for rejection have been made in view of Lannitti (US Patent Number US 20230100396 A1).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 2 are rejected under 35 U.S.C. 102(a)(1) as being unpatentable over Lannitti (US Patent Number US 20230100396 A1).
Regarding Claim 1, Iannitti teaches “1. (Currently amended) A system for detecting and extracting at least one checkbox symbol from a digital document, (see Iannitti [0033-0034] Post-fill may include information that may be input after the automatic populating is completed (before or after signature is inputted). For example, the system may auto-populate a list of creditors, but at closing, the lender and/or borrower may need to add additional creditors as post-fill. The post-fill may be embedded into the PDF via adding an XML layer into the document. This allows a program to review the document and determine that such post-fill data was added to the document. In other words, OCR is not needed to re-scan the document to determine the newly added data. [0034] In various embodiments, the system may perform mark detection (examiner interprets extract checkbox symbol as “perform mark detection”) by electronically tagging a document with tag data. The tag data may be metadata associated with the document or added to the document. The metadata may allow the system to interact with the document to effectuate an electronic transaction. The metadata may be data about one or more object fields and the process for executing the document in the object fields (e.g., electronic signature field, completing a checkbox, etc.). Electronically tagging a document may include, for example, conversion, OCR, object detection and vicinity assessment, as explained in more detail below. wherein the digital document comprises structured documents and unstructured documents, (see Iannitti [0006] The documents may include text files, image files, PDF, DOCX, DOC, TXT, PNG, HTML, JPEG or similar types of files. In various embodiments, the document may be a PDF document. The document may comprise multiple pages. The detecting of the words may comprise using OCR for detecting the words. the system comprises: one or more processors; (see Iannitti [0065] …processors…”) a non-transitory memory communicatively coupled to the one or more processors, (see Iannitti [0065] “The computing systems and processors discussed herein may include any type of computing system such as the exemplary computing system of FIG. 7. In various embodiments, and with reference to FIG. 7, computing system 700 may include, for example, any type of input device 705, a central processing unit (CPU) 710 and an output device 730. The CPU 710 may comprise a memory unit 715,”) wherein the one or more processors are configured to execute instructions stored in the non-transitory memory (see Iannitti [0071] “These computer program instructions may be loaded onto a general purpose computer…”) to detect and extract a checkbox symbol [0034] In various embodiments, the system may perform mark detection (examiner interprets extract checkbox symbol as “perform mark detection”) by electronically tagging a document with tag data.”) and a checkbox option from the digital document comprising one or more checkbox questions; and (see Iannitti [0006] “…The metadata in the tag data may include a type of tag that is being used and electronic data related to the object field. The metadata may enable interaction with the document in order to effectuate an electronic transaction. The metadata may include data about the object field. The metadata may include a process for executing the document in the object field. The object field may include an electronic signature field and/or completing a checkbox…”)(see Iannitti [0032] “…For example, the document may include instructions to “check one” or “check all that apply”… (examiner interprets checkbox question as “check all that apply””) a linking module configured to establish a relationship between the checkbox option and a corresponding checkbox question, (see Iannitti [0036] For example, as mentioned above, the system may recognize a notary section from the keywords “my commission expires on”. The system may use the keywords and text to detect candidate object fields (e.g., signature fields, checkboxes and other fields) that may include options for participant interaction. The system may separately detect lines, graphics, etc. using, for example, edge detection, as explained below. The system analyzes objects (e.g., lines) and the vicinity of keywords to those objects. The other object fields that may provide an option for participant interaction include, for example, checkboxes, bubbles, circles, shapes, symbols or other objects in the document. For a cell, a keyword may be at the top of a cell. For a line, the keyword may be under the line. For example, if the word “borrower” exists in text only, then the system will not consider the term as a candidate for a signature. However, if the term borrower is right below a line, then the system will determine a signature field is needed for the borrower's signature.”(examiner notes that the keywords establish a relationship with the options) wherein the one or more processors are configured to: detect, using a first machine learning model, (see Iannitti [0021] “…The system may use edge detection to determine the objects in a document. The system (e.g., using algorithms, artificial intelligence and/or machine learning) may use the keywords and objects to determine if a candidate object field exists”) the location of the checkbox symbol and the location of the corresponding checkbox option;”) (see Iannitti [0038] “This object detection algorithm may recognize the requirements and placements of the object field to facilitate more appropriately associating the electronic tag with the object field.”) (see Iannitti [0039] “…The system may also determine if keywords above or below the line are less or more important (e.g., set less or more weighting) than keywords to the left or right of the line. The algorithm may include a definition of a valid interactive object field (e.g., valid signature field). The system may include a database of definitions for the interactive object fields such as, for example, a notary object field, a borrower object field, etc. If a keyword near the line matches (or is similar to) a keyword in the definition (e.g., borrower is a keyword near the line and part of the definition for a borrower signature line), then the line may be considered a candidate for an interactive object field. After determining the signature field and other object fields, the system may “e-enable” the fields into interactive object fields that can accept electronic entries. The system may record the location of each interactive field as a longitude and latitude point (or set of points, zone, region, etc.) on the document. The system may then send the documents to the participants that the system determines are the appropriate people to sign in the active fields. The participants may use any known software routine to download an electronic signature at the correct location or allow the participant to manually sign at the correct location (e.g., during an e-closing process). perform, using a computer vision model, (see Iannitti [0051] “…computer vision…”) (see Iannitti [0006] “…OCR…”)visual processing on pixels of the digital document to detect a pictorial representation of non-textual information corresponding to the checkbox symbol by detecting closed contours in the digital document; (see Iannitti [0006] “The documents may include text files, image files, PDF, DOCX, DOC, TXT, PNG, HTML, JPEG or similar types of files. In various embodiments, the document may be a PDF document. The document may comprise multiple pages. The detecting of the words may comprise using OCR for detecting the words. The searching for the words may include filtering the words for the keywords. The keywords may comprise names of participants, names of participants that need to sign the document and/or notary language. The method may further include flagging the keywords. An object detection algorithm (examiner interprets visual processing on pixels as “object detection”) may be used in the determining the object fields. The object may include a geometric shape, line, field, parenthesis and/or colon. (examiner interprets closed contours as “geometric shape, line field parenthesis and/or colon.) The object field may include a checkbox, bubble, circle, shape and/or symbol. The tag data may include the metadata that is associated with the document and/or added to the document. The metadata in the tag data may include a type of tag that is being used and electronic data related to the object field. The metadata may enable interaction with the document in order to effectuate an electronic transaction. The metadata may include data about the object field. The metadata may include a process for executing the document in the object field. The object field may include an electronic signature field and/or completing a checkbox. The processor may implement parallel processing and/or include multiple servers. The determining the object field may be further based on requirements and placements of the object field.”) filter the detected closed contours using at least one threshold value to distinguish true checkbox symbols from false checkbox characters; (see Iannitti [0058] “If the system determines that the process produces a false positive about an object field in a particular location of a document…”) (see Iannitti [0059] “…mark the object field as a false positive object field...”) assign a visual token to each detected true checkbox symbol, (see Iannitti [0029] “In various embodiments, as set forth in FIG. 3 and method 300, the system may receive a PDF document (step 302). The system may use OCR to review the document to determine where text exists in the PDF document and consider certain text as keywords (step 304). The system may use edge detection to identify lines and shapes where an input may be needed (step 306). The system may also include rules, data or metadata about various documents. For example, the system may include in its knowledge database that a Promissory Note needs a borrower signature, so if a borrower signature line does not exist, the system may provide a notice that a borrower signature line is needed. The system may determine the type of signer (e.g., borrower, statutory agent, witness, loan officer, etc.) that should be associated with a signature line based on keywords in the vicinity of the signature line. The keywords may include, for example, the terms signor, borrower, statutory agent, witness, and/or loan officer. Such keywords may be included in a database. A known name of the signor for a particular document may also be a keyword. The known name may be associated with that particular document via tagging or other means. The name of a signor may have been input by a participant and/or the system may determine the name of the signor from analyzing other documents in the transaction. The system may ask the participant if the system should insert a borrower signature line. The system may instead just insert the borrower signature line, without participant input. The items in the PDF document that may indicate an object field including, for example, a signature line (step 308), box (step 310), underline (step 312) or line (step 314)..”) wherein each distinct symbolic representation of the detected checkbox symbol is assigned with the visual token that is unique to that symbolic representation, (see Iannitti [0036] “For example, as mentioned above, the system may recognize a notary section from the keywords “my commission expires on”. The system may use the keywords and text to detect candidate object fields (e.g., signature fields, checkboxes and other fields) that may include options for participant interaction. The system may separately detect lines, graphics, etc. using, for example, edge detection, as explained below. The system analyzes objects (e.g., lines) and the vicinity of keywords to those objects. The other object fields that may provide an option for participant interaction include, for example, checkboxes, bubbles, circles, shapes, symbols or other objects in the document. For a cell, a keyword may be at the top of a cell. For a line, the keyword may be under the line. For example, if the word “borrower” exists in text only, then the system will not consider the term as a candidate for a signature. However, if the term borrower is right below a line, then the system will determine a signature field is needed for the borrower's signature.”) wherein the visual token assigned to said symbolic representation is replicated for other checkbox symbols having the same symbolic representation across the digital document; (see Iannitti [0039] After all (or a subset) of the keywords and objects have been identified, in various embodiments, the system implements a grouping algorithm to provide a vicinity assessment. The grouping may also group multiple object fields that are similar. For example, the system may group together six signature fields for six different borrowers on the same document.”) (see Iannitti [0041] “Such changes or actions may be stored in association with the particular document, document type and/or participant account. In various embodiments, the input may be used to build a knowledge base (e.g., based on repeated inputs or actions) for each document, document type and/or participant account to use in future assessments of similar documents and/or document types by that participant.”) extract textual information from the checkbox option, and group with the corresponding checkbox symbol based on the assigned (see Iannitti [0039] “… The vicinity assessment determines if these keywords are close to certain objects or object fields (signature fields, checkboxes, etc.) (examiner interprets textual information as “keyword”) (examiner interprets group as “vicinity assessment”) in the area surrounding the keywords and objects.”) wherein the visual token is an anchor to group the textual information with the non-textual information corresponding checkbox symbol in the digital document; and (see Iannitti [0039] “…After determining the signature field and other object fields, the system may “e-enable” the fields into interactive object fields that can accept electronic entries. The system may record the location of each interactive field as a longitude and latitude point (or set of points, zone, region, etc.) on the document...”) link, using the linking module, the checkbox option with its corresponding checkbox question based on a context of the textual information corresponding to the checkbox option. (see Iannitti [0039] “After all (or a subset) of the keywords and objects have been identified, in various embodiments, the system implements a grouping algorithm to provide a vicinity assessment. The grouping may also group multiple object fields that are similar. For example, the system may group together six signature fields for six different borrowers on the same document. The vicinity assessment may assess the area in the vicinity of such keywords and objects. The vicinity assessment determines if these keywords are close to certain objects or object fields (signature fields, checkboxes, etc.) in the area surrounding the keywords and objects.”)
As to Claim 2, Iannitti teaches: 2. (Currently amended) The system according to claim 1,
Furthermore, Iannitti teaches wherein the one or more processors are configured to: detect a location of the checkbox symbol selected by a user present in the digital document, and determine whether the detected checkbox symbol contains a mark using the computer vision model, (see Iannitti [0036] “After the page has been converted into an image, in various embodiments, the system may use an OCR engine on each page to detect and/or distinguish all (or a subset of) text within the image. The system may also use OCR to detect the location of the text in the image. After the OCR engine identifies all (or a subset) of the text, the system may filter or search to determine if certain text should be considered as keywords. The keywords may include, for example, the names of certain participants, names of people that need to sign the document, notary language, legal language, language associated with certain titles (e.g., President, homeowner, buyer, borrower, lender), etc. The system may flag the keywords. For example, as mentioned above, the system may recognize a notary section from the keywords “my commission expires on”. The system may use the keywords and text to detect candidate object fields (e.g., signature fields, checkboxes and other fields) that may include options for participant interaction. The system may separately detect lines, graphics, etc. using, for example, edge detection, as explained below. The system analyzes objects (e.g., lines) and the vicinity of keywords to those objects. The other object fields that may provide an option for participant interaction include, for example, checkboxes, bubbles, circles, shapes, symbols or other objects in the document. “) wherein the computer vision model is configured to eliminate false checkboxes in the digital document; (see Iannitti [0041] “The system may work with a trained artificial intelligence system, wherein each input may be used to train the algorithm. The system may also include regular learning in that, if 90% of the participants indicate a candidate object field is a false positive, then the system may learn to not include that particular candidate object field in the document and the system may train the algorithm accordingly. ...”) store the mark presence in the non-transitory memory as a status; and (see Iannitti [0034] In various embodiments, the system may perform mark detection by electronically tagging a document with tag data. The tag data may be metadata associated with the document or added to the document. The metadata may allow the system to interact with the document to effectuate an electronic transaction. The metadata may be data about one or more object fields and the process for executing the document in the object fields (e.g., electronic signature field, completing a checkbox, etc.). Electronically tagging a document may include, for example, conversion, OCR, object detection and vicinity assessment, as explained in more detail below.”) utilize the first machine learning model to detect and validate the checkbox symbol in the digital document. (See Iannitti [0041] “While in some scenarios the detection process can be completed with very high accuracy, by its nature, this kind of algorithm can be susceptible to false positives and rules bias. With respect to rules bias, the system may include rules to detect a field, but those rules may be based on false assumptions, so system tries to avoid the rules bias. To overcome this limitation and increase the accuracy of the detection algorithm, the system may learn (e.g., a feedback loop) from a participant's response regarding validation of the interactive object candidates.”)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Lannitti (US Patent Number US 20230100396 A1), in view of WILLIAMSON (U.S. Patent Number US 20210012060 A1).
As to Claim 3, Iannitti teaches: 3. The system according to claim 2
Iannitti does not specifically teach, wherein the one or more processors are configured to train the first machine learning model by: receiving annotation indicating locations of the checkbox symbols in a training corpus, However Williamson does teach this limitiaton (See WILLIAMSON [0040] Match PDF Structure to User Experience 25. The raw PDF structure of fields and their position are sent to the machine learning service (3 of FIG. 1) to draw on historical data for like fields and usage. The service determines how to name the field and possible user interface data type. For example, Date of Birth would be detected with high confidence of being a Date and is matched to the Date user interface data type. Further relationships are made with surrounding fields on the form which may result in the algorithm returning a single datatype for many fields. For example, the detection of the text field named Address 1 in close proximity on the PDF to Address 2, City or State may return a high confidence that all of those fields can be represented by a single field named Address which is a compound user interface data type of Address. This intelligent resolution capability simplifies the review process and allows the generation in stage 27 to be a highly dynamic user experience.”) wherein in the training corpus, the checkbox symbols comprise of both user-selected and user-unselected checkbox symbols. (See WILLIAMSON [0012] “In some embodiments the method can further include the step of: providing an interactive user interface for a user to review the determination of input fillable fields. The step (b) further preferably can include: utilizing machine learning on a series of historical document examples to determine probabilistically if a document has input fillable fields. In some embodiments, upon completion of the creation of the second structured document, the second structured document can be added to the series of historical document examples.”) [0042] User confirms or modifies matches 27: The machine learning algorithm ranks it's results and provides the user with the best guesses based on the history contained in the database (4 of FIG. 1). The user is presented with a preview window showing the PDF file as an image marked-up with the various discovered fields.”)
Iannitti and WILLIAMSON are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti to incorporate wherein the one or more processors are configured to train the first machine learning model by: receiving annotation indicating locations of the checkbox symbols in a training corpus, wherein in the training corpus, the checkbox symbols comprise of both user-selected and user-unselected checkbox symbols of WILLIAMSON. This allows for the intelligent entry of information into fields as recognized by WILLIAMSON [0047].
As to Claim 4, Iannitti in view of Williamson teaches: 4. (Currently amended) The system according to claim 3,
Furthermore, Iannitti teaches, wherein the first machine learning model is trained to: detect the checkbox symbols and validate the checkbox symbols for a possible mark and (see Iannitti [0036] “After the page has been converted into an image, in various embodiments, the system may use an OCR engine on each page to detect and/or distinguish all (or a subset of) text within the image. The system may also use OCR to detect the location of the text in the image. After the OCR engine identifies all (or a subset) of the text, the system may filter or search to determine if certain text should be considered as keywords. The keywords may include, for example, the names of certain participants, names of people that need to sign the document, notary language, legal language, language associated with certain titles (e.g., President, homeowner, buyer, borrower, lender), etc. The system may flag the keywords. For example, as mentioned above, the system may recognize a notary section from the keywords “my commission expires on”. The system may use the keywords and text to detect candidate object fields (e.g., signature fields, checkboxes and other fields) that may include options for participant interaction. The system may separately detect lines, graphics, etc. using, for example, edge detection, as explained below. The system analyzes objects (e.g., lines) and the vicinity of keywords to those objects. The other object fields that may provide an option for participant interaction include, for example, checkboxes, bubbles, circles, shapes, symbols or other objects in the document. “) store the validation in the non-transitory memory as the status; and (see Iannitti [0034] In various embodiments, the system may perform mark detection by electronically tagging a document with tag data. The tag data may be metadata associated with the document or added to the document. The metadata may allow the system to interact with the document to effectuate an electronic transaction. The metadata may be data about one or more object fields and the process for executing the document in the object fields (e.g., electronic signature field, completing a checkbox, etc.). Electronically tagging a document may include, for example, conversion, OCR, object detection and vicinity assessment, as explained in more detail below.”) identify a corresponding location of the checkbox option with respect to the detected checkbox symbol. (See Iannitti [0041] “While in some scenarios the detection process can be completed with very high accuracy, by its nature, this kind of algorithm can be susceptible to false positives and rules bias. With respect to rules bias, the system may include rules to detect a field, but those rules may be based on false assumptions, so system tries to avoid the rules bias. To overcome this limitation and increase the accuracy of the detection algorithm, the system may learn (e.g., a feedback loop) from a participant's response regarding validation of the interactive object candidates.”)
Claims 5-9 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Lannitti (US Patent Number US 20230100396 A1), in view of WILLIAMSON (U.S. Patent Number US 20210012060 A1), and further in view of Khandekar (U.S. Patent Number US 20210042518 A1).
As to Claim 5, Iannitti teaches: 5. The system according to claim 1,
Iannitti does not specifically teach wherein the one or more processors are configured to: determine the context of the textual information corresponding to the checkbox option in the digital document using the textual processing, However, Williamson does teach this limitation (See WILLIAMSON [0024] “Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) (See WILLIAMSON [0042] “User confirms or modifies matches 27: The machine learning algorithm ranks it's results and provides the user with the best guesses based on the history contained in the database (4 of FIG. 1). The user is presented with a preview window showing the PDF file as an image marked-up with the various discovered fields.”) wherein the context of the textual information is determined by pre-processing the textual information in the digital document using an optical character recognition module by: converting the document into an image; (See WILLIAMSON [0013] “In some embodiments, when the structured document includes non fillable forms, the step (b) preferably can include rendering the structured document into a corresponding image, utilizing optical character recognition to determine corresponding textual information, and applying machine learning techniques to the textual information to determine corresponding input fillable fields in the PDF structured document.”) and feeding the sorted information to train a second machine learning model. (See WILLIAMSON [0025] The user is presented with this map and an opportunity to modify what the system has determined programmatically. The resulting map is simultaneously sent back to the machine learning algorithm as additional inputs for future system cycles as well as to a generation engine that processes the map into an adaptive user experience.
Iannitti in view of WILLIAMSON are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti to wherein the one or more processors are configured to: determine the context of the textual information corresponding to the checkbox option in the digital document using the textual processing, wherein the context of the textual information is determined by pre-processing the textual information in the digital document using an optical character recognition module by: converting the document into an image; and feeding the sorted information to train a second machine learning model of WILLIAMSON. This allows for the intelligent entry of information into fields as recognized by WILLIAMSON [0047].
Iannitti in view of WILLIAMSON does not specifically teach arranging words in the textual information into a two-dimensional sequence; However, Khandekar does teach this limitation. (See Khandekar “[0170] The system then scans the word coordinates to find the sequence of words that match all user-provided labels defining information-of-interest. For example, the system finds the sequence of words in word coordinates that match “Fiscal Year 2018” and “Capital Assets,” or their user-provided synonyms. These are the column-header and line identifying labels used by a person's eye to detect where the “Capital Assets” for “Fiscal Year 2018” amount is on the page. The system scans word coordinates multiple times to allow for detection of in-line or wrapped labels.”) obtaining a sorted information for each of the words present in the checkbox option; (see Khandekar [0228] “Minimum number of dark pixels in checked checkbox or radio button—Once the system finds the unknown location of the labels describing a checkbox or a radio button in a source document, it may use the actual location of the labels to find the checkbox or radio button itself and save its cropped image to a local folder. Then the system counts the dark pixels in that saved image using a well-known, readily available API called MICROSOFT System.Drawing. If the number of dark pixels in the locally saved image is greater than this user-provided control value, the system writes the value “True” in the output XML for this information-of-interest; otherwise it writes “False” in the output XML. This is just like a person's eyes noticing the dark area in a checked check box or a clicked radio button. For example, see FIG. 26 for an example of multiple checkboxes that are visually described by column headers and line identifiers. For example, the user will provide the visual relationship “Below Declarations//Borrower//Yes and RightOf Are_You_a_Party_To_a_Lawsuit as Image” to tell the system find the line/column visual intersection, crop that area's image, count the dark pixels and decide if the checkbox has a dark mark in it or not. A good cutoff value for this parameter is “40,” though it may be controlled by the user. In this example, the number of dark pixels in the cropped intersection image is 32, i.e., less than 40; hence, the system will correctly mark “False” in the output XML.”)
Iannitti in view of WILLIAMSON and Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of combination of Iannitti and WILLIAMSON to incorporate arranging words in the textual information into a two-dimensional sequence; obtaining a sorted information for each of the words present in the checkbox option of Khandekar. This allows for the system to prevent wrong data extraction from data sources that do not genuinely contain information-of-interest, as recognized by Khandekar [0167].
As to Claim 6, Iannitti in view of WILLIAMSON and further in view of Khandekar teaches: 6. The system according to claim 5,
Furthermore, Williamson teaches wherein the checkbox symbol is detected using the first machine learning model; (see WILLIAMSON [0029-0030] “Machine Learning service 3. Guided by a large and continuing growing database of previous matches (4) the machine learning service 3 takes the structure of the uploaded PDF and attempts to match the naming, data type and field relationships of the document. The model learns further from inputs from user experience front end 2, where the user has updated or modified the mapping. [0030] A Database 4 with historical data about PDF structure and mappings. This database can forms the input for a machine learning algorithm to determine the possible name, data type and relationship to other fields that exists for the input PDF. This database grows in proportion to the number of mappings performed by the embodiment, thereby growing more accurate over time.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of combination of Iannitti and WILLIAMSON and Khandekar to incorporate wherein the checkbox symbol is detected using the first machine learning model of WILLIAMSON. This allows for the intelligent entry of information into fields as recognized by WILLIAMSON [0047].
Furthermore, Khandekar teaches wherein the second machine learning model is trained by: creating a sequence of words from the sorted information of words present in the textual information; (see Khandekar [0170] The system then scans the word coordinates to find the sequence of words that match all user-provided labels defining information-of-interest. For example, the system finds the sequence of words in word coordinates that match “Fiscal Year 2018” and “Capital Assets,” or their user-provided synonyms. These are the column-header and line identifying labels used by a person's eye to detect where the “Capital Assets” for “Fiscal Year 2018” amount is on the page. The system scans word coordinates multiple times to allow for detection of in-line or wrapped labels.”) tagging words corresponding to the checkbox option corresponding to the checkbox symbol with a start point and an end point, (see Khandekar [0228] “Minimum number of dark pixels in checked checkbox or radio button—Once the system finds the unknown location of the labels describing a checkbox or a radio button in a source document, it may use the actual location of the labels to find the checkbox or radio button itself and save its cropped image to a local folder. Then the system counts the dark pixels in that saved image using a well-known, readily available API called MICROSOFT System.Drawing. If the number of dark pixels in the locally saved image is greater than this user-provided control value, the system writes the value “True” in the output XML for this information-of-interest; otherwise it writes “False” in the output XML. This is just like a person's eyes noticing the dark area in a checked check box or a clicked radio button. For example, see FIG. 26 for an example of multiple checkboxes that are visually described by column headers and line identifiers. For example, the user will provide the visual relationship “Below Declarations//Borrower//Yes and RightOf Are_You_a_Party_To_a_Lawsuit as Image” to tell the system find the line/column visual intersection, crop that area's image, count the dark pixels and decide if the checkbox has a dark mark in it or not. A good cutoff value for this parameter is “40,” though it may be controlled by the user. In this example, the number of dark pixels in the cropped intersection image is 32, i.e., less than 40; hence, the system will correctly mark “False” in the output XML.”) tagging the checkbox question corresponding to the checkbox option in the textual information; and tagging the checkbox symbol. (see Khandekar [0214] “Certain document types allow edits or updates to the document, for example, Web pages that allow a user to enter a search term, PDF and Word Forms that have fields that may be filled and saved by the user, and EXCEL worksheets where values may be entered in cells. There are well-known, readily available programming APIs provided by MICROSOFT and ADOBE, and automation solutions like RPA, to automate edits, entries, updates and clicks to user-enterable fields (for example, input boxes, drop-down lists, radio buttons and clickable button like ‘Submit” or “Save”) on Web pages, on PDF and WORD Forms, and in EXCEL worksheets. For example, a “submit” button on a Web page may be clicked automatically using the APIs to start a search on the Web page. However, initial examples must be shown for each Web page, each PDF and WORD form format, and each EXCEL worksheet that needs automated updates or entries, for the RPA to remember the location or other technical descriptors of each field where data needs to be entered automatically in the future. For example, the technical descriptors of an HTML tag, like “id,” “name,” “Tag-path-from-root” or “Xpath” are used by RPA to remember which data to put where on that Web page in the future. PDF and WORD forms have hidden technical field-descriptors, and EXCEL has “range” or cell row and column information, which the RPA remembers for future data entry. If the locations or technical descriptors change in the source document, the automation gives an error. If the future location is infinitely variable, for example, if an entire section of an EXCEL worksheet may move up or down, depending on the number of rows in an earlier section, the location of the user-enterable cell may not be pre-determined, making the “pre-taught examples” approach completely useless.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Lannittini WILLIAMSON and Khandekar to incorporate wherein the second machine learning model is trained by: creating a sequence of words from the sorted information of words present in the textual information; tagging words corresponding to the checkbox option corresponding to the checkbox symbol with a start point and an end point, tagging the checkbox question corresponding to the checkbox option in the textual information; and tagging the checkbox symbol of Khandekar. This allows for This ability allows the system to prevent wrong data extraction from data sources that do not genuinely contain information-of-interest, as recognized by Khandekar [0167].
As to Claim 7, Iannitti in view of WILLIAMSON and further in view of Khandekar teaches: 7. The system according to claim 6,
Furthermore, Khandekar teaches wherein the first machine learning model is trained to detect the checkbox symbol by identifying a plurality of pictorial representation of the checkbox symbol. (see Khandekar [0228] Minimum number of dark pixels in checked checkbox or radio button—Once the system finds the unknown location of the labels describing a checkbox or a radio button in a source document, it may use the actual location of the labels to find the checkbox or radio button itself and save its cropped image to a local folder. Then the system counts the dark pixels in that saved image using a well-known, readily available API called MICROSOFT System.Drawing. If the number of dark pixels in the locally saved image is greater than this user-provided control value, the system writes the value “True” in the output XML for this information-of-interest; otherwise it writes “False” in the output XML. This is just like a person's eyes noticing the dark area in a checked check box or a clicked radio button. For example, see FIG. 26 for an example of multiple checkboxes that are visually described by column headers and line identifiers. For example, the user will provide the visual relationship “Below Declarations//Borrower//Yes and RightOf Are_You_a_Party_To_a_Lawsuit as Image” to tell the system find the line/column visual intersection, crop that area's image, count the dark pixels and decide if the checkbox has a dark mark in it or not. A good cutoff value for this parameter is “40,” though it may be controlled by the user. In this example, the number of dark pixels in the cropped intersection image is 32, i.e., less than 40; hence, the system will correctly mark “False” in the output XML.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of lannitti WILLIAMSON and Khandekar to incorporate wherein the first machine learning model is trained to detect the checkbox symbol by identifying a plurality of pictorial representation of the checkbox symbol of Khandekar. This ability allows the system to prevent wrong data extraction from data sources that do not genuinely contain information-of-interest, as recognized by Khandekar [0167].
As to Claim 8, Iannitti in view of WILLIAMSON and further in view of Khandekar teaches: 8. The system according to claim 6,
Furthermore, WILLIAMSON teaches wherein the second machine learning model is further trained by: grouping the tagged information pertaining to the checkbox option, the checkbox question and the checkbox symbol by the visual token, (see WILLIAMSON [0023-0024] “For non-fillable PDF forms, the systems converts each page into its corresponding image, and leverages existing computer vision technology to find potential form fields visually. This process uses machine learning based on example tagged PDF forms to learn visual patterns to break up fields successfully. [0024] Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) (see WILLIAMSON [0029-0030] “Machine Learning service 3. Guided by a large and continuing growing database of previous matches (4) the machine learning service 3 takes the structure of the uploaded PDF and attempts to match the naming, data type and field relationships of the document. The model learns further from inputs from user experience front end 2, where the user has updated or modified the mapping. [0030] A Database 4 with historical data about PDF structure and mappings. This database can forms the input for a machine learning algorithm to determine the possible name, data type and relationship to other fields that exists for the input PDF. This database grows in proportion to the number of mappings performed by the embodiment, thereby growing more accurate over time.”) wherein words corresponding to the checkbox option are grouped with the checkbox symbol using the visual token; (see WILLIAMSON [0039-0040] This process uses a number of methods to detect fields in the form, including the ability to parse the PDF fields from fillable forms as well as using computer vision to visually detect fields, and to use system learned best matches to question text, intent and type of information being captured. [0040] Match PDF Structure to User Experience 25. The raw PDF structure of fields and their position are sent to the machine learning service (3 of FIG. 1) to draw on historical data for like fields and usage. The service determines how to name the field and possible user interface data type. For example, Date of Birth would be detected with high confidence of being a Date and is matched to the Date user interface data type. Further relationships are made with surrounding fields on the form which may result in the algorithm returning a single datatype for many fields. For example, the detection of the text field named Address 1 in close proximity on the PDF to Address 2, City or State may return a high confidence that all of those fields can be represented by a single field named Address which is a compound user interface data type of Address. This intelligent resolution capability simplifies the review process and allows the generation in stage 27 to be a highly dynamic user experience.”) and classify the checkbox option using the textual processing and the visual token. (see WILLIAMSON [0029] “Machine Learning service 3. Guided by a large and continuing growing database of previous matches (4) the machine learning service 3 takes the structure of the uploaded PDF and attempts to match the naming, data type and field relationships of the document. The model learns further from inputs from user experience front end 2, where the user has updated or modified the mapping. [0030] A Database 4 with historical data about PDF structure and mappings. This database can forms the input for a machine learning algorithm to determine the possible name, data type and relationship to other fields that exists for the input PDF. This database grows in proportion to the number of mappings performed by the embodiment, thereby growing more accurate over time.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti WILLIAMSON and Khandekar to incorporate wherein the second machine learning model is further trained by: grouping the tagged information pertaining to the checkbox option, the checkbox question and the checkbox symbol by the visual token, wherein words corresponding to the checkbox option are grouped with the checkbox symbol using the visual token; and classify the checkbox option using the textual processing and the visual token of WILLIAMSON. This allows for the intelligent entry of information into fields as recognized by WILLIAMSON [0047].
As to Claim 9, Iannitti in view of WILLIAMSON and further in view of Khandekar teaches: 9. The system according to claim 8,
Furthermore, WILLIAMSON teaches wherein the visual token corresponds to the non-textual information in the digital document which is utilized to classify the textual information in the digital document. (see WILLIAMSON [0023-0024] “For non-fillable PDF forms, the systems converts each page into its corresponding image, and leverages existing computer vision technology to find potential form fields visually. This process uses machine learning based on example tagged PDF forms to learn visual patterns to break up fields successfully. [0024] Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”) (see WILLIAMSON [0039] “This process uses a number of methods to detect fields in the form, including the ability to parse the PDF fields from fillable forms as well as using computer vision to visually detect fields, and to use system learned best matches to question text, intent and type of information being captured.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti WILLIAMSON and Khandekar to incorporate wherein the visual token corresponds to the non-textual information in the digital document which is utilized to classify the textual information in the digital document of WILLIAMSON. This allows for the intelligent entry of information into fields as recognized by WILLIAMSON [0047].
As to Claim 11, Iannitti in view of WILLIAMSON and further in view of Khandekar teaches: 11. The system according to claim 6,
Furthermore, WILLIAMSON teaches wherein the tagged information corresponding to the checkbox option and the checkbox question is extracted to be fed to a third machine learning model, wherein the third machine learning model is trained to: (See WILLIAMSON [0045] Final matches submitted 28: The overridden selections (and system generated selections) are sent to the machine learning service (3 of FIG. 1) to train future conversions, populating a history database (4 of FIG. 1) which will improve results and accuracy over time.”) link the checkbox option to the checkbox question using the linking module, (See WILLIAMSON [0037] PDF file structure analysis 23. This step is performed by the PDF structure analyser (5 of FIG. 1). The conversion subsystem parses the PDF file to detect fields/questions, as well as form input structures and data types.”) (See WILLIAMSON [0039] This process uses a number of methods to detect fields in the form, including the ability to parse the PDF fields from fillable forms as well as using computer vision to visually detect fields, and to use system learned best matches to question text, intent and type of information being captured.”) wherein the third machine learning model is trained to identify the relationship between the checkbox option and the checkbox question based on the tagged information pertaining to the checkbox option and the checkbox question. (See WILLIAMSON [0040] “Match PDF Structure to User Experience 25. The raw PDF structure of fields and their position are sent to the machine learning service (3 of FIG. 1) to draw on historical data for like fields and usage. The service determines how to name the field and possible user interface data type. For example, Date of Birth would be detected with high confidence of being a Date and is matched to the Date user interface data type. Further relationships are made with surrounding fields on the form which may result in the algorithm returning a single datatype for many fields. For example, the detection of the text field named Address 1 in close proximity on the PDF to Address 2, City or State may return a high confidence that all of those fields can be represented by a single field named Address which is a compound user interface data type of Address. This intelligent resolution capability simplifies the review process and allows the generation in stage 27 to be a highly dynamic user experience.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti WILLIAMSON and Khandekar to incorporate wherein the tagged information corresponding to the checkbox option and the checkbox question is extracted to be fed to a third machine learning model, wherein the third machine learning model is trained to: link the checkbox option to the checkbox question using the linking module, wherein the third machine learning model is trained to identify the relationship between the checkbox option and the checkbox question based on the tagged information pertaining to the checkbox option and the checkbox question of WILLIAMSON. This allows for the intelligent entry of information into fields as recognized by WILLIAMSON [0047].
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Lannitti (US Patent Number US 20230100396 A1), in view of WILLIAMSON (U.S. Patent Number US 20210012060 A1), and further in view of Khandekar (U.S. Patent Number US 20210042518 A1), and further in view of Hu (U.S. Patent Number US 9910842 B2),
As to Claim 10, Iannitti in view of WILLIAMSON and further in view of Khandekar teaches: 10. The system according to claim 8,
Furthermore, Hu teaches wherein a plurality of visual tokens are assigned based on corresponding plurality of pictorial representations of corresponding checkbox symbols across the digital documents. (see Hu (4:34-45) “(24) In some embodiments, the data structure is organized so that adjacent portions of an image map to adjacent elements of a data structure. (examiner interprets pictorial representations as “elements”) For example, data structure 230 of FIG. 2 is organized so that adjacent “portions” of image 220 map to adjacent elements of data structure 230. Each square of image 220 represents a “portion” of image 220, and each square of data structure 230 represents an element of data structure 230. Each corner of each square of image 220, such as the square at index (0,0), or the square at index (8,8), is coincident with a grid of image 220. Grid points 225 identifies examples of three grids, also referred to as grid points.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar and Hu are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti WILLIAMSON and Khandekar to incorporate wherein a plurality of unique visual tokens are assigned based on corresponding plurality of pictorial representations of corresponding checkbox symbols across the digital documents of Hu. This allows for an accurate determination of the checkbox location which results in improved a user experience with the document as recognized by Hu (6:28).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Lannitti (US Patent Number US 20230100396 A1), in view of WILLIAMSON (U.S. Patent Number US 20210012060 A1), and further in view of Khandekar (U.S. Patent Number US 20210042518 A1), and further in view of SHARMA (U.S. Patent Number US 20240095445 A1).
As to Claim 12, Iannitti in view of WILLIAMSON and further in view of Khandekar teaches: 12. The system according to claim 11,
Furthermore, WILLIAMSON teaches and extract the checkbox question with the checkbox option. (see WILLIAMSON [0024] “Both paths through the system result in a map of possible field names, data types and field relationships to represent the input PDF. This map can be passed into a machine learning algorithm to interpret intent to find the best possible match to dynamic user experience question types, such as date pickers, text fields, checkboxes and radio buttons.”)
Iannitti in view of WILLIAMSON and further in view of Khandekar are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti WILLIAMSON and Khandekar to incorporate and extract the checkbox question with the checkbox option of WILLIAMSON. This allows for the intelligent entry of information into fields as recognized by WILLIAMSON [0047].
Iannitti in view of WILLIAMSON and further in view of Khandekar does not specifically teach wherein the third machine learning model is trained by: masking checkbox questions and checkbox options randomly in a training corpus using a masking module; generating a plurality of links in the training corpus, wherein the plurality of links between the checkbox options and the checkbox questions are randomly masked; enabling the third machine learning model to predict a masked information in the training corpus, wherein the masked information pertains to the linking of the checkbox question to the checkbox option, However Sharma does teach this limitation (See SHARMA [0039] In one embodiment, the autocomplete model may be a semi-supervised model configured to suggest the next word to a user (e.g., a clinician) entering in data (e.g., text) in a data field. For example, server system 104 may receive a query or a string of text from a clinician operating an interactive GUI on a user device. The autocomplete model may tokenize the text in the query or the string of text, thereby reducing the query or string of text into smaller segments, which aids the autocomplete in interpreting the context of the query. In addition, one technique the autocomplete model may implement is a masking technique, wherein the text elements are masked (i.e., hidden) from the autocomplete model, thereby providing the autocomplete model with incomplete query, word, or string of text, and subsequently asking the autocomplete model to accurately generate a complete query, word, or string of text by predicting the masked query syntax elements. Accordingly, training may include predicting, via the autocomplete model, masked text in the query, word, or string of text. For example, the autocomplete model may receive a query with masked query syntax elements as input and attempt to predict the masked query syntax elements by bidirectionally analyzing the query and the non-masked query syntax elements for context. The autocomplete model can interpret context by applying attention weights to the non-masked query syntax elements adjacent to the masked query syntax elements, which influences the prediction process by applying a weight to every non-masked query syntax element. Additionally, the autocomplete model can analyze the query syntax elements in parallel, therefore allowing the autocomplete model the ability to predict one or more masked query syntax elements simultaneously. The autocomplete model may calculate loss of the autocomplete predictions. For example, the autocomplete model may evaluate how well it predicted the masked input. The autocomplete model may implement one or more loss functions in calculating the loss, such as, but not limited to, means squared error, likelihood loss, and log loss (cross entropy loss). The calculated loss may be fed into the autocomplete model to retrain the model.”)wherein, the third machine learning model with the help of the second machine learning model is trained to: analyze nearby words and context using position and sequence information of word vectors in word tokens between the checkbox options and the checkbox questions; (See SHARMA [0024] The document processing engine may asynchronously monitor, retrieve according to a schedule, and enable the submission of documents (e.g., clinical documents) received by the user device(s) 102. The server system 104, in response to receiving the one or more documents, converts the document(s) to a computer interpretable format via one or more computer vision techniques and extracts text from the document(s). Server system 104 additionally redacts protected health information from the document(s). In one or more embodiments, during the redaction process, server system 104 removes predetermined objects such as the patient's name, address, social security number, images, unique identifying characteristics, and the like. Server system 104 may then add the redacted document to a training dataset that includes a corpus of other redacted document(s) (e.g., clinical documents). Server system 104 may leverage a pre-trained language model and a trainer on the corpus of clinical documents, wherein the result of the training is a refined new version of a language model. The language model, via server system 104, may then map the text (i.e., words on each document within the corpus) to vectors, in an embedding process. The embeddings are then fed as the input to each task-specific NLP model. For example, the language model may feed the embeddings as the input to one or more task-specific NLP models, including but not limited to a classification model, search and rank model, autocomplete model, and topic model. In addition, server system 104 may enable usage of the task-specific NLP models by clinicians operating user device(s) 102 for various clinical purposes. Server system 104 may receive feedback (e.g., input indicative of a confirmation or correction) from clinicians operating the user device(s) 102 and leverage that feedback to fine-tune the task-specific NLP model(s) that were used by the clinician. The server system 104 may further generate instructions for displaying documents or portions of documents stored in the training dataset and actions that can be taken with said document(s), via a GUI that operates on the user device(s) 102. The aforementioned techniques provide accurate and automated solutions that improve upon prior methods for analyzing clinical documents.”) predict at least a correct link between the checkbox question and the checkbox option; (See SHARMA [0036] “The classification model component 216 may, given a text and/or document, classify the text and/or document into specific categories and or labels via a classification model. In addition, or alternatively, classification model component 216 may predict a specific attribute pertaining to the received text. In one embodiment, classification model component 216 may predict one or more clinical labels. In another embodiment, classification model component 216 may predict an ICD code for a patient diagnosis. In one example in order to classify or predict an attribute regarding the text, the classification model component 216 may have received a clinical document (e.g., a patient record) wherein the words within the document are split into vectors and the classification model component 216 attempts to predict the ICD code that was included on the received text. The classification model implemented by the classification model component 216 may be a supervised NLP task and further include its own neural network separate from language model component 214 and the other task-specific NLP models. There may be a predefined number of labels or categories by which the classification model may assign and label a text or document. Notably, the classification model may be evaluated based on how accurate its predictions are. Accordingly, the classification model component may leverage one or more loss functions to measure how far an estimated value is from its true value. Classification model component 216 may implement one or more loss functions including but not limited to binary cross entropy loss, categorical cross entropy loss, hinge loss, and/or Kullback Leibler Divergence Loss.”) (See SHARMA [0029] “Referring to FIG. 2, a composite clinical language modeling framework 200 is depicted, according to various embodiments of the present disclosure. Framework 200 provides components and processes for evaluating a document (e.g., clinical documents) using NLP and task-specific NLP models. These features provide an improvement of the prior art, which typically provided only basic electronic document retrieval. As shown, framework 200 includes a task scheduler component 204 (e.g., cronjob), which implements a task scheduler configured to schedule tasks to run periodically, at preset times, dates, and/or intervals. Framework 200 additionally includes a computer vision component 206 configured and capable of receiving a document 202 (e.g., a clinical document that may include protected health information) and redacting protected health information from the document. In one embodiment, the computer vision component 206 converts the document from a first format (e.g., PDF format) to a second format (e.g., JPEG format) readable by optical character recognition (OCR). Computer vision component 206 may be further configured to save the text read from the OCR engine as a single text file. In some instances, computer vision component 206 may be triggered in response to framework 200 receiving a document that is in a non-textual format (e.g., an image).”)
Iannitti in view of WILLIAMSON and further in view of Khandekar and SHARMA are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti WILLIAMSON Khandekar to incorporate the third machine learning model is trained by: masking checkbox questions and checkbox options randomly in a training corpus using a masking module; generating a plurality of links in the training corpus, wherein the plurality of links between the checkbox options and the checkbox questions are randomly masked; enabling the third machine learning model to predict a masked information in the training corpus, wherein the masked information pertains to a linking of the checkbox question to the checkbox option, wherein, the third machine learning model with the help of the second machine learning model is trained to: analyze nearby words and context using position and sequence information of word vectors in word tokens between the checkbox options and the checkbox questions; predict at least a correct link between the checkbox question and the checkbox option of SHARMA. This allows for ability to separate the labels/categories by capturing the count of positive predictions which are correct against the count of positive predictions that are incorrect at different thresholds as recognized by SHARMA [0046].
Claims 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Lannitti (US Patent Number US 20230100396 A1), in view of Prebble (U.S. Patent Number US 11721119 B2),
As to Claim 13, Iannitti teaches 13. (New) The system according to claim 1,
Furthermore lannitti teaches constructing a complete checkbox symbol by synthesizing the combined fragmented partial contours into a closed contour defining a single checkbox symbol; (See lannitti [0037] “In various embodiments, the system may use an object detection algorithm to search for object fields in order to process the page. The object detection algorithm may search for any object such as, for example, a geometric shape, line, field, parenthesis, colon or any other object (e.g., an object that does not include a string of letters). The object detection algorithm identifies any object in a document that may provide an option for participant interaction. The object detection algorithm may use edge detection to detect lines, circles, squares, blank (white) space, etc. The edge detection may find two lines for each actual line on the page (e.g., top of the line and bottom of the line) and may find lines that are part of letters (e.g., the letter “L” includes a vertical and horizontal line). As such, the algorithm may include a cleaning phase, a selection of sizes on the end of certain lines and/or removal of duplicate lines (e.g., a first line immediately above a second similar line). The system determines how the detected line is related to the rest of the shape or other artifact on the page. The systems may analyze the coordinates of the endpoints of the line and may compare the endpoints to the endpoints of other detected lines to determine if overlap of the endpoints (or lines) exists or the endpoints (or lines) are in a similar vicinity.”)
Iannitti does not specifically teach wherein the one or more processors are configured to perform the filtering by: processing a digital image using the computer vision model to detect pixel groupings corresponding to broken strokes and fragmented partial contours of checkbox symbols; (see Prebble (10:55-11:11) (53) In Step 213, a collection of selected connected components is generated, based on a predetermined criterion, from the combined connected components. Initially, the document image is analyzed to identify common connected components, which are connected components relating to predetermined types of foreground objects. In particular, the common connected components correspond to common protection areas of the document image. For example, the common connected components may include text connected components and natural image connected components. For example, any connected component that overlaps a text bounding box by more than a predetermined percentage (e.g., a predetermined percentage of 80%) of all pixels in the connected component is identified as a text connected component. Similarly, any connected component that overlaps a natural image bounding box by more than a predetermined percentage (e.g., a predetermined percentage of 80%) of all pixels in the connected component is identified as a natural image connected component. In one or more embodiments, the natural image bounding boxes are generated using the method described in reference to FIG. 2A above.”) employing multiple layers of the first machine learning model to the detected broken strokes and fragmented partial contours, wherein one or more higher layers of the first machine learning model combine spatially related broken strokes and fragmented partial contours that correspond to a same checkbox symbol; (see Prebble (10:8-24) “(47) In one or more embodiments, the document image is modified based on the final natural image bounding boxes for presenting to a user. For example, the document image may be modified for noise reduction where natural image areas are protected from being degraded by the noise reduction algorithm. An example of this application is described in reference to FIGS. 3A-3R above. In another example, the document image may be modified by applying image-specific processing (e.g., facial recognition) to natural image areas, such as generating descriptions of the natural images using machine learning or other techniques, extracting text from the natural images, and searching the natural images for specific types of content. In yet another example, the document image may be modified by applying document semantic analysis techniques to categorize a type of the document page, identify document topics within the document page, etc.”) and filtering incorrect checkbox detections by comparing parts of the checkbox symbol defined by the closed contour, including horizontal lines and vertical lines, against the at least one threshold value, (see Prebble (9:40-65) “(44) In Step 207, at least one final natural image bounding box is generated by expanding at least one candidate natural image bounding box. The expanded candidate natural image bounding box includes at least one combined CC that intersects the expanded candidate natural image bounding box. As noted above, any candidate background CCs are excluded from the collection of combined CCs where the intersecting combined CC is detected. In one or more embodiments, each final natural image bounding box is generated using an iteration cycle of iteratively expanding a candidate natural image bounding box of a new CC. Different final natural image bounding boxes are generated using separate iteration cycles. In each iteration, the candidate natural image bounding box of the new CC is expanded to include any intersecting combined CC that does not exceeds an enlarged boundary of the candidate natural image bounding box. The enlarged boundary of the candidate natural image bounding box is defined once before the iteration cycles. Any combined CC within the enlarged boundary and found to intersect with the expanded candidate natural image bounding box is merged into the expanded candidate natural image bounding box. Specifically, the expanded candidate natural image bounding box is further expanded to encompass the intersecting combined CC. In addition, the intersecting combined CC is removed from the collection of combined CCs at the end of each iteration.”) wherein the at least one threshold value is configured to distinguish true checkbox symbols from characters in the digital document that resemble portions of the checkbox symbol. (see Prebble (9:1-45) “(42) In Step 205, a number of candidate natural image CCs are generated using the collection of combined CCs. In one or more embodiments, the candidate natural image CCs are generated using a third predetermined criterion where the candidate background CCs are excluded from the combined CCs to generate the candidate natural image CCs. For example, the third predetermined criterion may include determining a combined CC as a candidate natural image CC when a size metric and a fill density metric of the combined CC (excluding any candidate background CC) exceeds a threshold. In one or more embodiments, the size metric includes one or more of a width, a height, and a size gap between an upper portion and a lower portion of the collection of combined CCs. In one or more embodiments, the threshold for comparing the fill density metric of the combined CC is adjusted based on the size metric of the combined CC and a percentage of background pixels in the combined CC. In one or more embodiments, the threshold for comparing the size metric of the combined CC is adjusted based on whether or not the combined CC is a candidate text CC. An example of generating candidate natural image CCs is illustrated in Block 6 of the example method described in reference to FIGS. 4A-4G below. (43) In Step 206, a number of candidate natural image bounding boxes of the candidate natural image CCs are generated. In one or more embodiments, a candidate natural image bounding box is generated for each candidate natural image CC using a maximum and minimum X-coordinate and Y-coordinate of the candidate natural image CC. In one or more embodiments, connected components of the candidate natural image bounding boxes are identified as new CCs. For any new CCs formed from multiple candidate natural image bounding boxes, these multiple candidate natural image bounding boxes are replaced with a single candidate natural image bounding box of the new CC.”)
Iannitti and Prebble are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti to incorporate the one or more processors are configured to perform the filtering by: processing a digital image using the computer vision model to detect pixel groupings corresponding to broken strokes and fragmented partial contours of checkbox symbols; employing multiple layers of the first machine learning model to the detected broken strokes and fragmented partial contours, wherein one or more higher layers of the first machine learning model combine spatially related broken strokes and fragmented partial contours that correspond to a same checkbox symbol; constructing a complete checkbox symbol by synthesizing the combined fragmented partial contours into a closed contour defining a single checkbox symbol; and filtering incorrect checkbox detections by comparing parts of the checkbox symbol defined by the closed contour, including horizontal lines and vertical lines, against the at least one threshold value, wherein the at least one threshold value is configured to distinguish true checkbox symbols from characters in the digital document that resemble portions of the checkbox symbol of Prebble. This allows distinguishing natural image areas having irregular pixel patterns and color variations in a complex document as recognized by Prebble (10:25-30).
As to Claim 14, Iannitti teaches 14. (New) The system according to claim 2,
Iannitti does not specifically teach wherein the one or more processors are configured to: determine the status of the checkbox symbol by calculating a pixel density within the closed contour of the detected checkbox symbol, (see Prebble (8:6-16) “ (37) In Step 201, one or more original connected components (CCs) are extracted from the original mask. In the original mask, adjacent marked pixels are collected into a cluster of marked pixels. Each cluster of marked pixels is a single original CC. The collection process iterates until each pixel in the original mask either belongs to a specific original CC or is an isolated pixel without any adjacent pixels. Each isolated pixel is designated as a single pixel original CC. An example of extracting the original CCs is described in reference to FIG. 3D below.”) wherein the one or more processors are configured to: assign a first status representing a selected state if the pixel density exceeds the threshold value indicating the presence of the mark; and (see Prebble (8:52-67) “(41) In Step 204, a number of candidate background CCs are generated from the collection of combined CCs. In one or more embodiments, the candidate background CCs are generated using a second predetermined criterion. For example, the second predetermined criterion may include determining a combined CC as a candidate background CC when a percentage of background pixels in the combined CC exceeds a threshold. The percentage of background pixels may be determined within the combined CC or within a convex hull of the combined CC. In one or more embodiments, the threshold for comparing the percentage of background pixels is adjusted based on a size metric of the combines CC. An example of generating candidate background CCs is illustrated in Block 6 of the example method described in reference to FIGS. 4A-4G below.”) assign a second status representing an unselected state if the pixel density is below the threshold value, indicating the absence of the mark, (see Prebble (9:23-39) “(43) In Step 206, a number of candidate natural image bounding boxes of the candidate natural image CCs are generated. In one or more embodiments, a candidate natural image bounding box is generated for each candidate natural image CC using a maximum and minimum X-coordinate and Y-coordinate of the candidate natural image CC. In one or more embodiments, connected components of the candidate natural image bounding boxes are identified as new CCs. For any new CCs formed from multiple candidate natural image bounding boxes, these multiple candidate natural image bounding boxes are replaced with a single candidate natural image bounding box of the new CC. Examples of new CCs are shown in FIG. 4D below. An example of a single candidate natural image bounding box replacing multiple candidate natural image bounding boxes of a new CC is shown in FIG. 4E below.”) wherein the assigned status is stored in the non-transitory memory as a data attribute employed by the linking module to extract the checkbox option. (see Prebble (1:58-2:19) “ (5) In general, in one aspect, the invention relates to a system for processing an image to identify a natural image in a document image. The system includes: a memory and a computer processor connected to the memory. The computer processor: generates, from the document image, a plurality of combined connected components (CCs); generates, from the plurality of combined CCs and based on a first predetermined criterion, a plurality of candidate text CCs; generates, from the plurality of combined CCs and based on a second predetermined criterion, a plurality of candidate background CCs; generates, from the plurality of combined CCs and based on a third predetermined criterion, a plurality of candidate natural image CCs where the plurality of candidate background CCs are excluded from the plurality of combined CCs to generate the plurality of candidate natural image CCs and where the third predetermined criterion is dependent on the plurality of candidate text CCs; generates a plurality of candidate natural image bounding boxes of the plurality of candidate natural image CCs; generates at least one final natural image bounding box by expanding at least one candidate natural image bounding box and including in the expanded at least one candidate natural image bounding box at least one combined CC of the plurality of combined CCs that intersects the expanded at least one candidate natural image bounding box; and modifies, based on the at least one final natural image bounding box, the document image and displays the modified document image to a user.”)
Iannitti and Prebble are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Iannitti to incorporate the one or more processors are configured to: determine the status of the checkbox symbol by calculating a pixel density within the closed contour of the detected checkbox symbol, wherein the one or more processors are configured to: assign a first status representing a selected state if the pixel density exceeds the threshold value indicating the presence of the mark; and assign a second status representing an unselected state if the pixel density is below the threshold value, indicating the absence of the mark, wherein the assigned status is stored in the non-transitory memory as a data attribute employed by the linking module to extract the checkbox option of Prebble. This allows distinguishing natural image areas having irregular pixel patterns and color variations in a complex document as recognized by Prebble (10:25-30).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KRISTEN MICHELLE MASTERS whose telephone number is (703)756-1274. The examiner can normally be reached M-F 8:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Louis Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KRISTEN MICHELLE MASTERS/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
Prosecution Insights