DETAILED ACTION
This action is in response to the application filed on April 29th, 2024. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a mental process without significantly more.
Independent claims 1, 14, and 20 recite:
“determining an order based on the image of the first page and one or more templates associated with the one or more pages of the ordering form, where in the order comprises one or more items selected on the first page of the ordering form, and each template comprises items and locations of indicator fields associated with a corresponding page of the ordering form”, which can be reasonably interpreted as a human observer viewing a filled in ordering form, and comparing this ordering form (mentally or physically) to a blank version of the ordering form, and determining which items are selected and/or written in.
This judicial exception is not integrated into a practical application because additional elements of:
“one or more computers” and “a non-transitory computer-readable storage medium storing instructions that when executed by one or more computer cause the one or more computer to performed operations” are generically recited computer elements that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea on a computer;
“obtaining, using an image sensor, an image of a fist page of an ordering form, wherein the ordering form comprises one or more pages, each page comprises names of items and indicator fields, and at least one indicator field of the first page is marked to indicate a corresponding item being ordered” is generically recited insignificant extra-solution activity of data gathering/inputting;
“transmitting the determine order to an order processing system” is generically recited insignificant extra-solution activity of data outputting.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because additional elements of:
“one or more computers” and “a non-transitory computer-readable storage medium storing instructions that when executed by one or more computer cause the one or more computer to performed operations” are mere instruction to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.06(f);
“obtaining, using an image sensor, an image of a fist page of an ordering form, wherein the ordering form comprises one or more pages, each page comprises names of items and indicator fields, and at least one indicator field of the first page is marked to indicate a corresponding item being ordered” is generically recited insignificant extra-solution activity of data gathering/inputting;
“transmitting the determine order to an order processing system” is generically recited insignificant extra-solution activity of data outputting.
Listed dependent claims do not remedy these deficiencies:
Claims 2, 11, and 15 recite mental process of determining which ordering menu is being used, which could be done by a human observer by visually comparing the filled in ordering form to one or more blank ordering forms.
Claims 3-4 and 16-17 recite mental processes involving visual perception of indicator fields.
Claims 5-6, 12, and 18-19 further link the abstract idea to the field of use of deep learning and optical character recognition.
Claims 7 and 13 recite mental processes of a user updating the order and correcting any mistakes.
Claims 8-9 further recite insignificant extra-solution activity of data gathering/input.
Claim 10 further links the abstract idea to the field of use of restaurant ordering menus.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 6-7, 14-15, and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by “Intelligent Document Processing” (herein after referred to by its primary author, Xu).
In regards to claim 1, Xu teaches a method performed by one or more computers, comprising: obtaining, using an image sensor, an image of a first page of an ordering form (Xu Figure 1 “Scanned form documents”), wherein the ordering form comprises one or more pages, each page comprises names of items and indicator fields, and at least one indicator field of the first page is marked to indicate a corresponding item being ordered (Xu Figures 3(a) and 3(b)); determining an order based on the image of the first page and one or more templates associated with the one or more pages of the ordering form (Xu Figure 1 “Text recognition” and “Form layout analysis”; Section 3.2.4 “Layout analysis is the last step of form information extraction. It tries to reconstruct the logical relationship among entities of input forms by utilizing results from foreground extraction and text recognition, as well as information from templates.”), wherein the order comprises one or more items selected on the first page of the ordering form, and each template comprises items and locations of indicator fields associated with a corresponding page of the ordering form (Xu Figure 2(b); Section 3.1 “Form registration is the process of creating a template which defines a new form type to which any forms belong can be extracted by the system. A template specifies region types and their spatial arrangement, including 1) key texts and corresponding content fields, and table headers and entries; 2) regions of plain texts; 3) regions of selection elements such as checkboxes; and 4) regions excluded from extraction.” Examiner note: This figure shows a template, with green boxes surrounding form fields.); and transmitting the determined order to an order processing system (Xu Figure 1 “Validation and export”; Figures 3(a) and 3(b) Examiner note: This figure shows another example form which this process could be applied to. This form is an invoice, which is analogous to an order as they both represent items which are purchased).
In regards to claim 2, Xu teaches the method of claim 1, wherein determining the order comprises: automatically and without user input, identifying, using a statistic comparison algorithm executed by the one or more computers, a template from the one or more templates that is corresponding to the image of the first page (Xu Section 3.2.1 “Form type identification is to match an input form to one of the templates registered in the form type library. During this step, the input form image is analyzed and features on the form are determined. The form type library is searched for the closest match based on the input feature set. A match is found if the discrepancy is within a certain threshold.”); determining items in the image of the first page based on items in the identified template; determining indicator fields in the image of the first page based on locations of indicator fields in the identified template (Xu Figure 1 “Text recognition” and “Form layout analysis”; Section 3.2.4 “Layout analysis is the last step of form information extraction. It tries to reconstruct the logical relationship among entities of input forms by utilizing results from foreground extraction and text recognition, as well as information from templates.” Examiner note: In this reference, text is identified, then for layout analysis is performed, which uses information from the template to determine the logical relationship between the form field and the identified text.); and for each indicator field in the image of the first page, determining an order quantity of an item indicated by the indicator field (Xu Figures 2(b) and 3(b) Examiner note: Figure 2(b) shows an example form which could be identified. This form includes fields such as DOB (date of birth) or postal code. These form fields are analagous to quantities, as they are both represented by numbers and regardless of the type of data extracted (quantity vs postal code), the process of extracting the data (represented by a number) is the same. Furthermore, figure 3(b) shows an invoice, which contains a “shipped” data field representing the quantity of an ordered item).
In regards to claim 6, Xu teaches the method of claim 2, wherein determining the order quantity of the item indicated by the indicator field comprises: recognizing a handwritten number filled in the indicator field using optical character recognition (OCR); and determining that the order quantity of the item is the recognized handwritten number (Xu Figure 1 “Text recognition”; Section 3.2.3 “Text recognition also employs Deep Learning to convert the field images to text strings, field by field. A field may be extracted by multiple images if its texts span multiple lines or are widely separated. Our latest AI-based OCR technology enables the system to achieve high recognition accuracy for both printed and handwritten texts in English and Japanese.”).
In regards to claim 7, Xu teaches the method of claim 2, wherein determining the order further comprises: displaying the order to a user; and updating the order in response to receiving an input from the user to modify the order (Xu Figure 1 “Validation and export”; Section 3.3 “aiDocuDroid’s validation tool provides users with a side-by-side view to verify the extraction results. The original form image is shown on the left and an output panel is shown on the right. Users can hover the cursor over any field text in the output panel to see the corresponding field image.”).
In regards to claim 14, Xu anticipates the claim language as in the consideration of claim 1.
In regards to claim 15, Xu anticipates the claim language as in the consideration of claim 2.
In regards to claim 19, Xu anticipates the claim language as in the consideration of claim 6.
In regards to claim 20, Xu anticipates the claim language as in the consideration of claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3-4 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of “PokeItUp Ordering Form”.
In regards to claim 3, Xu teaches the method of claim 2, wherein determining the order quantity of the item indicated by the indicator field comprises: determining whether the indicator field is selected (Xu Section 3.1 “A template specifies region types and their spatial arrangement, including… 3) regions of selection elements such as checkboxes;”); and determining the order quantity of the item in response to determining that the indicator field is selected (Xu Figure 2(b) Examiner note: This figure shows an example form which could be identified. This form includes a selection element (checkbox)).
Xu fails to teach determining that the order quantity of the item is one in response to determining that the indicator field is selected.
However, PokeItUp Ordering Form teaches determining that the order quantity of the item is one in response to determining that the indicator field is selected (PokeItUp Order Form Examiner note: This form shows that it is standard to check a box to indicate that one would like to order a portion of the indicated item. For example, if crab salad and spicy tuna are selected in step 2, it is expected that 1 scoop of crab salad and 1 scoop of spicy tuna are desired. See figure below).
PNG
media_image1.png
173
318
media_image1.png
Greyscale
PokeItUp Ordering Form is considered to be analogous to the claimed invention because they are both in the same field of food ordering menus. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Xu to include the teachings of PokeItUp Ordering Form, to provide the advantage of a clearer indication of a single order and easier recognition by the computer. Since a checkbox is used to indicate a single order, no OCR is needed, which saves time and processing power as the checkbox only needs to be recognized as checked, and no text inside need to be recognized.
In regards to claim 4, Xu in view of PokeItUp Ordering Form teaches the method of claim 3, wherein determining whether the indicator field is selected comprises: determining whether the indicator field is selected using an algorithm to detect a change in an image of the indicator field caused by a selection of the indicator field (Xu Section 3.2.2 “Foreground extraction is to separate form foreground pixels such as text, lines, table structure, logo, checkboxes, etc. from the background pixels. This step utilizes AI technologies to eliminate the color variation in backgrounds and text, and different types of image artifacts introduced during scanning and image compression.” Examiner note: This section shows a step of extracting the users input to the form fields. The background pixels are removed and the remaining form fields are detected. This process is analogous to detecting a change in the indicator fields, as the form fields are only detected when there is text within the field).
In regards to claim 16, Xu in view of PokeItUp Ordering Form renders obvious the claim language as in the consideration of claim 3.
In regards to claim 17, Xu in view of PokeItUp Ordering Form renders obvious the claim language as in the consideration of claim 4.
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of “PokeItUp Ordering Form”, and further in view of “Checkbox Detection on Rwandan Perioperative Flowsheets using Convolutional Neural Network” (herein after referred to by its primary author, Murphy).
In regards to claim 5, Xu in view of PokeItUp Ordering Form teaches the method of claim 3, but fails to teach wherein determining whether the indicator field is selected comprises: determining whether the indicator field is selected using a deep learning recognition algorithm and training data, wherein the training data comprises at least one of: an image of an unselected indicator field; an image of a selected indicator field; and an image of an indicator field that is selected by a mistake and has a correction to fix the mistake.
However, Murphy teaches wherein determining whether the indicator field is selected comprises: determining whether the indicator field is selected using a deep learning recognition algorithm and training data (Murphy Abstract “A checkbox image is cropped based on its location with template matching and then processed through a trained convolutional neural network (CNN) to classify it as checked or unchecked.”), wherein the training data comprises at least one of: an image of an unselected indicator field; an image of a selected indicator field; and an image of an indicator field that is selected by a mistake and has a correction to fix the mistake (Murphy Table 1 Examiner note: This table shows that training data includes positive and negative examples. The positive examples are images of a selected indicator field, while negative examples are images of an unselected indicator field).
Murphy is considered to be analogous to the claimed invention because they are both in the same field of form indicator field detection. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Xu in view of PokeItUp Ordering Form to include the teachings of Murphy, to provide the advantage of a network which is trained on meaningful data and is not overfit (Murphy Section IV B “Of the 3,231 identified checkboxes, approximately 75% of the images were unchecked. To decrease the possibility of predicting too many images as unchecked, the two classes were balanced and not all the unchecked images were used to fit the classification model. The training, validation, and test sets included 1,092, 365, and 159 images, respectively. Through the template matching process, the images had a consistent size of 354 x 69 pixels across the checkboxes, so this was used as the image size to train the model… Using early stopping and the checkpointer ensured the best model was selected and reduced the risk of the model overfitting to the train data.”).
In regards to claim 18, Xu in view of PokeItUp Ordering Form and Murphy renders obvious the claim language as in the consideration of claim 5.
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of “A document recognition system and its applications” (herein after referred to by its primary author, Yamashita).
In regards to claim 8, Xu teaches the method of claim 1, wherein the ordering form comprises forms of different types (Xu Figures 1, 2(a), and 3(a) Examiner note: Xu does not limit the type of forms which can be recognized, this can be seen through the different form types shown in figure 2(a) and 3(a), and the form type library in figure 1).
Xu fails to teach each of the forms of different types comprises at least one page.
However, Yamashita teaches each of the forms of different types comprises at least one page (Yamashita Introduction “In the field of document recognition applications, it is not possible to develop a comprehensive entry system to meet all of the requirements of varied applications. Some applications deal with simple reports, while others involve documents that have complicated layouts. Some require every page to be converted, while others require only parts of documents to be converted.” Examiner note: This reference, when considered in combination with Xu, teaches that a form can have multiple pages, and some document recognition applications are designed to recognize them all).
Yamashita is considered to be analogous to the claimed invention because they are both in the same field of food ordering menus. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Xu to include the teachings of Yamashita, to provide the advantage of a document recognition system which is flexible and customizable (Yamashita Introduction “This variety of requirements means that customization is indispensable in developing any given entry system for practical applications.”)
In regards to claim 9, Xu teaches the method of claim 1, and Xu in view of Yamashita teaches obtaining an image of a second page (Yamashita Introduction “In the field of document recognition applications, it is not possible to develop a comprehensive entry system to meet all of the requirements of varied applications. Some applications deal with simple reports, while others involve documents that have complicated layouts. Some require every page to be converted, while others require only parts of documents to be converted.”) of the ordering form; and automatically and without user input, identifying, using a statistic comparison algorithm executed by the one or more computers, a second template from the one or more templates that is corresponding to the image of the second page (Xu Section 3.2.1 “Form type identification is to match an input form to one of the templates registered in the form type library. During this step, the input form image is analyzed and features on the form are determined. The form type library is searched for the closest match based on the input feature set. A match is found if the discrepancy is within a certain threshold.”), wherein the order is determined further based on the image of the second page and the second template, and the order further comprises at least another item selected on the second page (Xu Figure 1 “Form layout analysis”; Section 3.2.4 “Layout analysis is the last step of form information extraction. It tries to reconstruct the logical relationship among entities of input forms by utilizing results from foreground extraction and text recognition, as well as information from templates. Spatial information is deducted from the detected lines and table structures from the foreground extraction, while textual and semantic information is obtained from recognized field texts. Matched with the logical structure defined in the template, aiDocuDroid can accurately extract the required data with its original semantic relationships.” Examiner note: Yamashita suggests that intelligent document system can recognize text from multiple pages of a document. Xu suggests that any page can be recognized, and have its own template. When considered together, these teachings suggest that multiple pages of a single document can be analyzed, with their respective page templates stored in the form type library).
Claims 10-13 are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of “Android Based Application for Recognition of Indonesian Restaurant Menus Using Convolution Neural Network” (herein after referred to by its primary author, Swastika)
In regards to claim 10, Xu teaches the method of claim 1, wherein each of the items comprises at least one of a product or a service (Xu Figures 3(a) and 3(b)).
Xu fails to teach wherein the product comprises a food, and the ordering form comprises a restaurant menu.
However, Swastika teaches wherein the product comprises a food, and the ordering form comprises a restaurant menu (Swastika Figures 5/7).
Swastika is considered to be analogous to the claimed invention because they are both in the same field of food ordering menus. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Xu to include the teachings of Swastika, to provide the advantage of including detailed information about the scanned food items which the user may not recognize (Swastika Abstract “Hence, it is required to facilitate foreign travelers who want to find out information about Indonesian food based on restaurant menus.”)
In regards to claim 11, Xu teaches the method of claim 1, and Xu in view of Swastika teaches wherein each template is generated by: obtaining an image of a corresponding page of the ordering form (Xu Section 3.1 “Templates can be created from either filled or empty forms.”); recognizing item names in the image of the corresponding page using OCR (Xu Section 3.2.3 “Text recognition also employs Deep Learning to convert the field images to text strings, field by field. A field may be extracted by multiple images if its texts span multiple lines or are widely separated. Our latest AI-based OCR technology enables the system to achieve high recognition accuracy for both printed and handwritten texts in English and Japanese.”); determining items based on the recognized item names and an item database (Swastika Figure 4 Examiner note: This figure shows a system performing OCR on selected text, then querying a database for a matching name, and determining the item based on the matching name in the database.); and determining locations of indication fields in the image of the corresponding page (Xu Section 3.1 “Our form registration requires a minimal amount of user guidance and is robust enough to deduce the rest. The registration consists of two steps. The first step is identification of region types, drawn by the users (Fig. 2(a), Fig. 3(a)); the second step is automatic field label association and table structure detection from the previous user-drawn regions. Users can verify the generated templates and renaming detected key (rekeying) using the form registration user interface (Fig. 2(b), Fig. 3(b)).”).
In regards to claim 12, Xu in view of Swastika teaches the method of claim 11, wherein determining the locations of the indication fields in the image of the corresponding page comprises: automatically detecting the indication fields in the image of the corresponding page based on at least one of a geometric shape detection algorithm or a deep learning object recognition algorithm (Xu Figure 2(b); Section 3.1 “If the form is empty (auto registration in Fig. 1), users only need to draw regions of exclusion (Fig. 2(a)), with the rest such as field labels, table headers, as well as the associated content regions being detected automatically (Fig. 2(b)).” Examiner note: The user draws an exclusion zone (red box in figure 2a) and the remaining fields are automatically given bounding boxes (green boxes in figure 2b)).
In regards to claim 13, Xu in view of Swastika teaches the method of claim 11, wherein determining the locations of the indication fields in the image of the corresponding page comprises: displaying the image of the corresponding page to a user; receiving a user input representing at least one indication field in the image of the corresponding page that is identified by the user; and determining the locations of the indication fields in the image of the corresponding page based on the at least one indication field identified by the user (Xu Figures 2(a) and 2(b); Section 3.1 “If a filled form is used for template creation (semiauto registration in Fig. 1), users also need to draw the regions of field labels such as table headers and regions of plain texts in addition to regions of exclusion. A single bounding box can be drawn to enclose all aligned field labels, such as those in a table header even without column delimiting lines.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
“Automatic Recognition Method for Checkbox in Data Form Image” teaches a method of recognizing checkboxes.
“Intelligent Forms Processing System” teaches a method of processing forms, including creating templates and using the template to create a difference image which is used to find the filled in form fields.
“Recurrent Neural Network Approach for Table Field Extraction in Business Documents” teaches a system which uses a neural network to process forms and extract relevant information.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677