Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
In response to the amendment of claims 8 and 18, the rejection under 35 U.S.C. § 112(b) is withdrawn.
In response to the amendment of claims 1, 6, 8, 11, and 18, the rejection of claims 1-20 under 35 U.S.C. §103 in view of Farivar as modified by Yamaguchi is withdrawn. However, upon further consideration in view of the amendment, a new ground of rejection is made of claims 1, 6, and 11 under 35 U.S.C. § 103 over Farivar in view of Yamaguchi and in further view of Zhu et al. (full citations in PTO-892 form).
Response to Arguments
The amended claims, taken in conjunction with the arguments directing Examiner’s attention to the specified subject matter, are persuasive as mentioned above. However, upon further consideration, a new ground of rejection is made of claims 1, 6, and 11 under 35 U.S.C. § 103 over Farivar in view of Yamaguchi and in further view of Zhu et al. (full citations in PTO-892 form).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6, 11-12, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Farivar et al. (US Pat. No. 10,599,952, hereinafter “Farivar”) in view of Yamaguchi et al. (US Pat. No. 11,151,413, hereinafter “Yamaguchi”) and in further view of Zhu et al. (Chinese PG Pub 112,287,653, hereinafter “Zhu”).
Regarding claim 1, Farivar discloses, in an optical/image character recognition system comprising an OICR engine and a machine learning system, a method of training the machine learning system, the method comprising receiving input text and/or image data (Col. 4 lines 53-57); altering the input text and/or image data to produce degraded data (Col. 6, lines 15-51, wherein the degraded data may be achieved using any or all of the distortion filter generators disclosed); training the machine learning system using the degraded data (Col. 7, lines 26-40 and col. 8 lines 47-55, disclosing the generation of a training set using the distortion filter generators and then the training of a machine learning model for removing textual distortions prior to sending the document image to the OCR engine); receiving the degraded data into the machine learning system (Col. 10, lines 1-17, wherein the distortion data is input to the neural network); correcting the degraded data with the machine learning system to produce corrected data (Col. 10, lines 1-17, wherein the distorted image data is input to the neural network which corrects the distortions, and wherein the corrected/distortion-less image data is input into an OCR engine to extract text); and adjusting one or more weights of nodes in the machine learning system (Col. 7 lines 26-40 for training a neural network model and Col. 18, line 50 – col. 19 line 22 for the description of the neural network, and wherein one having ordinary skill in the art would know that training a neural network would necessarily require adjustment of the weights and biases of at least one of the nodes within the system).
Specifically, Farivar discloses a method and system of training a machine learning network to remove distortions in document images pre-OCR by generating a degraded dataset and training a machine learning network to remove the distortions given the distorted data.
Although Farivar discloses reading the corrected data, an end to the training, and the machine learning architecture being separate from the OICR engine, Farivar does not explicitly disclose wherein the adjustment of the one or more weights of nodes in the machine learning system is in direct response to detecting that the adjustment of the machine learning system is required after reading the corrected data; repeating the correcting and adjusting until it is determined that adjustment no longer is required; or wherein the adjusting is carried out without requiring refinement or other alteration to the OICR engine.
However, Yamaguchi explicitly discloses wherein the adjustment of the one or more weights of nodes in the machine learning system is in direct response to detecting that the adjustment of the machine learning system is required after reading the corrected data (Col. 6 line 29 – col. 7 line 16, wherein the GAN components (discriminator and generator) are trained on both the loss signal between the corrected/generated image and the ground truth image as well as the certainty score collected from the corrected and read data); repeating the correcting and adjusting until it is determined that adjustment no longer is required (Col. 14, lines 17-28, wherein the calculated end condition is one or more of convergence, training completion of a subset of all data within the dataset, or training completion of the entirety of the training dataset); and wherein the adjusting is carried out without requiring refinement or other alteration to the OICR engine (Fig. 1 and col. 11, lines 30-60, wherein the OCR engine can be clearly seen as downstream of the trained machine learning model, and is in no way affected by the GAN components or training happening upstream).
Specifically, Yamaguchi discloses a GAN-based pre-OCR document filtering apparatus meant to remove noise and confounding features from document images before the images are processed downstream by an OCR engine. Therefore, both Farivar and Yamaguchi disclose methods and systems of pre-OCR document denoising using artificially-noised data and trained machine learning methods of denoising. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the disclosure of Yamaguchi (using the read corrected data to trigger node weight adjustment of the machine learning system, a specific adjustment end condition, and the explicit lack of modification to the OCR system) within the system of Farivar as the application of known techniques to a known device ready for improvement to yield the predictable improvements of a more effective training metric (using the read corrected data and certainty factor of Yamaguchi), a more effective indicator of a well-fit model (convergence or completion of execution on training pairs), and less necessary modification overall.
Although both Farivar and Yamaguchi disclose OCR pipelines for data generation and machine learning model training, the combination of Farivar and Yamaguchi nevertheless fails to explicitly disclose wherein the OICR system comprises a deep learning system.
However, Zhu discloses an OICR system comprising a deep learning system (paras. 0046-0047, “Compared with other OCR methods, end-to-end OCR based on deep learning is more suitable for the recognition of text with fixed format…FIG. 7 shows a schematic structural diagram of an end-to-end OCR model 700 based on a deep learning algorithm according to an embodiment of the present invention. As shown in FIG. 7 , the model 700 includes a shared convolutional layer 710 , a text detection layer 720 , a ROI (region of interest) rotation layer 730 , a text recognition layer 740 and a detection result layer 750”). Specifically, Zhu discloses a deep learning OCR-based template matching scheme, using a trained deep learning framework to identify and recognize text to compare recognized text to template text from a plurality of contract template objects.
Examiner notes that this combination keeps the upstream nature of the disclosure of Farivar in view of Yamaguchi, which itself addresses the claim limitation of the instant application, wherein the adjusting is carried out without requiring refinement or other alteration to the deep learning system in the OICR engine.
Therefore, Zhu discloses an applied OICR system utilizing deep learning for text identification and recognition, which is not only within the same field of endeavor as the combined disclosure of Farivar in view of Yamaguchi, but also provides a deep learning foundation for the downstream OICR engine of Farivar in view of Yamaguchi. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the deep learning-based OICR system of Zhu within the method and system of Farivar in view of Yamaguchi as the application of a known technique to a known method to yield the predictable result of a robust training data generation and training methodology using the disclosure of Farivar in view of Yamaguchi for the deep learning system of Zhu requiring no direct adjustment to the deep learning-based OCR framework of Zhu, increasing overall computational efficiency and accuracy on fixed-format based OCR tasks.
Claim 11 is rejected, mutatis mutandis, for reasons similar to claim 1.
Regarding claim 6, Farivar discloses, in an optical/image character recognition (OICR) system comprising an OICR engine and a machine learning system a method of restoring degraded data, the method comprising, in the machine learning system: receiving the degraded data (Col. 10, lines 1-17); and correcting the degraded data with the machine learning system to produce corrected data (Col. 10, lines 1-17, wherein the distorted image data is input to the neural network which corrects the distortions, and wherein the corrected/distortion-less image data is input into an OCR engine to extract text).
Farivar does not disclose, in response to detecting that adjustment of the machine learning system is required after reading the corrected data, adjusting one or more weights of nodes in the machine learning system; and repeating the correcting and adjusting until it is determined that adjustment no longer is required.
However, Yamaguchi discloses, in response to detecting that adjustment of the machine learning system is required after reading the corrected data, adjusting one or more weights of nodes in the machine learning system (Col. 6 line 29 – col. 7 line 16, wherein the GAN components (discriminator and generator) are trained on both the loss signal between the corrected/generated image and the ground truth image as well as the certainty score collected from the corrected and read data); and repeating the correcting and adjusting until it is determined that adjustment no longer is required (Col. 14, lines 17-28, wherein the calculated end condition is one or more of convergence, training completion of a subset of all data within the dataset, or training completion of the entirety of the training dataset).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosures of Farivar and Yamaguchi according to the rationale of claim 1.
Although both Farivar and Yamaguchi disclose OCR pipelines for data generation and machine learning model training, the combination of Farivar and Yamaguchi nevertheless fails to explicitly disclose wherein the OICR system comprises a deep learning system.
However, Zhu discloses an OICR system comprising a deep learning system (paras. 0046-0047, “Compared with other OCR methods, end-to-end OCR based on deep learning is more suitable for the recognition of text with fixed format…FIG. 7 shows a schematic structural diagram of an end-to-end OCR model 700 based on a deep learning algorithm according to an embodiment of the present invention. As shown in FIG. 7 , the model 700 includes a shared convolutional layer 710 , a text detection layer 720 , a ROI (region of interest) rotation layer 730 , a text recognition layer 740 and a detection result layer 750”).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosures of Farivar in view of Yamaguchi and Zhu according to the rationale of claim 1.
Regarding claim 16, Farivar in view of Yamaguchi and in further view of Zhu discloses all limitations of claim 11. The remaining limitations of claim 16 are rejected, mutatis mutandis, for reasons similar to claim 6. Farivar further discloses outputting the corrected end user data (Col. 10 lines 15-20, outputting machine-encoded text which was corrected for the distortion and storing it).
Regarding claims 2 and 12, Farivar and Yamaguchi and Zhu disclose all limitations of claims 1 and 11, respectively. Yamaguchi further discloses wherein the machine learning system uses additional data besides the degraded data for training (Col. 5 line 63 – col. 6 line 13, wherein the training data for the GAN includes both degraded training images and corrected ground truth images to be discriminated). Thus, it would have been obvious to combine the additional training data of Yamaguchi within the method of Farivar as modified by Yamaguchi and Zhu according to the rationale of claim 1.
Claims 3-5, 7, 9-10, 13-15, 17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Farivar in view of Yamaguchi and Zhu and in further view of Alonso et al. (“Adversarial Generation of Handwritten Text Images Conditioned on Sequences”, hereafter referred to as Alonso).
Regarding claims 3, 9, 13, and 19, Farivar and Yamaguchi and Zhu disclose all limitations of claims 1, 6, 11, and 16, respectively. The combination of Farivar, Yamaguchi, and Zhu does not disclose wherein the machine learning system is a convolutional recurrent neural network (CRNN), the CRNN comprising a convolutional neural network (CNN) and a recurrent neural network (RNN).
However, Alonso discloses wherein the machine learning system is a convolutional recurrent neural network (CRNN), the CRNN comprising a convolutional neural network (CNN) and a recurrent neural network (RNN) (Figure 2, specifically wherein the generator neural network is a CNN and the recognizer network is a recurrent network, working in series).
Specifically, Alonso discloses a method of handwritten text recognition within the OCR-sphere using an adversarial architecture employing generation of artificial and noisy handwriting samples to be decoded by a recognition network and classified as real or fake by a discrimination network. Thus, Alonso discloses an OCR-enabled method of generating noised image data to be cleaned, recognized, and classified as artificial or real, in the same field of endeavor and operation as both Farivar and Yamaguchi and Zhu. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the CRNN architecture of Alonso within the method and system of Farivar as modified by Yamaguchi and Zhu as the application of a known technique to a known method ready for improvement to yield the predictable result of a more robust data generation and discrimination mechanism as the CRNN architecture allows for efficient recognition of image-based sequences.
Regarding claims 4, 10, 14, and 20, Farivar and Yamaguchi and Zhu and Alonso disclose all limitations of claims 3, 9, 13, and 19, respectively. Alonso further discloses wherein the CNN produces the degraded data (Figures 2 and 3, specifically the generator block of figure 2, wherein the desired data is encoded and inserted into the ResBlock Up blocks and the normalization and convolution blocks; and the ResBlock architecture of figure 3, wherein multiple steps convolving text with noise for degradation occur), and the RNN produces the corrected data (Figure 2, specifically the recognizer element, where the corrected data is the output of the LSTM decoder, returning the decoded, noiseless text). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the specific roles of the CNN and RNN within the CRNN as disclosed by Alonso within the method of Farivar in view of Yamaguchi and Zhu and in further view of Alonso according to the rationale of claim 3.
Regarding claims 5 and 15, Farivar and Yamaguchi and Zhu and Alonso disclose all limitations of claims 3 and 13, respectively. Alonso further discloses wherein the CNN is trained with generative adversarial network (GAN) loss, and the RNN is trained with connectionist temporal categorical (CNN) loss (Figure 2, wherein the CNN generator network is trained on adversarial loss from the discriminator network, and the RNN recognizer network is trained with the CTC loss from the real data). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the disclosure of Alonso regarding the types of loss used to train the CRNN’s sub-networks within the method of Farivar in view of Yamaguchi and Zhu and in further view of Alonso according to the rationale of claim 3.
Regarding claims 7 and 17, Farivar and Yamaguchi and Zhu disclose all limitations of claims 6 and 16, respectively. Farivar and Yamaguchi and Zhu do not disclose wherein the degraded data specifically comprises characters with one or more of merged or missing strokes and background noise. However, Alonso discloses a handwritten text generation/recognition method and system. Specifically, real handwriting samples likely include at least noise and often include both noise and stray marks, as can be seen in Alonso Fig. 1 and Fig. 5 (the Dimanche) noise and/or stray marks (Dimanche in Fig. 1; Fig. 5). It would have been obvious to one of ordinary skill in the art to use imperfect samples of handwritten text (which would include noise and likely missing/extra markings) in training the modified system of Alonso in order to replicate real-world conditions necessary to training OICR systems. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the disclosure of Alonso regarding the inclusion of degraded characters with background noise and one or more of merged or missing strokes within the method of Farivar in view of Yamaguchi and Zhu and in further view of Alonso according to the rationale of claim 3.
Allowable Subject Matter
Claims 8 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROHAN TEJAS MUKUNDHAN whose telephone number is (571)272-2368. The examiner can normally be reached Monday - Friday 9AM - 6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 5712723838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROHAN TEJAS MUKUNDHAN/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights