DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1-17 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 6, 10, and 15 recite “associated ones of unlabeled portions.” As written in the claims it is unclear what “ones” refers to, therefore the stated claims and all of the following dependents are rejected for being indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-17 are rejected under 35 U.S.C. 103 as being unpatentable over “OCR-VQA: Visual Question Answering by Reading Text in Images” (hereinafter referred to as “Mishra”) in further view of “Recognition and Classifying Sales Flyers Using Semi-Supervised Learning” (hereinafter referred to as “Mosquera”).
Regarding Claim 1, Mishra teaches separating label data from portions of an image [See section IV. Part A where text blocks are extracted from portions of an image of book covers]
PNG
media_image1.png
481
607
media_image1.png
Greyscale
And generating candidate labeled data based on associated ones of unlabeled portions of the image and optical character recognition (OCR) data [See above where the image undergoes OCR and features are obtained, indicating that labeled data is generated. See also figure 4 where various info is collected from the text of the image];
PNG
media_image2.png
644
1408
media_image2.png
Greyscale
generating key performance indicator (KPI) metric values based on a comparison between the candidate labeled data and a second data set [See section IV part D. where the model is by propagating the cross entropy loss using an optimizer]; and adjust weights of a model based on the KPI metric values
PNG
media_image3.png
392
376
media_image3.png
Greyscale
[See above where optimization is done using back propagation of ross entropy loss, indicating that the weights are adjusted in the process].
Mishra does not explicitly teach that the model incorporates interface circuitry; machine-readable instructions; and at least one processor circuit to be programmed by the machine-readable instructions to carry out the instructions.
Mosquera does teach a processer, therefore it also teaches interface circuitry and machine readable instructions [See section III part j. where the neural network is trained on GTX1070]
PNG
media_image4.png
231
855
media_image4.png
Greyscale
Mosquera also teaches separating labels from a data set, and using OCR multiple portions of the image using semi-supervised learning [see fig 1 below].
PNG
media_image5.png
822
613
media_image5.png
Greyscale
Therefore it would have been obvious to one with ordinary skill in the art before the effective filing date to combine the model of Mishra, with the processer and identifying of products of Mosquera, as they are in the same field of endeavor of using machine learning and OCR to classify images. The motivation to combine would be to increase the accuracy of being able to recognize products and the information associated with them (see section IV the conclusion of Mosquera)
PNG
media_image6.png
760
581
media_image6.png
Greyscale
Regarding Claim 2, Mosquera and Mishra teach the apparatus as defined in claim 1, wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to generate the first data set and the second data set based on labeled image data associated with the image [See Figure 6 of Mosquera where the image is updated with bounding boxes to showcase the product image with the associated label. See also section IV part D of Mishra which discloses back propagation, indicating that the data generated becomes part of the data that trains the model.
PNG
media_image7.png
490
620
media_image7.png
Greyscale
Regarding claim 3, Mosquera and Mishra teach the apparatus as defined in claim 2, wherein the second data set retains the label data, the retained label data unmodified from an original format [see section III part C. of Mosquera where a ground truth is defined for the crop of the image. Mishra also defines ground truth associated with titles of a book in section III part A. stage 2].
PNG
media_image8.png
294
620
media_image8.png
Greyscale
PNG
media_image9.png
160
506
media_image9.png
Greyscale
Regarding claim 4, Mishra and Mosquera teach the apparatus as defined in claim 3, wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to compare the candidate labeled data with the retained label data associated with the second data set [See Mosquera section III part M. where the labeled data from the product image is compared with the text data which acts as the ground truth].
PNG
media_image10.png
821
632
media_image10.png
Greyscale
Regarding Claim 5, Mosquera teaches generating first polygons corresponding to the unlabeled portions; and generate second polygons corresponding to the OCR data [See section III part I. of Mosquera where polygons are generated of the text on a product flyer, as well as text from product images see also Fig 3. Of Mosquera which depicts the bounding boxes on a store flyer].
PNG
media_image11.png
696
615
media_image11.png
Greyscale
PNG
media_image12.png
485
605
media_image12.png
Greyscale
Regarding claim 6, Mosquera teaches the apparatus as defined in claim 5, wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to identify the associated ones of the unlabeled portions and the OCR data based on respective intersections of the first polygons and the second polygons [See figure 6 above and section III part M of Mosquera where the OCRed text is associated with a product image to create a bounding box consisting of both the product and the related text of a section]
PNG
media_image13.png
97
591
media_image13.png
Greyscale
PNG
media_image14.png
808
579
media_image14.png
Greyscale
Regarding claim 7, Mishra and Mosquera teach the apparatus as defined in claim 1, wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to unlink label data in the first data set [See section III paragraph F and the title of Mosquera where a semi-supervised training method is described. This indicates that not all of the training images have a ground truth linked to them, meaning that the label data is unlinked. See also section III part A. stage 4 of Mishra where a data split is disclosed, indicating that a data set that is used for training is unlinked from its labels (ground truth)]; the second data set including originally labeled data associated with the crops of the image [see section III part C. above of Mosquera where a ground truth is defined for the crop of the image. Mishra also defines ground truth associated with titles of a book in section III part A. stage 2 above].
PNG
media_image15.png
416
608
media_image15.png
Greyscale
PNG
media_image16.png
292
504
media_image16.png
Greyscale
Regarding claim 8, both Mishra and Mosquera teach that the model is a machine learning model, and Misra additionally teaches adjusting the weights [see above section IV part D. of Mishra and the conclusion of Mosquera above where CNNs are part of both models, therefore making them machine learning models. Additionally the above stated paragraph of Mishra discusses adjusting the weights].
Regarding claim 9, Mosquera teaches that portions of the image represent separate product images within the image [See the conclusion of Mosquera above where it states that the model is used to identify portions of images from a sales flyer, indicating that the portions are products].
Claim 10 is similarly analyzed to Claim 1
Claim 11 is similarly analyzed to Claim 2
Claim 12 is similarly analyzed to Claim 3
Claim 13 is similarly analyzed to Claim 4
Claim 14 is similarly analyzed to Claim 5
Claim 15 is similarly analyzed to Claim 6
Claim 16 is similarly analyzed to Claim 7
Claim 17 is similarly analyzed to Claim 8
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANUSHA KASHYAPA whose telephone number is (571)272-8766. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANUSHA KASHYAPA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights