DETAILED ACTION
Notice of Pre-AIA or AIA Status.
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. Claims 1-20 filed and preliminary amended on 05/16/2024 are pending and being examined. Claims 1, 13, and 18 are independent form.
Claim Rejections - 35 USC § 103
3. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1-7, 10-16, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Delgado del Hoyo et al (US 2025/0005947, hereinafter “Delgado del Hoyo”) in view of Hirai (US 2006/0197928, hereinafter “Hirai”).
Regarding claim 1, Delgado del Hoyo discloses a non-transitory machine-readable medium having machine executable instructions for an equipment verification system that causes a processor core to execute operations (the apparatus and the method for recognizing products using visual and textual information; see Abstract and fig.5), the operations comprising:
receiving an image comprising an equipment identifier tag (see para.34: “In particular, example product recognition disclosed herein identifies respective product identifiers for the product candidates in the image-based set of product candidates, [...] Example product recognition disclosed herein can compare the reference text with product text extracted from shelf image to determine text similarity”; also see, the second product 414 image in fig.4 and para.140: “In particular, the first, second, and third product facings 406, 408, 410 depict an example first product 412, an example second product 414, and an example third product 416, respectively.” It should be noticed that “a product” there includes “an equipment” recited in the claim.);
identifying a set of text regions in the image (see the text boxes 520 included in the second product image 414 in fig.5 and para.145: “The OCR output 518 includes machine readable words (e.g., text) 520 represented by example text boxes.”);
adaptively applying an image processing algorithm to a sub-image of the image to generate a processed clustered text region, wherein the sub-image comprises the clustered text region and the image processing algorithm is selected from a set of image processing algorithms based on the sub-image (wherein the algorithm(s) selected for detecting text regions to which the OCR performs include CNN, RPN, RCNN, and YOLO algorithms; see para.71);
generating a predicted tag identifier for the processed clustered text region; and determining a tag identifier for the equipment identifier tag based on a comparison between the predicted tag identifier for the processed clustered text region and a set of tag identifiers comprising the tag identifier (see para.57: “The product recognition circuitry 104 generates or otherwise obtains textual information (e.g., words and their corresponding bounding boxes) from the image using OCR techniques, which can be compared to characteristics of products in the set of product candidates. In particular, the product recognition circuitry 104 is communicatively coupled to an example products database 112, which is structured to store product reference data”. Also see para.34: “Example product recognition disclosed herein can compare the reference text with product text extracted from shelf image to determine text similarity.” It should be noticed that the product identifier identified by the product recognition includes: “a product identifier(s) (e.g., a numeric identifier, an alphanumeric identifier, a universal product code (UPC), a European Article Number (EAN), a stock keeping unit (SKU), etc.” See para.24).
As explained above, the mere difference is that, Delgado del Hoyo does not explicitly disclose “generating a clustered text region associated with the equipment identifier tag from the set of text regions, wherein the clustered text region comprises a subset of the set of text regions” recited in the claim. However, the technique of using a pixel clustering, a word clustering, and/or a text region cluster is well-known and widely used in the field of text recognition. As evidence, Hirai teaches, generating a clustered text region from the set of text regions, wherein the clustered text region comprises a subset of the set of text regions, see fig.4 and para.63; wherein a text region is clustered from the words “sheet AAA” as shown in fig.4. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made to incorporate the teachings of Hirai into the teachings of Delgado del Hoyo and utilize the text region cluster technique taught by Hirai for image-based product identifier recognition taught by Delgado del Hoyo. Suggestion or motivation for doing so would have been to extract text regions from an image including text information as taught by Hirai, see fig.4. Therefore, the claim is unpatentable over Delgado del Hoyo in view of Hirai.
Regarding claim 2, 14, 19, the combination of Delgado del Hoyo and Hirai, the operations further comprising: determining whether the tag identifier is a target tag identifier associated with a work order; and transmitting an indication of whether to proceed with the work order based on whether the tag identifier is the target tag identifier (Delgado del Hoyo, whether in the products are ordered in, e.g., the 1st product 412, the 2nd product 414, and the 3rd product 416 as shown in image 400 of fig.5, and the corresponding identified results are output in 526 of fig.5).
Regarding claim 3, 15, the combination of Delgado del Hoyo and Hirai, wherein a text region of the set of text regions has associated bounding box coordinates for the text region, and wherein the clustered text region is generated based on proximity between bounding box coordinates of the text regions of the subset of the set of text regions (Hirai, see bounding boxes (BBs) in the right-side image in fig.4).
Regarding claim 4, the combination of Delgado del Hoyo and Hirai discloses the non-transitory machine-readable medium of claim 1, wherein the image processing algorithm is selected by a trained machine learning algorithm based on the sub-image (Delgado del Hoyo, wherein the algorithm(s) selected for detecting text regions to which the OCR performs include CNN, RPN, RCNN, and YOLO algorithms; see para.71).
Regarding claim 5, the combination of Delgado del Hoyo and Hirai discloses the non-transitory machine-readable medium of claim 1, wherein the set of image processing algorithms comprises a binary thresholding algorithm and a grayscale algorithm (Hirai, see para.62: “An image to be processed is binarized to a monochrome image, and a cluster of pixels bounded by black pixels is extracted by outline tracing.”).
Regarding claim 6, the combination of Delgado del Hoyo and Hirai discloses the non-transitory machine-readable medium of claim 1, wherein the comparison comprises determining a similarity score between the predicted tag identifier and the tag identifier (Delgado del Hoyo, see para.34: “Example product recognition disclosed herein can compare the reference text with product text extracted from shelf image to determine text similarity.”).
Regarding claim 7, the combination of Delgado del Hoyo and Hirai discloses the non-transitory machine-readable medium of claim 1, wherein generating the predicted tag identifier comprises: generating a text prediction for the processed clustered text region; and filtering the text prediction to determine the predicted tag identifier (Delgado del Hoyo, see para.147: “the prediction circuitry 224 generates an example final set of product candidates 526 based on the merged scores and the image based confidence score. For example, the final set of product candidates 526 includes example SKU02, which includes an example final confidence score of 0.55, example SKU01, which includes an example final confidence score of 0.30, example SKU03, which includes an example text-based confidence score of 0.05, etc. In some examples, the final confidence scores are based on a maximum between a respective merged score and a respective image-based confidence score.”).
Regarding claim 10, the combination of Delgado del Hoyo and Hirai discloses the non-transitory machine-readable medium of claim 1, wherein determining the tag identifier for the equipment identifier tag is further based on at least one of a property of the image or a property of the equipment identifier tag (Delgado del Hoyo, see para.34: “Example product recognition disclosed herein can compare the reference text with product text extracted from shelf image to determine text similarity” to recognize the SKUs of the products listed in 526 in fig.5).
Regarding claim 11, the combination of Delgado del Hoyo and Hirai discloses the non-transitory machine-readable medium of claim 1, wherein generating the predicted tag identifier is further based on user feedback (Delgado del Hoyo, see para.147: “the prediction circuitry 224 generates an example final set of product candidates 526 based on the merged scores and the image-based confidence score. For example, the final set of product candidates 526 includes example SKU02, which includes an example final confidence score of 0.55, example SKU01, which includes an example final confidence score of 0.30, example SKU03, which includes an example text-based confidence score of 0.05, etc. In some examples, the final confidence scores are based on a maximum between a respective merged score and a respective image-based confidence score.”).
Regarding claim 12, the combination of Delgado del Hoyo and Hirai discloses the non-transitory machine-readable medium of claim 1, wherein identifying the set of text regions in the image comprises employing a trained machine learning algorithm to identify the set of text regions (Delgado del Hoyo, wherein the algorithm(s) selected for detecting text regions to which the OCR performs include CNN, RPN, RCNN, and YOLO algorithms; see para.71).
Regarding claim 13, 18, each of them is an inherent variation of claim 1, thus it is interpreted and rejected for the reasons set forth in the rejection of claim 1.
Regarding claim 16, the combination of Delgado del Hoyo and Hirai discloses the equipment verification system of claim 13, wherein the sub-image is generated based on cropping the image to comprise the subset of the set of text regions (Delgado del Hoyo, see cropping image 400 to obtain image 414 in fig.5; Hirai, see cropping the left image shown in the left side of fig.4 to obtain the text regions shown in the right side of fig.4).
Allowable Subject Matter
6. After an updated search of applicable prior art, the subject matter of claims 8-9, 17, and 20, in combination with the base claim and intervening claims, were not found. Claims 8-9, 17, and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
7. Any inquiry concerning this communication or earlier communications from the examiner should be directed to RUIPING LI whose telephone number is (571)270-3376. The examiner can normally be reached 8:30am--5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HENOK SHIFERAW can be reached on (571)272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit https://patentcenter.uspto.gov; https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center, and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RUIPING LI/Primary Examiner, Ph.D., Art Unit 2676
Prosecution Insights