DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Remarks page 8, filed 06/22/2026, with respect to the Rejections of claims 1-20 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The Rejections of claims 1-20 under 35 U.S.C. 112(b) have been withdrawn.
Applicant's arguments, see Remarks pages 9-10, filed 06/22/2026, with respect to the Rejections of claims 1, 10, and 19 under 35 U.S.C. 103 have been fully considered but they are not persuasive.
On pages 9-10 of Remarks, Applicant argues:
PNG
media_image1.png
618
576
media_image1.png
Greyscale
Examiner respectfully disagrees.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Col 11, Lines 24-56 of Migdal disclose: “In embodiments where the image restoration model(s) structured identifier image 538 to obtain the structured 342 is a deep learning-based model, the deep learning-based model may be trained by creating datasets consisting of high quality original images and low quality corrupted images. The datasets may be created from cropped regions of larger images. The cropped regions may be chosen from a combination of set of images that contain structured identifiers and a set of images that do not contain structured identifiers. The low quality counterparts to the cropped regions may be created using a variety of image augmentation transforms, including, but not limited to, focal and motion blur, JPEG artifacts, shutter effects, brightness, contrast, and other image processing corruption methods…To train the deep learning-based model, the images 702 and 704 are fed in pairs consisting of the high quality cropped original (e.g., high quality image 702) and its low quality corrupted counterpart (e.g., low quality image 704). The output is a model trained to take as input a low quality, possibly corrupted image, and reconstruct a high quality representation of the image.”. Wherein the high quality original images constitute a first set of images, and the low quality corrupted images, obtained by performing a variety of image augmentations through predetermined corruption methods, constitute a second set of images. Wherein the neural network, being trained to reconstruct a high quality representation from a low quality input, constitutes “identifying the one or more predefined perspective distortions and generating the plurality of transformation parameters based on the one or more predefined perspective distortions”.
Thus, Migdal discloses the claim limitations: “a first set of images including objects that are the same category of physical object as the received image and that satisfy at least one readability threshold; and a second set of images generated by applying one or more predefined perspective distortions to one or more images of the first set of images, wherein training the neural network includes identifying the one or more predefined perspective distortions and generating the plurality of transformation parameters based on the one or more predefined perspective distortions”.
However Migdal fails to disclose: “receiving, from the neural network, the plurality of transformation parameters for the received image; determining whether the plurality of transformation parameters exceed the at least one readability readable threshold, wherein the at least one readability readable threshold includes a number of degrees askew from an axis; and performing, in response to determining that the plurality of transformation parameters exceed the at least one readability threshold, the image transformation on the received image, based on for the plurality of transformation parameters for the received image, to generate a rectified image comprising the graphical data oriented within the at least one readability threshold.”.
Section: 3 Our Approach of Hansen discloses: “Through some small scale test, we found out that rotating the 1D barcodes such that the bars are vertical and rotating QR barcodes so that the sides of the small squares align with the x and y axis can benefit the performance of the decoding. For 1D barcodes, there is a speedup in time and a higher decoding rate, whereas for the QR barcodes the de coding will take longer, but the decoding success rate is higher. To find the amount of rotation needed a regression network is used to predict a rotation value between 0 and 1. The value will be mapped to an angle going from 0 to 180 for 1D and 45 and 135 for QR barcodes…Each of these barcodes is then put through the Angle prediction network which predicts a rotation and the predicted rotation is then used to rotate the image before it is tried decoded by a decoding framework.”. Thus, a regression network processes a barcode image and outputs a prediction regarding an amount of rotation needed for the correction of the bar code image. Wherein, if an amount of rotation is deemed necessary, the image is rotated by its predicted rotation amount.
Thus, Hansen discloses: receiving, from the neural network, the transformation parameters for the received image; determining whether the transformation parameters exceed the at least one readability readable threshold, wherein the at least one readability readable threshold includes a number of degrees askew from an axis; and performing, in response to determining that transformation parameters exceed the at least one readability threshold, the image transformation on the received image, based on the transformation parameters for the received image, to generate a rectified image comprising the graphical data oriented within the at least one readability threshold.
Given the teachings above, in addition to the rejection of claim 1 under 35 U.S.C. 103 below, one of ordinary skill in the art would have been motivated to implement the known technique of detecting the rotation angle of an image through a neural network prior to performing image rectification based on a selected value taught by Hansen by thresholding the image defect values determined by the image restoration model(s) disclosed by Migdal prior to selectively applying the different image enhancement techniques.
Therefore, Migdal in view of Hansen discloses the claim 1 limitation: “a second set of images generated by applying one or more predefined perspective distortions to one or more images of the first set of images, wherein training the neural network includes identifying the one or more predefined perspective distortions and generating the plurality of transformation parameters based on the one or more predefined perspective distortions.".
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Migdal et al. (US 12,131,224 B1) hereinafter referenced as Migdal, in view of Hansen et al. (Real-Time Barcode Detection and Classification using Deep Learning) hereinafter referenced as Hansen.
Regarding claim 1, Migdal discloses: A computer-implemented method, comprising: receiving an image of a physical object comprising graphical data having an undefined perspective distortion (Migdal: Col 2: Lines 47-58: “embodiments described herein provide improved systems and techniques for reading structured identifiers on items using camera-based scanning systems…the structured identifier within the image that is captured via a camera-based scanning system may be non-decodable due to one or more image defects (e.g., image noise, blurriness, low sharpness, low lighting, low contrast, compression artifacts, etc.).”;
Col 7: Lines 39-42: “…geometric transformations may be used to rectify the shape of a structured identifier due to perspective and lens distortion, package deformation, structured identifier wrinkling, etc.”; Wherein the structured identifier constitutes graphical data and distorted shape of the identifier constitutes the perspective distortion);
determining that an image transformation is to be performed on the received image (Migdal: Figure 4; Col 12: Lines 38-47: “…the analysis tool 322 may provide the structured identifier images 336 to the decoding tool 326. If the decoding tool 326 is able to successfully decode a structured identifier from the structured identifier images 336, then the enhancement tool 324 may not be applied or implemented. On the other hand, if the decoding tool 326 is unable to successfully decode a structured identifier from the structured identifier images 336, then the decoding tool 326 can send the structured identifier images 336 to the enhancement tool 324 for an enhancement process.”);
providing the received image to a neural network configured to generate a plurality of transformation parameters corresponding to the undefined perspective distortion (Migdal: Figure 4; Col 11: Lines 7-9: “ the image restoration model(s) 342 may be configured to use deep learning-based image restoration as an image enhancement technique.”;
Col 11-12: Lines 66-05: “ the enhancement tool 324 can be configured to selectively apply different image enhancement techniques (via the image restoration model(s) 342 ) based on the type of image defect (or corruption). For example, if the detected structured identifier is heavily distorted, then the enhancement tool 324 may apply geometric transforms to rectify the structured identifier.”; Wherein the deep learning models extract transformation parameters used to apply transformations to rectify defects within the identifier),
wherein the neural network was trained on a set of training data comprising: a first set of images including objects that are the same category of physical object as the received image and that satisfy at least one readability threshold; and a second set of images generated by applying one or more predefined perspective distortions to one or more images of the first set of images, wherein training the neural network includes identifying the one or more predefined perspective distortions and generating the plurality of transformation parameters based on the one or more predefined perspective distortions (Migdal: Figure 7; Col 3: Lines 17-23: “Additionally, note that, as used herein, the "quality" of an image may refer to (or be based on) one or more image attributes, such as sharpness, noise, dynamic range, resolution, contrast, etc. A "high quality" image, for example, may include a higher sharpness, lower noise, higher dynamic range, higher resolution, etc. relative to a "low quality" image.”;
Col 11: Lines 32-56: “The low quality counterparts to the cropped regions may be created using a variety of image augmentation transforms, including, but not limited to, focal and motion blur, JPEG artifacts, shutter effects, brightness, contrast, and other image processing corruption methods…The high quality images 702 1 - 4 are high quality crops taken from imagery consisting of a mixture of structured identifiers and other regions…The low quality images 704 1 - 4 are generated by using an image augmentation transform on the high quality images 702 1 - 4, respectively… To train the deep learning-based model, the images 702 and 704 are fed in pairs consisting of the high quality cropped original (e.g., high quality image 702) and its low quality corrupted counterpart (e.g., low quality image 704). The output is a model trained to take as input a low quality, possibly corrupted image, and reconstruct a high quality representation of the image.”; Wherein the high quality barcode images constitute the first set of images, and the low quality barcode images constitute the second set of images. Also, wherein the augmented first set image data constitutes the predetermined parameters.);
performing the image transformation on the received image to generate a rectified image comprising the graphical data oriented within the at least one readability threshold (Migdal: Figure 4; Col 7-8: Lines 43-02: “In one embodiment, at least one image restoration model 342 is configured to perform a deep learning-based image restoration to enhance the structured identifier image(s) 336…The decoding tool 326 is generally configured to extract information associated with the structured identifier 175 (referred to herein as structured identifier information 340 )…the decoding tool 326 extracts information associated with the structured identifier 175 from the enhanced structured identifier images 338.”; Wherein the image restoration models extract the transformation parameters from the identifier in the image and rectifies the image using them, then is able to read the identifiers using the decoding tool).
Migdal does not disclose expressly: receiving, from the neural network, the plurality of transformation parameters for the received image; determining whether the plurality of transformation parameters exceed the at least one readability threshold, wherein the at least one readability threshold includes a number of degrees askew from an axis; and performing, in response to determining that the plurality of transformation parameters exceed the at least one readability threshold, the image transformation on the received image, based on the plurality of transformation parameters for the received image, to generate a rectified image comprising the graphical data oriented within the at least one readability threshold.
Hansen discloses: receiving, from a neural network, transformation parameters for a received image; determining whether the transformation parameters exceed at least one readability threshold, wherein the at least one readability threshold includes a number of degrees askew from an axis (Hansen: Figure 1; Section: 3 OUR APPROACH: “we found out that rotating the 1D barcodes such that the bars are vertical and rotating QR barcodes so that the sides of the small squares align with the x and y axis can benefit the performance of the decoding. For 1D barcodes, there is a speedup in time and a higher decoding rate, whereas for the QR barcodes the decoding will take longer, but the decoding success rate is higher. To find the amount of rotation needed a regression network is used to predict a rotation value between 0 and 1. The value will be mapped to an angle going from 0 to 180 for 1D and 45 and 135 for QR barcodes. At fig. 2, the method on how the angle is measured is shown.”; Wherein the determination of a rotation angle needed, such that it is determined that the barcode requires rotation, constitutes an exceeded threshold.); and
performing, in response to determining that the transformation parameters exceed the at least one readability threshold, the image transformation on the received image, based on the transformation parameters for the received image, to generate a rectified image comprising the graphical data oriented within the at least one readability threshold (Hansen: Figure 1; Section: 3 OUR APPROACH: “Each of these barcodes is then put through the Angle prediction network which predicts a rotation and the predicted rotation is then used to rotate the image before it is tried decoded by a decoding framework”; Wherein the image is rectified by rotating the image based on a predicted rotation).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the Angle prediction network disclosed by Hansen into the image enhancement tool disclosed by Migdal prior to selectively applying the different image enhancement techniques. The suggestion/motivation for doing so would have been “The developed system can also find the rotation of both the 1D and QR barcodes, which gives the opportunity of rotating the detection accordingly which is shown to benefit the decoding process in a positive way. Both the detection and the rotation prediction shows real-time performance.” (Hansen: Abstract.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
In addition, before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of detecting the rotation angle of an image through a neural network prior to performing image rectification based on a selected value taught by Hansen by thresholding the image defect values determined by the image restoration model(s) disclosed by Migdal prior to selectively applying the different image enhancement techniques. The suggestion/motivation for doing so would have been “Through some small scale test, we found out that rotating the 1D barcodes such that the bars are vertical and rotating QR barcodes so that the sides of the small squares align with the x and y axis can benefit the performance of the decoding.” (Hansen: 3 OUR APPROACH; Wherein the selection of values for parameters allows for adjustment for improved performance.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Migdal with Hansen to obtain the invention as specified in claim 1.
Regarding claim 2, Migdal in view of Hansen discloses: The computer-implemented method of claim 1, further comprising: providing the rectified image to a system configured to read the graphical data oriented within the at least one readability threshold (Migdal: Figure 4; Col 12: Lines 10-15: “As shown in FIG. 4, the enhancement tool 324 provides the enhanced structured identifier images 338 to the decoding tool 326, which is configured to determine structured identifier information 340, based on reading the structured identifier(s) 175 in the enhanced structured identifier images 338.”).
Regarding claim 3, Migdal in view of Hansen discloses: The computer-implemented method of claim 1, wherein the neural network is configured to perform pixel analysis on pixels of the graphical data of the received image (Hansen: Section: 3 OUR APPROACH: “At fig. 2, the method on how the angle is measured is shown. The regression network is based on the Darknet19 classification network1 where the softmax layer is removed, and the number of filters in the last convolutional layer is set to one. Furthermore, three different activation functions are tried in the last convolutional as well, Leaky ReLU, Logistic and ReLU.”)
(Migdal: Col 11: Lines 50-56: “To train the deep learning-based model, the images 702 and 704 are fed in pairs consisting of the high quality cropped original (e.g., high quality image 702) and its low
quality corrupted counterpart (e.g., low quality image 704). The output is a model trained to take as input a low quality, possibly corrupted image, and reconstruct a high quality representation of the image.”).
Regarding claim 4, Migdal in view of Hansen discloses: The computer-implemented method of claim 3, wherein the graphical data comprises one of a barcode or a QR (quick response) code (Migdal: Col 2: Lines 8-13: “Such a computer-vision system can use the camera-based structured identifier reader(s) to detect and read (or decode) structured identifiers (e.g., linear barcodes, one-dimensional (1D) barcodes, two-dimensional (2D) barcodes, etc.) on various items (or packages) in the facility.”).
Regarding claim 5, Migdal in view of Hansen discloses: The computer-implemented method of claim 4, wherein the barcode comprises parallel lines (Migdal: Figures 6A-6G and 7; Wherein the barcodes contain parallel lines).
Regarding claim 6, Migdal in view of Hansen discloses: The computer-implemented method of claim 1, further comprising: identifying a plurality of graphical objects on the received image (Migdal: Col 6: Lines 41-45: “In a particular embodiment, the detector tool 320 uses a ML model (e.g., ML-based localizer model) to determine the bounding box of each structured identifier 175 within a respective image 330.”; Wherein determining a bounding box for each identifier within an image means that multiple identifiers may be present in an image); selecting a first graphical object from the plurality of graphical objects; and providing the first graphical object to the neural network, wherein the performing comprises performing the image transformation on the plurality of graphical objects based on the plurality of transformation parameters of the first graphical object (Hansen: Section: 3 OUR APPROACH: “The system first receives an input image, and then it is fed through the YOLO detection system which produces a number of detections depending on the number of barcodes in the image. Each of these barcodes is then put through the Angle prediction network which predicts a rotation and the predicted rotation is then used to rotate the image before it is tried decoded by a decoding framework.”)
(Migdal: Figure 4; Col 11-12: Lines 66-05: “the enhancement tool 324 can be configured to selectively apply different image enhancement techniques (via the image restoration model(s) 342 ) based on the type of image defect (or corruption). For example, if the detected structured identifier is heavily distorted, then the enhancement tool 324 may apply geometric transforms to rectify the structured identifier.”; Wherein the DCT CNN defect detector outputs identified defects within the structural identifier image on a patch by patch basis, and the enhancement tool applies image restoration models based on the identified distortions.).
Regarding claim 7, Migdal in view of Hansen discloses: The computer-implemented method of claim 1, further comprising providing, in response to determining that the plurality of transformation parameters do not exceed the at least one readability threshold, the received image to an object reader without performing the image transformation (Migdal: Figure 4; Col 11-12: Lines 66-05: “the enhancement tool 324 can be configured to selectively apply different image enhancement techniques (via the image restoration model(s) 342 ) based on the type of image defect (or corruption). For example, if the detected structured identifier is heavily distorted, then the enhancement tool 324 may apply geometric transforms to rectify the structured identifier.”; Wherein the image enhancement techniques are applied based on corruptions/defects detected.).
Regarding claim 8, Migdal in view of Hansen discloses: The computer-implemented method of claim 1, wherein the graphical data comprises alphanumeric text with a known pattern (Migdal: Figures 6A-6G; Col 3: Lines 11-16: “the structured identifier can include any type of globally unique identifier used to identify an item. Examples of such structured identifiers can include visual and/or geometric features of a label (e.g., ridges, edges, pixel value intensity changes), text, 1D barcodes, two-dimensional (2D) barcodes, etc.”;
Col 7: Lines 56-61: “The structured identifier information 340 may include symbology used for the structured identifier 175, the detected corners of the structured identifier 175, the decoded text or binary information associated with the structured identifier 175 (e.g., the alphanumeric string or binary information), or combinations thereof.”; Wherein the alphanumeric string/text constitutes an alphanumeric text with a known pattern).
Regarding claim 9, Migdal in view of Hansen discloses: The computer-implemented method of claim 1, wherein the second set of images includes at least a subset of the first set of images re-oriented according to the one or more predefined perspective distortions (Migdal: Col 11: Lines 25-36: “the deep learning-based model may be trained by creating datasets consisting of high quality original images and low quality corrupted images…The low quality counterparts to the cropped regions may be created using a variety of image augmentation transforms, including, but not limited to, focal and motion blur, JPEG artifacts, shutter effects, brightness, contrast, and other image processing corruption methods.”;
Col 11: Lines 47-49: “The low quality images 704 1 - 4 are generated by using an image augmentation transform on the high quality images 702 1 - 4 , respectively.”).
As per claim(s) 10, arguments made in rejecting claim(s) 1 are analogous. In addition, Figure 3 and Col 15-16: Lines 65-20 of Migdal disclose at least one processor coupled to memory.
As per claim(s) 11, arguments made in rejecting claim(s) 2 are analogous.
As per claim(s) 12, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 13, arguments made in rejecting claim(s) 4 are analogous.
As per claim(s) 14, arguments made in rejecting claim(s) 5 are analogous.
As per claim(s) 15, arguments made in rejecting claim(s) 6 are analogous.
As per claim(s) 16, arguments made in rejecting claim(s) 7 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 8 are analogous.
As per claim(s) 18, arguments made in rejecting claim(s) 9 are analogous.
As per claim(s) 19, arguments made in rejecting claim(s) 1 are analogous. In addition, Col 15-16: Lines 65-20 of Migdal disclose a non-transitory computer-readable medium having instructions stored thereon that are executed by a processor.
As per claim(s) 20, arguments made in rejecting claim(s) 2 are analogous.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/
Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights