Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejection – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 7, 11-12, 17, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garlapati (US11164300B2), hereinafter referenced as Garlapati, in view of the following: Sun (US9858496B2) hereinafter referenced as Sun, Halstead (US2011254840A1) hereinafter referenced as Halstead, Goswami (US20140164146A1) hereinafter referenced as Goswami, and Burton (US20170280130A1) hereinafter referenced as Burton.
Regarding claim 1, Garlapati teaches
A system comprising a processor and a non-transitory computer-readable medium storing computing instructions that, when executed on the processor, cause the processor to perform operations comprising:
“An exemplary system configured according to the concepts disclosed herein can include: a processor configured to perform image analysis; and a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations” (¶ 6, Garlapati); “An exemplary non-transitory computer-readable storage medium configured according to this disclosure can have instructions stored which, when executed by a computing device configured to perform image processing, cause the computing device to perform operations” (¶ 8, Garlapati);
auto-validating that a visual resolution level of the 2D silo image falls within a predetermined acceptance rate; and
“The system can provide confidence of the classification of the images, confidence that the description is correct, and/or that the order of the images is correct. If one or more of these indications is low, the system can prompt manual review for the low confidence items (112). Finally, the system can provide the results of the assessments and algorithms to an automated catalogue management (110).” (¶ 19, Garlapati); “the system can determine, based on a business unit to which the image(s) will be assigned, cutoffs for the similarity score and/or test data to determine if the image quality is at an acceptable, predetermined quality, or if the images must be revised or otherwise corrected.” (¶ 26, Garlapati);
Garlapati teaches of an automated quality gate score for each image and compares it to a predetermined cutoff point either passing or rejecting it.
2D silo image
“extracting the shape information from the product data, said extracting including obtaining a 2D outline shape (i.e., the image mask) of the consumer good product and obtaining a 3D shape class of the subject consumer good product (e.g., as assigned or otherwise provided)” (¶ 18, Garlapati);
Garlapati teaches of extracting including obtaining a 2D outline shape (i.e., the image mask) of the consumer good product (reads on 2D silo image).
Garlapati fails to teach the following: identifying a 2D silo image of a geometric item based on a probability value exceeding a predetermined probability threshold; segmenting artifacts from the 2D silo image to isolate first pixels of a border of the geometric item; trimming second pixels along the border of the geometric item; performing an aspect ratio validation on the 2D silo image to validate that the 2D silo image corresponds to a shape of the geometric item; and generating a 3D view image from the 2D silo image of the geometric item enabled for use in virtual environments when the visual resolution level falls within the predetermined acceptance rate.
But Sun does. Sun teaches the following:
identifying a 2D silo image of a geometric item based on a probability value exceeding a predetermined probability threshold;
“Fast Region-Based Convolutional Neural Network (FRCN) proposal classifier to determine a class of each object in the image and a confidence score associated therewith.” (Abstract, Sun); “, the RPN can calculate an objectness score. In such examples, if the objectness score is above a threshold, the RPN can determine that an object or portion thereof is located at a particular point.” (¶ 66, Sun); “the proposal classifier can generate a confidence score associated with each object category. In various examples, the confidence score can be based on a similarity between the object in the proposal and an object associated with a pre-defined object category. In such examples, the object associated with the pre-defined object category may be learned from a training image. The similarity can be based on object curves, size, aspect, angles, or other characteristics of the objects. In some examples, the confidence score may be calculated for each proposal. In other examples, the confidence score may be calculated for each proposal except for those designated as background.” (¶ 59, Sun);
Sun determines the class of each object in the image (reads on identifying a 2D silo image of a geometric item) based on if a confidence and objectness score is above a threshold (reads on based on a probability value exceeding a predetermined probability threshold).
Sun BASE is analogous art with respect to Garlapati because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati with the feature of Sun to identify a 2D silo image of a geometric item based on a probability value exceeding a predetermined probability threshold by determining a class of each object in the image based on a confidence and objectness score is above a threshold. A person of ordinary skill in the art would do such in order to improve the accuracy of the output object classifications and confidence scores.
Garlapati in view of Sun fail to teach the following: segmenting artifacts from the 2D silo image to isolate first pixels of a border of the geometric item; trimming second pixels along the border of the geometric item; performing an aspect ratio validation on the 2D silo image to validate that the 2D silo image corresponds to a shape of the geometric item; and generating a 3D view image from the 2D silo image of the geometric item enabled for use in virtual environments when the visual resolution level falls within the predetermined acceptance rate.
But Halstead does. Halstead teaches the following:
segmenting artifacts from the 2D silo image to isolate first pixels of a border of the geometric item;
“Step 112 uses the exported raw product 2D shape information and forms a product image mask file. The product image mask file content is numeric data describing the 2D outline shape (2D image mask) of the product package extracted at pixel level from the product photograph(s)/images referenced in product table 10 . Masks are calculated for both width (X coordinate) and height (Y coordinate) dimensions. Step 112 also calculates an additional width mask for cutout regions to determine the location of holes or other special features which occur inside the product 2D outline boundaries.” (¶ 71, Halstead);
Halstead teaches of segmenting background artifacts to isolate first pixels of a border.
trimming second pixels along the border of the geometric item;
“Lastly step 112 extends and blurs edges of the product image. This smoothes edges on images and effectively crops extra white space.” (¶ 72, Halstead);
Halstead teaches of a border-cleanup that removes residual white pixels along the product edge.
generating a 3D view image from the 2D silo image of the geometric item enabled for use in virtual environments when an initial image is provided.
“FIGS. 7A-7C are schematic illustrations of the invention modeling technique from an initial image of the subject (a wine glass), to a mesh texture file of the subject, to the 3D model with initial image projected onto the subject mesh.” (¶ 43, Halstead); “A 3D model of the subject consumer good product results and is configured for interactive display on web pages and in other user-interactive environments.” (Abstract, Halstead); “the output 3D model file 25 is configured for use in serving Web pages of retail store/online shopping, video gaming and other user-interactive applications.” (¶ 88, Halstead);
Halstead teaches of a 3D model built for interactive online use (reads on generating a 3D view image from the 2D silo image of the geometric item). The 3D model is bult from the 2D product image (reads on item enabled for use in virtual environments) when an initial image is provided.
Halstead BASE is analogous art with respect to Garlapati in view of Sun because they are from the same field of endeavor, namely computer vision and machine learning to analyze 2D images, identify physical items, and extract actionable data. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of Sun with the feature of Halstead to incorporate the following: a segment background artifact to isolate first pixels of a border; a border-cleanup that removes residual white pixels along the product edge; and a 3D model built for interactive online use. A person of ordinary skill in the art would do such in order to improve engagement in e-commerce and prototyping.
Garlapati in view of Sun and Halstead fail to teach the following: performing an aspect ratio validation on the 2D silo image to validate that the 2D silo image corresponds to a shape of the geometric item; and generating a 3D image from the 2D image when the visual resolution level falls within the predetermined acceptance rate.
Goswami does. Goswami teaches the following:
performing an aspect ratio validation on the 2D silo image to validate that the 2D silo image corresponds to a shape of the geometric item;
“Other examples of image characteristics include size factors (e.g., area, aspect ratio, height, or width). Moreover, characteristics assessed by the image assessment machine may include factors used in segmenting a foreground shape, from its background within an image (e.g., foreground uniformity, background uniformity, image colorfulness, or foreground colorfulness). Where the image is segmented, the image assessment machine may calculate a ratio of foreground pixels to background pixels within the segmented image.” (¶ 18, Goswami); “the depiction of the item in the image tends to have a reasonable size (e.g., relative to the image as a whole) and tends to be in focus. In addition, such images are likely to have a reasonable aspect ratio (e.g., within a predetermined threshold distance from 1).” (¶ 21, Goswami);
Goswami teaches of an automated system, the image assessment machine, (read on performing an aspect ratio validation on the 2D silo image) the quality and composition of a product image in which the image is evaluated based on the shape, size, and proportions of specific elements within it to be within a predetermined threshold distance (reads on validate that the 2D silo image corresponds to a shape of the geometric item) to determine if the image is clean, well-focused photos where a clearly defined item is centered, appropriately sized, and set against a simple background.
Goswami BASE is analogous art with respect to Garlapati in view of Sun and Halstead because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of Sun and Halstead with the feature of Goswami to incorporate an aspect ratio validation on the 2D silo image to validate that the 2D silo image corresponds to a shape of the geometric item by an automated system image assessment machine to evaluated based on the shape, size, and proportions of specific elements within it to be within a predetermined threshold distance in order to determine if the image is clean, well-focused photos where a clearly defined item is centered, appropriately sized, and set against a simple background. A person of ordinary skill in the art would do such in order to improve image quality.
Garlapati in view of the following: Sun, Halstead, and Goswami fail to teach generating a 3D image from the 2D image when the visual resolution level falls within the predetermined acceptance rate. But Burton does. Burton teaches the following:
generating a 3D image from the 2D image when the visual resolution level falls within the predetermined acceptance rate.
“Selected sets of 2D image frames that satisfy the feature count criteria, the pose criteria and the image quality criteria, are validated. A set of validated 2D image frames is provided to a three-dimensional (3D) reconstruction system (216) to generate a 3D model of a physical scene.” (Abstract, Burton); “If the number of features in the candidate 2D image frame N is greater than the threshold number of features, then the method 300 moves to 110. Otherwise, the candidate 2D image frame is deemed unsuitable for 3D model generation, and the method 300 moves to 324.” (¶ 35, Burton); “step 312 may precede steps 308 and/or 310, and feature considerations will only be made for those frames having sufficient quality parameters/score.” (¶ 40, Bruton);
Burton teaches selecting 2D image frames based on feature count, pose, and image quality to generate 3D models (reads on generating a 3D view image from the 2D image). Candidate frames lacking sufficient features and quality are rejected to ensure accurate 3D scene reconstruction (reads on when the visual resolution level falls within the predetermined acceptance rate).
Burton BASE is analogous art with respect to Garlapati in view of the following: Sun, Halstead, and Goswami because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, and Goswami with the feature of Burton to incorporate 2D image frames selected based on feature count, pose, and image quality to generate 3D models. A person of ordinary skill in the art would do such in order to improve image and model quality.
Regarding claim 2, Garlapati in view of the following: Sun, Halstead, Goswami, and Burton teaches the system of claim 1, and additionally teaches the following. Garlapati teaches
extracting, using an image classification model, images of items from a catalog into multiple classes of images; and
“the system deploys an algorithm to classify the images according to the different views (such as Front, Side, and Back views), and order the images (108).” (¶ 27, Garlapati);
Garlapati teaches of a CNN classifier which sorts and catalog items into multiple view classes.
determining a predicted label and the probability value for each image of the images.
“the concepts disclosed herein can include: receiving a plurality of images of an item; identifying, via a processor configured to perform image analysis, and within each image in the plurality of images, the item“ (¶ 6, Garlapati ); “The predictors can also be used as inputs into a Convolution Neural Network (CNN) model trained to identify the distinct classifications of images.” (¶ 19, Garlapati); “the system deploys an algorithm to classify the images according to the different views (such as Front, Side, and Back views), and order the images (108). The system can provide confidence of the classification of the images, confidence that the description is correct, and/or that the order of the images is correct. If one or more of these indications is low, the system can prompt manual review for the low confidence items (112).” (¶ 27, Garlapati);
Garlapati teaches of a CNN classifier outputs a class label with a confidence value after sorting for each image in the plurality of images.
Regarding claim 7, Garlapati in view of the following: Sun, Halstead, Goswami, and Burton teach the system of claim 1 and performing the aspect ratio validation on the 2D silo image, and additionally teaches the following. Goswami teaches
wherein performing the aspect ratio validation on the 2D silo image comprises: comparing an aspect ratio of each 2D silo image of the geometric item within a predetermined tolerance level, wherein the aspect ratio validation comprises a first quality check point that is automatically implemented.
“Other examples of image characteristics include size factors (e.g., area, aspect ratio, height, or width). Moreover, characteristics assessed by the image assessment machine may include factors used in segmenting a foreground shape, from its background within an image (e.g., foreground uniformity, background uniformity, image colorfulness, or foreground colorfulness). Where the image is segmented, the image assessment machine may calculate a ratio of foreground pixels to background pixels within the segmented image.” (¶ 18, Goswami); “the depiction of the item in the image tends to have a reasonable size (e.g., relative to the image as a whole) and tends to be in focus. In addition, such images are likely to have a reasonable aspect ratio (e.g., within a predetermined threshold distance from 1).” (¶ 21, Goswami);
Goswami teaches of an automated system, the image assessment machine, (read on first quality check point that is automatically implemented) the quality and composition of a product image in which the image is evaluated based on the shape, size, and proportions of specific elements within it to be within a predetermined threshold distance (reads on comparing an aspect ratio of each 2D silo image of the geometric item within a predetermined tolerance level) to determine if the image is clean, well-focused photos where a clearly defined item is centered, appropriately sized, and set against a simple background (reads on the aspect ratio validation).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Goswami to incorporate an automated system image assessment machine to evaluated based on the shape, size, and proportions of specific elements within it to be within a predetermined threshold distance in order to determine if the image is clean, well-focused photos where a clearly defined item is centered, appropriately sized, and set against a simple background. A person of ordinary skill in the art would do such in order to improve image quality.
comparing an aspect ratio of each 2D silo image against a physical aspect ratio
“the computing system can assign a positive label to an anchor 508 that has a highest Intersection-over-Union (IoU) overlap with a ground-truth item, and an anchor 508 that has an IoU overlap higher than a pre-determined percentage with a ground-truth box.” (¶ 89, Sun);
Sun teaches comparing the 2D silo image against an anchor image (reads on comparing an each 2D silo image against a physical image.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Sun to incorporate comparing the 2D silo image against an anchor image. A person of ordinary skill in the art would do such in order to improve image quality.
Claim 11 is rejected using the same rationale or bases as applied to claim 1.
Claim 12 is rejected using the same rationale or bases as applied to claim 2.
Claim 17 is rejected using the same rationale or bases as applied to claim 7.
Claim 19 is rejected using the same rationale or bases as applied to claim 1.
Claim 20 is rejected using the same rationale or bases as applied to claim 2.
Claim(s) 3-4, and 13-14, is/are rejected under 35 U.S.C. 103 as being unpatentable Garlapati in view of the following: Sun, Halstead, Goswami, Burton and Fu (CN111724396A), hereinafter referenced as Fu.
Regarding claim 3, Garlapati in view of the following: Sun, Halstead, Goswami, and Burton teach the system of claim 1 and segmenting the artifacts from the 2D silo image, and additionally teaches the following. Halstead teaches
wherein segmenting the artifacts from the 2D silo image comprises: isolating, using a computer network, the geometric item of interest in the 2D silo image without eroding edges along the 2D silo image.
“This Alpha layer/image mask layer separates the photograph into a foreground and a background. The foreground includes any pixel that is part of the product 2D image and the background is the remaining pixels (any pixel that is not part of the product 2D image) where the background or surrounding space in the photographic scene is visible.” (¶ 48, Halstead); “Lastly step 112 extends and blurs edges of the product image. This smoothes edges on images and effectively crops extra white space. Step 114 receives as input the product image mask file from step 112. As a function of shape class of the product, Step 114 refers to the corresponding class template 17 (geometry data generic to the class) and models the shape map (outline) curve with a small number of line segments. Step 114 converts the product outline curve from the product image mask file into best fit line segments (in pixels). Step 114 employs an iterative process to optimize fit and reduce the number of line segments to a target goal.” (¶ 72-73, Halstead); “FIG. 8A illustrates a computer network or similar digital processing environment in which the present invention may be implemented.” (¶ 89, Halstead);
Halstead teaches using a computer network to isolate the alpha-layer image masks (reads on isolating the geometric item of interest in the 2D silo image) into foreground comprising of every pixel belonging to the product and background comprising of everything else. Halstead preserves the item boundary and removes only the surplus background wherein the item’s edges are extended rather than shrunk and only the excess white space background is cropped rather than the item’s boundary. (reads on without eroding edges along the 2D silo image).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Halstead to clean up a product photo and convert its outline into a simplified geometric shape. A person of ordinary skill in the art would do such in order to improve image quality.
Garlapati in view of the following: Sun, Halstead, Goswami, and Burton fail to explicitly teach using an image segmentation model.
Fu teaches the following:
using an image segmentation model
“The image segmentation module is used to segment the colorless regions in the replaced grayscale image using a background threshold map to obtain the background image and foreground image of the image to be processed.” (¶ 40, Fu);
Fu teaches using an image segmentation model to segment the background and foreground of an image.
Fu BASE is analogous art with respect to Garlapati in view of the following: Sun, Halstead, Goswami, and Burton because they are from the same field of endeavor, namely image segmentation. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Fu to incorporate an image segmentation model to segment the background and foreground of an image. A person of ordinary skill in the art would do such in order to improve the accuracy of the foreground image.
Regarding claim 4, Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Fu teaches the system of claim 3 and segmenting the artifacts from the 2D silo image, and additionally teaches the following. Halstead teaches
wherein segmenting the artifacts from the 2D silo image further comprises: removing the artifacts of portions of background pixels of the 2D silo image, wherein the portions of the background pixels comprise white pixels.
“The image mask layer separates the photograph into a foreground and a background. The foreground comprises any pixel that is part of the product 2D image. The background comprises any pixel that is not part of the product 2D image where the background or surrounding space in the photographic scene is visible.” (¶ 13, Halstead); “Lastly step 112 extends and blurs edges of the product image. This smoothes edges on images and effectively crops extra white space.” (¶ 72, Halstead);
Halstead teaches of using an image mask (reads on artifact) to separate a product from its background (reads on removing the artifacts of portions of background pixels of the 2D silo image), followed by blurring and edge extension to remove excess whitespace (reads on the portions of the background pixels comprise white pixels), preparing the image for 3D modeling.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Fu with the feature of Halstead to incorporate uses an image mask to separate a product from its background, followed by blurring and edge extension to remove excess whitespace, preparing the image for 3D modeling. A person of ordinary skill in the art would do such in order to improve object rendering scalability and efficiency.
Claim 13 is rejected using the same rationale or bases as applied to claim 3.
Claim 14 is rejected using the same rationale or bases as applied to claim 4.
Claim(s) 6 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable Garlapati in view of the following: Sun, Halstead, Goswami, Burton, Fu, and Li (US20070019257A1) hereinafter referenced as Li.
Regarding claim 6, Garlapati in view of the following: Sun, Halstead, Goswami, and Burton teaches the system of claim 1, and removing portions of the background along the 2D silo image, but fail to teach the following determining a respective color intensity threshold for the 2D silo image based on the background pixels; and filtering out, using the respective color intensity threshold, the portions of the background pixels when the portions of the background pixels exceed the respective color intensity threshold. But Fu does. Fu teaches the following:
wherein removing portions of the background along the 2D silo image further comprises: determining a respective color intensity threshold for the 2D silo image based on the background pixels; and
“This invention provides an image segmentation method and apparatus. Firstly, it calculates the mean map of the image to be processed based on a preset first color space, and then calculates the color saliency map of the image to be processed based on the mean map. Next, it calculates a segmentation threshold for the color saliency map and segments the current pixel of the image to be processed according to the segmentation threshold, obtaining colored regions with saliency and colorless regions without saliency.” (¶ 46, Fu)
Fu teaches calculating the segmentation threshold for the color saliency map (reads on a respective color intensity threshold for the 2D silo image) according to the segmentation threshold and the mean map of the image based on a preset first color space (reads on background pixels).
filtering out, using the respective color intensity threshold, the portions of the background pixels when given a portion of the background.
“Firstly, it calculates the mean map of the image to be processed based on a preset first color space, and then calculates the color saliency map of the image to be processed based on the mean map. Next, it calculates a segmentation threshold for the color saliency map and segments the current pixel of the image to be processed according to the segmentation threshold, obtaining colored regions with saliency and colorless regions without saliency. Then, it replaces the colored regions in the grayscale image of the image to be processed using preset pixel values, and performs Gaussian filtering on the replaced grayscale image to obtain a background threshold map. Finally, it uses the background threshold map to segment the colorless regions in the replaced grayscale image, obtaining a background image and a foreground image of the image to be processed.” (¶ 46, Fu)
Fu teaches replacing the colored regions in the grayscale image of the image to be processed using preset pixel values (filtering out using the respective color intensity threshold) based on the background threshold map in order to segment the colorless regions in the replaced grayscale image (uses a portions of the background pixels).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, Burton and Fu with the feature of Fu to incorporate the segmentation threshold for the color saliency map and replacing the colored regions in the grayscale image of the image to be processed using preset pixel values based on the background threshold map in order to segment the colorless regions in the replaced grayscale image. A person of ordinary skill in the art would do such in order to improve the accuracy of the foreground image.
Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Fu fail to explicitly teach the following: the portions of the background pixels exceed the respective color intensity threshold exceed the respective color intensity threshold.
But Li does. Li teaches:
the portions of the background pixels exceed the respective color intensity threshold exceed the respective color intensity threshold
“determining the background value of a document includes compiling a histogram of the image intensity values from pixels within the selected document area. The histogram background peak, the standard deviation from the peak, and a white pixel or background pixel or luminance threshold Tw are determined 114. The background peak value is the gray scale level with greatest number of pixels having an intensity related to the background level value or the white pixels values of the image being scanned. Optionally, or alternatively, the white pixel threshold Tw is a predetermined value.” (¶ 28, Li); “A white pixel determining processor or algorithm or means 140 classifies 142 each pixel as a white pixel or a non-white pixel. More specifically, a pixel is determined as a "white" pixel if it meets the following white pixel criteria 146: L≥Tw-[Delta]T, and max(a-[Delta]A,b-[Delta]B)+[min(a-[Delta]A,b-[Delta]B)/2]<TC” (¶ 9, Li);
Li determines the background value (reads on the portions of the background pixels) which is greater than or equal to the predetermined white pixel threshold Tw (reads on pixels exceed the respective color intensity threshold exceed the respective color intensity threshold).
Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Fu BASE is analogous art with respect to Li because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Fu with the feature of Li to incorporate the background value which is greater than or equal to the predetermined white pixel threshold Tw. A person of ordinary skill in the art would do such in order to improve image quality.
Claim 16 is rejected using the same rationale or bases as applied to claim 6.
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable Garlapati in view of the following: Sun, Halstead, Goswami, Burton and OpenCV (https://github.com/opencv/opencv/blob/315d0f581e322effba15fa43f49045e79634252c/modules/imgproc/src/contours.cpp#L1808), hereinafter referenced as OpenCV.
Regarding claim 5, Garlapati in view of the following: Sun, Halstead, Goswami, Burton teaches the system of claim 1 and trimming the second pixels along the border of the geometric item, and additionally teaches the following. Halstead teaches
wherein trimming the second pixels along the border of the geometric item comprises using an algorithm to capture the geometric item within the 2D silo image; and
“Step 112 uses the exported raw product 2D shape information and forms a product image mask file. The product image mask file content is numeric data describing the 2D outline shape (2D image mask) of the product package extracted at pixel level from the product photograph(s)/images referenced in product table 10.” (¶ 71, Halstead);
Halstead teaches of an algorithm step to capture the geometric item within the 2D silo image.
removing portions of a background along the 2D silo image by separating the portions of the background into two clusters, wherein the two clusters comprise background pixels and main object pixels.
“The image mask layer separates the photograph into a foreground and a background. The foreground comprises any pixel that is part of the product 2D image. The background comprises any pixel that is not part of the product 2D image where the background or surrounding space in the photographic scene is visible.” (¶ 13, Halstead); “Step 114 receives as input the product image mask file from step 112. As a function of shape class of the product, Step 114 refers to the corresponding class template 17 (geometry data generic to the class) and models the shape map (outline) curve with a small number of line segments. Step 114 converts the product outline curve from the product image mask file into best fit line segments (in pixels). Step 114 employs an iterative process to optimize fit and reduce the number of line segments to a target goal.” (¶ 73, Halstead);
Halstead teaches of a product outline contour and portioning pixels into two groups: foreground and background (removing portions of a background along the 2D silo image by separating the portions of the background into two clusters). The foreground comprises any pixel that is part of the product 2D image. The background comprises any pixel that is not part of the product 2D image (reads on wherein the two clusters comprise background pixels and main object pixels)
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Halstead to incorporate a product outline contour and portioning pixels into two groups: foreground and background. A person of ordinary skill in the art would do such in order to improve object rendering scalability and efficiency.
Garlapati in view of the following: Sun, Halstead, Goswami, and Burton fail to explicitly teach a contour finding algorithm. But OpenCV does. OpenCV teaches the following:
a contour finding algorithm
PNG
media_image1.png
565
780
media_image1.png
Greyscale
(OpenCV /modules/imgproc/src/contours.cpp#L1808)
OpenCV BASE is analogous art with respect to Garlapati in view of the following: Sun, Halstead, Goswami, and Burton because they are from the same field of endeavor, namely computer vision. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of OpenCV to incorporate the function cvFindContours to find contours. A person of ordinary skill in the art would do such in order to improve efficiency and efficiently find shape contours.
Claim 15 is rejected using the same rationale or bases as applied to claim 5.
Claim(s) 8 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable Garlapati in view of the following: Sun, Halstead, Goswami, Burton and Yen (US7668369B2), hereinafter referenced as Yen.
Regarding claim 8, Garlapati in view of the following: Sun, Halstead, Goswami, and Burton teach the system of claim 1, and additionally teaches the following. Garlapati teaches
2D silo image
“extracting the shape information from the product data, said extracting including obtaining a 2D outline shape (i.e., the image mask) of the consumer good product” (¶ 18, Garlapati);
Garlapati of a 2D outline shape the consumer good product are extracted (reads on 2D silo image).
Goswami teaches
predicting the shape of the geometric item by measuring proportions of different sections
“to determine (e.g., detect) a shape of the item (e.g., pants) depicted in the image 111, where the shape of the item is the foreground portion of the image 111 . In other words, the foreground portion may depict the item, while the background portion depicts no part of the item. In addition, unless explicitly stated otherwise, all functionality of the analysis module 420 and the score module 430 described with respect to the entirety of the image 111 are similarly applicable to one or more portions of the image 111 (e.g., applicable to the foreground portion, to the background portion, or to both, separately or together).” (¶ 76, Goswami);
Goswami teaches of isolating an object (the foreground portion) from its surroundings (the background portion) and determining the shape of the item (reads on predicting the shape of the geometric item) by measuring proportions of different sections.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Goswami to incorporate predicting the shape of the geometric item by measuring proportions of different sections by isolating an object from its surroundings and determining the shape of the item by measuring proportions of different sections. A person of ordinary skill in the art would do such in order to improve image quality.
Halstead teaches
forming a 2D outline shape by computing the aspect ratio of each portion of the 2D silo image; and
“Step 112 uses the exported raw product 2D shape information and forms a product image mask file. The product image mask file content is numeric data describing the 2D outline shape (2D image mask) of the product package extracted at pixel level from the product photograph(s)/images referenced in product table 10. Masks are calculated for both width (X coordinate) and height (Y coordinate) dimensions. Step 112 also calculates an additional width mask for cutout regions to determine the location of holes or other special features which occur inside the product 2D outline boundaries.” (¶ 71, Halstead);
Halstead teaches of extracting a mask of the silhouetted product image (reads on predicting the shape of the geometric item) by computing the width and height dimensions (reads on aspect ratio) of the cutout regions (reads each portion of the 2D silo image).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Halstead to incorporate extracting a mask of the silhouetted product image by computing the width and height dimensions of the cutout regions. A person of ordinary skill in the art would do such in order to improve image quality.
Garlapati in view of the following: Sun, Halstead, Goswami, and Burton fail to teach the filtering out texture images using metadata corresponding to the image. But Yen does. Yen teaches the following:
filtering out texture images using metadata corresponding to the image
“In some implementations, respective conditions are applied on two types of metadata (i.e., a light received metadata and a brightness metadata) that together provide high classification accuracy with respect to scene type classes, such as night scenes and snow scenes, from which image content features, such as color and texture, typically cannot be extracted” (¶ 16, Yen); “FIG. 1 shows an embodiment of a scene classification system 10 that includes a metadata-based classifier 12, which classifies a digital image 14 into a particular scene type class (i.e., scene type class A) that is defined by a metadata predicate 16 that defines (or specifies) the scene type class. In operation, the metadata-based classifier 12 determines whether to accept the image 14 into the specified scene type class or to reject the image 14 from the specified scene type class.” (¶ 21, Yen);
Yen teaches of filtering an image into or out of a designated class using image texture data found within metadata of the image (reads on filtering out texture images using metadata corresponding to the image).
Yen BASE is analogous art with respect to Garlapati in view of the following: Sun, Halstead, Goswami, and Burton because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Yen to incorporate filtering an image into or out of a designated class using image texture data found within metadata of the image. A person of ordinary skill in the art would do such in order to improve image quality.
Claim 18 is rejected using the same rationale or bases as applied to claim 8.
Claim(s) 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable Garlapati in view of the following: Sun, Halstead, Goswami, Burton, Hong (US8805112B2) hereinafter referenced as Li, and Jai (US20240127438A1) hereinafter referenced as Jai.
Regarding claim 9, Garlapati in view of the following: Sun, Halstead, Goswami, and Burton teaches the system of claim 1, and additionally teaches the following. Sun teaches
wherein the operations further comprise performing a surface overlap validation on the 2D silo image to catch new artifacts forming on the 2D silo image of the geometric item, wherein the new artifacts comprise deviations from shape predictions, and wherein the surface overlap validation comprises a second quality check point on each 2D silo image that is automatically implemented
“the computing system can train the RPN to recognize if an anchor overlaps an object in the image, and if so, how to shift and scale the anchor to most effectively cover the object...In the current example, machine learning can be used to improve the proposal generation (e.g., candidate object detection) in images.” (86, Sun); “the computing system can assign a positive label to an anchor 508 that has a highest Intersection-over-Union (IoU) overlap with a ground-truth item, and an anchor 508 that has an IoU overlap higher than a pre-determined percentage with a ground-truth box.” (¶ 89, Sun); “All of the methods and processes described above can be embodied in, and fully automated via, software code modules executed by one or more general purpose computers or processors.” (¶ 167, Sun);
Sun teaches of an automatic computation for an Intersection-over-Union (IoU), an overlap spatial metric, between a predicted region/shape and the ground-truth shape (this overlap computation reads on performing a surface overlap validation on the 2D silo image). A low IoU result indicates the region (this region is the new artifacts) deviates from the true shape (reads on the new artifacts comprise deviations from shape predictions).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Sun with the feature of Garlapati in view of the following: Sun, Halstead, Goswami, and Burton to incorporate an automatic computation for an Intersection-over-Union between a predicted region/shape and the ground-truth shape. A person of ordinary skill in the art would do such in order to improve image quality.
Garlapati in view of the following: Sun, Halstead, Goswami, and Burton fail to teach the following: performing a sharpness validation on the 2D silo image to validate a degree of resolution of the 2D silo image, where the sharpness validation comprises a quality check point that is automatically implemented, and a second quality check point, and a third quality check point.
But Hong does. Hong teaches the following:
and performing a sharpness validation on the 2D silo image to validate a degree of resolution of the 2D silo image, where the sharpness validation comprises a quality check point that is automatically implemented.
“predicting whether a test image (318) is sharp or blurred includes the steps of: providing a sharpness classifier (316) that is trained to discriminate between sharp and blurred images; computing a set of sharpness features (322) for the test image (318) by (i) generating a high pass image (404) from the test image (318), (ii) generating a band pass image (406) from the test image (318), (iii) identifying textured regions (408) in the high pass image, (iv) identifying texture regions (410) in the band pass image, and (v) evaluating the identified textured regions in the high pass image and the band pass image to compute the set of test sharpness features (412); and evaluating the sharpness features using the sharpness classifier (324) to estimate if the test image (318) is sharp or blurry (20)” (Abstract, Li); “the control system 24 uses a learning-based image sharpness classification algorithm that automatically detects whether the image 14, 16 is sharp or blurred.” (¶ 18, Li)
Hong teaches of an automated classifier that validates whether an image is sharp or blurry (reads on performing a sharpness validation on the 2D silo image) by computing a set of sharpness features (reads on a degree of resolution of the 2D silo image) without manual inspection (reads on a quality check point that is automatically implemented).
Hong BASE is analogous art with respect to Garlapati in view of the following: Sun, Halstead, Goswami, and Burton because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, and Burton with the feature of Hong to incorporate an automated classifier that validates whether an image is sharp or blurry. A person of ordinary skill in the art would do such in order to improve image quality.
Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Hong fail to teach a second quality check point, and a third quality check point. But Jai does. Jai teaches the following:
a second quality check point, and a third quality check point
“The at least one validation model may be configured to verify the image processing model. That is, the at least one validation model may be configured to determine whether the trained image processing model can accurately perform a corresponding image processing. In some embodiments, the at least one validation model may include a first validation model and/or a second validation model.” (¶ 105, Jia); “For example, the processing device 140 may obtain a processing result by processing the target image using the updated image processing model, generate a modified processing result by modifying the processing result based on a modification instruction inputted by the user, and determine a third validation score of the processing result and a fourth validation score of the modified processing result based on the at least one updated validation model” (¶ 129, Jai);
Jai teaches of a second and a third quality validation model.
Jai BASE is analogous art with respect to Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Hong because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Garlapati in view of the following: Sun, Halstead, Goswami, Burton, and Hong with the feature of Jai to incorporate a second and a third quality validation model. A person of ordinary skill in the art would do such in order to improve image quality.
Regarding claim 10, Garlapati in view of the following: Sun, Halstead, Goswami, Burton, Li, and Jai teaches the system of claim 9, and additionally teaches the following. Garlapati teaches
wherein the sharpness validation is based on a perceptual similarity metric and a structural similarity metric.
“performing, via the processor, a structural similarity analysis of the item, to yield a structural similarity score” (¶ 5, Garlapati); “To account for human perception, the system utilizes a structural similarity index to which the object is compared, where the structural similar index can take into account the impact of changes in luminescence, contrast, and structure within the image being considered. The structural similarity index can be a single score which takes into account all of the individual factors (such as luminescence, contrast, etc.).” (¶ 13, Garlapati);
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUNE NGUYEN whose telephone number is (571)272-8919. The examiner can normally be reached M-TH 7:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUNE NGOC NGUYEN/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights