Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Status of the Claims
1, 3-11, and 13-20 are pending in the subject application, with claims 1 and 11 being the independent claims.
Claims 1 and 11 are amended.
Claims 2,12 are cancelled.
Response to Applicant Remarks
Applicant’s well-articulated remarks have been considered but are unpersuasive for the reasons below.
Applicant argues that the cited art does not teach the features of the claimed invention. (Applicant’s 8/3/26 remarks, p.12, “Chaubard does not disclose that the item parameters associated with each Object are extracted to determine whether the item parameters when combined matches corresponding item parameters stored in an item identification database in which each Object is identified when the item parameters when combined match different combinations of item parameters stored in the item database and fail to identify each corresponding when the item parameters from the images fails to match corresponding item parameters stored in the item identification database.
Rather, Chaubard discloses that the item parameters captured from the previous timestep that are then matched with the item parameters captured from the current timestep based on a match score to identify the item to then generate a bounding box for each unknown item.”) The examiner respectfully disagrees.
Chaubard clearly discloses that multiple stored parameters of objects could be evaluated in order to determine a match. (Chaubard, col.8ln.5-col.10,ln.9, “(29) The system performs product detection and recognition that outputs a virtual bounding box around every product in the image and then recognizes the contents of each bounding box by its visual cues, by its “likeness” or “visual affinity” to a template, a classifier, or known/labeled image, or by recognizing the barcode, or by matching to a specific known shape or material, or by reading the information on the packaging of the product via OCR, or some combination of all the aforementioned.
(30) The preferred embodiment works in the following way. First a Convolutional Neural Network (CNN) is run to infer if there exists a product at each part of the image, and if so impose a bounding shape to fit around each distinct product (perhaps a box or polygon, and if using depth, a bounding cuboid or prism or frustrum. Each bounding shape is processed individually, running optical character recognition (OCR) on it to attempt to read any words on the packaging of the product and using a barcode detection algorithm to try to find and recognize barcodes if in the image, if such exist or are able to be read. In addition, a template matching algorithm is applied, taking a database of labeled “known” products or by using a classifier that was already trained to detect those “known” products and attempting to match each one to the pixels in the bounding shape. This template matching process takes in a known image and the bounding box image and outputs a probability of match. This could be done with a Siamese CNN or by a more well studied, more classical, descriptor and homography based approach. Another approach is to train a CNN to output the probability per product over all the possible products. However, maintaining and training such a large CNN is difficult, especially since we would need to retrain this every time one of the products changes or we want to add one new product. This much retraining may not be scalable. Thus, the system can employ an “embedding” Convolutional Neural Network that is a function that maps from a set of pixels, video, or depth pixels to a compact description or embedding vector that can be used to quickly compare the new unknown embedding vector to millions of other “known” embedding vectors that came from images with labels from the master image library with a distance function like cosine distance or Lp-Norm or otherwise. Such a comparison from between the unknown embedding vector to millions of known embedding vectors can be done in milliseconds on a Graphical Processing Unit (GPU) or other parallelized processing system.
(31) The output of this portion of the processing per product bounding box is then a series of bounding shapes, each with an associated set of “features”, which could be written words and their locations in the images, such as “Tide” or “Coca-Cola”, a bounding box around the barcode inside the product bounding shape if it is there, and the results of the template matching process which is a list of probabilities or distances of a match with each of the known classes. Then all of this information about this product within the bounding box is taken together and the system tries to infer the correct SKU (Stock Keeping Unit), UPC, or other product identifier of the product. To do this, the system preferably performs a “water-fall” classification scheme that works in the following way. First, if the barcode is read, then it is taken as truth and the process is finished. In some embodiments, the system will still continue the processing even after detecting a barcode to detect a form of theft called “tag switching”, a trick that people looking to steal from the store use, which consists of adding a tag of an inexpensive item (like ground beef) on an expensive item (like filet mignon) to get a lower price. If the barcode is not detected, the system tries to match the text read from the OCR to a database of known information about all the “known” products and to match the text to a dictionary or lexicon via Jaccard Similarity or otherwise. If there is a match above a certain threshold, then the process is finished and we have our match. In some embodiments, we will continue our processing to increase system accuracy. If there is no text or the Jaccard Similarity is not high enough, then the template matching pipeline or embedding vector method is run to produce a set of classifications, scores, or distances of all “known” products and the system will merge that information with other known information such as previous “track information” (which we will define further below) or the OCR Jaccard Similarity information or barcode information or any other information to produce a best estimated probability across all “known” products and take the argmax of those probabilities over all known products/SKUs to make a final guess as to what the product in the bounding shape is. Finally, the cleaned output of this pipeline is a list of bounding shapes for each product in the image, a guess of a product identifier per bounding shape, and an associated probability per guess per bounding shape. If this occurs at time (t), we will call this output Ft”)
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 13-21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 is dependent on now cancelled claim 2.
Applicant has cancelled claim 12 and the matter therein has been incorporated into independent claim 11. However, the current claim set recites claims 13-21, which appear to correspond to claims 12-20 of the prior claim set. The examiner understands it was Applicant’s intention to cancel claims 2, 12 and a numbering error may have been introduced. Accordingly, for the purpose of this action, the examiner will consider claims 14-21 of the current claim set to be effectively numbered 13-20 and claim 12 to be cancelled.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1,3-7,10-11,13-17,20 are rejected under 35 U.S.C. 102a1 as being anticipated by
Chaubard US11481751B1
Regarding Claim 1,
a plurality of cameras positioned at the POS system configured to capture the plurality of images of the plurality of items positioned at the POS system,
wherein each camera captures a corresponding image with a corresponding Field of View (FOV) of the plurality of items thereby capturing different item parameters associated with each item; and
Chaubard is directed to a checkout station that uses sensors and imagery to detect products. (Chaubard, abstract, “A retail store automated checkout system uses images, video, or depth data to recognize products being purchased to expedite the checkout process and improve accuracy. All of a store's products, including those not sold in packages, such as fruits and vegetables, are imaged from a series of different angles and in different lighting conditions, to produce a library of images for each product. This library is used in a checkout system that takes images, video, and depth sensor readings as the products pass through the checkout area and remove bottlenecks later in the checkout process. Recognition of product identifiers or attributes such as barcode, QR code or other symbols, as well as OCR of product names, as well as the size and material of the product, can be additional or supplemental devices for identifying products being purchased.”; col.4 lns. 1-26, “In another implementation of the invention, an existing retail self-checkout station is retrofitted with equipment of the invention. In the upgraded form, the self-checkout station will recognize, by computer vision, produce and other non-coded items. One or more cameras are positioned at the station, at different viewing angles, for imaging each item brought forth by the customer. A scale is already included for produce sold by weight, and preferably an interactive touchscreen is provided for confirmation by the customer in the event, for example, the system does not recognize to a sufficiently high probability that selected produce is oranges or grapefruits, or between two different types of oranges or apples. Again, this further information can be added to the library once the customer has confirmed product identity.”))
at least one processor;
a memory coupled with the at least one processor, the memory including instructions that, when executed by the at least one processor cause the at least one processor to:
extract the plurality of item parameters associated with each item positioned at the POS system from the plurality of images of each item captured by the plurality of cameras positioned at the POS system, wherein the item parameters associated with each item when combined are indicative as to an identification of each corresponding item thereby enabling the identification of each corresponding item,
analyze the plurality of item parameters associated with each item positioned at the POS system to determine whether the item parameters when combined matches corresponding item parameters stored in an item parameter identification database, wherein the item parameter identification database stores different combinations of item parameters with each different combination of item parameters associated with a corresponding item thereby identifying each corresponding item based on a different combination of item parameters associated with each corresponding item.
Chaubard discloses that multiple stored parameters of objects could be evaluated in different combinations to determine a match. (Chaubard, col.8ln.5-col.10,ln.9, “(29) The system performs product detection and recognition that outputs a virtual bounding box around every product in the image and then recognizes the contents of each bounding box by its visual cues, by its “likeness” or “visual affinity” to a template, a classifier, or known/labeled image, or by recognizing the barcode, or by matching to a specific known shape or material, or by reading the information on the packaging of the product via OCR, or some combination of all the aforementioned.
(30) The preferred embodiment works in the following way. First a Convolutional Neural Network (CNN) is run to infer if there exists a product at each part of the image, and if so impose a bounding shape to fit around each distinct product (perhaps a box or polygon, and if using depth, a bounding cuboid or prism or frustrum. Each bounding shape is processed individually, running optical character recognition (OCR) on it to attempt to read any words on the packaging of the product and using a barcode detection algorithm to try to find and recognize barcodes if in the image, if such exist or are able to be read. In addition, a template matching algorithm is applied, taking a database of labeled “known” products or by using a classifier that was already trained to detect those “known” products and attempting to match each one to the pixels in the bounding shape. This template matching process takes in a known image and the bounding box image and outputs a probability of match. This could be done with a Siamese CNN or by a more well studied, more classical, descriptor and homography based approach. Another approach is to train a CNN to output the probability per product over all the possible products. However, maintaining and training such a large CNN is difficult, especially since we would need to retrain this every time one of the products changes or we want to add one new product. This much retraining may not be scalable. Thus, the system can employ an “embedding” Convolutional Neural Network that is a function that maps from a set of pixels, video, or depth pixels to a compact description or embedding vector that can be used to quickly compare the new unknown embedding vector to millions of other “known” embedding vectors that came from images with labels from the master image library with a distance function like cosine distance or Lp-Norm or otherwise. Such a comparison from between the unknown embedding vector to millions of known embedding vectors can be done in milliseconds on a Graphical Processing Unit (GPU) or other parallelized processing system.
(31) The output of this portion of the processing per product bounding box is then a series of bounding shapes, each with an associated set of “features”, which could be written words and their locations in the images, such as “Tide” or “Coca-Cola”, a bounding box around the barcode inside the product bounding shape if it is there, and the results of the template matching process which is a list of probabilities or distances of a match with each of the known classes. Then all of this information about this product within the bounding box is taken together and the system tries to infer the correct SKU (Stock Keeping Unit), UPC, or other product identifier of the product. To do this, the system preferably performs a “water-fall” classification scheme that works in the following way. First, if the barcode is read, then it is taken as truth and the process is finished. In some embodiments, the system will still continue the processing even after detecting a barcode to detect a form of theft called “tag switching”, a trick that people looking to steal from the store use, which consists of adding a tag of an inexpensive item (like ground beef) on an expensive item (like filet mignon) to get a lower price. If the barcode is not detected, the system tries to match the text read from the OCR to a database of known information about all the “known” products and to match the text to a dictionary or lexicon via Jaccard Similarity or otherwise. If there is a match above a certain threshold, then the process is finished and we have our match. In some embodiments, we will continue our processing to increase system accuracy. If there is no text or the Jaccard Similarity is not high enough, then the template matching pipeline or embedding vector method is run to produce a set of classifications, scores, or distances of all “known” products and the system will merge that information with other known information such as previous “track information” (which we will define further below) or the OCR Jaccard Similarity information or barcode information or any other information to produce a best estimated probability across all “known” products and take the argmax of those probabilities over all known products/SKUs to make a final guess as to what the product in the bounding shape is. Finally, the cleaned output of this pipeline is a list of bounding shapes for each product in the image, a guess of a product identifier per bounding shape, and an associated probability per guess per bounding shape. If this occurs at time (t), we will call this output Ft”)
identify each corresponding item positioned at the POS system when the plurality of item parameters from the plurality of images of each item captured by the plurality of cameras when combined matches corresponding item parameters as stored in an item identification database …
(Chaubard,col.15,ln.5-col.16,ln.30, “4. Per Object, Perform the Following Analysis a. Measure Length, Width, and Height of the Object and x, y, z of the centroid in global coordinate system This is very straightforward to do once you have a merged track. b. 3D CNN to attempt to classify the Object as a specific UPC and deliver a probability distribution over the possible UPCs This is very straightforward to do once you have a merged track with template matching or a CNN trained on those UPCs. c. 3D CNN to identify the superclass of the Object as a type of product, the colors of the product, the material it is made of, the shape of it, etc. We would train a 3D CNN to output superclass information like this. d. OCR to read any text on the Object We would use an Optical Character Recognition (OCR) algorithm on each image captured at time t. There are many OCR algorithms to choose from. In one embodiment, we would use a CNN to infer a rotated bounding box [x1, y1, x2, y2, angle] for every word in the image. We would then input each bounding box crop in the image into a Convolutional Recurrent Neural Network (CRNN) to output the character sequence inside the box. In some embodiments, we may use a lexicon of possible words and use a Jaccard Similarity between each word in the lexicon and the outputted word to find the true word. If the detected words are near each other, we may merge them into a phrase. In all embodiments, we would take each detected set of words or phrases and assign it to the Object from which the bounding box has the highest Intersection Over Union with. e. Barcode Detection on all sides of the Object Same as OCR except we look for barcodes, not text. f. 3D CNN and 2D CNN to compute an embedding vector for the Object In some embodiments we would use a 3D CNN to compute an embedding vector per Object. This 3D CNN would be trained by taking a database of known Objects some of the same UPCs. At each stage of training we would select two training Objects of the same UPC and one training Object of a different UPC, and use a loss function like triplet embedding loss to give penalty to the 3D CNN if the model does not recognize the same Objects as the same and the different Object as different. In other embodiments we would use a 2D CNN to detect general purpose bounding boxes per camera. Then we would match each bounding box across all cameras to a specific Object. For each bounding box, we would compute an embedding vector that would encode and represent the set of pixels in the bounding box as a vector of n numbers. The 2D CNN would be trained in a similar process as the 3D CNN but instead the training Objects would be training bounding boxes. In some embodiments we will use both approaches”)
and fail to identify each corresponding item when the plurality of item parameters from the plurality of images of each item when combined fails to match corresponding item parameters,
(Chaubard, summary, “If an item in the checkout area is recognized to a sufficiently high probability, meeting a preset threshold of, e.g., 90%, that item is added to the checkout list and charged to the customer. If probability is too low, below the preset threshold, the system may prompt the customer or cashier to reposition the item if it appears stacking or occlusion has occurred, or to remove and scan that item using the store's existing scanning system.”)
map a plurality of image pixels associated with each unknown item as extracted from each image of each unknown item as captured by the plurality of cameras to a plurality of real world coordinates associated with each unknown item as extracted from a position of each unknown item as positioned at the POS system,
(Chaubard, col.14,lns.10-38, “Every camera in the system has its own coordinate system. To fuse this information for each camera in a coherent way, the system will have a fixed and known position and orientation (position=x,y,z, orientation=α, β, γ and scaling=s.sub.x, s.sub.y, s.sub.z) from the global origin to each cameras coordinate system. These 9 numbers will be theoretically static for the life of the system but may require a recalibration from time to time. These parameters can be used to compute an Extrinsic Matrix, H, per camera which includes the following transformations: Rotation, Translation, and Scale. Depending on the type of camera used, each camera may also have a Intrinsic Matrix, K, which defines the deformation from the real world 3D space to the 2D space on the cameras sensor that is not modeled by H. Then H and K together will be used to convert each camera's specific point cloud in its own coordinate system to the global coordinate system, giving us a final fused point cloud across all cameras D.sub.t=[[R, G, B, X, Y, Z] . . . ], which if the cameras are positioned correctly and the products are not stacked, give us a full 360 volumetric coverage of each product.
(56) Once we have the global fused point cloud D.sub.t, we can then run a clustering algorithm on D.sub.t to detect the existence of distinct objects. There are many clustering algorithms that can be used for this task. In one embodiment, we use a 3D CNN or RNN to cluster each specific point p=(x, y, z, R, G, B) to one of an unknown number of centroids. In other embodiments, we may be able to achieve the same thing with a simpler model like Density-based Scanning (DB-Scan) or Mean Shift.”; col.14,lns.42-50, : In another embodiment, we apply an Oriented 3D volume detection from Pixel-wise neural network predictions where we take each cameras pixels and push them through a fully-convolutional neural network as the backbone and merge this data with a Header Network that is tasked with localizing the Objects size (height, length, width), shape (classification, embedding, or otherwise), and heading (angle/orientation in the global coordinate system in α, β, and γ).”)
generate a corresponding bounding polygon for each unknown item that encapsulates each unknown item within the corresponding bounding polygon based on the image pixels with each unknown item as mapped to the real world coordinates associated with each unknown item, and
project the corresponding bounding polygon onto each unknown item positioned at the POS system to encapsulate each unknown item thereby providing visual feedback for each unknown item positioned at the POS system.
(Chaubard, col.5,ln.55-col.6,ln.2, “If there are overlapping items or stacked items that occlude the camera or cameras, or if there are items in the tunnel that the cameras failed to capture or identify to the threshold probability, or if the tunnel otherwise made an error, then the non-identified items will be shown visually on the screen 20. As an example, a red bounding box can be shown around the unidentified item. The shopper or cashier or auditor in the store, can then manually correct the error by taking the item or items that were not identified and moving them (dashed line 25) by the pre-existing barcode scanner 24 or another scanner, as indicated at 26, therefore correcting the list of items and the amount owed. Alternatively, the system can prompt the customer or cashier to select from a list of screen-shown choices to confirm the correct product from a list of system-generated guesses.”)
Regarding Claim 3, Chaubard discloses the system of claim 2.
calibrate each camera positioned at the POS system to determine the plurality of real world coordinates of the POS system relative to a corresponding position of each camera, wherein the calibration of each camera at the POS system enables the real world coordinates of the POS system to be mapped to the plurality of image pixels as extracted from each image captured by each camera.
(Chaubard, col.14, lns.10-17, “Every camera in the system has its own coordinate system. To fuse this information for each camera in a coherent way, the system will have a fixed and known position and orientation (position=x,y,z, orientation=α, β, γ and scaling=s.sub.x, s.sub.y, s.sub.z) from the global origin to each cameras coordinate system. These 9 numbers will be theoretically static for the life of the system but may require a recalibration from time to time.”)
Regarding Claim 4, Chaubard discloses the system of claim 1.
extract a plurality of metrology features as included in the plurality of item parameters of each item from each image captured by the plurality of cameras for each unknown item positioned
at the POS system, wherein the plurality of metrology features is indicative as to a physical appearance of each unknown item positioned at the POS system.
See prior art rejection of claim 1.
Regarding Claim 5, Chaubard discloses the system of claim 4.
map the plurality of pixels associated with each unknown item as extracted from each image of each unknown item of each unknown item to the plurality of real world coordinates associated with each unknown item from the position of each unknown item based on the metrology features of each unknown item, wherein the metrology features of each unknown item enable the plurality of pixels to be mapped to the real world coordinates of each unknown item based on the physical appearance of each unknown item at the POS system.
(Chaubard, col.14,lns.42-50, : In another embodiment, we apply an Oriented 3D volume detection from Pixel-wise neural network predictions where we take each cameras pixels and push them through a fully-convolutional neural network as the backbone and merge this data with a Header Network that is tasked with localizing the Objects size (height, length, width), shape (classification, embedding, or otherwise), and heading (angle/orientation in the global coordinate system in α, β, and γ).”)
Regarding Claim 6, Chaubard discloses the system of claim 5.
generate the corresponding bounding polygon for each unknown item that encapsulates each unknown item within the corresponding bounding polygon based on the image pixels of the metrology features associated with each unknown item as mapped to the real world coordinates of the metrology features associated with each unknown item thereby enabling the corresponding bounding polygon to encapsulate the physical appearance of each unknown item at the POS system; and
project the corresponding bounding onto each unknown item positioned at the POS system to encapsulate the physical appearance of each unknown item based on the metrology features of the image pixels associated with each unknown item mapped to the metrology features of the real world coordinates of each unknown item thereby providing visual feedback for each unknown item positioned at the POS system.
(Chaubard, col.8,ln.65-col.9,ln.15, “The preferred embodiment works in the following way. First a Convolutional Neural Network (CNN) is run to infer if there exists a product at each part of the image, and if so impose a bounding shape to fit around each distinct product (perhaps a box or polygon, and if using depth, a bounding cuboid or prism or frustrum. Each bounding shape is processed individually, running optical character recognition (OCR) on it to attempt to read any words on the packaging of the product and using a barcode detection algorithm to try to find and recognize barcodes if in the image, if such exist or are able to be read. In addition, a template matching algorithm is applied, taking a database of labeled “known” products or by using a classifier that was already trained to detect those “known” products and attempting to match each one to the pixels in the bounding shape. This template matching process takes in a known image and the bounding box image and outputs a probability of match.”)
Regarding Claim 7, Chaubard discloses the system of claim 6.
analyze an accumulation of metrology features and item parameters as associated with each corresponding item in the item parameter identification database to determine the metrology features of each unknown item positioned at the POS system;
map the plurality of pixels associated with each unknown item as extracted from each image of each unknown item of each unknown item to the plurality of real world coordinates associated with each unknown item from the position of each unknown item based on the accumulation of metrology features and item of each unknown item as stored in the item parameter identification database, wherein the accumulation metrology features and item parameters of each unknown item enable the plurality of pixels to be mapped to the real world coordinates of each unknown item based on the physical appearance of each unknown item at the POS system; and
generate the corresponding bounding polygon for each unknown item that encapsulates each unknown item within the corresponding bounding polygon based on the image pixels of the accumulation of metrology features and item parameters associated with each unknown item as mapped to the real world coordinates of the metrology features associated with each unknown item thereby enabling the corresponding bounding polygon to encapsulate the physical appearance of each unknown item at the POS system.
(Chaubard, col.15,ln.35-col.16,ln.37, “4. Per Object, Perform the Following Analysis a. Measure Length, Width, and Height of the Object and x, y, z of the centroid in global coordinate system This is very straightforward to do once you have a merged track. b. 3D CNN to attempt to classify the Object as a specific UPC and deliver a probability distribution over the possible UPCs This is very straightforward to do once you have a merged track with template matching or a CNN trained on those UPCs. c. 3D CNN to identify the superclass of the Object as a type of product, the colors of the product, the material it is made of, the shape of it, etc. We would train a 3D CNN to output superclass information like this. d. OCR to read any text on the Object We would use an Optical Character Recognition (OCR) algorithm on each image captured at time t. There are many OCR algorithms to choose from. In one embodiment, we would use a CNN to infer a rotated bounding box [x1, y1, x2, y2, angle] for every word in the image. We would then input each bounding box crop in the image into a Convolutional Recurrent Neural Network (CRNN) to output the character sequence inside the box. In some embodiments, we may use a lexicon of possible words and use a Jaccard Similarity between each word in the lexicon and the outputted word to find the true word. If the detected words are near each other, we may merge them into a phrase. In all embodiments, we would take each detected set of words or phrases and assign it to the Object from which the bounding box has the highest Intersection Over Union with. e. Barcode Detection on all sides of the Object Same as OCR except we look for barcodes, not text. f. 3D CNN and 2D CNN to compute an embedding vector for the Object In some embodiments we would use a 3D CNN to compute an embedding vector per Object. This 3D CNN would be trained by taking a database of known Objects some of the same UPCs. At each stage of training we would select two training Objects of the same UPC and one training Object of a different UPC, and use a loss function like triplet embedding loss to give penalty to the 3D CNN if the model does not recognize the same Objects as the same and the different Object as different. In other embodiments we would use a 2D CNN to detect general purpose bounding boxes per camera. Then we would match each bounding box across all cameras to a specific Object. For each bounding box, we would compute an embedding vector that would encode and represent the set of pixels in the bounding box as a vector of n numbers. The 2D CNN would be trained in a similar process as the 3D CNN but instead the training Objects would be training bounding boxes. In some embodiments we will use both approaches!
(67) 5. Ensemble of all Features to Classify the Object Ft Features for this Object
(68) The key to detecting and identifying each Object with high certainty is to take a number of separate approaches and ensemble the results.
(69) The output of this process will be a predicted SKU per Object with a corresponding confidence score.”)
Regarding Claim 10, Chaubard discloses the system of claim 1.
automatically display via a user interface a notification of each unknown item positioned at the POS system as encapsulated within the corresponding bounding polygon as projected onto each unknown item positioned at the POS system.
See prior art rejection of claim 1.
Regarding Claims 11,13-17,20
See prior art rejections of claim 1,3-7,10
Allowable Subject Matter
Claims 8,9,18,19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALLEN C CHEIN whose telephone number is (571)270-7985. The examiner can normally be reached Monday-Friday 8am -5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Florian Zeender can be reached at (571) 272-6790. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALLEN C CHEIN/Primary Examiner, Art Unit 3627
Prosecution Insights