DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-5, 7-8, 10-15, and 17-23 are pending for examination in the application filed 04/02/2026. Claims 1-2, 5, 10, 12-13, and 17-18 have been amended, claims 6, 9, and 16 have been cancelled, and claims 21-23 are new.
Priority
Acknowledgement is made of Applicant’s claim to priority of provisional application 63/358,786, filing date 07/06/2022.
Response to Arguments and Amendments
The 35 U.S.C. 112(b) rejection of claims 1, 12, and 17 has been withdrawn in view of the amendments.
Applicant’s arguments with regards to the limitation “retraining the product model using the product images extracted from within the potential product embedding group of the vector database” in claims 1, 12, and 17 on pages 12-13 of the Remarks filed 04/02/2026 have been considered but are moot because the new ground of rejection does not rely on the combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument, as facilitated by the newly added amendments.
Applicant’s arguments with respect to the “brand axis value” on page 13 of the Remarks filed 04/02/2026 have been considered but they are not persuasive. Applicant argues that Adato does not disclose a brand axis value. Examiner disagrees. Adato teaches using a Support Vector Machine for the product model for product identification ([0250] In yet another example, the product model may include support vectors that may be used by a Support Vector Machine to identify products). A Support Vector Machine transforms data onto the axes of the feature space. Adato also teaches that the product may be identified by the brand identifier ([0123] Additionally or alternatively, the term “identifying a product” may refer to determining a unique identifier associated with a specific brand of products that allows inventory managers to uniquely refer to products, e.g., based on a specific brand in a product catalogues). Furthermore, please see the 35 U.S.C. 112(a) rejection below in regards to this newly added limitation.
Applicant states on page 13 of the Remarks that Savvides, Adato, Haghighat, and Tiwari do not teach the amended limitations of claim 2. Examiner disagrees. Adato teaches transmitting a payment in response to the compliance exceeding a compliance threshold ([0238] FIG. 11E illustrates an example GUI 1150 for output device 145D used by an online customer of retail store 105. Traditional online shopping systems present online customers with a list of products. Products selected for purchase may be placed into a virtual shopping cart until the customers complete their virtual shopping trip. [0733] Reference is now made to FIG. 40A, which illustrates a timeline associated with the exemplary solution of the present disclosure. Initially, the suggested method includes receiving image data 4000 to determine a current inventory in retail store 105, and receiving product supply information 4002 to determine a predicted inventory. Thereafter, at the end of a first time period, a customer 4004 of a virtual store may complete an online order. [0734] The following demonstrates the two example embodiments. Assume customer 4004 is considering buying avocados online. At the time customer 4004 is ordering products from a virtual store (i.e., during the first time period), she wants to know if the store has avocados. Independently from or in connection with the activities of customer 4004, the virtual store may receive information indicating that at the time the order of customer 4004 is expected to be collected (i.e., during the second time period), retail store 105 is expected to have avocados in-stock but their quality is estimated to be below average (e.g., based on a determined average shelf time for in stock avocados and based on predetermined metrics such as average time to ripeness, based on predicted future restocking events, etc.). According to the first embodiment, the virtual store may present to customer 4004 at the first time period an indication that avocados are in-stock. According to the second embodiment, the virtual store may present to customer 4004 at the first time period an indication that the avocados' quality is estimated to be below average. Armed with this knowledge, customer 4004 can complete her online shopping and decide whether to buy or not to buy avocados at the virtual store. [0751] Predicted inventory determination module 4106 may also calculate a confidence level associated with the predicted inventory and may initiate a first automated action (e.g., updating virtual store 4116, adjust a price, etc.) when the confidence level is above a threshold. [0752] In one example, when the predicted quality of the product type is below the threshold, the at least one processor may prevent virtual store 4116 from selling the at least one product type). Please see below for the updated 35 USC § 103 rejections of claims 1-5, 7-8, 10-15, and 17-23 in view of the newly added amendments.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-5, 7-8, 10-15, and 17-23 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claims 1, 12, and 17 include the amended limitation “each product embedding comprising a product identifier and a plurality of axis values for the vector database, wherein the product identifier is one of unidentified and linked to product data and the plurality of axis values comprise at least a plurality of product image axis values, a plurality of product placement axis values, and a brand axis value”. The Remarks filed 04/02/2026 refer to [0053-0054] of the specification for support of this newly added amendment. The specification describes [0053] Figure 2E is a schematic block diagram illustrating one embodiment of the product embedding 235. The product embedding 235 may include an embedding identifier 239, the product image 107 from which the product embedding 235 is created, a corresponding product identifier 241, a novel distance 243, and one or more axis values 237. The specification also describes [0056] The axis values 237 of each product embedding 235 position the product embedding 235 within the virtual latent space of the vector database 105. [0063] The axis value neurons 439 generate the axis values 437 for the product image 107. Although for simplicity only two axis value neurons 439 are shown, any number of axis value neurons 439 may be employed. The specification does not describe the plurality of axis values comprise at least a plurality of product image axis values, a plurality of product placement axis values, and a brand axis value, as recited in amended claims 1, 12, and 17. Claims 2-5, 7-8, 10-11, 13-15, and 18-23 depend from independent claims 1, 12, and 17 and are therefore also rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 7-8, 10-15, and 17-23 are rejected under 35 U.S.C. 103 as being unpatentable over Savvides (US20220058425A1) in view of Adato (US20190149725A1), Haghighat (US20240370740A1), and Asif (US20220101184A1).
Regarding claim 1, Savvides teaches a method ([0002] The present invention is related to the field of automated inventory monitoring in a commercial retail setting and, in particular, is directed to systems, processes and methods for automatically tracking products displayed in the retail setting) comprising: training, by use of a processor (data processing module 136), a product model comprising a product detector ([0072] In preferred embodiments, product detector 402 is a machine learning model trained on images of products), a price detector ([0077] Shelf labels are detected in a similar manner using label detector 406, shown as a component of pipeline 400 in FIG. 4. Shelves in stores typically have two kinds of labels. The first type of label is a price label for the products, which are referred to herein as “shelf labels”), a shelf detector ([0021] FIG. 8 is an example of the output of the shelf segment classifier of the present invention, showing a binary mask having highlighted areas indicating which pixels in the panoramic image are located on a shelf), a dimension estimator ([0071] In some embodiments, the bounding box is represented in a data structure as a tuple of data of the form BB={x, y, w, h}. A tuple may comprise, for example, the x,y coordinates of a corner of the bounding box as well as the width (w) and the height (h) of the bounding box), and an orientation classifier ([0011] The system is then able to match the placement of and, optionally, the identity of the products on the fixtures with the expected positions of the products to determine that the products are shelved properly, are miss-shelved or are out-of-stock),
wherein the product model embeds product embeddings of a same product close to another product in a latent space of a vector database (product library) ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't. [0012] The extracted features from the product images are compared to extracted features in a product library and a best-fit is obtained);
generating a product embedding for a plurality of product images ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't) of segmented products using the product model ([0143] The process 2000 for automatically enrolling new products is shown in flowchart form in FIG. 20. Given an image of a shelf, for example, panoramic image 300, at step 2002 of process 2000, product detection model 402 is run on panoramic image 300 to infer regions of interest in the shelf image, indicated by bounding boxes 504 around the product images 502. An example of the output of the product detection model 402 is shown in FIG. 5. Bounding boxes 504 identified by product detection model 402 are the input regions of interest for the next step of process 2000. Preferably, each region of interest will contain an image of a product. At step 2004 of process 2000, feature extractor 1704 is then run on each region of interest and features are extracted from the region of interest in accordance with the training of feature extractor 1704);
generating the vector database of the product embeddings for the plurality of the product images, wherein the vector database comprises product embeddings of known products and unknown products ([0124] The product library comprises features of images representing variations of a product, for example, pose or labeling variations, and an associated prime ID. The group of features of the product images are obtained using a trained machine learning model (i.e., a feature extractor) which is trained to extract the features from the images and to output a prime ID associated with the product images. [0125] Product images extracted from the acquisition image are feature matched against product source images in the product library and ranked based on the highest confidence in the association between product images from the acquisition image and source product images. Product images with the same product and label primary association are grouped together, for example, by placing the image features into one folder or assigning the same digital designation (or tag) for one particular product or a type of product. [0144] When a new product has been discovered, it is enrolled, at step 2012 of process 2000, in product library 1708, along with identifying information. In some embodiments, the new products may be assigned identifying information which may be random, or which may be obtained from another source);
generating a new product embedding for a new product or different views or packaging of already known products ([0128] In another aspect of this embodiment, a method of building a product library that can be used for general machine learning training is disclosed. In this aspect, a system is provided to take images of new or re-packaged products and learns to recognize the new product with the new packaging or appearance. This ongoing learning feature makes it easy for the system to stay updated with new products or products having a new appearance);
querying the vector database with the new product embedding as a centroid for a proximity query, wherein the new product embedding is a novel distance from other product embeddings in the vector database ([0143] At step 2006 of process 2000, the extracted features are used to search for a best-fit (i.e., a closest match) in product library 1708. If, at step 2008 of process 2000, the distance between the extracted features and the features of the best-fit in product library 1708 are above a predetermined distance threshold, the object in the region of interest is determined to be a new product);
adding the new product to the product detector using product images extracted from within a product embedding group of the vector database ([0144] When a new product has been discovered, it is enrolled, at step 2012 of process 2000, in product library 1708, along with identifying information. In some embodiments, the new products may be assigned identifying information which may be random, or which may be obtained from another source. In other embodiments, the bounding boxes 504 containing the product images may be associated with a shelf label 1108 by product-label association algorithm 422, as shown in FIGS. 10 and 11, and the identifying information extracted from the shelf label 1108);
and detecting the new product using the product detector ([0013] In yet another aspect of the invention, a system and method for dynamically enrolling new object images in the product library through zero- or low-shot training is disclosed. [0120] In some aspects of this embodiment, the product library may be initiated, expanded, and optimized automatically. The product library can further be deployed into a product recognition and detection process that significantly improves the accuracy of out-of-stock detection).
Savvides does not teach a Stock Keeping Unit (SKU) classifier, a brand classifier, a refrigerator detector; and the plurality of axis values comprise at least a plurality of product placement axis values, and a brand axis value.
Adato, in the same field of endeavor of classifying products from images, teaches a Stock Keeping Unit (SKU) classifier ([0818] The processing device may also be configured to determine stock keeping units (SKUs) for the plurality of differing products based on the unique identifiers (other than SKU bar codes) in the image), a brand classifier ([0400] In some aspects, system 100 may differentiate between products based on a brand name associated with a product…a system may determine that a subset of the milk is associated with a particular supplier, e.g., Hiland Dairy®, by detecting a logo or brand name associated therewith), a refrigerator detector [0301] By way of another example, when image processing unit 130 identifies that the shelf in the image has “a glass door in the front,” image processing unit 130 may determine the shelf in the image is in a refrigerator);
and the plurality of axis values comprise at least a plurality of product placement axis values, and a brand axis value ([0123] Embodiments of the present disclosure further include analyzing images to detect and identify different products. As used herein, the term “detecting a product” may broadly refer to determining an existence of the product. For example, the system may determine the existence of a plurality of distinct products displayed on a store shelf…Additionally or alternatively, the term “identifying a product” may refer to determining a unique identifier associated with a specific brand of products that allows inventory managers to uniquely refer to products, e.g., based on a specific brand in a product catalogue. [0250] In yet another example, the product model may include support vectors that may be used by a Support Vector Machine to identify products).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Adato to use an SKU classifier, a brand classifier, and a refrigerator detector and use product placement axis values and brand axis values "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Savvides does not teach each product embedding comprising a product identifier and a plurality of axis values for the vector database, wherein the product identifier is one of unidentified and linked to product data and the plurality of axis values comprise at least a plurality of product image axis values; wherein the vector database comprises product embeddings of known products and unknown products at the axis values for each product embedding; generating a new product embedding in a potential product embedding group for a new product or different views or packaging of already known products; labeling close product embeddings from the vector database as the new product in the potential product embedding group, wherein the close product embeddings are generated from previously captured product images, have an unidentified product identifier that is not linked to product data, are not labeled as the new product, and are within a novel distance threshold to a centroid of the potential product embedding group; adding the new product to the product detector using product images extracted from within the potential product embedding group of the vector database.
Haghighat, in the same field of endeavor of classifying products from images, teaches each product embedding comprising a product identifier and a plurality of axis values for the vector database, wherein the product identifier is one of unidentified and linked to product data and the plurality of axis values comprise at least a plurality of product image axis values; wherein the vector database comprises product embeddings of known products and unknown products at the axis values for each product embedding; generating a new product embedding in a potential product embedding group for a new product or different views or packaging of already known products ([0027] The present CNN 22, as shown in FIG. 4 and, further, in FIGS. 5-7, is configured to operate without using a Softmax probability, but rather to “embed” vectors 26 corresponding with the initial data set on which the CNN 22 is trained, as well as new vectors 26 derived from later images 24 analyzed by the CNN 22, in a feature space having a number of dimensions equal to the number of features included in the vectors 26 realized by the feature learning layers of the CNN 22…In this respect, the value for a given feature (i.e., cell) in a given vector 26 can indicate the position of the vector 26 within the corresponding dimension, with the complete vector 26 locating the embedded entry within the complete feature space. [0029] Because of the training process applied to the CNN 22, any new vectors 40 will be embedded in the hypersphere 34 according to the similarity in perception of the images 24 to the existing vectors 26. In this respect, if the new image 24 is of a item corresponding with a known class-in the present example of a cooking appliance and, in particular, oven 10, then the resulting new vector 40 will be embedded within or close to the cluster 36 consisting of the other vectors 26 representing earlier images 24, including the training images. In this respect, the CNN 22 is configured to output an identification of the image as corresponding with a known class according to the closest cluster 36 to the newly-embedded vector 40, within a predetermined threshold distance or in combination with a probability that that the newly-embedded vector 40 fits within the class of the closest cluster 36. If the new vector 40 is not sufficiently close to an existing cluster 36 (i.e., is positioned within the open space 38 surrounding the clusters 36) and/or has a high probability of not fitting within the closest existing cluster 36, the CNN 22 can return an output indicating that the vector 40 and originating image 24 do not correspond with one of the known classes);
labeling close product embeddings from the vector database as the new product in the potential product embedding group, wherein the close product embeddings are generated from previously captured product images, have an unidentified product identifier that is not linked to product data, are not labeled as the new product, and are within a novel distance threshold to a centroid of the potential product embedding group ([0030] Accordingly, the CNN 22 can, in various configurations, treat new vectors 40 within the open space 38 as comprising or being within a new cluster 42 that corresponds with a new, previously-unknown food type. Notably, such treatment is generally inherent in the CNN 22 configuration described herein, as the new vectors 40 and original (trained) vectors 26 are treated the same during use of the CNN 22 to perceive and identify subsequent images 24. The CNN 22, however, can be configured to require a certain number of new vectors 40 within a specified distance from each other or with specified relative distribution characteristics (as discussed further below) before taking steps to specifically designate the particular new vectors 40 as a new cluster 42, such as “registering” the cluster as a specific class and/or querying the user for the name or classification of the new cluster, for example. [0031] Subsequently, upon embedding the new vector 40 in the hypersphere 34, the CNN 22 can compute the Mahalanobis distance between the new vector 40 and, by nature of the calculation, a centroid 44 of each known cluster 36…If no cluster 36 is within the threshold Mahalanobis distance, then the new vector 40 is registered as a new class. In yet another variation, the Mahalanobis distance of the new vector 40 can be assessed in terms of the standard deviation of the vectors 26 of the particular cluster 36 to determine, for example, if the new vector 40 is an outlier with respect to even the closest cluster 36, which may indicate that the new vector 40 corresponds with a new class. [0038] When the specific requirements of the particular appliance 10 are met (which may include a user indication via HMI 46 that a single new vector 40 should be registered as a new food type) (step 128), the vector 40 or plurality of proximate, unregistered new vectors 40 are registered as a new food product type that is added to the plurality of known food product types accessible by the cooking appliance 10 (step 134);
adding the new product to the product detector using product images extracted from within the potential product embedding group of the vector database ([0038] As discussed above, this step may include cataloging the vector 40 as a new, but still unidentified, food type and not yet registering the vector 40 as a cluster until a subsequent operation embeds another new vector 40 in close proximity to that previously-registered new vector 40 (or another predetermined number of other new, unregistered vectors 40). When the specific requirements of the particular appliance 10 are met (which may include a user indication via HMI 46 that a single new vector 40 should be registered as a new food type) (step 128), the vector 40 or plurality of proximate, unregistered new vectors 40 are registered as a new food product type that is added to the plurality of known food product types accessible by the cooking appliance 10 (step 134). This may include prompting the user to enter a name of the new food type for association with the new cluster 42 during registration (step 130). Notably, and as discussed above, the addition of the new food type to the CNN 22 is done without retraining the identification model 31 or the perception model 27).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Haghighat to label close product embeddings from the vector database as the new product embeddings in a potential product embedding group and add the new product to the product detector "to identify a food type as unknown and register the food type as new without retraining the identification model of the convolutional neural network" [Haghighat 0001] because "Even at the manufacturer or programmer level, training a new identification model takes a considerable amount of time and computation resources" [Haghighat 0003].
Savvides does not teach retraining the product model using the product images extracted from within the potential product embedding group of the vector database.
Asif, in the same field of endeavor of classification models, teaches retraining the model using the images extracted from within the potential embedding group of the vector database ([0024] Further aspects of the present disclosure relate to systems and methods to improve model performance based on measured data. More particular aspects of the present disclosure relate to a system to receive unlabeled data, compare the unlabeled data to known data, label the unlabeled data based on the comparison, and retrain the model based on the newly-labeled data. [0090] FIG. 6 illustrates a high-level method 600 of updating a machine learning model based on de-identified feature data derived from unlabeled input data, consistent with several embodiments of the present disclosure. In general, method 600 includes several operations that may be substantially similar to operations performed in method 400. However, method 600 retrains a model based on unlabeled “de-identified” features (i.e., feature data) that are themselves based on collected data (i.e., input data). [0097] Method 600 further comprises retraining the machine learning model via the newly labeled (“weak”-labeled) feature data at operation 614. In some embodiments, operation 614 may further include retraining via the hard-labeled feature data and feature data from one or more teacher models. Operation 614 may include, for example, inputting the weak-labeled features to the “next layer” of the model. As an example, the now-labeled feature data generated at operation 606 may be output from a 37th layer of a 50-layer model. Operation 614 may include inputting the weak-labeled feature data into the 38th layer of the model, receiving an output from the model, calculating an error between the output and the label, adjusting one or more of layers 38-50 of the model based on the error, and repeating until the error no longer decreases. [0028] As an example, input data may be an image, wherein a first layer may identify edges in the image, producing a list (“vector”) of data describing all the edges detected in the image (“features”). The features generated by the first layer are then fed into a second layer, which performs further manipulation and produces a second feature vector, and so on).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product model of Savvides with the teachings of Asif to retrain the model using the images extracted from the potential embedding group because "The machine learning model may have already been trained to analyze or classify input data such as the data collected via operation 402. As the nature of the input data changes, the model may lose accuracy over time, which can be alleviated by retraining" [0076].
Regarding claim 2, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Savvides further teaches receiving a shelf image; determining a product placement; and determining compliance with placement requirements ([0011] In preferred embodiments of the invention, the system analyzes the panoramic images to detect the presence of and, optionally, to determine the identity of products placed on the fixtures. Additionally, the system can analyze the panoramic images to identify shelf labels indicating which products are expected to be at various positions on the fixtures. The system is then able to match the placement of and, optionally, the identity of the products on the fixtures with the expected positions of the products to determine that the products are shelved properly, are miss-shelved or are out-of-stock. The system is further functional to flag misplaced and out-of-stock products and alert the store's staff such that the misplacement may be corrected or such that the product may be re-stocked).
Savvides does not explicitly teach transmitting a payment in response to the compliance exceeding a compliance threshold.
Adato teaches transmitting a payment in response to the compliance exceeding a compliance threshold ([0238] FIG. 11E illustrates an example GUI 1150 for output device 145D used by an online customer of retail store 105. Traditional online shopping systems present online customers with a list of products. Products selected for purchase may be placed into a virtual shopping cart until the customers complete their virtual shopping trip. [0733] Reference is now made to FIG. 40A, which illustrates a timeline associated with the exemplary solution of the present disclosure. Initially, the suggested method includes receiving image data 4000 to determine a current inventory in retail store 105, and receiving product supply information 4002 to determine a predicted inventory. Thereafter, at the end of a first time period, a customer 4004 of a virtual store may complete an online order. [0734] The following demonstrates the two example embodiments. Assume customer 4004 is considering buying avocados online. At the time customer 4004 is ordering products from a virtual store (i.e., during the first time period), she wants to know if the store has avocados. Independently from or in connection with the activities of customer 4004, the virtual store may receive information indicating that at the time the order of customer 4004 is expected to be collected (i.e., during the second time period), retail store 105 is expected to have avocados in-stock but their quality is estimated to be below average (e.g., based on a determined average shelf time for in stock avocados and based on predetermined metrics such as average time to ripeness, based on predicted future restocking events, etc.). According to the first embodiment, the virtual store may present to customer 4004 at the first time period an indication that avocados are in-stock. According to the second embodiment, the virtual store may present to customer 4004 at the first time period an indication that the avocados' quality is estimated to be below average. Armed with this knowledge, customer 4004 can complete her online shopping and decide whether to buy or not to buy avocados at the virtual store. [0751] Predicted inventory determination module 4106 may also calculate a confidence level associated with the predicted inventory and may initiate a first automated action (e.g., updating virtual store 4116, adjust a price, etc.) when the confidence level is above a threshold. [0752] In one example, when the predicted quality of the product type is below the threshold, the at least one processor may prevent virtual store 4116 from selling the at least one product type).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Adato to transmit a payment in response to the compliance exceeding a threshold because "common problems of traditional online shopping systems arise when the list of products on the website does not correspond with the actual products on the shelf. For example, an online customer may order a favorite cookie brand without knowing that the cookie brand is out-of-stock. Consistent with some embodiments, system 100 may use image data acquired by capturing devices 125 to provide the online customer with a near real-time display of the retail store and a list of the actual products on the shelf based on near real-time data" [0238].
Regarding claim 3, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Savvides further teaches wherein the product detector detects products, empty space, and specified products in product image ([0011] In preferred embodiments of the invention, the system analyzes the panoramic images to detect the presence of and, optionally, to determine the identity of products placed on the fixtures. [0008] To implement machine vision technology relying on a planogram, one or more fixed position cameras can be used throughout a store to monitor aisles, with large gaps in shelf space being checkable against the planogram or shelf labels and flagged as “out-of-stock” if necessary).
Regarding claim 4, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Adato teaches wherein the SKU classifier classifies a SKU of a product ([0818] The processing device may also be configured to determine stock keeping units (SKUs) for the plurality of differing products based on the unique identifiers (other than SKU bar codes) in the image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Adato to use an SKU classifier "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Regarding claim 7, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Adato teaches wherein the brand family classifier classifies a brand of a product ([0400] In some aspects, system 100 may differentiate between products based on a brand name associated with a product, a logo associated with the product, a text associated with the product, a color associated with the product, a position of the product, and so forth and further differentiate between the products based on price ranges in which the products fall. For example, continuing with the milk example, a system may determine that a subset of the milk is associated with a particular supplier, e.g., Hiland Dairy®, by detecting a logo or brand name associated therewith).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Adato to use a brand classifier "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Regarding claim 8, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Savvides further teaches wherein the shelf detector detects shelves and product placement within shelves ([0081] As part of the process of identifying pegged products, is necessary to identify where shelves are located on image 300. At box 414 of pipeline 400, the image 300 is processed by a classifier 414 that classifies each pixel of the image 300 determine if the pixel is part of a shelf or not part of a shelf, to produce a binary mask, having pixels located on shelves flagged as a binary “1” in pixels not located on shelves flagged as a binary “0”. [0084] Once a location of a shelf is inferred, it is also possible to determine which product bounding boxes, discovered by product detector 402, are positioned on the shelf by comparing the location of the bottom of the products bounding box with the location of the top of the shelves bounding box).
Regarding claim 10, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Savvides further teaches wherein the dimension estimator maps pixel dimensions of a product image to physical dimensions ([0047] In some embodiments, the depth sensors are associated with image cameras and depth pixels registered to image pixels. This provides depth information for pixels in the image of the shelves. This depth information measures the distances of the image camera to the shelf lip and to the products. [0071] Product detector 402 produces, as an output, the image with a bounding box as shown in FIG. 5. In some embodiments, the bounding box is represented in a data structure as a tuple of data of the form BB={x, y, w, h}. A tuple may comprise, for example, the x,y coordinates of a corner of the bounding box as well as the width (w) and the height (h) of the bounding box. Other information may be included in the tuple, for example, depth information. In some embodiments, the tuple may represent bounding boxes within a 3D point cloud).
Savvides does not explicitly teach wherein the refrigerator detector detects a refrigerator door on a shelf.
Adato teaches wherein the refrigerator detector detects a refrigerator door on a shelf ([0301] By way of another example, when image processing unit 130 identifies that the shelf in the image has “a glass door in the front,” image processing unit 130 may determine the shelf in the image is in a refrigerator).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Adato to use a refrigerator detector "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Regarding claim 11, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Savvides further teaches wherein the orientation classifier determines a side a product is facing ([0011] The system is then able to match the placement of and, optionally, the identity of the products on the fixtures with the expected positions of the products to determine that the products are shelved properly, are miss-shelved or are out-of-stock. [0117] However, product recognition is a challenging problem. First, products can look very similar and yet be entirely different products. Further, the same product can look very different under different conditions, for example, pose variations (e.g., the same product viewed from different angles). [0124] The product library comprises features of images representing variations of a product, for example, pose or labeling variations, and an associated prime ID).
Regarding claim 12, Savvides teaches an apparatus ([0002] The present invention is related to the field of automated inventory monitoring in a commercial retail setting and, in particular, is directed to systems, processes and methods for automatically tracking products displayed in the retail setting) comprising: a processor (data processing module 136) executing code stored in a memory to perform ([0054] Electronic control unit 130 may also provide image processing using a camera control and data processing module 136. The camera control and data processing module 136 can include a separate data storage module 138): training a supervised learning product model ([0119] One aspect of this embodiment of the invention involves the creation of a product library through a supervised or unsupervised training process) comprising a product detector ([0072] In preferred embodiments, product detector 402 is a machine learning model trained on images of products), a price detector ([0077] Shelf labels are detected in a similar manner using label detector 406, shown as a component of pipeline 400 in FIG. 4. Shelves in stores typically have two kinds of labels. The first type of label is a price label for the products, which are referred to herein as “shelf labels”), a shelf detector ([0021] FIG. 8 is an example of the output of the shelf segment classifier of the present invention, showing a binary mask having highlighted areas indicating which pixels in the panoramic image are located on a shelf), a dimension estimator ([0071] In some embodiments, the bounding box is represented in a data structure as a tuple of data of the form BB={x, y, w, h}. A tuple may comprise, for example, the x,y coordinates of a corner of the bounding box as well as the width (w) and the height (h) of the bounding box), and an orientation classifier ([0011] The system is then able to match the placement of and, optionally, the identity of the products on the fixtures with the expected positions of the products to determine that the products are shelved properly, are miss-shelved or are out-of-stock),
wherein the product model embeds product embeddings of a same product close to another product in a latent space of a vector database (product library) ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't. [0012] The extracted features from the product images are compared to extracted features in a product library and a best-fit is obtained);
generating a product embedding for a plurality of product images ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't) of segmented products using the product model ([0143] The process 2000 for automatically enrolling new products is shown in flowchart form in FIG. 20. Given an image of a shelf, for example, panoramic image 300, at step 2002 of process 2000, product detection model 402 is run on panoramic image 300 to infer regions of interest in the shelf image, indicated by bounding boxes 504 around the product images 502. An example of the output of the product detection model 402 is shown in FIG. 5. Bounding boxes 504 identified by product detection model 402 are the input regions of interest for the next step of process 2000. Preferably, each region of interest will contain an image of a product. At step 2004 of process 2000, feature extractor 1704 is then run on each region of interest and features are extracted from the region of interest in accordance with the training of feature extractor 1704);
generating the vector database of the product embeddings for the plurality of the product images, wherein the vector database comprises product embeddings of known products and unknown products ([0124] The product library comprises features of images representing variations of a product, for example, pose or labeling variations, and an associated prime ID. The group of features of the product images are obtained using a trained machine learning model (i.e., a feature extractor) which is trained to extract the features from the images and to output a prime ID associated with the product images. [0125] Product images extracted from the acquisition image are feature matched against product source images in the product library and ranked based on the highest confidence in the association between product images from the acquisition image and source product images. Product images with the same product and label primary association are grouped together, for example, by placing the image features into one folder or assigning the same digital designation (or tag) for one particular product or a type of product. [0144] When a new product has been discovered, it is enrolled, at step 2012 of process 2000, in product library 1708, along with identifying information. In some embodiments, the new products may be assigned identifying information which may be random, or which may be obtained from another source);
generating a new product embedding for a new product or different views or packaging of already known products ([0128] In another aspect of this embodiment, a method of building a product library that can be used for general machine learning training is disclosed. In this aspect, a system is provided to take images of new or re-packaged products and learns to recognize the new product with the new packaging or appearance. This ongoing learning feature makes it easy for the system to stay updated with new products or products having a new appearance);
querying the vector database with the new product embedding as a centroid for a proximity query, wherein the new product embedding is a novel distance from other product embeddings in the vector database ([0143] At step 2006 of process 2000, the extracted features are used to search for a best-fit (i.e., a closest match) in product library 1708. If, at step 2008 of process 2000, the distance between the extracted features and the features of the best-fit in product library 1708 are above a predetermined distance threshold, the object in the region of interest is determined to be a new product);
adding the new product to the product detector using product images extracted from within a product embedding group of the vector database ([0144] When a new product has been discovered, it is enrolled, at step 2012 of process 2000, in product library 1708, along with identifying information. In some embodiments, the new products may be assigned identifying information which may be random, or which may be obtained from another source. In other embodiments, the bounding boxes 504 containing the product images may be associated with a shelf label 1108 by product-label association algorithm 422, as shown in FIGS. 10 and 11, and the identifying information extracted from the shelf label 1108);
and detecting the new product using the product detector ([0013] In yet another aspect of the invention, a system and method for dynamically enrolling new object images in the product library through zero- or low-shot training is disclosed. [0120] In some aspects of this embodiment, the product library may be initiated, expanded, and optimized automatically. The product library can further be deployed into a product recognition and detection process that significantly improves the accuracy of out-of-stock detection).
Savvides does not teach a Stock Keeping Unit (SKU) classifier, a brand classifier, a refrigerator detector; and the plurality of axis values comprise at least a plurality of product placement axis values, and a brand axis value.
Adato, in the same field of endeavor of classifying products from images, teaches a Stock Keeping Unit (SKU) classifier ([0818] The processing device may also be configured to determine stock keeping units (SKUs) for the plurality of differing products based on the unique identifiers (other than SKU bar codes) in the image), a brand classifier ([0400] In some aspects, system 100 may differentiate between products based on a brand name associated with a product…a system may determine that a subset of the milk is associated with a particular supplier, e.g., Hiland Dairy®, by detecting a logo or brand name associated therewith), a refrigerator detector [0301] By way of another example, when image processing unit 130 identifies that the shelf in the image has “a glass door in the front,” image processing unit 130 may determine the shelf in the image is in a refrigerator);
and the plurality of axis values comprise at least a plurality of product placement axis values, and a brand axis value ([0123] Embodiments of the present disclosure further include analyzing images to detect and identify different products. As used herein, the term “detecting a product” may broadly refer to determining an existence of the product. For example, the system may determine the existence of a plurality of distinct products displayed on a store shelf…Additionally or alternatively, the term “identifying a product” may refer to determining a unique identifier associated with a specific brand of products that allows inventory managers to uniquely refer to products, e.g., based on a specific brand in a product catalogue. [0250] In yet another example, the product model may include support vectors that may be used by a Support Vector Machine to identify products).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the apparatus of Savvides with the teachings of Adato to use an SKU classifier, a brand classifier, and a refrigerator detector and use product placement axis values and brand axis values "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Savvides does not teach each product embedding comprising a product identifier and a plurality of axis values for the vector database, wherein the product identifier is one of unidentified and linked to product data and the plurality of axis values comprise at least a plurality of product image axis values; wherein the vector database comprises product embeddings of known products and unknown products at the axis values for each product embedding; generating a new product embedding in a potential product embedding group for a new product or different views or packaging of already known products; labeling close product embeddings from the vector database as the new product in the potential product embedding group, wherein the close product embeddings are generated from previously captured product images, have an unidentified product identifier that is not linked to product data, are not labeled as the new product, and are within a novel distance threshold to a centroid of the potential product embedding group; adding the new product to the product detector using product images extracted from within the potential product embedding group of the vector database.
Haghighat, in the same field of endeavor of classifying products from images, teaches each product embedding comprising a product identifier and a plurality of axis values for the vector database, wherein the product identifier is one of unidentified and linked to product data and the plurality of axis values comprise at least a plurality of product image axis values; wherein the vector database comprises product embeddings of known products and unknown products at the axis values for each product embedding; generating a new product embedding in a potential product embedding group for a new product or different views or packaging of already known products ([0027] The present CNN 22, as shown in FIG. 4 and, further, in FIGS. 5-7, is configured to operate without using a Softmax probability, but rather to “embed” vectors 26 corresponding with the initial data set on which the CNN 22 is trained, as well as new vectors 26 derived from later images 24 analyzed by the CNN 22, in a feature space having a number of dimensions equal to the number of features included in the vectors 26 realized by the feature learning layers of the CNN 22…In this respect, the value for a given feature (i.e., cell) in a given vector 26 can indicate the position of the vector 26 within the corresponding dimension, with the complete vector 26 locating the embedded entry within the complete feature space. [0029] Because of the training process applied to the CNN 22, any new vectors 40 will be embedded in the hypersphere 34 according to the similarity in perception of the images 24 to the existing vectors 26. In this respect, if the new image 24 is of a item corresponding with a known class-in the present example of a cooking appliance and, in particular, oven 10, then the resulting new vector 40 will be embedded within or close to the cluster 36 consisting of the other vectors 26 representing earlier images 24, including the training images. In this respect, the CNN 22 is configured to output an identification of the image as corresponding with a known class according to the closest cluster 36 to the newly-embedded vector 40, within a predetermined threshold distance or in combination with a probability that that the newly-embedded vector 40 fits within the class of the closest cluster 36. If the new vector 40 is not sufficiently close to an existing cluster 36 (i.e., is positioned within the open space 38 surrounding the clusters 36) and/or has a high probability of not fitting within the closest existing cluster 36, the CNN 22 can return an output indicating that the vector 40 and originating image 24 do not correspond with one of the known classes);
labeling close product embeddings from the vector database as the new product in the potential product embedding group, wherein the close product embeddings are generated from previously captured product images, have an unidentified product identifier that is not linked to product data, are not labeled as the new product, and are within a novel distance threshold to a centroid of the potential product embedding group ([0030] Accordingly, the CNN 22 can, in various configurations, treat new vectors 40 within the open space 38 as comprising or being within a new cluster 42 that corresponds with a new, previously-unknown food type. Notably, such treatment is generally inherent in the CNN 22 configuration described herein, as the new vectors 40 and original (trained) vectors 26 are treated the same during use of the CNN 22 to perceive and identify subsequent images 24. The CNN 22, however, can be configured to require a certain number of new vectors 40 within a specified distance from each other or with specified relative distribution characteristics (as discussed further below) before taking steps to specifically designate the particular new vectors 40 as a new cluster 42, such as “registering” the cluster as a specific class and/or querying the user for the name or classification of the new cluster, for example. [0031] Subsequently, upon embedding the new vector 40 in the hypersphere 34, the CNN 22 can compute the Mahalanobis distance between the new vector 40 and, by nature of the calculation, a centroid 44 of each known cluster 36…If no cluster 36 is within the threshold Mahalanobis distance, then the new vector 40 is registered as a new class. In yet another variation, the Mahalanobis distance of the new vector 40 can be assessed in terms of the standard deviation of the vectors 26 of the particular cluster 36 to determine, for example, if the new vector 40 is an outlier with respect to even the closest cluster 36, which may indicate that the new vector 40 corresponds with a new class. [0038] When the specific requirements of the particular appliance 10 are met (which may include a user indication via HMI 46 that a single new vector 40 should be registered as a new food type) (step 128), the vector 40 or plurality of proximate, unregistered new vectors 40 are registered as a new food product type that is added to the plurality of known food product types accessible by the cooking appliance 10 (step 134);
adding the new product to the product detector using product images extracted from within the potential product embedding group of the vector database ([0038] As discussed above, this step may include cataloging the vector 40 as a new, but still unidentified, food type and not yet registering the vector 40 as a cluster until a subsequent operation embeds another new vector 40 in close proximity to that previously-registered new vector 40 (or another predetermined number of other new, unregistered vectors 40). When the specific requirements of the particular appliance 10 are met (which may include a user indication via HMI 46 that a single new vector 40 should be registered as a new food type) (step 128), the vector 40 or plurality of proximate, unregistered new vectors 40 are registered as a new food product type that is added to the plurality of known food product types accessible by the cooking appliance 10 (step 134). This may include prompting the user to enter a name of the new food type for association with the new cluster 42 during registration (step 130). Notably, and as discussed above, the addition of the new food type to the CNN 22 is done without retraining the identification model 31 or the perception model 27).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the apparatus of Savvides with the teachings of Haghighat to label close product embeddings from the vector database as the new product embeddings in a potential product embedding group and add the new product to the product detector "to identify a food type as unknown and register the food type as new without retraining the identification model of the convolutional neural network" [Haghighat 0001] because "Even at the manufacturer or programmer level, training a new identification model takes a considerable amount of time and computation resources" [Haghighat 0003].
Savvides does not teach retraining the product model using the product images extracted from within the potential product embedding group of the vector database.
Asif, in the same field of endeavor of classification models, teaches retraining the model using the images extracted from within the potential embedding group of the vector database ([0024] Further aspects of the present disclosure relate to systems and methods to improve model performance based on measured data. More particular aspects of the present disclosure relate to a system to receive unlabeled data, compare the unlabeled data to known data, label the unlabeled data based on the comparison, and retrain the model based on the newly-labeled data. [0090] FIG. 6 illustrates a high-level method 600 of updating a machine learning model based on de-identified feature data derived from unlabeled input data, consistent with several embodiments of the present disclosure. In general, method 600 includes several operations that may be substantially similar to operations performed in method 400. However, method 600 retrains a model based on unlabeled “de-identified” features (i.e., feature data) that are themselves based on collected data (i.e., input data). [0097] Method 600 further comprises retraining the machine learning model via the newly labeled (“weak”-labeled) feature data at operation 614. In some embodiments, operation 614 may further include retraining via the hard-labeled feature data and feature data from one or more teacher models. Operation 614 may include, for example, inputting the weak-labeled features to the “next layer” of the model. As an example, the now-labeled feature data generated at operation 606 may be output from a 37th layer of a 50-layer model. Operation 614 may include inputting the weak-labeled feature data into the 38th layer of the model, receiving an output from the model, calculating an error between the output and the label, adjusting one or more of layers 38-50 of the model based on the error, and repeating until the error no longer decreases. [0028] As an example, input data may be an image, wherein a first layer may identify edges in the image, producing a list (“vector”) of data describing all the edges detected in the image (“features”). The features generated by the first layer are then fed into a second layer, which performs further manipulation and produces a second feature vector, and so on).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product model of Savvides with the teachings of Asif to retrain the model using the images extracted from the potential embedding group because "The machine learning model may have already been trained to analyze or classify input data such as the data collected via operation 402. As the nature of the input data changes, the model may lose accuracy over time, which can be alleviated by retraining" [0076].
Regarding claim 13, Savvides, Adato, Haghighat, and Asif teach the apparatus of claim 12. Savvides further teaches receiving a shelf image; determining a product placement; and determining compliance with placement requirements ([0011] In preferred embodiments of the invention, the system analyzes the panoramic images to detect the presence of and, optionally, to determine the identity of products placed on the fixtures. Additionally, the system can analyze the panoramic images to identify shelf labels indicating which products are expected to be at various positions on the fixtures. The system is then able to match the placement of and, optionally, the identity of the products on the fixtures with the expected positions of the products to determine that the products are shelved properly, are miss-shelved or are out-of-stock. The system is further functional to flag misplaced and out-of-stock products and alert the store's staff such that the misplacement may be corrected or such that the product may be re-stocked).
Savvides does not explicitly teach transmitting a payment in response to the compliance exceeding a compliance threshold.
Adato teaches transmitting a payment in response to the compliance exceeding a compliance threshold ([0238] FIG. 11E illustrates an example GUI 1150 for output device 145D used by an online customer of retail store 105. Traditional online shopping systems present online customers with a list of products. Products selected for purchase may be placed into a virtual shopping cart until the customers complete their virtual shopping trip. [0733] Reference is now made to FIG. 40A, which illustrates a timeline associated with the exemplary solution of the present disclosure. Initially, the suggested method includes receiving image data 4000 to determine a current inventory in retail store 105, and receiving product supply information 4002 to determine a predicted inventory. Thereafter, at the end of a first time period, a customer 4004 of a virtual store may complete an online order. [0734] The following demonstrates the two example embodiments. Assume customer 4004 is considering buying avocados online. At the time customer 4004 is ordering products from a virtual store (i.e., during the first time period), she wants to know if the store has avocados. Independently from or in connection with the activities of customer 4004, the virtual store may receive information indicating that at the time the order of customer 4004 is expected to be collected (i.e., during the second time period), retail store 105 is expected to have avocados in-stock but their quality is estimated to be below average (e.g., based on a determined average shelf time for in stock avocados and based on predetermined metrics such as average time to ripeness, based on predicted future restocking events, etc.). According to the first embodiment, the virtual store may present to customer 4004 at the first time period an indication that avocados are in-stock. According to the second embodiment, the virtual store may present to customer 4004 at the first time period an indication that the avocados' quality is estimated to be below average. Armed with this knowledge, customer 4004 can complete her online shopping and decide whether to buy or not to buy avocados at the virtual store. [0751] Predicted inventory determination module 4106 may also calculate a confidence level associated with the predicted inventory and may initiate a first automated action (e.g., updating virtual store 4116, adjust a price, etc.) when the confidence level is above a threshold. [0752] In one example, when the predicted quality of the product type is below the threshold, the at least one processor may prevent virtual store 4116 from selling the at least one product type).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the apparatus of Savvides with the teachings of Adato to transmit a payment in response to the compliance exceeding a threshold because "common problems of traditional online shopping systems arise when the list of products on the website does not correspond with the actual products on the shelf. For example, an online customer may order a favorite cookie brand without knowing that the cookie brand is out-of-stock. Consistent with some embodiments, system 100 may use image data acquired by capturing devices 125 to provide the online customer with a near real-time display of the retail store and a list of the actual products on the shelf based on near real-time data" [0238].
Regarding claim 14, Savvides, Adato, Haghighat, and Asif teach the apparatus of claim 12. Savvides further teaches wherein the product detector detects products, empty space, and specified products in product image ([0011] In preferred embodiments of the invention, the system analyzes the panoramic images to detect the presence of and, optionally, to determine the identity of products placed on the fixtures. [0008] To implement machine vision technology relying on a planogram, one or more fixed position cameras can be used throughout a store to monitor aisles, with large gaps in shelf space being checkable against the planogram or shelf labels and flagged as “out-of-stock” if necessary).
Regarding claim 15, Savvides, Adato, Haghighat, and Asif teach the apparatus of claim 12. Adato teaches wherein the SKU classifier classifies a SKU of a product ([0818] The processing device may also be configured to determine stock keeping units (SKUs) for the plurality of differing products based on the unique identifiers (other than SKU bar codes) in the image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the apparatus of Savvides with the teachings of Adato to use an SKU classifier "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Regarding claim 17, Savvides teaches a computer program product comprising a non-transitory storage medium storing code executable by a processor ([0054] The camera control and data processing module 136 can include a separate data storage module 138. Data storage model 138 may be, for example, a solid-state hard drive or other form of flash memory. Data storage model 138 is connected to a processing module 140. The communication module 134 is connected to the processing module 140 to transfer product availability and/or identification data or panoramic images to remote locations, including store servers or other supported camera systems, and optionally receive inventory information to aid in product identification and localization) to perform: training a product model comprising a product detector ([0072] In preferred embodiments, product detector 402 is a machine learning model trained on images of products), a price detector ([0077] Shelf labels are detected in a similar manner using label detector 406, shown as a component of pipeline 400 in FIG. 4. Shelves in stores typically have two kinds of labels. The first type of label is a price label for the products, which are referred to herein as “shelf labels”), a shelf detector ([0021] FIG. 8 is an example of the output of the shelf segment classifier of the present invention, showing a binary mask having highlighted areas indicating which pixels in the panoramic image are located on a shelf), a dimension estimator ([0071] In some embodiments, the bounding box is represented in a data structure as a tuple of data of the form BB={x, y, w, h}. A tuple may comprise, for example, the x,y coordinates of a corner of the bounding box as well as the width (w) and the height (h) of the bounding box), and an orientation classifier ([0011] The system is then able to match the placement of and, optionally, the identity of the products on the fixtures with the expected positions of the products to determine that the products are shelved properly, are miss-shelved or are out-of-stock),
wherein the product model embeds product embeddings of a same product close to another product in a latent space of a vector database (product library) ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't. [0012] The extracted features from the product images are compared to extracted features in a product library and a best-fit is obtained);
generating a product embedding for a plurality of product images ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't) of segmented products using the product model ([0143] The process 2000 for automatically enrolling new products is shown in flowchart form in FIG. 20. Given an image of a shelf, for example, panoramic image 300, at step 2002 of process 2000, product detection model 402 is run on panoramic image 300 to infer regions of interest in the shelf image, indicated by bounding boxes 504 around the product images 502. An example of the output of the product detection model 402 is shown in FIG. 5. Bounding boxes 504 identified by product detection model 402 are the input regions of interest for the next step of process 2000. Preferably, each region of interest will contain an image of a product. At step 2004 of process 2000, feature extractor 1704 is then run on each region of interest and features are extracted from the region of interest in accordance with the training of feature extractor 1704);
generating the vector database of the product embeddings for the plurality of the product images, wherein the vector database comprises product embeddings of known products and unknown products ([0124] The product library comprises features of images representing variations of a product, for example, pose or labeling variations, and an associated prime ID. The group of features of the product images are obtained using a trained machine learning model (i.e., a feature extractor) which is trained to extract the features from the images and to output a prime ID associated with the product images. [0125] Product images extracted from the acquisition image are feature matched against product source images in the product library and ranked based on the highest confidence in the association between product images from the acquisition image and source product images. Product images with the same product and label primary association are grouped together, for example, by placing the image features into one folder or assigning the same digital designation (or tag) for one particular product or a type of product. [0144] When a new product has been discovered, it is enrolled, at step 2012 of process 2000, in product library 1708, along with identifying information. In some embodiments, the new products may be assigned identifying information which may be random, or which may be obtained from another source);
generating a new product embedding for a new product or different views or packaging of already known products ([0128] In another aspect of this embodiment, a method of building a product library that can be used for general machine learning training is disclosed. In this aspect, a system is provided to take images of new or re-packaged products and learns to recognize the new product with the new packaging or appearance. This ongoing learning feature makes it easy for the system to stay updated with new products or products having a new appearance);
querying the vector database with the new product embedding as a centroid for a proximity query, wherein the new product embedding is a novel distance from other product embeddings in the vector database ([0143] At step 2006 of process 2000, the extracted features are used to search for a best-fit (i.e., a closest match) in product library 1708. If, at step 2008 of process 2000, the distance between the extracted features and the features of the best-fit in product library 1708 are above a predetermined distance threshold, the object in the region of interest is determined to be a new product);
adding the new product to the product detector using product images extracted from within a product embedding group of the vector database ([0144] When a new product has been discovered, it is enrolled, at step 2012 of process 2000, in product library 1708, along with identifying information. In some embodiments, the new products may be assigned identifying information which may be random, or which may be obtained from another source. In other embodiments, the bounding boxes 504 containing the product images may be associated with a shelf label 1108 by product-label association algorithm 422, as shown in FIGS. 10 and 11, and the identifying information extracted from the shelf label 1108);
and detecting the new product using the product detector ([0013] In yet another aspect of the invention, a system and method for dynamically enrolling new object images in the product library through zero- or low-shot training is disclosed. [0120] In some aspects of this embodiment, the product library may be initiated, expanded, and optimized automatically. The product library can further be deployed into a product recognition and detection process that significantly improves the accuracy of out-of-stock detection).
Savvides does not teach a Stock Keeping Unit (SKU) classifier, a brand classifier, a refrigerator detector; and the plurality of axis values comprise at least a plurality of product placement axis values, and a brand axis value.
Adato, in the same field of endeavor of classifying products from images, teaches a Stock Keeping Unit (SKU) classifier ([0818] The processing device may also be configured to determine stock keeping units (SKUs) for the plurality of differing products based on the unique identifiers (other than SKU bar codes) in the image), a brand classifier ([0400] In some aspects, system 100 may differentiate between products based on a brand name associated with a product…a system may determine that a subset of the milk is associated with a particular supplier, e.g., Hiland Dairy®, by detecting a logo or brand name associated therewith), a refrigerator detector [0301] By way of another example, when image processing unit 130 identifies that the shelf in the image has “a glass door in the front,” image processing unit 130 may determine the shelf in the image is in a refrigerator);
and the plurality of axis values comprise at least a plurality of product placement axis values, and a brand axis value ([0123] Embodiments of the present disclosure further include analyzing images to detect and identify different products. As used herein, the term “detecting a product” may broadly refer to determining an existence of the product. For example, the system may determine the existence of a plurality of distinct products displayed on a store shelf…Additionally or alternatively, the term “identifying a product” may refer to determining a unique identifier associated with a specific brand of products that allows inventory managers to uniquely refer to products, e.g., based on a specific brand in a product catalogue. [0250] In yet another example, the product model may include support vectors that may be used by a Support Vector Machine to identify products).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product of Savvides with the teachings of Adato to use an SKU classifier, a brand classifier, and a refrigerator detector and use product placement axis values and brand axis values "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Savvides does not teach each product embedding comprising a product identifier and a plurality of axis values for the vector database, wherein the product identifier is one of unidentified and linked to product data and the plurality of axis values comprise at least a plurality of product image axis values; wherein the vector database comprises product embeddings of known products and unknown products at the axis values for each product embedding; generating a new product embedding in a potential product embedding group for a new product or different views or packaging of already known products; labeling close product embeddings from the vector database as the new product in the potential product embedding group, wherein the close product embeddings are generated from previously captured product images, have an unidentified product identifier that is not linked to product data, and are within a novel distance threshold to a centroid of the potential product embedding group; adding the new product to the product detector using product images extracted from within the potential product embedding group of the vector database.
Haghighat, in the same field of endeavor of classifying products from images, teaches each product embedding comprising a product identifier and a plurality of axis values for the vector database, wherein the product identifier is one of unidentified and linked to product data and the plurality of axis values comprise at least a plurality of product image axis values; wherein the vector database comprises product embeddings of known products and unknown products at the axis values for each product embedding; generating a new product embedding in a potential product embedding group for a new product or different views or packaging of already known products ([0027] The present CNN 22, as shown in FIG. 4 and, further, in FIGS. 5-7, is configured to operate without using a Softmax probability, but rather to “embed” vectors 26 corresponding with the initial data set on which the CNN 22 is trained, as well as new vectors 26 derived from later images 24 analyzed by the CNN 22, in a feature space having a number of dimensions equal to the number of features included in the vectors 26 realized by the feature learning layers of the CNN 22…In this respect, the value for a given feature (i.e., cell) in a given vector 26 can indicate the position of the vector 26 within the corresponding dimension, with the complete vector 26 locating the embedded entry within the complete feature space. [0029] Because of the training process applied to the CNN 22, any new vectors 40 will be embedded in the hypersphere 34 according to the similarity in perception of the images 24 to the existing vectors 26. In this respect, if the new image 24 is of a item corresponding with a known class-in the present example of a cooking appliance and, in particular, oven 10, then the resulting new vector 40 will be embedded within or close to the cluster 36 consisting of the other vectors 26 representing earlier images 24, including the training images. In this respect, the CNN 22 is configured to output an identification of the image as corresponding with a known class according to the closest cluster 36 to the newly-embedded vector 40, within a predetermined threshold distance or in combination with a probability that that the newly-embedded vector 40 fits within the class of the closest cluster 36. If the new vector 40 is not sufficiently close to an existing cluster 36 (i.e., is positioned within the open space 38 surrounding the clusters 36) and/or has a high probability of not fitting within the closest existing cluster 36, the CNN 22 can return an output indicating that the vector 40 and originating image 24 do not correspond with one of the known classes);
labeling close product embeddings from the vector database as the new product in the potential product embedding group, wherein the close product embeddings are generated from previously captured product images, have an unidentified product identifier that is not linked to product data, are not labeled as the new product, and are within a novel distance threshold to a centroid of the potential product embedding group ([0030] Accordingly, the CNN 22 can, in various configurations, treat new vectors 40 within the open space 38 as comprising or being within a new cluster 42 that corresponds with a new, previously-unknown food type. Notably, such treatment is generally inherent in the CNN 22 configuration described herein, as the new vectors 40 and original (trained) vectors 26 are treated the same during use of the CNN 22 to perceive and identify subsequent images 24. The CNN 22, however, can be configured to require a certain number of new vectors 40 within a specified distance from each other or with specified relative distribution characteristics (as discussed further below) before taking steps to specifically designate the particular new vectors 40 as a new cluster 42, such as “registering” the cluster as a specific class and/or querying the user for the name or classification of the new cluster, for example. [0031] Subsequently, upon embedding the new vector 40 in the hypersphere 34, the CNN 22 can compute the Mahalanobis distance between the new vector 40 and, by nature of the calculation, a centroid 44 of each known cluster 36…If no cluster 36 is within the threshold Mahalanobis distance, then the new vector 40 is registered as a new class. In yet another variation, the Mahalanobis distance of the new vector 40 can be assessed in terms of the standard deviation of the vectors 26 of the particular cluster 36 to determine, for example, if the new vector 40 is an outlier with respect to even the closest cluster 36, which may indicate that the new vector 40 corresponds with a new class. [0038] When the specific requirements of the particular appliance 10 are met (which may include a user indication via HMI 46 that a single new vector 40 should be registered as a new food type) (step 128), the vector 40 or plurality of proximate, unregistered new vectors 40 are registered as a new food product type that is added to the plurality of known food product types accessible by the cooking appliance 10 (step 134);
adding the new product to the product detector using product images extracted from within the potential product embedding group of the vector database ([0038] As discussed above, this step may include cataloging the vector 40 as a new, but still unidentified, food type and not yet registering the vector 40 as a cluster until a subsequent operation embeds another new vector 40 in close proximity to that previously-registered new vector 40 (or another predetermined number of other new, unregistered vectors 40). When the specific requirements of the particular appliance 10 are met (which may include a user indication via HMI 46 that a single new vector 40 should be registered as a new food type) (step 128), the vector 40 or plurality of proximate, unregistered new vectors 40 are registered as a new food product type that is added to the plurality of known food product types accessible by the cooking appliance 10 (step 134). This may include prompting the user to enter a name of the new food type for association with the new cluster 42 during registration (step 130). Notably, and as discussed above, the addition of the new food type to the CNN 22 is done without retraining the identification model 31 or the perception model 27).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product of Savvides with the teachings of Haghighat to label close product embeddings from the vector database as the new product embeddings in a potential product embedding group and add the new product to the product detector "to identify a food type as unknown and register the food type as new without retraining the identification model of the convolutional neural network" [Haghighat 0001] because "Even at the manufacturer or programmer level, training a new identification model takes a considerable amount of time and computation resources" [Haghighat 0003].
Savvides does not teach retraining the product model using the product images extracted from within the potential product embedding group of the vector database.
Asif, in the same field of endeavor of classification models, teaches retraining the model using the images extracted from within the potential embedding group of the vector database ([0024] Further aspects of the present disclosure relate to systems and methods to improve model performance based on measured data. More particular aspects of the present disclosure relate to a system to receive unlabeled data, compare the unlabeled data to known data, label the unlabeled data based on the comparison, and retrain the model based on the newly-labeled data. [0090] FIG. 6 illustrates a high-level method 600 of updating a machine learning model based on de-identified feature data derived from unlabeled input data, consistent with several embodiments of the present disclosure. In general, method 600 includes several operations that may be substantially similar to operations performed in method 400. However, method 600 retrains a model based on unlabeled “de-identified” features (i.e., feature data) that are themselves based on collected data (i.e., input data). [0097] Method 600 further comprises retraining the machine learning model via the newly labeled (“weak”-labeled) feature data at operation 614. In some embodiments, operation 614 may further include retraining via the hard-labeled feature data and feature data from one or more teacher models. Operation 614 may include, for example, inputting the weak-labeled features to the “next layer” of the model. As an example, the now-labeled feature data generated at operation 606 may be output from a 37th layer of a 50-layer model. Operation 614 may include inputting the weak-labeled feature data into the 38th layer of the model, receiving an output from the model, calculating an error between the output and the label, adjusting one or more of layers 38-50 of the model based on the error, and repeating until the error no longer decreases. [0028] As an example, input data may be an image, wherein a first layer may identify edges in the image, producing a list (“vector”) of data describing all the edges detected in the image (“features”). The features generated by the first layer are then fed into a second layer, which performs further manipulation and produces a second feature vector, and so on).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product model of Savvides with the teachings of Asif to retrain the model using the images extracted from the potential embedding group because "The machine learning model may have already been trained to analyze or classify input data such as the data collected via operation 402. As the nature of the input data changes, the model may lose accuracy over time, which can be alleviated by retraining" [0076].
Regarding claim 18, Savvides, Adato, Haghighat, and Asif teach the product of claim 17. Savvides further teaches receiving a shelf image; determining a product placement; and determining compliance with placement requirements ([0011] In preferred embodiments of the invention, the system analyzes the panoramic images to detect the presence of and, optionally, to determine the identity of products placed on the fixtures. Additionally, the system can analyze the panoramic images to identify shelf labels indicating which products are expected to be at various positions on the fixtures. The system is then able to match the placement of and, optionally, the identity of the products on the fixtures with the expected positions of the products to determine that the products are shelved properly, are miss-shelved or are out-of-stock. The system is further functional to flag misplaced and out-of-stock products and alert the store's staff such that the misplacement may be corrected or such that the product may be re-stocked).
Savvides does not explicitly teach transmitting a payment in response to the compliance exceeding a compliance threshold.
Adato teaches transmitting a payment in response to the compliance exceeding a compliance threshold ([0238] FIG. 11E illustrates an example GUI 1150 for output device 145D used by an online customer of retail store 105. Traditional online shopping systems present online customers with a list of products. Products selected for purchase may be placed into a virtual shopping cart until the customers complete their virtual shopping trip. [0733] Reference is now made to FIG. 40A, which illustrates a timeline associated with the exemplary solution of the present disclosure. Initially, the suggested method includes receiving image data 4000 to determine a current inventory in retail store 105, and receiving product supply information 4002 to determine a predicted inventory. Thereafter, at the end of a first time period, a customer 4004 of a virtual store may complete an online order. [0734] The following demonstrates the two example embodiments. Assume customer 4004 is considering buying avocados online. At the time customer 4004 is ordering products from a virtual store (i.e., during the first time period), she wants to know if the store has avocados. Independently from or in connection with the activities of customer 4004, the virtual store may receive information indicating that at the time the order of customer 4004 is expected to be collected (i.e., during the second time period), retail store 105 is expected to have avocados in-stock but their quality is estimated to be below average (e.g., based on a determined average shelf time for in stock avocados and based on predetermined metrics such as average time to ripeness, based on predicted future restocking events, etc.). According to the first embodiment, the virtual store may present to customer 4004 at the first time period an indication that avocados are in-stock. According to the second embodiment, the virtual store may present to customer 4004 at the first time period an indication that the avocados' quality is estimated to be below average. Armed with this knowledge, customer 4004 can complete her online shopping and decide whether to buy or not to buy avocados at the virtual store. [0751] Predicted inventory determination module 4106 may also calculate a confidence level associated with the predicted inventory and may initiate a first automated action (e.g., updating virtual store 4116, adjust a price, etc.) when the confidence level is above a threshold. [0752] In one example, when the predicted quality of the product type is below the threshold, the at least one processor may prevent virtual store 4116 from selling the at least one product type).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product of Savvides with the teachings of Adato to transmit a payment in response to the compliance exceeding a threshold because "common problems of traditional online shopping systems arise when the list of products on the website does not correspond with the actual products on the shelf. For example, an online customer may order a favorite cookie brand without knowing that the cookie brand is out-of-stock. Consistent with some embodiments, system 100 may use image data acquired by capturing devices 125 to provide the online customer with a near real-time display of the retail store and a list of the actual products on the shelf based on near real-time data" [0238].
Regarding claim 19, Savvides, Adato, Haghighat, and Asif teach the product of claim 17. Savvides further teaches wherein the product detector detects products, empty space, and specified products in product image ([0011] In preferred embodiments of the invention, the system analyzes the panoramic images to detect the presence of and, optionally, to determine the identity of products placed on the fixtures. [0008] To implement machine vision technology relying on a planogram, one or more fixed position cameras can be used throughout a store to monitor aisles, with large gaps in shelf space being checkable against the planogram or shelf labels and flagged as “out-of-stock” if necessary).
Regarding claim 20, Savvides, Adato, Haghighat, and Asif teach the product of claim 17. Adato teaches wherein the SKU classifier classifies a SKU of a product ([0818] The processing device may also be configured to determine stock keeping units (SKUs) for the plurality of differing products based on the unique identifiers (other than SKU bar codes) in the image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product of Savvides with the teachings of Adato to use an SKU classifier "for processing images captured in a retail store and automatically identifying misplaced products…For example, the product may have been moved by a potential purchaser who did not return the product to its appropriate location, such as a shelf containing similar products or products of the same brand…notifications of misplaced products may be selectively provided to store employees based on an urgency level of restoring the misplaced product to its proper location. For example, certain products (milk, ice cream, etc.) may spoil if they are not located in a proper storage location, such as a refrigerated area" [Adato 0471].
Regarding claim 21, Savvides, Adato, Haghighat, and Asif teach the method of claim 1. Savvides further teaches wherein product embeddings are generated from a product classifier by feeding the product classifier one of pairs and triplets of product images ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't. [0012] The extracted features from the product images are compared to extracted features in a product library and a best-fit is obtained).
Savvides does not explicitly teach wherein the product embeddings are generated by pushing product embeddings of the same product close to each other in the latent space.
Haghighat teaches wherein the product embeddings are generated by pushing product embeddings of the same product close to each other in the latent space ([0028] Prior to training of the CNN 22, the embedded vectors 26 are scattered and intermixed along the feature space 34, as shown in FIG. 5… The training of the CNN 22, rather, is based on the overall distances in the feature space 34 between the vectors 26 within the same class. With an appropriately-trained embedding layer 32 based on minimizing the in-class distances of vectors 26 (i.e., appropriate location of vectors 26 within the feature space 34 based on the image perception), the convolution and RELU layers 28 can be trained similarly to those used in a classifier CNN, including using known machine-learning libraries such as Tensorflow, Keras, or PyTorch, to accurately extract relevant features from the image data 16. After the CNN 22 is trained, the vectors 26 will be arranged in clusters 36 of the known food types on which the model is trained, as shown in FIG. 6. Sufficient training of the CNN 22 will also introduce and increase the presence of open space 38 between the clusters 36. This effect is illustrated in FIG. 7, which depicts the hypersphere 34 in three dimensions, reflecting the fitting of the vectors 26 with three features about the surface of the sphere).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Haghighat to push product embeddings of the same product close to each other in the latent space because "If the new vector 40 is not sufficiently close to an existing cluster 36 (i.e., is positioned within the open space 38 surrounding the clusters 36) and/or has a high probability of not fitting within the closest existing cluster 36, the CNN 22 can return an output indicating that the vector 40 and originating image 24 do not correspond with one of the known classes…, the clusters 36 can, more particularly, correspond with the known food types associated with the original vectors 26 on which the CNN 22 is trained" [0029].
Regarding claim 22, Savvides, Adato, Haghighat, and Asif teach the apparatus of claim 12. Savvides further teaches wherein product embeddings are generated from a product classifier by feeding the product classifier one of pairs and triplets of product images ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't. [0012] The extracted features from the product images are compared to extracted features in a product library and a best-fit is obtained).
Savvides does not explicitly teach wherein the product embeddings are generated by pushing product embeddings of the same product close to each other in the latent space.
Haghighat teaches wherein the product embeddings are generated by pushing product embeddings of the same product close to each other in the latent space ([0028] Prior to training of the CNN 22, the embedded vectors 26 are scattered and intermixed along the feature space 34, as shown in FIG. 5… The training of the CNN 22, rather, is based on the overall distances in the feature space 34 between the vectors 26 within the same class. With an appropriately-trained embedding layer 32 based on minimizing the in-class distances of vectors 26 (i.e., appropriate location of vectors 26 within the feature space 34 based on the image perception), the convolution and RELU layers 28 can be trained similarly to those used in a classifier CNN, including using known machine-learning libraries such as Tensorflow, Keras, or PyTorch, to accurately extract relevant features from the image data 16. After the CNN 22 is trained, the vectors 26 will be arranged in clusters 36 of the known food types on which the model is trained, as shown in FIG. 6. Sufficient training of the CNN 22 will also introduce and increase the presence of open space 38 between the clusters 36. This effect is illustrated in FIG. 7, which depicts the hypersphere 34 in three dimensions, reflecting the fitting of the vectors 26 with three features about the surface of the sphere).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the apparatus of Savvides with the teachings of Haghighat to push product embeddings of the same product close to each other in the latent space because "If the new vector 40 is not sufficiently close to an existing cluster 36 (i.e., is positioned within the open space 38 surrounding the clusters 36) and/or has a high probability of not fitting within the closest existing cluster 36, the CNN 22 can return an output indicating that the vector 40 and originating image 24 do not correspond with one of the known classes…, the clusters 36 can, more particularly, correspond with the known food types associated with the original vectors 26 on which the CNN 22 is trained" [0029].
Regarding claim 23, Savvides, Adato, Haghighat, and Asif teach the product of claim 17. Savvides further teaches wherein product embeddings are generated from a product classifier by feeding the product classifier one of pairs and triplets of product images ([0106] Auto encoder 1206 can involve use of deep models where deep features are learned from the images and matched. In auto encoder 1206, embeddings for each of the images are learned and followed with training a pair-wise deep classifier 1208 on the autoencoder features. The pair-wise classifier 1208 provides a decision of “1” if the pair of images match and “0” if they don't. [0012] The extracted features from the product images are compared to extracted features in a product library and a best-fit is obtained).
Savvides does not explicitly teach wherein the product embeddings are generated by pushing product embeddings of the same product close to each other in the latent space.
Haghighat teaches wherein the product embeddings are generated by pushing product embeddings of the same product close to each other in the latent space ([0028] Prior to training of the CNN 22, the embedded vectors 26 are scattered and intermixed along the feature space 34, as shown in FIG. 5… The training of the CNN 22, rather, is based on the overall distances in the feature space 34 between the vectors 26 within the same class. With an appropriately-trained embedding layer 32 based on minimizing the in-class distances of vectors 26 (i.e., appropriate location of vectors 26 within the feature space 34 based on the image perception), the convolution and RELU layers 28 can be trained similarly to those used in a classifier CNN, including using known machine-learning libraries such as Tensorflow, Keras, or PyTorch, to accurately extract relevant features from the image data 16. After the CNN 22 is trained, the vectors 26 will be arranged in clusters 36 of the known food types on which the model is trained, as shown in FIG. 6. Sufficient training of the CNN 22 will also introduce and increase the presence of open space 38 between the clusters 36. This effect is illustrated in FIG. 7, which depicts the hypersphere 34 in three dimensions, reflecting the fitting of the vectors 26 with three features about the surface of the sphere).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the product of Savvides with the teachings of Haghighat to push product embeddings of the same product close to each other in the latent space because "If the new vector 40 is not sufficiently close to an existing cluster 36 (i.e., is positioned within the open space 38 surrounding the clusters 36) and/or has a high probability of not fitting within the closest existing cluster 36, the CNN 22 can return an output indicating that the vector 40 and originating image 24 do not correspond with one of the known classes…, the clusters 36 can, more particularly, correspond with the known food types associated with the original vectors 26 on which the CNN 22 is trained" [0029].
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Savvides in view of Adato, Haghighat, Asif, and Tiwari (US20240395000A1).
Regarding claim 5, Savvides, Adato, Haghighat, and Asif teach the method of claim 4. Savvides does not explicitly teach wherein the SKU classifier comprises a beer model, a wine and spirits model, and a non-alcoholic beverage model and the price detector classifies price tags, price boxes with price tags, and price digits within price boxes.
Adato teaches the price detector classifies price tags, price boxes with price tags, and price digits within price boxes ([0272] In some embodiments, image processing unit 130 may recognize the price tag and/or barcode on the products in the image. Based on the price tag and/or barcode, image processing unit 130 may determine the second candidate type of product. [0406] A detected label may be further analyzed to identify a price indicator printed on the price label (e.g., by using an OCR algorithm, using an artificial neural network configured to identify printed price indicators, and so forth). The price indicator may be any string of numerical, alphabetical, or alphanumeric characters that identifies the price of an item (e.g., “$2.49,” “2 for $10,” “50 cents,” “Two-Dollars,” etc.). The price indicator may also include, for example, promotional or temporary pricing information (e.g., “50% off,” “½ Off,” “Buy 1 get 2 Free,” etc.). By way of example, with reference to box 2310 of FIG. 23, system 100 may detect price label A3 and price label B3).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Adato to use a price detector because "The first type of product and the second type of product may be associated with different price ranges…The method may include analyzing the at least one image to determine a price associated with the detected product. Further, the method may include determining that the detected product is of the first type of product when the determined price falls within the first price range" [Adato 0017].
Tiwari, in the same field of endeavor of classifying products from images, teaches wherein the SKU classifier comprises a beer model, a wine and spirits model, and a non-alcoholic beverage model ([0023] In addition, or as an alternative, one or more sensors 146 may comprise a radio receiver and/or transmitter configured to read an electronic tag, such as, for example, a radio-frequency identification (RFID) tag. Each item may be represented in supply chain network 100 by an identifier, including, for example, Stock-Keeping Unit (SKU), Universal Product Code (UPC), serial number, barcode, tag, RFID, or other like encodings of identifying information. One or more imaging devices 140 may generate a mapping of one or more items in supply chain network 100 by scanning an identifier or object associated with an item and identifying the item based, at least in part, on the scan. [0050] Product category data 234 may comprise category data for each of the products stored in product placement data 232. Product categories may organize products into any number of separate or overlapping categories. By way of example only and not by way of limitation, product category data 234 may comprise, for a series of drink products, alcoholic and non-alcoholic categories, beer, wine, and spirits categories).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Savvides with the teachings of Tiwari to use different models for different beverage types because "Simulation engine 218 may access database 114, including but not limited to historical data 230, product placement data 232, product category data 234, product sales history data 236, planogram dimension data 238, and/or other data, and may generate one or more planogram options for specific product category and store combinations based on changes detected in product packaging" [Tiwari 0045].
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666 /Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights