DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-15 are pending for examination.
Claims 1 is independent Claims.
Claims 1-15 are rejected under 35 U.S.C. §103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8-13 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Glaser et al. (U.S. 2019/0333039 hereinafter Glaser) in view of Gong et al. (U.S. 2023/0296802 hereinafter Gong) in further view of Kalyuzhner et al. (U.S. 2021/0365719 hereinafter Kalyuzhner).
As Claim 1, Glaser teaches an artificial intelligence identification scale system based on autonomous incremental learning, comprising a weighing system, a weight sensor, a visual sensor and an identification system, wherein,
the weighing system is configured to receive a weight of a target commodity (Glaser (¶0161 line 6-7, fig. 3 item S310), “collecting weight information from a scale when the first set of items is present at a scale S310”), and receive and send an image of the target commodity to the identification system (Glaser (¶0161 line 1-5, fig. 2 tiem 220), “collecting image data Sl00; detecting selection of a first set of items S210 by a first user through computer vision; applying computer vision to the image data and detecting an item identity of the first set of items S220”);
receive at least one candidate commodity identifier of the target commodity, determine a commodity identifier of the target commodity therefrom (Glaser (¶0088 line 1-3), “detecting item identity of a set of items through computer vision processing of the image data”), and send the commodity identifier of the target commodity to the identification system (Glaser (¶0088 last 5 lines), “A product database may store product information indexed or searchable by the item identifier. For example, detecting item identity can lead to obtaining item density data, volumetric data, and the like which can be used in further CV processing or checkout processing.”); and
generate a commodity label based on the weight and the commodity identifier of the target commodity (Glaser (¶0077 last 5 lines), “generating an item price factoring in weight information and the item identity”);
the weight sensor is configured to obtain and send the weight of the target commodity to the weighing system (Glaser (¶0161 line 6-7, fig. 3 item S310, fig. 8), “collecting weight information from a scale when the first set of items is present at a scale S310”);
the visual sensor is configured to obtain and send the image of the target commodity to the weighing system (Glaser (¶0078 line 4-5, fig. 8), “applying volumetric interpretation of a first set of items that is at least partially based on image data S320”);
determine and output a candidate commodity identifier corresponding to the at least one feature vector to the weighing system (Glaser (¶0078 line 4-5, fig. 8), “applying volumetric interpretation of a first set of items that is at least partially based on image data S320”); and
with a trademark identifier of the target commodity (Glaser (¶0044 line 6-8), “Preferably, a product can be classified and associated with a product SKU (stock keeping unit) or PLU (price look up) identifier”. Glaser (¶0048 line 3-5), “item related properties may additionally be tracked such as color characterization, inventory source (trademark), date of stocking, history of location/placement, and the like”)
Glasser may not explicitly disclose:
the identification system comprises an identification algorithm module, an incremental learning algorithm module and a data module, wherein, the identification algorithm module is configured to deal the image of the target commodity and acquire a feature vector of the image;
calculate a similarity between the feature vector and feature vectors in a feature vector database to obtain at least one feature vector exceeding a similarity threshold; and
Gong teaches:
the identification system comprises an identification algorithm module (Gong (¶0050 line 4-5), “The determination may use the food image recognition ML model 350 based on the series of images”), an incremental learning algorithm module and a data module (Gong (¶0048 last 5 lines), “These locally collected data may be labeled periodically by the operator (like every day, every week). In other words, the food image recognition ML model 350 on the device is self-evolving by performing these periodic "on-device training" cycles”), wherein, the identification algorithm module is configured to deal the image of the target commodity and acquire a feature vector of the image (Gong (¶0047, bottom portion), “Based on a distance between the matching probabilities and the label of the food image, the parameters of the neural network may be adjusted to minimize the distance. In some embodiments, the distance may be determined as a Euclidean distance.”);
calculate a similarity between the feature vector and feature vectors in a feature vector database to obtain at least one feature vector exceeding a similarity threshold (Gong (¶0050 line 6-9), “The output of the food image recognition ML model 350 may include one or more food candidates similar to the food in the image with corresponding matching probabilities”); and
Glaser discloses a system/method to estimate food item weight based on item recognition. Gong discloses a system/method to use machine learning for food item recognition. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify object recognition module of Glaser instead be a machine learning object recognition module taught by Gong, with a reasonable expectation of success. The motivation would be so that “based on the estimated portion-based food information, the user may timely adjust the portion size before placing the food onto a plate or container.” (Gong (¶0031 line 14-16))
Glaser in view of Gong may not explicitly disclose:
the incremental learning algorithm module is configured to judge whether incremental learning is needed after receiving the commodity identifier determined by the target commodity, and if so, extract a feature vector after performing enhancement processing on the image, combine the extracted feature vector and store into the feature vector database of the data module.
Kalyuhzner teaches:
the incremental learning algorithm module is configured to judge whether incremental learning is needed after receiving the commodity identifier determined by the target commodity, and if so, extract a feature vector after performing enhancement processing on the image (Kalyuzhner (¶0034 line 1-4), “Subsequently to identifying the candidate objects, the processor augments the training set with images containing those of the identified candidates that are most likely true positives.”), combine the extracted feature vector and store into the feature vector database of the data module (Kalyuzhner (¶0048 (line 1-3), “For each of the candidate images, candidate-image feature-vector extractor 68 extracts a respective feature vector for each candidate object identified in the image”).
Glaser discloses a system/method to estimate food item weight based on item recognition. Gong discloses a system/method to use machine learning for food item recognition. The system of Gong updates the machine learning model at every set period. Kalyzhner discloses a system/method for detect new image(s)/sample for incremental learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify object recognition module of Glaser in view of Gong instead be a few-shot machine learning module taught by Gong, with a reasonable expectation of success. The motivation would be to “to provide the model with additional training images in which objects of other classes are identified.” (Kalyuzhner (¶0029 last 3 lines))
As Claim 2, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the weighing system comprises:
a configuration component configured to configure the weight sensor (Glaser (¶0161 line 6-7, fig. 3 item S310), “collecting weight information from a scale when the first set of items is present at a scale S310”), the visual sensor (Glaser (¶0078 line 4-5, fig. 8), “applying volumetric interpretation of a first set of items that is at least partially based on image data S320”), the identification system and a cashier system which are connected (Glaser (¶0083 last 5 lines), “For example, a storage bin of produce may be monitored with one set of cameras in a first region and then a scale and/or POS system may be monitored with a second set of cameras in a second region”);
a touch screen (Gong (¶0032, last 3 lines), “The display 120 may be used as an input device when it is a touch screen and receives user input.”) configured to display the commodity identifier and the weight of the target commodity (Glaser (¶0088 last 6 lines), “A product database may store product information indexed or searchable by the item identifier. For example detecting item identity can lead to obtaining item density data, volumetric data, and the like which can be used in further CV processing or checkout processing.”);
an interaction module configured to receive the weight of the target commodity (Glaser (¶0161 line 6-7, fig. 3 item S310, fig. 8), “collecting weight information from a scale when the first set of items is present at a scale S310”), and receive and send the image of the target commodity to the identification system (Glaser (¶0088 line 1-3), “detecting item identity of a set of items through computer vision processing of the image data”);
receive the at least one candidate commodity identifier of the target commodity (Glaser (¶0088 last 5 lines), “A product database may store product information indexed or searchable by the item identifier. For example, detecting item identity can lead to obtaining item density data, volumetric data, and the like which can be used in further CV processing or checkout processing.”);
determine the commodity identifier of the target commodity therefrom and send the commodity identifier to the identification system (Glaser (¶0088 last 5 lines), “A product database may store product information indexed or searchable by the item identifier. For example, detecting item identity can lead to obtaining item density data, volumetric data, and the like which can be used in further CV processing or checkout processing.”);
perform a search operation after receiving a search instruction input by a user (Glaser (¶0088 last 5 lines), “A product database may store product information indexed or searchable by the item identifier. For example, detecting item identity can lead to obtaining item density data, volumetric data, and the like which can be used in further CV processing or checkout processing.”);
call the identification system to identify the target commodity after receiving a re-identification instruction from the user (Glaser (¶0094 line 4-8), “Detecting item identity of a set of items preferably includes detecting selection of a set of items S210 from storage by a first user through computer vision and applying computer vision to the image data and detecting an item identity of the set of items S220”); and
receive a commodity identifier confirmed by the user (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”); and
a settlement module configured to generate the commodity label based on the weight and the commodity identifier of the target commodity (Glaser (¶0077 last 5 lines), “generating an item price factoring in weight information and the item identity”); and
settle the target commodity through the connected cashier system (Glaser (¶0088 last 5 lines), “A product database may store product information indexed or searchable by the item identifier. For example detecting item identity can lead to obtaining item density data, volumetric data, and the like which can be used in further CV processing or checkout processing.”).
As Claim 3, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the interaction module of the weighing system is further configured to:
receive a commodity identifier input by a user when the received candidate commodity identifier of the target commodity is null (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”); and
take the commodity identifier input by the user as the determined commodity identifier (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”).
As Claim 4, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the identification algorithm module comprises:
a target detection module configured to identify a commodity area in the image of the target commodity (Glaser (¶0044 line 2-5), “detect and apply an identifier to an object. The item detection module preferably performs a combination of object detection, segmentation, classification, and/or identification.”);
an image segmentation module configured to segment the commodity area from the image of the target commodity (Glaser (¶0087 line 6-9), “image data from multiple distinct views of objects in the environment can be converted to a 3D model of a portion of the environment.”); and
an image identification algorithm module configured to identify the commodity area to obtain the feature vector of the image (Kalyuzhner (¶0048 line 1-3), “For each of the candidate images, candidate-image feature-vector extractor 68 extracts a respective feature vector for each candidate object identified in the image”);
calculate the similarity between the feature vector and the feature vectors in the feature vector database to obtain at least one feature vector (Kalyuzhner (¶0048 last 5 lines), “The extracted feature vectors, together with a reference data item indicating the candidate image and object to which each feature vector corresponds, are passed to a similarity-score calculator 72”) exceeding the similarity threshold (Kalyuzhner (¶0050), “checks whether the score passes a predefined threshold. (In some embodiments, the similarity-score assessor, prior to assessing any of the scores, calculates the threshold based on the scores.) The similarity-score assessor further updates the reference data item to indicate each score that passes the threshold, and then passes the reference data item to a training-set augmenter 76”); and
determine and output the candidate commodity identifier corresponding to the at least one feature vector to the weighing system (Kalyuzhner (¶0032 line 4-7), “the processor identifies candidate objects of the class (i.e., groups of pixels having a relatively high likelihood of representing objects of the class) in the images belonging to the image repository.”).
As Claim 5, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the incremental learning algorithm module is specifically configured to:
determine that incremental learning is needed in the following cases:
among the candidate commodity identifier identified corresponding to the at least one feature vector, the commodity identifier determined by the target commodity ranks first, and a confidence of the target commodity is lower than a confidence threshold (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”);
among the candidate commodity identifiers identified corresponding to the at least one feature vector, the commodity identifier determined by the target commodity does not rank first (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”); and
the identified candidate commodity identifier is null (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”).
As Claim 6, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the incremental learning algorithm module is specifically configured to:
extract the feature vector after performing enhancement processing on the image using the following steps:
performing enhancement processing on the image of the target commodity, the enhancement processing comprising at least one selected from the group of rotation, flipping and color transformation (Kalyuzhner (¶0044 last 3 lines), “For embodiments in which the object is segmented, pixels outside the segmented boundary of the object may be set to a constant color”;)
calling the identification algorithm module to deal the image after the enhancement processing to obtain a feature vector (Kalyuzhner (¶0048 lines 1-3), “For each of the candidate images, candidate-image feature-vector extractor 68 extracts a respective feature vector for each candidate object identified in the image”;); and
removing an item with a differentiation less than a differentiation threshold from the feature vector to obtain the extracted feature vector (Kalyuzhner (¶0044 last 3 lines), “For embodiments in which the object is segmented, pixels outside the segmented boundary of the object may be set to a constant color”;).
As Claim 8, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the visual sensor is further configured to obtain a video stream of the target commodity, and send the video stream to the identification system through the weighing system; and the identification algorithm module of the identification system is further configured to extract the image of the target commodity from the video stream (Glaser (¶0083 line 1-3), “which includes collecting image data, functions to collect video, pictures, or other imagery of a region containing objects of interest (e.g., product items)”).
As Claim 9, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the commodity identifier comprises a commodity name and a commodity code (Glaser (¶0088 line 5-8), “The item identity is preferably associated with a descriptor of that item which may be or be further associated with a product identifier such as a PLU, SKU ID, or the like”).
As Claim 10, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner teaches an identification method for an artificial intelligence identification scale based on autonomous incremental learning, wherein the method is applied to the system according to claim 1, comprising:
receiving and dealing an image of a target commodity sent by a weighing system to obtain a feature vector of the image (Glaser (¶0161 line 6-7, fig. 3 item S310, fig. 8), “collecting weight information from a scale when the first set of items is present at a scale S310”);
calculating a similarity between the feature vector and feature vectors in a feature vector database to obtain at least one feature vector (Kalyuzhner (¶0048 last 5 lines), “The extracted feature vectors, together with a reference data item indicating the candidate image and object to which each feature vector corresponds, are passed to a similarity-score calculator 72”) exceeding a similarity threshold (Kalyuzhner (¶0050), “checks whether the score passes a predefined threshold. (In some embodiments, the similarity-score assessor, prior to assessing any of the scores, calculates the threshold based on the scores.) The similarity-score assessor further updates the reference data item to indicate each score that passes the threshold, and then passes the reference data item to a training-set augmenter 76”);
determining and outputting a candidate commodity identifier corresponding to the at least one feature vector to the weighing system (Kalyuzhner (¶0032 line 4-7), “the processor identifies candidate objects of the class (i.e., groups of pixels having a relatively high likelihood of representing objects of the class) in the images belonging to the image repository.”), wherein the weighing system is configured to receive a commodity identifier of the target commodity determined by a user from the at least one candidate commodity identifier, and generate a commodity label based on the weight of the target commodity and the determined commodity identifier (Glaser (¶0077 last 5 lines), “generating an item price factoring in weight information and the item identity”);
judging whether incremental learning is needed after receiving the commodity identifier determined by the target commodity sent by the weighing system (Kalyuzhner (¶0034 line 1-4), “Subsequently to identifying the candidate objects, the processor augments the training set with images containing those of the identified candidates that are most likely true positives.”);
if so, extracting a feature vector after performing enhancing processing on the image (Kalyuzhner (¶0048 (line 1-3), “For each of the candidate images, candidate-image feature-vector extractor 68 extracts a respective feature vector for each candidate object identified in the image”); and
combining the extracted feature vector with a trademark identifier of the target commodity and storing into the feature vector database of the data module (Kalyuzhner (¶0048 (line 1-3), “For each of the candidate images, candidate-image feature-vector extractor 68 extracts a respective feature vector for each candidate object identified in the image”).
As Claim 11, besides Claim 10, Glaser in view of Gong in further view of Kalyuzhner teaches wherein the receiving and dealing the image of the target commodity sent by the weighing system to obtain the feature vector of the image comprises:
identifying a commodity area in the image of the target commodity, or segmenting the commodity area from the image of the target commodity (Glaser (¶0087 line 6-9), “image data from multiple distinct views of objects in the environment can be converted to a 3D model of a portion of the environment.”); and
identifying the commodity area to obtain the feature vector of the image (Kalyuzhner (¶0048 line 1-3), “For each of the candidate images, candidate-image feature-vector extractor 68 extracts a respective feature vector for each candidate object identified in the image”).
As Claim 12, besides Claim 10, Glaser in view of Gong in further view of Kalyuzhner teaches wherein judging whether incremental learning is needed comprises:
determine that incremental learning is needed in the following cases:
among the candidate commodity identifier identified corresponding to the at least one feature vector, the commodity identifier determined by the target commodity ranks first, and a confidence of the target commodity is lower than a confidence threshold (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”);
among the candidate commodity identifiers identified corresponding to the at least one feature vector, the commodity identifier determined by the target commodity does not rank first (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”); and
the identified candidate commodity identifier is null (Gong (¶0049, bottom portion), “when the difference between the top two or more matching probabilities is below a threshold, the controller may display the corresponding top two or more food candidates on the built-in display 360 for the user to confirm the identification of the food”).
As Claim 13, besides Claim 10, Glaser in view of Gong in further view of Kalyuzhner teaches wherein extracting the feature vector after performing enhancing processing on the image comprises:
performing enhancement processing on the image of the target commodity, the enhancement processing comprising at least one selected from the group of rotation, flipping and color transformation (Kalyuzhner (¶0044 last 3 lines), “For embodiments in which the object is segmented, pixels outside the segmented boundary of the object may be set to a constant color”;);
calling an identification algorithm module to deal the image after the enhancement processing to obtain a feature vector (Kalyuzhner (¶0048 lines 1-3), “For each of the candidate images, candidate-image feature-vector extractor 68 extracts a respective feature vector for each candidate object identified in the image”;); and
removing an item with a differentiation less than a differentiation threshold from the feature vector to obtain the extracted feature vector (Kalyuzhner (¶0044 last 3 lines), “For embodiments in which the object is segmented, pixels outside the segmented boundary of the object may be set to a constant color”;).
As Claim 15, the Claim is rejected for the same reasons as Claim 10.
Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Glaser and Gong in view of Kalyuzhner in further view of Yang et al. (U.S. 2014/0379617 hereinafter Yang).
As Claim 7, besides Claim 1, Glaser in view of Gong in further view of Kalyuzhner may not explicitly disclose:
wherein the incremental learning algorithm module is specifically configured to:
increase a first record of selected times of the commodity identifier determined by the target commodity by 1;
if the commodity identifier determined by the target commodity does not rank first, increase a second record by 1; and
calculate a confidence of the target commodity based on the first record and the second record.
Yang teaches wherein the incremental learning algorithm module is specifically configured to:
increase a first record of selected times of the commodity identifier determined by the target commodity by 1 (Yang (¶0022, middle portion), “for a buyer-user on an e-commerce transaction platform, this "quality" embodies buyer-user positive evaluation rates (positive evaluation rates relate to the ratio of positive reviews to total reviews (including good reviews, average reviews, negative reviews, etc”);
if the commodity identifier determined by the target commodity does not rank first, increase a second record by 1 (Yang (¶0022, middle portion), “for a buyer-user on an e-commerce transaction platform, this "quality" embodies buyer-user positive evaluation rates (positive evaluation rates relate to the ratio of positive reviews to total reviews (including good reviews, average reviews, negative reviews, etc”)); and
calculate a confidence of the target commodity based on the first record and the second record (Yang (¶0022, middle portion), “making recommendations based on the operating behavior information of some specific first users.”).
Glaser in view of Gong in further view of Kalyuzhner discloses a system/method to suggest food recognition. Yang disclose a system/method to suggest product based on user’s reviews. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify object suggestion module of Glaser in view of Gong in further view of Kalyuzhner instead be a suggestion module taught by Yang, with a reasonable expectation of success. The motivation would be to allow “recommendations can reduce search efforts of users. If the recommendations are sufficiently accurate, the user can directly click on a link to perform a purchase or other operation, thus the recommendations can increase transaction opportunities for both buying and selling users.” (Yang (¶0003 last 5 lines))
As Claim 14, the Claim is rejected for the same reasons as Claim 7.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Matsumoto (U.S. 2012/0239599) disclose a system/method for gene.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHAT HUY T NGUYEN whose telephone number is (571)270-7333. The examiner can normally be reached M-F: 12:00-8:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHAT HUY T NGUYEN/Primary Examiner, Art Unit 2147