Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments, see Remarks Pgs. 10-11, filed 7/6/2026 have been fully considered but they are not persuasive.
Applicant argues that the prior art does not disclose the claimed limitation as written in claim 1. Examiner respectfully disagrees.
Applicant argues (Pg. 11, second paragraph):
For example, Applicant submits that Sawada fails to teach or suggest detecting whether the camera’s view into the storage space of a cart or basket is obstructed nor the claimed “Occluded” status classification.
Examiner responds:
Sawada discloses (Para. 123, "The settlement apparatus with the above-described structure recognizes upper commodities among the commodities which are disposed in an overlapping manner. In addition, the settlement apparatus prompts the user to take out the upper commodities which are recognizable, thereby causing lower commodities to be exposed."). This prompt is a status classification that there is an occlusion of the storage space. One of ordinary skill in the art would have understood that detecting whether an upper commodity is blocking visibility of a lower commodity constitutes a detection of whether the camera’s view into the storage space of a cart or basket is obstructed or “Occluded” in view of the ordinary and customary meaning of the term occlude: to block, obstruct, or prevent from being seen. Applicant further argues, see Remarks Pg. 11, paragraph 1, that the “Occluded” status classification addresses situations where “the mesh walls of the shopping cart obstruct a view inside the basket” or “a portion of the body of the shopper blocks a view of a space defined by the basket”, pointing to their specification (Paras. 101 and 135) for support. However, the claim as written requires none of these limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 11, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Sawada (previously cited) in view of Carter et al. (US Patent Pub. No. 2021/0287013 A1, published 2021).
Regarding claim 1, Sawada teaches a retail store self-checkout station (Fig. 1), comprising: a scanner configured to enable a shopper to scan machine-readable indicia on items being purchased (Para. 82, “For example, the processor 21 reads a bar code in which the commodity code indicating the commodity is encoded. For example, the processor 21 reads the bar code by raster-scanning the ROI image.”); a camera positioned and oriented (i) to capture images of an area in which a shopping cart or shopping basket at the self-checkout station is to be positioned by the shopper during self-checkout, and (ii) to generate image signals of a self-checkout area at the self-checkout station (Para. 20, “The housing 2 is formed such that the basket 10 can be disposed thereon.”; Para. 21, “The camera 3 photographs commodities in the basket 10.”); and a processor in communication with the camera and scanner (Para. 32, “As illustrated in FIG. 2, the settlement apparatus 1 includes the camera 3, display 4, operation unit 5, weight scale 6, processor 21, a ROM 22…”), the processor configured to: process the self-checkout area to determine a status classification including one of the following:(i) No Cart: a shopping cart or shopping basket is not in the self- checkout area (Para. 108, “To start with, the processor 21 of the settlement apparatus 1 determines whether the basket 10 was disposed on the weight scale 6 (ACT 11). If the processor 21 determines that the basket 10 was not disposed on the weight scale 6 (ACT 11, NO), the processor 21 returns to ACT 11.”; (ii) Empty: a shopping cart or shopping basket in the self-checkout area is empty; (iii) Not Empty: a shopping cart or shopping basket in the self-checkout area currently includes an item for purchase (Para. 114, “Upon recognizing the commodities, the processor 21 acquires registered weights of the recognized commodities (ACT 22). Upon acquiring the registered weights, the processor 21 calculates a difference weight, based on the first weight and the second weight (ACT 23).”; Para. 115, “Upon calculating the difference weight, the processor 21 determines whether the total of the registered weights agrees with the difference weight (ACT 24). If the processor 21 determines that the total of the registered weights agrees with the difference weight (ACT 24, YES), the processor 21 determines whether the basket 10 is empty or not (ACT 25).”); and(iv) Occluded: an occlusion of a storage space of the shopping cart or shopping basket exists for the camera (Para. 123, “The settlement apparatus with the above-described structure recognizes upper commodities among the commodities which are disposed in an overlapping manner. In addition, the settlement apparatus prompts the user to take out the upper commodities which are recognizable, thereby causing lower commodities to be exposed.”); in response to determining the status classification, communicate a notification indicative of the status classification to the shopper (Para. 63, “the processor 21 may display a guidance which prompts take-out of the commodities in the extracted commodity areas from the basket 10. In the example of FIG. 4, the processor 21 displays, on the display 4 or the like, a guidance indicating take-out of the commodities in the frame 31A and frame 32A. For example, the processor 21 displays a message such as “Please take out commodities, which are surrounded by the frames, from the basket, and put them in a disposable plastic bag or in your shopping bag”.”).
Sawada does not explicitly disclose performing the status classification of (i) No Cart, (ii) Empty, or (iii) Not Empty solely by processing a captured image of the self-checkout area. However, they do utilize a camera to perform processing on captured images of the self-checkout area.
Carter teaches to process a captured image of the area to determine a status classification including one of the following:(i) No Cart: a shopping cart or shopping basket is not in the area (Fig. 1A, shows the detection of a cart, implicitly determining if there is a cart there or not); (ii) Empty: a shopping cart or shopping basket in the area is empty (Fig. 1B); (iii) Not Empty: a shopping cart or shopping basket in the area currently includes an item for purchase (Fig. 1A).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sawada to incorporate the teachings of Carter to include determining a status classification by processing a captured image. Sawada teaches to perform status classifications on carts or baskets within a self-checkout area, as well as using a camera to image the cart and identify items inside it. However, Sawada uses a scale to perform the majority of status classifications, rather than a captured image. Carter teaches to analyze images inside of a store to make a status classification on the many carts moving throughout it. One of ordinary skill in the art would have recognized that including the image classification technique of Carter into the self-checkout status classification method of Sawada would predictably supplement its weight-based detection with a more robust image-derived classification system, improving classification accuracy and providing critical redundancy to the self-checkout system.
Regarding claim 2, Sawada as modified in view of Carter teaches all of the elements of claim 1, as stated above as well as wherein the processor, in processing the captured image, is further configured to: identify, using artificial intelligence, that a shopping cart or shopping basket is in the self-checkout area (Para. 49, “To begin with, the processor 21 includes a function of acquiring an image captured by photographing a predetermined place where a plurality of commodities are disposed. Here, the processor 21 acquires an image captured by photographing a plurality of commodities existing in the basket 10.”; Para. 50, “For example, the processor 21 detects that the basket 10 was disposed in a predetermined area.”; Carter; Para. 37, “The cart containment system can use cameras installed in the store (and/or cameras mounted to the shopping carts) to image shopping cart baskets and can use computer vision and machine learning techniques to analyze the images”; Para. 38, “If the system detects that an at least partially loaded cart is attempting to exit the store”, Sawada aims the camera at a self-checkout area, but does not process the captured image to identify its position, instead using a scale. Carter discloses identifying a cart position when a cart is attempting to exit, leaving it obvious to implement this area specific identification for the self-checkout area); responsive to identifying that a shopping cart or shopping basket is in the self-checkout area, determine whether an occlusion of a storage space defined by either the shopping cart or shopping basket exists from the captured image by determining whether a wall of the shopping cart or shopping basket is obstructing visibility of the camera into at least a part of storage space (Para. 123, “The settlement apparatus with the above-described structure recognizes upper commodities among the commodities which are disposed in an overlapping manner… As a result, the settlement apparatus can effectively recognize the commodities which are disposed in an overlapping manner.”, A person of ordinary skill in the art would recognize that, when capturing images from non-orthogonal angles (as disclosed in both Sawada), the structural elements such as the side walls of a shopping cart can block visibility into parts of the cart’s interior. It would have been obvious to further utilize the occlusion determination method as taught by Sawada to determine any occlusions caused by a cart wall); responsive to determining that an occlusion does not exist, determine, based on the captured image, whether the shopping cart or shopping basket in the self-checkout area is empty or not empty (Para. 115, “Upon calculating the difference weight, the processor 21 determines whether the total of the registered weights agrees with the difference weight (ACT 24). If the processor 21 determines that the total of the registered weights agrees with the difference weight (ACT 24, YES), the processor 21 determines whether the basket 10 is empty or not”; Carter; Fig. 1A, 1B); and generate the status classification of the captured image based on the determinations as to whether or not:(i) a shopping cart or shopping basket is in the self-checkout area (Para. 49, “To begin with, the processor 21 includes a function of acquiring an image captured by photographing a predetermined place where a plurality of commodities are disposed. Here, the processor 21 acquires an image captured by photographing a plurality of commodities existing in the basket 10.”;Para. 109, “If the processor 21 determines that the basket 10 was disposed on the weight scale 6 (ACT 11, YES), the processor 21 acquires a first image by using the camera 3 (ACT 12).”; Carter; Para. 38, “If the system detects that an at least partially loaded cart is attempting to exit the store”), (ii) an occlusion of the shopping cart or basket exists for the camera (Para. 123, “The settlement apparatus with the above-described structure recognizes upper commodities among the commodities which are disposed in an overlapping manner.”), or (iii) the shopping cart or shopping basket is empty (Para. 117, “If the processor 21 determines that the basket 10 is empty (ACT 25, YES), the processor 21 settles the recognized commodities (ACT 28).”; Carter; Fig. 1B).
Regarding claim 3, Sawada as modified teaches all of the elements of claim 1, as stated above, as well as wherein the processor, in communicating a notification to the shopper, includes communicating an audible notification to the shopper indicative of the status classification (Para. 105, “The processor 21 also includes a function of outputting, if the processor 21 determines that the total weight disagrees with the difference weight, an error indicating that the commodity recognition failed. For example, the processor 21 displays a message prompting an alternative action for the user, such as prompting the user to perform the settlement process once again, prompting the user to perform a settlement process at a cash register that is operated by a salesclerk, or prompting the user to call a salesclerk. Incidentally, the processor 21 may transmit the error to an external apparatus.”, One of ordinary skill would understand that an audible notification would be an obvious substitution to a message prompt).
Regarding claim 11, the recited method performs substantially the same function as that of claim 1. It is rejected under the same analysis.
Regarding claim 16, the recited elements perform substantially the same function as that of claim 2. It is rejected under the same analysis.
Claim(s) 4-9, 12-15, 17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Sawada as modified in view of Carter above, further in view of Moteki (US Patent Pub. No. 2023/0094892, previously cited).
Regarding claim 4, Sawada as modified in view of Carter teaches all of the elements of claim 1, as stated above, as well as to identify an action indicative of an error or misbehavior by the shopper (Para. 124, “Thus, the settlement apparatus can detect the actually taken-out commodities among the upper commodities which are recognizable. As a result, the settlement apparatus can prevent an unlawful act, such as take-out of a non-recognized commodity by the user, compared to a method in which a commodity, after recognized, is taken out.”).
Sawada as modified in view of Carter does not explicitly disclose to analyze a sequence of captured images inclusive of the shopper while scanning items at the self-checkout station or responsive to identifying an action indicative of an error or misbehavior, generate a signal to notify the shopper and/or personnel at the retail store that an identification of a potential error or misbehavior by the shopper has been made.
Moteki teaches wherein the processor is further configured to: analyze a sequence of captured images inclusive of the shopper while scanning items at the self-checkout station (Moteki, Para. 39, “Furthermore, the estimation device 10 uses an existing object detection technology to identify, from the captured images, a customer staying in the store (may simply be referred to as “person” hereinafter), a shopping basket (may simply be referred to as “basket” hereinafter) or a shopping cart held by the person, and the user terminal 100” ); identify an action indicative of an error or misbehavior by the shopper (Moteki, Para. 50, “Therefore, the fraud detection system recognizes behaviors of the customers from the videos captured by the camera device 200 and detects the fraudulent behavior.”); and responsive to identifying an action indicative of an error or misbehavior, generate a signal to notify the shopper and/or personnel at the retail store that an identification of a potential error or misbehavior by the shopper has been made (Moteki, Para. 43, “The clerk terminal 300 receives an alert from the estimation device 10, when the estimation device 10 detects a fraudulent behavior of the customer, such as omitting scanning an item” ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sawada and Carter to incorporate the teachings of Moteki to include images inclusive of the shopper while scanning items at the self-checkout station and when identifying an action indicative of an error or misbehavior, to generate a signal to notify the shopper and/or personnel at the retail store that an identification of a potential error or misbehavior by the shopper has been made. Both Sawada and Carter disclose methods to prevent theft and increase security around the store and the self-checkout area. Moteki teaches to image the shopper while at the self-checkout area in order to identify actions indicative fraudulent behavior. One of ordinary skill in the art would have recognized that implementing the method of Moteki into the modified system of Sawada and Carter would predictably improve security at the self-checkout area, reducing the risk of theft.
Regarding claim 5, Sawada as modified teaches all of the elements of claim 4, as stated above, as well as to identify and track hands of the shopper in the sequence of images to determine motions of the hands of the shopper (Moteki, Fig. 8, shows the full skeleton being tracked, including the hands of the shopper); identify primitive motions of the shopper based on the motion of the hands of the shopper (Moteki, Para. 105, “Furthermore, as illustrated on the right side of FIG. 16, when the absolute three-dimensional coordinate position of the right wrist or left wrist of the person enters inside the three-dimensional coordinate position of the cuboid area of the cart, for example, the estimation device 10 determines that the person puts his or her hand into the cart.”); determine, using a visual scan analysis, that the primitive motions of the shopper are indicative of a potential error or misbehavior (Moteki, Para. 105, “the estimation device 10 can also detect the actions of the person and perform behavior recognition without being affected by the angle of the captured image. Then, the estimation device 10 can store the behaviors such as purchasing items by the behavior recognition of the person, and determine whether there is any fraudulent behavior to detect a fraudulent behavior.”); and communicate a notification signal to notify the shopper and/or personnel at the retail store that an identification of the potential error or misbehavior of the shopper has been made (Moteki, Para. 123, “if a fraudulent behavior of the person is recognized by the action detection executed at step S208, the estimation device 10 notifies the clerk terminal 300 or the like of an alert.”).
Regarding claim 6, Sawada as modified teaches all of the elements of claim 5, as stated above, as well as a data repository configured to store reference model motions of hands performing errors and/or misbehaviors (Moteki, Para. 66, “For example, the storage unit 30 stores therein a human skeleton 3D model for estimating three-dimensional skeleton information of a person, an object 3D model for estimating the pose of an object, and the like.”; Para. 69, “The above information stored in the storage unit 30 is only an example, and the storage unit 30 can store therein various kinds of other information in addition to the information described above.”; Para. 80, “Based on the absolute three-dimensional skeleton information of the person as well as the three-dimensional position and pose data of the cart, the action determination unit determines and detects the action of the person”, a storage unit storing various 3D models and pose information is disclosed, as well as an action determination unit. Given that fraudulent behavior detection is disclosed, it is necessary for the determination unit to be trained on fraudulent behavior in order to detect it); and wherein the processor is further configured to: identify, using a trained neural network, potential errors or misbehaviors, from the primitive motions of the shopper from the sequence of images (Moteki, Para. 39, “Furthermore, the estimation device 10 uses an existing skeleton detection technology to generate skeleton information of the person identified from the captured image to estimate the position, pose, and scale of the person, and detects an action such as grasping the cart, putting items into a basket or cart, and the like.”; Para. 54, “existing skeleton estimation algorithms are skeleton estimation algorithms that use deep learning such as HumanPoseEstimation like DeepPose and OpenPose, for example.”); and responsive to identifying that the motions of the shopper are indicative of an error or misbehavior, generating the signal to notify the shopper and/or personnel at the retail store that an identification of a potential error or misbehavior by the shopper has been made (Moteki, Para. 51, “Then, based on the person, the cart, and the like identified from the captured image 250, the estimation device 10 determines the behavior of the person, such as whether the person has put an item into the cart and scanned the item, for example. The estimation device 10 then detects the person acting improperly, such as omitting scanning the item, as the target and notifies the clerk terminal 300 of the alert.”).
Regarding claim 7, Sawada as modified in view of Carter teaches all of the elements of claim 1, as stated above, and when further modified in view of Moteki also teaches wherein the processor, in processing the captured image, is configured to render a 3D image from 2D images of the shopping cart or shopping basket (Moteki, Para. 90, “As illustrated in FIG. 11, the estimation device 10 inputs a partial image in the bounding box of the cart in the captured image and the CAD of the cart to a machine learning model to acquire a three-dimensional rectangular area that is a cuboid area indicating the three-dimensional position and the pose of the cart.”) to determine whether an occlusion of the storage space of the shopping cart or shopping basket exists for the camera (Para. 123, “The settlement apparatus continues the same operation until there remains no commodity. As a result, the settlement apparatus can effectively recognize the commodities which are disposed in an overlapping manner.”; Moteki, Para. 53, “In addition to persons and baskets, for example, items, the user terminal 100, sales areas of items such as aisles and shelves of the items, clothing of the persons, and the like may also be detected from the captured images.”, Moteki discloses a 3D render of the cart is disclosed, as well as the detection of items inside the cart. In view of Sawada, performing occlusion detection would be obvious).
Regarding claim 8, Sawada as modified in view of Carter teaches all of the elements of claim 1, as stated above, and when further modified in view of Moteki also teaches a database including shopping cart and shopping basket model information that describes physical attributes of the shopping cart and shopping basket at different angles (Moteki, Para. 62, “Here, since information regarding the length of the cart 160 handled in the store is known in advance and can be held in the estimation device 10”; Para. 67, “The storage unit 30 also stores therein the length information of the cart, such as the length of the grip part of the cart.”); and wherein the processor, in determining whether a shopping cart or shopping basket is in the self-checkout area (Sawada, Para. 50, “For example, the processor 21 detects that the basket 10 was disposed in a predetermined area.”), the processor being configured to: access the database inclusive of the shopping cart and shopping basket model information; and identify, using the shopping cart or shopping basket model information, whether a shopping cart or shopping basket is captured in the image signals (Moteki, Para. 39, “Furthermore, the estimation device 10 uses an existing object detection technology to identify, from the captured images… a shopping basket (may simply be referred to as “basket” hereinafter) or a shopping cart held by the person”).
Regarding claim 9, Sawada as modified in view of Carter teaches all of the elements of claim 1, as stated above, and when further modified in view of Moteki also teaches wherein the camera is an overhead, non- orthogonal camera (Sawada; Para. 21, “The camera 3 may be disposed in a manner to photograph the inside of the basket 10 obliquely from above. The position and direction for disposition of the camera 3 are not restricted to a specific configuration.”; Moteki; Fig. 4, a non-orthogonal image is shown).
Regarding claim 12, the recited elements perform substantially the same function as that of claim 9. It is rejected under the same analysis.
Regarding claim 13, the recited elements perform substantially the same function as that of claim 5. It is rejected under the same analysis.
Regarding claim 14, Sawada as modified in view of Carter teaches all of the elements of claim 11, as stated above, and when further modified in view of Moteki also teaches determining, by the processor, whether a machine-readable indicia that is scanned is associated with an item that the shopper is scanning as captured in the image signals (Moteki, Para. 50, “The estimation device 10 identifies persons and objects from the captured image 250. Then, based on the person, the cart, and the like identified from the captured image 250, the estimation device 10 determines the behavior of the person, such as whether the person has put an item into the cart and scanned the item, for example”); responsive to determining that the item being scanned is associated with or not associated with the machine-readable indicia scanned by the shopper, setting, by the processor, a third status classification (Para. 124, “Additionally, after the user took out commodities, the settlement apparatus executes a recognition process of the taken-out commodities, based on the ROI images that are the images of the taken-out commodities. Thus, the settlement apparatus can detect the actually taken-out commodities among the upper commodities which are recognizable. As a result, the settlement apparatus can prevent an unlawful act, such as take-out of a non-recognized commodity by the user, compared to a method in which a commodity, after recognized, is taken out.”); and communicating, by the processor, a notification to the shopper and/or retail store personnel (Moteki, Para. 50, “The estimation device 10 then detects the person acting improperly, such as omitting scanning the item, as the target and notifies the clerk terminal 300 of the alert.”, the estimation device detects improper actions and notifies the clerk, leaving it obvious that a status associated with that action is set).
Regarding claim 15, Sawada as modified teaches all of the elements of claim 14, as stated above, as well as wherein determining whether a machine-readable indicia that is scanned is associated with the item that the shopper is scanning includes identifying the item, by the processor, based on shape, color, and/or weight of the item (Moteki, Para. 52, “Note here that an existing object detection algorithm is an object detection algorithm using deep learning such as faster R-convolutional neural network (CNN), for example. It may also be an object detection algorithm such as you only look once (YOLO) or single shot multibox detector (SSD).”; Para. 53, “In addition to persons and baskets, for example, items… and the like may also be detected from the captured images.”, existing object detection algorithms take shape and color into account).
Regarding claim 17, the recited elements perform substantially the same function as that of claim 7. It is rejected under the same analysis.
Regarding claim 19, Sawada as modified in view of Carter teaches all of the elements of claim 16, as stated above, and when further modified in view of Moteki also teaches wherein determining whether the shopping cart or shopping basket is empty includes utilizing a trained neural network to identify position and orientation of the shopping cart or shopping basket in the images (Moteki, Para. 72, “The object pose estimation unit inputs a partial image in the bounding box of the cart in the captured image and computer aided design (CAD) data of the cart into a machine learning model, for example, to acquire three-dimensional position and pose data of the cart.”).
Regarding claim 20, Sawada as modified in view of Carter teaches all of the elements of claim 11, as stated above, and when further modified in view of Moteki also teaches further comprising preventing the shopper from tendering payment at the self-checkout station in response to determining that the basket is not empty or an occlusion of the basket from a camera exists (Para. 125, “Additionally, the settlement apparatus continues the recognition process if the pre-registered weight of the commodity agrees with the weight of the taken-out commodity. As a result, the settlement process can prevent an unlawful act, such as take-out of a non-recognized commodity by the user.”; Moteki, Para. 50, “In particular, it is easy for the clerks or the like to detect fraudulent behaviors when the number of items is small, but it is difficult for the clerks or the like to detect such behaviors when the number of items is large and some of the items are not scanned, for example. Therefore, the fraud detection system recognizes behaviors of the customers from the videos captured by the camera device 200 and detects the fraudulent behavior.”, if improper or fraudulent behavior is detected (such as not emptying the basket of items that have not been scanned, or occluding items in the cart), the shopper would not be allowed to pay and a clerk is notified).
Claim(s) 10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Sawada as modified in view of Carter and Moteki, and further in view of Cho (NPL, “Partially Occluded Object-Specific Segmentation in View-Based Recognition”, previously cited).
Regarding claim 10, Sawada as modified in view of Carter teaches all of the elements of claim 1, as stated above, and when further modified in view of Moteki also teaches wherein the processor, in determining that the status classification is Occluded, is further configured to: determine whether the occluded region is due to a side wall of the basket of the shopping cart blocking a space in which items are placed in the basket of the shopping cart (Para. 22, “In this case, the plural cameras 3 may be disposed in a manner to photograph commodities in the basket 10 at different positions and angles.”; Para. 123, “In addition, the settlement apparatus prompts the user to take out the upper commodities which are recognizable, thereby causing lower commodities to be exposed. The settlement apparatus recognizes the lower commodities. The settlement apparatus continues the same operation until there remains no commodity. As a result, the settlement apparatus can effectively recognize the commodities which are disposed in an overlapping manner.”, Occlusion determination is performed on overlapping commodities. One of ordinary skill in the art would understand that accounting for occlusions caused by the wall of a cart would be a routine application of the already disclosed occlusion determination method.) and in response to determining that an item is contained in the basket of the shopping cart, setting the status classification to Not Empty (Para. 115, “ If the processor 21 determines that the total of the registered weights agrees with the difference weight (ACT 24, YES), the processor 21 determines whether the basket 10 is empty or not (ACT 25).”; Para. 116, “If the processor 21 determines that the basket 10 is not empty (ACT 25, NO), the processor 21 sets the second image as the first image (ACT 26)”).
Sawada as modified does not explicitly disclose dividing an image of the basket of the shopping cart into a visible region and an occluded region; performing a small blob analysis to identify items in the space that items are visible via any openings defined by the side wall of the basket of the shopping cart; form a blob inclusive of the small blobs to determine that an item is contained within the basket of the shopping cart; and determining that the item is in the basket based on the formed blob.
PNG
media_image1.png
614
671
media_image1.png
Greyscale
Cho teaches to divide an image of an obstructed item into a visible region and an occluded region; perform a small blob analysis to identify items in the space that items are visible via any openings available through the obstructed item; and form a blob inclusive of the small blobs to determine that an item is present (Fig. 6, reprinted below, showcases the division of an item into a visible and occluded region, the performance of a “small blob” analysis which detects the visible parts of the item through available openings, and forms a “blob” inclusive of the “smalls blobs” to determine an items presence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Sawada, Carter and Moteki to incorporate the teachings of Cho to include dividing an image of the basket of the shopping cart into a visible region and an occluded region; performing a small blob analysis to identify items in the space that items are visible via any openings defined by the side wall of the basket of the shopping cart; form a blob inclusive of the small blobs to determine that an item is contained within the basket of the shopping cart; and determining that the item is in the basket based on the formed blob. Cho teaches a method of object identification under partial occlusion by assembling visible “blobs” and comparing them to stored model views of objects. Although Cho applies this in a general object recognition context, one of ordinary skill in the art, in view of Moteki’s disclosed preprocessing of using stored model cart information, and Sawada’s teaching of a retail self-checkout environment, would find it obvious to apply Cho’s method to self-checkout carts, where the inventory of potential items is fixed and known in advance. The skilled artisan would recognize the benefit of accurately identifying partially occluded objects in a cart, especially when considering that occlusion scenarios are a known challenge in self-checkout systems (Sawada; Para. 3), and would reasonably adapt Cho’s model-based approach using stored views to improve classification accuracy in this context.
Regarding claim 18, Sawada as modified teaches all of the elements of claim 16, as stated above, and when further modified in view of Moteki also teaches wherein determining whether an item is visible via spaces defined by a wall of the shopping cart or shopping basket includes applying a size- based content classification process relative to the size of the spaces defined by the wall of the shopping cart or shopping basket (Cho, Fig. 6, reprinted above, shows the classification of foreground obstructions as well as the size of the visible region).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A WAMBST whose telephone number is (703)756-1750. The examiner can normally be reached M-F 9-6:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ALEXANDER WAMBST/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights