DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This communication is a first office action non-final rejection on the merits. Claim(s) 1-20, as filed on 02/07/2025, are currently pending and have been fully considered below.
Claim Objections
Claim 15 is objected to because of the following informalities:
The limitation “by the computer system” of line 6 should read “by a computer system”;
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 20 is rejected under 35 U.S.C. § 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more and thus do not satisfy the criteria for subject matter eligibility.
Step 1
Claim(s) 20 is all directed to a statutory category (e.g., a process, machine, manufacture, or composition of matter).
Step 2A Prong One: Yes
Exemplary claim(s) 20 recite(s) the following abstract concepts that are found to include “abstract idea”:
A method comprising: triggering a
accessing an image output
identifying a tag, arranged on the shelving structure, depicted in a first region of the image;
detecting a set of features in the first region of the image, the set of features representing a product descriptor;
accessing a set of template features of a product associated with the product descriptor;
identifying a slot, proximal the tag, depicted in a second region of the image;
in response to detecting features analogous to the set of template features in the second region of the image, confirming presence of a first product unit of the product in the slot;
detecting a quantity of facings of the product in the image;
accessing a target quantity of facings assigned to the slot; and
in response to the quantity of facings falling below the target quantity of facings, generating a prompt to restock product units of the product in the slot.
Claim(s) 2 disclose(s) an abstract idea of product inventory, which falls into the grouping of “Certain Methods of Organizing Human Activities”. More specifically, the claims limitations above have concepts related to: receive data (A, B, E, I), identify data (C, D, F, H), confirm data (G), message data (J);
Claim(s) 20 recite(s) an abstract idea.
Step 2A Prong Two: No
Claim(s) 20 additional elements are:
Claim 20: “a robotic system, deployed in a store, to autonomously”;
The claimed additional elements that perform limitation(s) A, B, E, I are claimed at a high level of generality and are considered merely data gathering, and thus are considered nothing more than insignificant extra-solution activity; the additional elements that perform limitation(s) C, D, F, H are claimed at a high level of generality and are considered data identification without the recitation of an technical or technology improvement; the additional elements that perform limitation(s) G is claimed at a high level of generality and is considered data being check, and thus are considered instructions to implement an abstract idea on a computer; the additional elements that perform limitation(s) J is claimed at a high level of generality and is considered a message being generated from data such as no stock and stock, and thus are considered instructions to implement an abstract idea on a computer. When view in combination, the additional elements merely describe how to generally “apply” the abstract idea in a generic or general-purpose computer, and generality links the use of the judicial exception to a particular technological environment or field of use, and thus do not integrate the abstract idea into a practical application, and claim(s) 20 is directed to the judicial exception.
Claim 20 is directed to an abstract idea.
Claim 20 is not including additional element(s) that are sufficient to amount to significantly more than the judicial exception. As discussed with respect to Step 2A Prong Two, to generally links the use of the judicial exception to a particular technological environment or field of use, and thus are not significantly more to the abstract idea.
Examiner takes Official Notice that a Robotic system being used to capture image in a retail facility is an old and well known method of capturing images of shelf in a retail facility.
In addition, the courts have found computer functions claimed at high level of generality as not sufficient to show an improvement in computer-functionality (see MPEP 2106.05(a)), and well‐understood, routine, and conventional functions (see MPEP 2106.05(d)), applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e)
Even when viewed as a whole, nothing in the claims adds significantly more to the abstract idea.
Claim 20 is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4, 7-13, 15-16, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zimmerman (US 20080077511 A1) in view of Calhoon et al. (US 10007964 B1, hereinafter Calhoon).
Regarding claim 1, Zimmerman discloses:
A method comprising:
by a robotic system deployed in a store: Figures 1, 3-9 and para. 47, 59-60, 67 - mobile inventory robot 20;
autonomously navigating along a shelving structure within the store; (para. 59 the mobile inventory robot 20 navigates the store and detects out-of-stock items autonomously and automatically; further see para. 48-50, 53 and Figures 1-3 );
capturing an image of the shelving structure within a field of view of an optical sensor arranged in the robotic system; and (para. 59-60, 68 - captures images of items on shelves; further see para. 48-50, 53 and Figures 1-3);
transmitting the image to a computer system; and (para. 50 the mobile inventory robot 20 transmits images to the inventory control system 15; further see para. 48-50, 53 and Figures 1-3);
by the computer system (para.50 and 53 and Figure 2 - the inventory control system 15);
receiving the image from the robotic system; (para. 50, 53 the mobile inventory robot 20 transmits images to the inventory control system 15; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
identifying a tag, arranged on the shelving structure, depicted in the image; (para. 54 - The product image segmentation module 205 segments the received images of individual products and extract barcodes from the images of barcode placed on shelves below the items and barcodes of items; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
detecting a set of features, the set of features representing a product descriptor; (para. 54 – extract barcodes from the images of barcode placed on shelves below the items and barcodes of items, the barcode that is use to retrieve the product visual descriptor; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
accessing a set of template features of a product associated with the product descriptor; (para. 54 - uses the image of an item corresponding to the extracted barcode as the product visual descriptor found in the product database 225; Further see para. 48-54, 57-58, 68-69 and Figures 1-99);
in response to absence of features analogous to the set of template features in the image, detecting absence of the product in the slot; (para. 69 out-of-stock detector 215 determines whether the product visual descriptor in the product database 225 sufficiently matches the shelf image captured by the imaging device 30, if does not match the inventory control module 215 flags for the expect product at the product location; Please not this process is made for all regions and all products of the shelf, Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
in response to detecting absence of the product in the slot, generating a prompt to restock the slot with a quantity of units of the product. (para. 69, when the product is out of the stock, the inventory control module 230 sends an out-of-stock message to the store product ordering system; para. 59 – the report with the misplaced, out of stock; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
Zimmerman does not specifically disclose the first region and second region; also, does not disclose identifying a slot, proximal the tag, depicted in a second region of the image; also, does not disclose accessing a quantity of units of the product assigned to the slot by a product database; and
Calhoon discloses: 5:60-67, 6:1-67; 7:1-35 “a shelf label may indicate a Campbell's Chicken Noodle Soup region of shelving. This identifier—in conjunction with the store's planogram—may indicate that the imagery is taken from aisle 7B (i.e., the north side of aisle 7), 42 feet from the east end, on the 3d shelf (numbered from the bottom)”; “The planogram may indicate that the product region for Campbell's Chicken Noodle Soup is to be eight inches in width—enabling stocking of three adjacent cans of soup, where each can is 2.625″ in diameter by 4″ tall”; Please note: the soupe region has 3 slots, therefore they can be considered different regions of the same soup / features; “the left edge of the Campbell's Chicken Noodle Soup shelf label serves as the left edge of an overlaid dashed bounding box that extends eight inches to the right, and the full height of the shelf space. Again, the known width of the shelf label can serve as a measuring reference. That is, the bounding box has a width (in pixels of captured imagery), equal to the width of the shelf label (in pixels), times the ratio of the product region width to the label width.”; “After the bounding box is virtually located in the captured FIG. 3A imagery, the server checks the watermark data. In the FIG. 3A case, the watermarks show 6 cans of soup within the bounded area (as indicated by 6 different instances of non-geometrically consistent product watermarks found within the box). However, one of the watermarks indicates the can is of Cream of Mushroom soup—not Chicken Noodle soup. This error is added to a punch list of planogram exceptions that results from the analysis. A store employee uses the punch list to make needed corrections to product placements”; “In like fashion, the rest of the shelf is similarly analyzed. FIG. 3B depicts analysis of the next product region to the right, which is an eight inch segment of shelf that is planogram-assigned to Campbell's Cream of Mushroom Soup. Analysis of this region of the captured imagery shows only four cans of such soup (by detection of four watermark patterns that are not geometrically consistent). All indicate the expected variety of soup. So the punch list simply instructs the store employee to restock cans of Campbell's Cream of Mushroom Soup in the noted region.”; “Each frame of imagery is examined, e.g., by a GPU configured as a stream processor, to identify rectangular features of a certain range of pixel scales, with a 5:2 aspect ratio. These regions can be segmented-out, and applied to a watermark decoder. The resulting shelf label payload is checked against a list of action-triggering label payloads stored in a data structure. If a match against the stored list is found, then an action stored in association with that shelf label payload is undertaken (e.g., switching spectral illumination to activate an infrared LED light source).”
; 15:14-20 - planogram database;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
Regarding claim 2, Zimmerman and Calhoon already discloses the limitations “identifying a tag, arranged on the shelving structure, depicted in a first region of the image; o detecting a set of features in the first region, the set of features representing a product descriptor; o accessing a set of template features of a product associated with the product descriptor; o identifying a slot, proximal the tag, depicted in a second region of the image; o in response to absence of features analogous to the set of template features in the second region of the image, detecting absence of the product in the slot; o accessing a quantity of units of the product assigned to the slot by a product database; and o in response to detecting absence of the product in the slot, generating a prompt to restock the slot with the quantity of units of the product.” of claim 1. The limitations “further comprising: o identifying a second tag, arranged on the shelving structure, depicted in a third region of the image; o detecting a second set of features in the third region, the second set of features representing a second product descriptor; o accessing a second set of template features of a second product associated with the second product descriptor; o and o detecting presence of a set of product units of the second product in the second slot based on features analogous to the second set of template features in the fourth region of the image.” are considered a duplication of the limitations of claim 1. Therefore, the limitations of claim 2 does not patentably distinguish the claimed invention from the prior art, as a mere duplication of parts has no patentable significance unless a new and unexpected result is produced, as in In re Harza, 274 F.2d 669, 124 USPQ 378 (CCPA 1960).
Regarding claim 4, Zimmerman discloses:
wherein transmitting the image to the computer system comprises wirelessly transmitting the image to the computer system comprising a remote server. Claim 13 and para. 49-50, 53;
Regarding claim 7, Zimmerman discloses:
wherein capturing the image comprises capturing the image at a first time; and wherein transmitting the image to the computer system comprises transmitting the image to the computer system at a second time succeeding the first time. (para. 50 and para. 51 - captures image and perform image pre-processing; therefore, the mobile inventory robot 20 captures image and transmits image on different time;)
Regarding claim 8, Zimmerman discloses:
wherein autonomously navigating along the shelving structure by the robotic system comprises autonomously navigating along the shelving structure by the robotic system comprising: o a base; o a mast extending vertically from the base; and o a camera arranged on the mast at a first height. (para. 47-52, 60-67 and Figures 3-2; para. 59 autonomies navigate in the retail store and detects out of stock item on shelf;)
Regarding claim 9, Zimmerman discloses:
wherein autonomously navigating along the shelving structure by the robotic system comprising the camera arranged on the mast at the first height comprises autonomously navigating along the shelving structure by the robotic system comprising the camera arranged on the mast at the first height less than a second height of the slot within the shelving structure. (para. 47-52, 60-67 and Figures 3-2; para. 59 autonomies navigate in the retail store and detects out of stock item on shelf; para. 48 – the cameras 30 are correspond to the height of the shelf)
Regarding claim 10, Zimmerman discloses:
wherein autonomously navigating along the shelving structure by the robotic system comprising the camera arranged on the mast at the first height comprises autonomously navigating along the shelving structure by the robotic system comprising the camera arranged on the mast at the first height corresponding to a second height of the slot within the shelving structure. (para. 47-52, 60-67 and Figures 3-2; para. 59 autonomies navigate in the retail store and detects out of stock item on shelf; para. 48 – the cameras 30 are correspond to the height of the shelf)
Regarding claim 11, Zimmerman discloses:
further comprising: detecting a second set of features second product descriptor; " accessing a second set of template features of a second product associated with the second product descriptor; " identifying a second slot, proximal the second tag, depicted in second set of template features in second product in the second slot; "detecting a quantity of facings of the second product in the image; "accessing a target quantity of facings assigned to the second slot; and " in response to the quantity of facings falling below the target quantity of facings: o detecting an improper stocking condition of the second product in the second slot; and o generating a prompt to restock units of the second product in the second slot. (para. 13, 50, 54-59 – discloses imaging of products from various angles on shelves (multiple product, multiple shelves), image segmentation of all products and tags in the shelf, see Figures 7A-7B, the decode of the barcodes of tags placed on the shelf and the products, the retrieval of the product visual descriptors and comparisons, the counting of items on the shelf compared to the expected inventory threshold, when below restocking is need, the flagging of items after being compared from various angles - wrong spot, out-of-stock, missed, misplaced, need to be relocated, disarray, report generation to employees with those information for employees to arrange the shelfs; In addition, Figure 6 discloses multiple shelf tags are processed in the same workflow. Further see para. 47-54, 57-58, 68-69 and Figures 1-9;
Zimmerman does not specifically disclose the third region and fourth region; also, does not disclose identifying a second tag, arranged on the shelving structure, depicted in a third region of the image;
Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 15:14-20 and Figures 2B-3B – multiple regions;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
In addition, Zimmerman and Calhoon already discloses the limitations “further comprising: "identifying a second tag, arranged on the shelving structure, depicted in a third region of the image; " detecting a second set of features in the third region, the second set of features representing a second product descriptor; " accessing a second set of template features of a second product associated with the second product descriptor; " identifying a second slot, proximal the second tag, depicted in a fourth region of the image; " in response to detecting features analogous to the second set of template features in the fourth region of the image,” on claim 1 without the word second, third, fourth; those limitations are considered a duplication of the limitations of claim 1. Further, adding the word second “confirming presence of a unit of the second product in the second slot; "detecting a quantity of facings of the second product in the image; "accessing a target quantity of facings assigned to the second slot; and " in response to the quantity of facings falling below the target quantity of facings: o detecting an improper stocking condition of the second product in the second slot; and o generating a prompt to restock units of the second product in the second slot”, also do no patentably distinguish the claimed invention from the prior art; those limitations are considered a duplication of what Zimmerman and Calhoon already taught. Therefore, the limitations of claim 11 do not patentably distinguish the claimed invention from the prior art, as a mere duplication of parts has no patentable significance unless a new and unexpected result is produced, as in In re Harza, 274 F.2d 669, 124 USPQ 378 (CCPA 1960).
Regarding claim 12, Zimmerman discloses:
further comprising: o in response to detecting features analogous to the second set of template features in the e, confirming presence of a unit of the second product in the second slot; detecting the unit of the second product in an orientation in the second slot based on the second set of template features; and o in response to the orientation of the second product deviating from a target orientation of units of the second product associated with the second slot, generating a prompt to straighten products in the second slot (see figure 7B for the orientation; para. 13,55-58 – items are flagged for being in the wrong spot, out-of-stock, missed, misplaced, relocate items, disarray items when number are adequate and report to employees in a report; - Zimmerman discloses an image are segmented and compared to product visual descriptor of various angles to determine out of stock, misplaced, or disarranged items of shelfs. In addition, Figure 6 discloses multiple shelf tags are processed in the same workflow. In addition, the robot captures additional images views of the shelf in order to determine a sufficient quantity of items on a shelf. Therefore, disclosing multiple shelf tags are processed in the same image, met by para. 59 - "an inventory control report comprises indications of shelves that are in disarray in which an adequate number of items are on the Further see para. 48-54, 57-58, 68-69 and Figures 1-9;
Zimmerman does not specifically disclose the third region and fourth region; also, does not disclose identifying a second slot, proximal the second tag, depicted in a fourth region of the image;
Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 15:14-20 and Figures 2B-3B – multiple regions;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
In addition, Zimmerman and Calhoon already discloses the limitations “further comprising: o identifying a second slot, proximal the second tag, depicted in a fourth region of the image; o in response to detecting features analogous to the second set of template features in the third region of the image, confirming presence of a unit of the second product in the second slot;” on claim 1 without the word second, third, fourth; those limitations are considered a duplication of the limitations of claim 1. Further, adding the word second “detecting the unit of the second product in an orientation in the second slot based on the second set of template features; and o in response to the orientation of the second product deviating from a target orientation of units of the second product associated with the second slot, generating a prompt to straighten products in the second slot”, also do no patentably distinguish the claimed invention from the prior art; those limitations are considered a duplication of what Zimmerman and Calhoon already taught. Therefore, the limitations of claim 12 do not patentably distinguish the claimed invention from the prior art, as a mere duplication of parts has no patentable significance unless a new and unexpected result is produced, as in In re Harza, 274 F.2d 669, 124 USPQ 378 (CCPA 1960).
Regarding claim 13, Zimmerman discloses:
wherein accessing the set of template features of the product comprises accessing a set of template images depicting representative units of the product in a range of orientations; and "wherein detecting absence of the product in the slot comprises: o detecting a second set of features from the second region of the image; and o detecting absence of the product in the slot in response to absence of correlation between the second set of features and the set of template images. (para. 13, 50, 54-59 – discloses imaging of products captured from various angles on shelves (multiple product, multiple shelves), images segmentation of all products and tags in the shelf, see Figures 7A-7B, the decode of the barcodes of tags placed on the shelf and the products, the retrieval of the product visual descriptors and comparisons from various angles, the counting of items on the shelf compared to the expected inventory threshold, when below restocking is need, the flagging of items after being compared from various angles - wrong spot, out-of-stock, missed, misplaced, need to be relocated, disarray, report generation to employees with those information for employees to arrange the shelfs; In addition, Figure 6 discloses multiple shelf tags are processed in the same workflow. Further see para. 47-54, 57-58, 68-69 and Figures 1-9;
Zimmerman does not specifically disclose the second region;
Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 15:14-20;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
Regarding claim 15, Zimmerman discloses:
A method comprising:
by a robotic system deployed in a store: Figures 1, 3-9 and para. 47, 59-60, 67
autonomously navigating along a shelving structure within the store; and (para. 59 and Figures 1-3);
capturing a set of images of a shelving structure within a field of view of an optical sensor arranged in the robotic system; and (para. 59; further see para. 48-50, 53 and Figures 1-3);
by the computer system: (para. 50, 50, 53, 48-49 and Figure 2)
accessing an image representing the set of images captured by the robotic system; (para. 50, 53; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
identifying a tag, arranged on the shelving structure, represented in image; (para. 54; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
extracting a set of features in the image, the set of features representing a product descriptor; (para. 54, 58; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
accessing a set of template features of a product associated with the product descriptor; (para. 54, 58; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
in response to detecting features analogous to the set of template features in the image, confirming presence of a product unit of the product in the slot; and ; (para. 54, 58, 68-69 – determines whether the product visual descriptor in the product database 225 sufficiently matches the shelf image captured by the imaging device 30, if does match the inventory control module 215 marks the found barcode as old and go check the next product; Please not this process is made for all regions and all products of the shelf, Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
in response to a first orientation of the product unit deviating from a target orientation of product units of the product (see figure 7B for the orientation; para. 13,55-58 – items are flagged for being in the wrong spot, out-of-stock, missed, misplaced, relocate items, disarray items when number are adequate and report to employees in a report; - Zimmerman discloses an image are segmented and compared to product visual descriptor of various angles to determine out of stock, misplaced, or disarranged items of shelfs. In addition, Figure 6 discloses multiple shelf tags are processed in the same workflow. In addition, the robot captures additional images views of the shelf in order to determine a sufficient quantity of items on a shelf. Therefore, disclosing multiple shelf tags are processed in the same image, met by para. 59 - "an inventory control report comprises indications of shelves that are in disarray in which an adequate number of items are on the Further see para. 48-54, 57-58, 68-69 and Figures 1-9;
generating a prompt to straighten product units in the slot. (para. 58-59, 69 - generate a report for the employer to proper stack or arrange items on shelf that are adequate, Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
Zimmerman does not specifically disclose the first region and second region; also, does not disclose identifying a slot, proximal the tag, depicted in a second region of the image; also, does not disclose accessing a quantity of units of the product assigned to the slot by a product database; and that the image is composite.
Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 – stitch images; 15:14-20 14:1-15
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
Regarding claim 16, Zimmerman discloses:
further comprising: " by the robotic system, transmitting the set of images to the computer system; and " (para. 50-54, 58; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
Zimmerman does not specifically disclose by the computer system, transforming the set of images into the composite image.
Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 – stitch images; 15:14-20 14:1-15
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
Regarding claim(s) 20, Zimmerman discloses:
• triggering a robotic system, deployed in a store, to autonomously scan shelving structures within the store; (para. 59, 50, 53, 48-49 and Figures 1-3);
• accessing an image output by the robotic system and depicting a shelving structure in the store, the shelving structure within a field of view of an optical sensor arranged in the robotic system; (para. 50, 54-53; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
• identifying a tag, arranged on the shelving structure, depicted in the image; (para. 54; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
detecting a set of features in the image, the set of features representing a product descriptor; (para. 54; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
• accessing a set of template features of a product associated with the product descriptor; (para. 54, 58; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
• in response to detecting features analogous to the set of template features in the image, confirming presence of a first product unit of the product in the slot; (para. 54, 58; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
• detecting a quantity of facings of the product in the image; accessing a target quantity of facings assigned to the slot; and (para. 58 “The product counter 220 counts the number of items on the shelf and determines whether the number of items present in the shelf is below an inventory threshold, requiring restocking of the shelf before the item is out of stock. The mobile inventory robot 20 is mobile, allowing capture of additional points of view of the same item to determine whether additional items are on the shelf”; Further see para. 48-54, 57-58, 68-69 and Figures 1-9)
• in response to the quantity of facings falling below the target quantity of facings, generating a prompt to restock product units of the product in the slot. (para. 58-59 - generate a report for the employer to restock; Further see para. 48-54, 57-58, 68-69 and Figures 1-9);
Zimmerman does not specifically disclose the first region and second region; also, does not disclose identifying a slot, proximal the tag, depicted in a second region of the image;
Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 15:14-20 14:1-15;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
Claim(s) 3, 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Zimmerman and Calhoon combination as applied to claim 1, and further in view of Letsky (US 20120265391 A1).
Regarding claim 3, the combination, specifically Zimmerman discloses:
wherein capturing the image of the shelving structure comprises capturing the image of the shelving structure (para. 40 – the inventory control system is remote from the mobile inventory robot 20; (para. 50 and para. 51 - captures image and perform image pre-processing; therefore, the mobile inventory robot 20 captures image and transmits image on different time;))
The combination does not disclose sending data at the same time, and integrated into a charging dock for the robotic system, at a third time succeeding the second time.
Letsky discloses: Figure 17 and para. 76 - robot , docking station , and server exchange data via wireless; para. 162 – real-time data exchanged between robot and server; para. 171;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to include the above limitations as taught by Letsky, in order to effectively perform tasks in an acceptable speed, see Letsky para. 6.
Regarding claim 5, the combination, specifically Zimmerman discloses:
further comprising, by the robotic system, autonomously navigating to a charging dock configured to transiently couple to the robotic system to supply power to the robotic system, the charging dock installed within a region of the store; and wherein transmitting the image to the computer system comprises transmitting the image to the computer system comprising. (para. 62 and 65 the mobile platform comprises a battery and a dicking pot for charging the battery with power supply; the mobile robot can automatically self-dock and self-charging when required);
The combination does not disclose sending data for a local computer system integrated into the charging dock.
Letsky discloses: Figure 17 and para. 76 - robot , docking station with a central processing unit for processing any data , and server exchange data via wireless; para. 162 – real-time data exchanged between robot and server; para. 171;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to include the above limitations as taught by Letsky, in order to effectively perform tasks in an acceptable speed, see Letsky para. 6.
Regarding claim 6, the combination, specifically Zimmerman discloses:
wherein capturing the image comprises capturing the image; and wherein transmitting the image to the computer system comprises transmitting the image to the computer system. (para. 50 and para. 51 - captures image and perform image pre-processing; therefore, the mobile inventory robot 20 captures image and transmits image on different time;)
The combination does not disclose sending data at the same first time.
Letsky discloses: Figure 17 and para. 76 - robot , docking station , and server exchange data via wireless; para. 162 – real-time data exchanged between robot and server; para. 171;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to include the above limitations as taught by Letsky, in order to effectively perform tasks in an acceptable speed, see Letsky para. 6.
Claim(s) 14 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zimmerman and Calhoon combination as applied to claims 1 and 15, and further in view of Official Notice.
Regarding claim 14, the combination, specifically Zimmerman discloses:
wherein identifying the tag, arranged on the shelving structure, comprises identifying the tag, arranged on the shelving structure, depicted in the first region of the image; and " wherein detecting the set of features in the first region of the image comprises: o detecting a set of features in the first region of the image; and o deriving the product descriptor of the product based on the set of text characters. (para. 13, 50, 54-59 – discloses imaging of products captured from various angles on shelves (multiple product, multiple shelves), images segmentation of all products and tags in the shelf, see Figures 7A-7B, the decode of the barcodes of tags placed on the shelf and the products, the retrieval of the product visual descriptors and comparisons from various angles, the counting of items on the shelf compared to the expected inventory threshold, when below restocking is need, the flagging of items after being compared from various angles - wrong spot, out-of-stock, missed, misplaced, need to be relocated, disarray, report generation to employees with those information for employees to arrange the shelfs; In addition, Figure 6 discloses multiple shelf tags are processed in the same workflow. Further see para. 47-54, 57-58, 68-69 and Figures 1-9;
The combination does not disclose the tags are paper tags, and text character recognition is used as the technique to retrieved product visual descriptors; Examiner takes Official Notice that it is old and well-known in the art to use paper tags on grocery stores shelfs, and text character recognition as one of innumerous visual recognition existing in the visual recognition business.
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to include the above limitations as taught by Official Notice, in order to have the same brand and type of product recognized even if they have different SKU such as cereal from different flavors, and save money with tags since paper tags are cheaper than electronic tags.
Regarding claim 17, the combination discloses Zimmerman discloses:
further comprising, by the robotic system: o transmitting the image to the computer system; and "wherein accessing the composite image. (para. 50-54, 58; Further see para. 48-54, 57-58, 68-69 and Figures 1-9); and Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 – stitch images; 15:14-20 14:1-15
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify Zimmerman to include the above limitations as taught by Calhoon, in order to assure compliance with store planograms, see Calhoon 1:65-67, 2:1-10.
The combination does not disclose the robot system transforms the image into composite image and transmits to the computer system as composite image - o transforming the set of images into the composite image and receiving the composite image from the robotic system; Examiner takes Official Notice that any computer such as in form of robot, server, table, etc can transform images into composite image;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to have the robot to transform the composite image, instead the server, and send the composite image to the computer system as taught by Official Notice; in order to send the images as panoramic and have the computer system spend less time processing images.
Having the computer system or the robot system transforming the images into panoramic is a design choice.
Claim(s) 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zimmerman and Calhoon combination as applied to claim 15, and further in view of Letsky (US 20120265391 A1) and Official Notice.
Regarding claim 18, the combination, specifically Zimmerman discloses:
wherein capturing the set of images comprises capturing the set of images; and wherein accessing the composite image, the computer system comprising a remote server. (para. 62 and 65 the mobile platform comprises a battery and a dicking pot for charging the battery with power supply; the mobile robot can automatically self-dock and self-charging when required; para. 50 and para. 51 - captures image and perform image pre-processing; therefore, the mobile inventory robot 20 captures image and transmits image on different time; para. 40 – the inventory control system is remote from the mobile inventory robot 20;) and Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 – stitch images; 15:14-20 14:1-15;
The combination does not disclose sending data at the same first time.
Letsky discloses: Figure 17 and para. 76 - robot , docking station , and server exchange data via wireless; para. 162 – real-time data exchanged between robot and server; para. 171;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to include the above limitations as taught by Letsky, in order to effectively perform tasks in an acceptable speed, see Letsky para. 6.
The combination does not disclose the robot system transforms the image into composite image and transmits to the computer system as composite image - receiving the composite image from the robotic system; Examiner takes Official Notice that any computer such as in form of robot, server, table, etc can transform images into composite image;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to have the robot to transform the composite image, instead the server, and send the composite image to the computer system as taught by Official Notice; in order to send the images as panoramic and have the computer system spend less time processing images.
Having the computer system or the robot system transforming the images into composite images is a design choice.
Regarding claim 19, the combination, specifically Zimmerman discloses:
further comprising, by the robotic system, autonomously navigating to a charging dock configured to transiently couple to the robotic system to supply power to the robotic system, the charging dock installed within a region of the store; wherein capturing the set of images comprises capturing the set of images at a first time; and wherein accessing the composite image at a second time succeeding the first time. (para. 62 and 65 the mobile platform comprises a battery and a docking port for charging the battery with power supply; the mobile robot can automatically self-dock and self-charging when required, the docking system is installed in the retail store; para. 50 and para. 51 - captures image and perform image pre-processing; therefore, the mobile inventory robot 20 captures image and transmits image on different time) and Calhoon discloses: 5:60-67, 6:1-67; 7:1-67 – stitch images; 15:14-20 14:1-15;
The combination does not disclose the computer system comprising a local computer system integrated into the charging dock.
Letsky discloses: Figure 17 and para. 76 - robot , docking station with a central processing unit for processing any data exchanging data; para. 171;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to include the above limitations as taught by Letsky, in order to effectively perform tasks in an acceptable speed, see Letsky para. 6.
The combination does not disclose the robot system transforms the image into composite image and transmits to the computer system as composite image - receiving the composite image from the robotic system; Examiner takes Official Notice that any computer such as in form of robot, server, table, etc can transform images into composite images;
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention, to modify the combination to have the robot to transform the composite image, instead the server, and send the composite image to the computer system as taught by Official Notice; in order to send the images as panoramic and have the computer system spend less time processing images.
Having the computer system or the robot system transforming the images into composite images is a design choice.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VANESSA DELIGI whose telephone number is (571)272-0503. The examiner can normally be reached on Monday-Friday 07:30AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Florian (Ryan) Zeender can be reached on (571) 272-6790. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center to authorized users only. Should you have questions about access to the USPTO patent electronic filing system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/VANESSA DELIGI/Patent Examiner, Art Unit 3627
/FLORIAN M ZEENDER/ Supervisory Patent Examiner, Art Unit 3627
Prosecution Insights