DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Prosecution on the merits of this application is reopened on claims 31-50 considered unpatentable for the reasons indicated below:
The indicated allowability of claims 31-50 is withdrawn in view of the newly discovered reference(s) to Adato et al (US PG. Pub. 2019/0213535); Deyle (US PG. Pub. 2020/0061839); Brandt (US Patent No. 2003/005782). Rejections based on the newly cited reference(s) follow.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 31-41, and 43-50 is/are rejected under 35 U.S.C. 103 as being unpatentable over Adato et al (US PG. Pub. 2019/0213535) in view of Deyle et al (US PG. 2020/0061839). Relative to claims 31-41, Adato discloses:
claim 31) a control system (100) for managing and optimizing a location of one or more containers in one or more stacks of containers (see objects or products, 608, which may be containers, see Fig. 1, 6A) in an inventory management facility (Para. 0120, products may be moved and arranged in stacks, Para. 0193), the control system (100) comprising:
one or more container handlers (container handlers may be an automated system such as an autonomous robotic device equipped with one or more cameras, see Para. 0149, for performing product-related tasks, including moving or rearranging products; Para. 0115; 0628; container handlers may also be employees that move products, the employees may have mobile device, such as a smartphone, etc. equipped with capturing device) operable within the inventory management facility (retail store, Para. 0114)(Fig. 1) and configured to move the one or more containers from a first location to a second location (see Para. 0628; 0664; employees or automated devices perform product related tasks, such as moving products within the store, see also “product sorting robot”, Para. 0538, robotic employee, Para. 0665), the or more container handlers: includes:
one or more image sensors (capturing devices 125, Para. 0115; 0140)(Fig. 1) on the one or more container handlers (image capturing devices 125 may be located on the autonomous robotic devices, moving through the environment to perform tasks, Para. 0114; 0149, image sensors may also be on the mobile device worn or carried by the employee), the one or more image sensors (125) configured to capture one or more images of one or more parts of at least one of the one or more containers (Para. 0115);
one or more geolocation devices (positioning sensor may be integrated with capturing device 125, such as GPS, GLObal NAvigation Satellite System (GLONASS), etc., which may be on the autonomous robotic device) on the one or more container handlers (Para. 0140), the one or more geolocation devices (positioning sensors) configured to determine one or more geolocation values for at least one of:
the one or more container handlers (autonomous robotic device)(Para. 0140), or
the one or more containers (products), or a combination thereof (Para. 0140; 0160);
one or more wireless transceivers on the one or more container handlers (system uses wireless technology to transmit image data from the image capturing device 125 on the on the autonomous robotic device, therefore the autonomous robotic device inherently includes, but does not show, a wireless transceiver with the image capturing device 125; Para. 0115, 0132) and configured to transmit at least one of:
image data from the one or more image sensors (125), or
geolocation data from the one or more geolocation devices (Para. 0140), or
a combination thereof (see Para. 0122, 0132); and
one or more controllers (see processing device 302)(Fig. 3, 5) on the one or more container handlers (see the robotic device with the image capturing device 125), the one or more controllers (processors) coupled to the one or more image sensors (125), the one or more geolocation devices (positioning sensor), and the one or more wireless transceivers (Para. 0151),
the one or more controllers (302) configured to transmit at least one of the image data, or the geolocation value, or the combination thereof (Para. 0140; 0151); and
one or more servers (135) in communication with the one or more wireless transceivers (Para. 0244),
the one or more servers (135) include one or more processors and one or more non-transitory computer- readable storage mediums storing instructions (Para. 0121; 0146) comprising one or more algorithms that when executed by the one or more processors cause the one or more processors (see machine learning algorithm, Para. 0121) to perform steps to:
apply one or more trained neural network models on the image data to determine an identification of at least one container of the one or more containers (system uses machine learning algorithms to identify products, Para. 0121, 0244);
generate a database (140) comprising the identification, the image data, and the geolocation data of the at least one container of the one or more containers (system captures images of products, and determines and associates the shelf associated with the products and location in the retail store, Para. 0244); and
apply one or more trained neural network models to the database (140) to optimize a location of the at least one container of the one or more containers (products) in the one or more stacks of the inventory management facility (tasks include rearranging and restocking products on shelves, models are used to optimize placement of products Para. 0136; 0571, 0661);
claim 32) the step of optimizing the location of the at least one container of the one or more containers (products) in the one or more stacks of the inventory management facility includes at least one of:
predicting one or more optimal locations of the container of the one or more containers (products) in the one or more stacks of the inventory management facility based on one or more predetermined metrics, or one or more predetermined factors, or a combination thereof (layout is optimized, Para. 0571, 0661, for instance, the store may rearrange products to make more efficient use of the space in the retail store, vacant spaces are analyzed, system may aggregate identified spaces between products, Para. 0644; product assortment rules, Para. 0649, products may be stacked, Para. 0212); or
generating one or more operational values for a container handler of the one or more container handlers (autonomous robotic device) to position the container of the one or more containers (products) at the one or more optimal locations in the one or more stacks of the inventory management facility; or a combination thereof (Para. 0231; 0571);
claim 33) the one or more servers (135) are configured to transmit the one or more optimal locations, or the one or more operational values, or a combination thereof, to the container handler of the one or more container handlers (tasks are submitted to the autonomous robotic device or employee) to position the container of the one or more containers (products) at a predetermined location within the inventory management facility (system using scores and other metrics to determine optimal locations of products, Para. 0231; 0296; 0661);
claim 34) the predetermined location is a predetermined location in a stack of containers (products) within the inventory management facility (Para. 0212);
claim 35) the one or more servers (135) are configured to transmit the one or more optimal locations, or the one or more operational values, or a combination thereof, to the container handler of the one or more container handlers (autonomous robotic devices or employees) to re-position the container of the one or more containers (products) at one or more second predetermined locations in the one or more stacks of the inventory management facility based on one or more other predetermined metrics, or other predetermined factors, or a combination thereof (products may be rearranged or relocated based on various metrics, Para. 0661);
claim 36) the one or more servers (135) are configured to track one or more of a location, or a movement, or a combination thereof, of the one or more containers (products) in the one or more stacks of the inventory management facility (Para. 0033; 0036);
claim 37) the one or more geolocation values comprises a latitude value, a longitude value, and an altitude value for at least one of:
the one or more containers (products); or one or more parts of the one or more container handlers (autonomous robotic device); or a combination thereof (Para. 0265; 0321);
claim 38) the one or more geolocation devices on the one or more container handlers (autonomous robotic device) comprises:
a first geolocation device on a first part of a container handler of the one or more container handlers (see positioning sensor on autonomous robotic device), the first geolocation device is configured to determine a first geolocation value for the first part of the container handler (Para. 0140); and at least the first geolocation value, or the second geolocation value, or a combination thereof, comprises a latitude value, a longitude value, and an altitude value (Para. 0265; 0321);
claim 39) the one or more geolocation devices (positioning sensor) on the one or more container handlers (autonomous robotic device) comprises:
a first geolocation device (see positioning sensor) on a first part of a container handler of the one or more container handlers (autonomous robotic device), the first geolocation device is configured to determine a first geolocation value for the first part of the container handler (Para. 0140, location of autonomous robotic device and employee holding mobile device, each having the image capturing device may be determined);
at least the first geolocation value, the second geolocation value, or the third geolocation value, or a combination thereof, comprises a latitude value, a longitude value, and an altitude value (Para. 0265);
claim 41) the step of applying the one or more trained neural network models on the image data to determine the identification of the at least one of the one or more containers (products) comprises: executing at least one of:
a first machine learning model comprising a neural network trained to predict a location of text sequences in the image data (Para. 0121, system can recognize text and logos on products and can use the images and machine learning models to identify products, system can also recognize missing text on products; Para. 0270, label is missing, Para. 0657, see “a need for labeling may be detected when a label is missing”); and performing optical character recognition (OCR) on the image data (Para. 0121);
claim 43) the step of optimizing the location of the container of the one or more containers (products) in the one or more stacks of the inventory management facility includes at least one of:
predicting one or more optimal locations of a plurality of containers (products) of the one or more containers (products) in the one or more stacks of the inventory management facility based on one or more predetermined metrics, or one or more predetermined factors, or a combination thereof (Para. 0231; 0661; 0766); or
generating one or more operational values for the one or more container handlers (autonomous robotic device) to position each of the plurality of containers (products) at a respective predetermined location in the one or more stacks of the inventory management facility; or a combination thereof (system uses scores to analyze planogram compliance and generate tasks to improve store execution, Para. 0231; 0661; 0766);
claim 44) the one or more servers (135) are configured to transmit the one or more optimal locations, or the one or more operational values, or a combination thereof, to the one or more container handlers (autonomous robotic devices or employees) to position each of the plurality of containers (products) at the respective predetermined location in a stack of containers (products) within the inventory management facility (employees or autonomous robotic devices may tasked to restock or rearrange products according to an arrangement determined by the system to achieve optimal layout, Para. 0231; 0661; 0766);
claim 45) the one or more servers (135) are configured to transmit the one or more optimal locations, or the one or more operational values, or a combination thereof, to the one or more container handlers (autonomous robotic device) to re-position one or more of the plurality of containers (products) at one or more second predetermined locations in the one or more stacks of the inventory management facility based on one or more other predetermined metrics, or other predetermined factors, or a combination thereof (Para. 0231; 0661; 0766);
claim 46) the one or more servers (135) are configured to continuously apply the one or more trained neural network models to the database to continuously optimize the location of the at least one container of the one or more containers (products) in the one or more stacks of the inventory management facility (Para. 0226; 0678, products are continuously monitored using timely images in retail store; image capturing devices obtain data in real-time, Para. 0150; server generates actionable tasks using data in real-time, as well as update assignments Para. 0384; 0678); and
claim 47) the one or more servers (135) are configured to transmit the one or more optimal locations, or the one or more operational values, or a combination thereof, to the one or more container handlers (autonomous robotic device) to continuously re-position one or more of the plurality of containers (products) at one or more additional predetermined locations in the one or more stacks of the inventory management facility based on one or more additional predetermined metrics, or other predetermined factors, or a combination thereof (Para. 0226; 0678).
Adato does not expressly disclose:
claim 31) the container handlers are configured to engage at least one of the one or more containers to move the at least one of the one or more containers from a first location to a second location;
claim 38) the one or more geolocation devices on the one or more container handlers comprises:
a second geolocation device on a second part of the container handler of the one or more container handlers, the second geolocation device is configured to determine a second geolocation value for the second part of the container handler,
claim 39) the one or more geolocation devices on the one or more container handlers comprises:
a second geolocation device on a second part of the container handler of the one or more container handlers, the second geolocation device is configured to determine a second geolocation value for the second part of the container handler; and
a third geolocation device on a third part of the container handler of the one or more container handlers, wherein the third geolocation device is configured to determine a third geolocation value for the third part of the container handler,
claim 40) the one or more geolocation devices are configured to determine the one or more geolocation values at least one of:
when one or more parts of the one or more container handlers is within a predetermined distance from one or more parts of a container of the one or more containers; or
when the one or more parts of the one or more container handlers engages with the one or more parts of the container of the one or more containers; or
when the one or more parts of the one or more container handlers disengages from the one or more parts of the container of the one or more containers; or a combination thereof;
claim 48) a container handler of the one or more container handlers comprises one of a side loader, a reach loader, a reach stacker, a Rubber Tired Gantry Cranes (RTG), or a crane loader.
Deyle teaches:
claim 31) the container handlers (mobile robots 2710)(Para. 0338) are configured to engage at least one of the one or more containers to move the at least one of the one or more containers from a first location to a second location (mobile robots may pick up items and perform tasks, Para. 0338);
claim 38) the one or more geolocation devices on the one or more container handlers comprises:
a second geolocation device on a second part of the container handler of the one or more container handlers, the second geolocation device is configured to determine a second geolocation value for the second part of the container handler (robots 100 may include multiple GPS receivers, which includes having one GPS receiver on one part and another GPS receiver on another part of robot 100),
claim 39) the one or more geolocation devices (GPS receiver) on the one or more container handlers (robot 100) comprises:
a second geolocation device on a second part of the container handler (robot 100, 2710) of the one or more container handlers, the second geolocation device is configured to determine a second geolocation value for the second part of the container handler (Para. 0119); and
a third geolocation device on a third part of the container handler (robot 100, 2710) of the one or more container handlers, the third geolocation device is configured to determine a third geolocation value for the third part of the container handler (robot may include multiple GPS receivers, which includes three GPS receivers; Para. 0119, this includes determining the position of the part at that GPS receiver),
claim 40) the one or more geolocation devices (GPS receiver) are configured to determine the one or more geolocation values at least one of:
when one or more parts of the one or more container handlers is within a predetermined distance from one or more parts of a container of the one or more containers (objects, such as containers in a retail store, may be detected to be within a proximity of the robot, RFID tags on objects may be detected to be within a threshold distance; Para. 0103; 0286; 0302); or
when the one or more parts of the one or more container handlers engages with the one or more parts of the container of the one or more containers; or
when the one or more parts of the one or more container handlers disengages from the one or more parts of the container of the one or more containers; or a combination thereof;
claim 48) a container handler of the one or more container handlers comprises one of a side loader, a reach loader, a reach stacker, a Rubber Tired Gantry Cranes (RTG), or a crane loader (robot 100 may have arms 740 to receive objects from shelves or deliver objects; Para. 0118, robots 100, 2710 reach to retrieve the object from a location, and is therefore a reach loader; 0338).
Deyle teaches the: container handlers are configured to engage the containers; second and third geolocation devices on the container handler; determining when one or more parts of the container handlers is within a predetermined distance of a container, or has engaged or disengaged with a container; predicting one or more missing characters in the one or more container identification codes; and the container handler comprises one of a: side loader, a reach loader, a reach stacker, a Rubber Tired Gantry Cranes (RTG), or a crane loader, as described above, for the purpose of providing a mobile robot for use in a commercial and industrial settings, that can effectively and frequently communicate with other robots (Para. 0002).
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the system of Adato with the: container handlers are configured to engage the containers; second and third geolocation devices, determining when the container handler is within a predetermined distance of a container, or has engaged or disengaged with a container, predicting one or more missing characters in a code, and the container handler comprises one of a: side loader, a reach loader, a reach stacker, a Rubber Tired Gantry Cranes (RTG), or a crane loader, as taught in Deyle, for the purpose of a mobile robot for use in a commercial and industrial settings, that can effectively and frequently communicate with other robots (Para. 0002).
Relative to claims 49-50, the disclosure of Adato includes:
A server (135)(Fig. 2) in a control system (100)(Fig. 1), and method of operating a server (135) in a control system for managing and optimizing a location of one or more containers (products) in one or more stacks of containers (products) in an inventory management facility (retail facility)(Fig. 1), the control system (100) comprising:
one or more container handlers (automated system such as autonomous robotic device) operable within the inventory management facility (retail facility) and configured to move the at least one of the one or more containers (products) from a first location to a second location (Para. 0115; 0149), the one or more container handlers (autonomous robotic device) includes:
one or more image sensors (125) on the one or more container handlers (autonomous robotic device), the one or more images sensors (125) configured to capture one or more images of one or more parts of at least one of the one or more containers (products)(Para. 0115; 0140);
one or more geolocation devices (positioning sensor) on the one or more container handlers (autonomous robotic device), the one or more geolocation devices configured to determine one or more geolocation values for at least one of (Para. 0140):
the one or more container handlers (autonomous robotic device), or the one or more containers (products), or a combination thereof (Para. 0140; 0160);
one or more wireless transceivers (inherently included with autonomous robotic device having the image capturing device 125) on the one or more container handlers (autonomous robotic device)(Para. 0115; 0132) and configured to transmit at least one of:
image data from the one or more image sensors (125), or geolocation data from the one or more geolocation devices, or a combination thereof (Para. 0140); and
one or more controllers (302) on the one or more container handlers (autonomous robotic devices), the one or more controllers (302) coupled to the one or more image sensors (125), the one or more geolocation devices, and the one or more wireless transceivers (Para. 0151),
the one or more controllers (302) configured to transmit at least one of the image data, or the geolocation value, or the combination thereof (Para. 0140; 0151); and
the server (135) comprising: one or more processors and one or more non-transitory computer-readable storage mediums storing instructions comprising one or more algorithms that when executed by the one or more processors cause the one or more processors to perform steps to (Para. 0121; 0146; 0244):
apply one or more trained neural network models on the image data to determine an identification of at least one container of the one or more containers (products)(Para. 0121);
generate a database (140) comprising the identification, the image data, and the geolocation data of the at least one container of the one or more containers (products)(Para. 0244); and
apply one or more trained neural network models to the database (140) to optimize a location of the at least one container (products) of the one or more containers (products) in the one or more stacks of the inventory management facility (Para. 0136; 0571, 0661)
the step of optimizing the location of the at least one container of the one or more containers (products) in the one or more stacks of the inventory management facility includes at least one of:
predicting one or more optimal locations of the container of the one or more containers (products) in the one or more stacks of the inventory management facility (retail facility) based on one or more predetermined metrics, or one or more predetermined factors, or a combination thereof (Para. 0571; 0661); or
generating one or more operational values for a container handler of the one or more container handlers (autonomous robotic device) to position the container of the one or more containers (products) at the one or more optimal locations in the one or more stacks of the inventory management facility (Para. 0571; 0661); or
transmitting the one or more optimal locations, or the one or more operational values, or a combination thereof, to the container handler of the one or more container handlers (autonomous robotic device) to position the container of the one or more containers (products) at a predetermined location within the inventory management facility; or a combination thereof (Para. 0231; 0296; 0661).
Adato does not expressly disclose: the container handler is configured to engage at least one of the one or more containers to move the at least one of the one or more containers from a first location to a second location.
Deyle teaches: the container handler (robot, 2710, 100) is configured to engage at least one of the one or more containers (objects) to move the at least one of the one or more containers from a first location to a second location (Para. 0338), for the purpose of providing a mobile robot for use in a commercial and industrial settings, that can effectively and frequently communicate with other robots (Para. 0002).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Adato so that the container handler is configured to engage at least one of the one or more containers, as taught in Deyle, for the purpose of providing a mobile robot for use in a commercial and industrial settings, that can effectively and frequently communicate with other robots.
Claim(s) 42 is/are rejected under 35 U.S.C. 103 as being unpatentable over Adato and Deyle as applied to claim 31 above, and further in view of Brandt et al (US Patent No. 2003/0057282). Relative to claim 42, Adato in view of Deyle discloses all claim limitations mentioned above, including: the step of applying the one or more trained neural network models on the image data to determine the identification of the at least one of the one or more containers (products) comprises: detecting one or more container identification codes on one or more surfaces of the one or more containers (products)(Para. 0121, see optical character recognition).
Adato in view of Deyle does not expressly disclose: predicting one or more missing characters in the one or more container identification codes.
Brandt teaches: predicting one or more missing characters in the one or more container identification codes (Para. 0126, missing characters are reconstructed), for the purpose of providing a system and method for reconstructing complete bar code label information from partial scan information obtained under of variety of non-ideal situations that is more efficient and accurate (Para. 0004; 0015).
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the system of Adato in view of Deyle with the predicting one or more missing characters in the one or more container identification codes, as taught in Brandt for the purpose of providing a system and method for reconstructing complete bar code label information from partial scan information obtained under of variety of non-ideal situations that is more efficient and accurate.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sharp (US PG. Pub. 2020/0247611): relocates vessels 102, carrying containers (Para. 0103), 0093; Para. 0095; determines locations of vessels, Para. 0099, 0121; determines optimal locations, Para. 0099; uses machine learning to relocate vessels, Para. 0140)
Takehara (US PG. Pub. 2003/0190057): Para. 0100; 0108; 0094; 0117; 0119
Nazarian et al (US PG. Pub. 2018/0374036): robots 110 pick and transports containers.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YOLANDA RENEE CUMBESS whose telephone number is (571)270-5527. The examiner can normally be reached M-F 10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gene Crawford can be reached at 571-272-6911. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YOLANDA R CUMBESS/Primary Examiner, Art Unit 3651
Prosecution Insights