Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments dated 06/25/2026 have been received.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 regarding the generation of plans and selecting of plans have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments regarding claim 1-20 that the drive elements cannot move in any direction, orientation, and path have been fully considered but they are not persuasive. The examiner agrees that the Galluzzo’s robot is more restrictive in movement than applicant’s. However, the claims do not capture this requirement in mobility in a way that Galluzzo does not read on. The claims read that the drive elements move the robot “in any direction, orientation, and path” but do not require them all to be done at the same time. Galluzzo’s robot can move in any direction, if it is facing the correct direction. It can move in any orientation, if it is facing the right orientation. The term “any… path” is also not clearly defined. While the examiner recognizes the intent is to represent the idea of flexible movement on the part of the robot, this element is not particularly clear. For example, it is not clear if a path between a narrow gap that the robot can not fit would count as “any path” . Presumably this is not the case, because there are paths that would be impossible for any robot (a path through a wall, for example), ergo no robot would be capable of driving along “any path” . If the claims are more referring to “any path that it is provided” then that just means that it can follow paths in general. In the interest of compact prosecution, the examiner would note that if the claims did have language that focused on the flexibility of movement in a way that Galluzzo did not meet, another reference might be necessary. For example, Murphy et al (US Pub 2022/0305641 A1) has this passage:
[0049] In some embodiments, each wheel of a mobile base is independently steerable and independently drivable. In such embodiments, each wheel is associated with at least two actuated degrees of freedom (e.g., rotation about a drive axis, and rotation about a steering axis). In the embodiment of FIGS. 3A and 3B, each wheel 204 is associated with both a steering actuator 206 and a driving actuator 208, as described in the preceding paragraphs. As such, the mobile base 200 of FIGS. 3A and 3B includes four wheels 204a-204d and eight associated actuators (i.e., steering actuators 206a-206d and driving actuators 208a-208d). In embodiments of a mobile base with different numbers of wheels, a mobile base with independently steerable and independently drivable wheels may be associated with twice as many actuators as the number of wheels.
See Figure 3A.
While this element if brought into the claims would be unlikely to make the claims allowable on their own, it may necessitate another reference to be used in the rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 18, and 20 cite “to move the mobile logistics robot in any direction, orientation, and path within the space bounded at least in part…”
This claim is not particularly pointed out or distinctly claimed. It is not particularly pointed out or distinctly claimed what “any path” is referring to. The examiner believes that it is the intention to capture flexibility of movement in the robot, in that the wheels do not need to drive backwards or forwards like a car may have to, but that the wheels can steer themselves and turn on a dime. However, this is not particularly pointed out and distinctly claimed. The paths might refer to paths through walls, or through narrow gaps smaller than the robot. They might refer to any path that the robot is capable of following, or any path the robot would be provided with. They could potentially be 2 dimensional, or three dimensional. Some interpretations are impossible, whereas others are trivial. For the sake of rejection, the examiner is treating the claim as that the drive elements must be capable of moving the robot along paths within the space bounded. Thus the examiner considers this element not particularly pointed out or distinctly claimed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 12-14 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Galluzzo et al (US Pub 2020/0242544 A1), hereafter known as Galluzzo in light of Oka et al (US Pub 2020/0391385 A1), hereafter known as Oka.
For Claim 1, Galluzzo teaches A mobile logistics robot, comprising:
a mobile chassis having a plurality of independently controllable drive elements; ([0071] As shown in FIGS. 1A, 1B and 2, in certain embodiments the individual manipulation robots 100 may have a wheeled mobile base 160, internal batteries 190, and an onboard computer processor 218 with memory storage 216. The robots may also have at least one temporary storage bed 140 for picked items and at least one robotic manipulator arm 120. The onboard computer processor 218 may be configured to run a set of programs with algorithms capable of performing navigation and picking. Further, the onboard computer processor 218 utilizes data from sensors (150, 110), to output control signals to the mobile base 160 and manipulator arm 120 for navigation and picking, respectively.
[0161] The manipulator robots 100 have a mobile base 160 that is controlled by the onboard computer processor 218. The mobile base may have two main drive wheels 167, each driven by a servo motor. Each drive wheel 167 may have an encoder that provides motion feedback, which is used to precisely control the speed of each wheel in order to achieve the desired rotation and translation velocities of the robot 100. The feedback data is also used for odometry to estimate the motion of the robot 100 relative to the facility. The odometry is responsible for guiding the robot 100 navigation at times when visual markers 420 are out of sensor (150, 110) range. The mobile base 160 may also use passive wheels, such as casters 165, for stability and weight distribution.
Figure 1A and 1B)
one or more robotic arms mounted on the mobile chassis; ([0101] A robot manipulator arm 620 may be used in the presently disclosed manipulation robot 600 to pick totes 650 or bins within a logistics facility. As shown in FIGS. 7A and 7B, the manipulator arm 620 may be mounted to the containment area 640 at a position distal from the mobile base 660 of the manipulation robot 600. The vertical reach of the manipulator arm 620 may be extended, for example, by mounting a proximal end of the containment area 640 on a track that may provide vertical motion of the platform with respect to the mobile base 660.
[0102] The manipulator arm 620, which is at the distal end portion of the containment area 640, may be mounted on a vertical actuator stage which may raise and lower the manipulator arm 620 relative to the containment area 640. This may provide the clearance necessary to enable the manipulator arm 620 to transfer more than one tote 650 or bin onto the containment area 640, such as stacked one on another, or as detailed above (multiple levels on the transport platform). Further, the articulated segments of the manipulator arm 620 may provide the clearance necessary to enable the manipulator arm 620 to transfer more than one tote 650 or bin onto the containment area 640 without the need or use of a vertical actuator stage.
Figure 1A and 1B)
a processor configured to control the one or more robotic arms and the plurality of independently controllable drive elements as needed to pick items from a set of one or more source locations and place each item in a corresponding destination location included in a set of one or more destination locations, including by using the independently controllable drive elements to move the mobile chassis within a space bounded at least in part by the one or more source locations and the one or more destination locations. ([0014] The presently disclosed invention overcomes many of the shortcomings of the prior art by providing systems, devices and methods for robotic piece picking or put-away directly from existing stock item locations in a logistics facility. The presently disclosed invention provides a mobile robotic system that includes sensors and manipulator arm(s) to perceive, localize, reach, grasp and transfer SKUs from a storage rack to a transport container for piece picking, or conversely, from a transport container to a storage rack for put-away. This system and method allow existing facility infrastructure to remain intact and further allows the facility to use both manual picking and robotic picking interchangeably.
[0016] According to its major aspects, and briefly stated, the presently disclosed invention includes a system for piece picking or put-away within a logistics facility comprising a central server and at least one mobile manipulation robot. The logistics facility may be a warehouse, distribution center, manufacturing facility, or retail facility. The central server comprises a server communication interface, one or more server processors, and a server memory. Each of the mobile manipulation robots comprise a mobile base, at least one articulated manipulator arm having an end effector, at least one piece containment area, a plurality of sensors, a remote communication interface, a robot memory configured to store robot specific information, and one or more robot processors coupled to the sensors, the robot memory, the mobile base, and the at least one articulated manipulator arm.
[0112] As shown in zone 830, manipulation robots 874 may pick items, totes, or bins from standard shelving 840 and may transport those picks to a different zone within the logistics facility, such as to a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). The manipulation robots 873 may pick items, totes, or bins and may transfer those items to a transport robot 868. The transport robot 868 may then travel to a different location to deliver the item. Exemplary locations include a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). Alternatively, the transport robot 868 may transfer the items to another transport robot 867 for delivery to any of those locations.
[0128] In order to perform pick work, the manipulator robots 100 may move and navigate between pick locations in the work zone 330 and an order transfer area 360. During navigation the sensor data may be processed by the onboard computer processor 218 in a navigation software module 212 to extract two modalities of information. The first modality may be local mapping information that indicates which areas around the manipulation robot 100 are traversable and which areas contain obstacles. The ground facing sensors 150 on the manipulation robot 100 are primarily used to generate this mapping information and collision detection information. There may be two ground facing sensors 150, a front-facing one and a rear-facing one. This unique design allows the manipulation robot 100 to navigate while driving both forwards and backwards, which in certain picking scenarios, eliminates the need for the manipulation robot 100 to turn around, thus reducing travel time and increasing picking efficiency.
[0125] The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.
[0019] In certain embodiments of the system, the server memory may comprise computer program instructions executable by the one or more server processors to receive data from a warehouse management system and dispatch the at least one mobile manipulation robot. The server communication interface may connect with the remote communication interface to send and receive piece picking data which may include a unique identification for each piece to be picked, a location within the logistics facility of the pieces to be picked, and a route for the at least one mobile manipulation robot to take within the logistics facility. The unique identification for the piece to be picked may comprise a shape of the piece, a size of the piece, a weight of the piece, a color of the piece, a property of the construction material of the piece, such as roughness, porosity, and deformability, a visual marking on the piece, a barcode on the piece, or any combination thereof. Further, the connection between the server communication interface and the robot communication interface may be via one or more wired or wireless networks, or a combination thereof.
Figures 3-6 clearly show shelves and storage that would obstruct movement of the mobile chassis.)
wherein the processor is configured to control the plurality of independently controllable drive elements to move the mobile logistics robot in any direction, orientation, and path within the space bounded at least in part by the one or more source locations and the one or more destination locations; and
[0043] The present invention further relates to a method for autonomous robot navigation and region of interest localization, the method comprising: receiving data captured by a sensor coupled to a robotic device during navigation of the robotic device; analyzing the received data to detect at least one identifier corresponding to a region of interest; for each detected identifier: using the data to determine a current pose of the robotic device within a logistics facility; and generating a navigation instruction for navigation of the robotic device to a location of an item, the navigation instruction based on the current pose of the robotic device and a location of a region of interest at which the item is located.
[0117] The system's central server 200 may be used to process order information that is transacted with a WMS 201 and may coordinate the fulfillment of orders with a plurality of manipulation robots 100. All computation on the server 200 may be executed by one or more internal processors 220. In certain embodiments, the server may have two software modules that enable this order fulfillment coordination. The first processor may be a task dispatch module 228, which analyzes orders received from a WMS 201, and determines which of the plurality of manipulation robots 100 is to be assigned to an order. After a manipulation robot 100 is selected for picking an order, the task dispatcher 228 instructs the robot 100 with high-level order picking information, such as, route navigation paths, SKU locations, and an order drop-off location. The task dispatcher 228 works closely with a system state monitor 230 to obtain key feedback information from the system. The system state monitor 230 may communicate with the manipulation robots 100 to keep track of their current physical location within the facility, along with status information, which may include but is not limited to: whether the robot 100 is currently assigned an order, any faults or error modes, health information, such as remaining battery power, or charging status.
[0161] The manipulator robots 100 have a mobile base 160 that is controlled by the onboard computer processor 218. The mobile base may have two main drive wheels 167, each driven by a servo motor. Each drive wheel 167 may have an encoder that provides motion feedback, which is used to precisely control the speed of each wheel in order to achieve the desired rotation and translation velocities of the robot 100. The feedback data is also used for odometry to estimate the motion of the robot 100 relative to the facility. The odometry is responsible for guiding the robot 100 navigation at times when visual markers 420 are out of sensor (150, 110) range. The mobile base 160 may also use passive wheels, such as casters 165, for stability and weight distribution.
Figure 1A and 1B, the robot can clearly be controlled to move in any direction or orientation.
Galluzzo does not explicitly teach wherein the processor is further configured to generate for each item a set of plans to move the item from an associated source location to a corresponding destination location, each plan comprising a sequence of locations and poses, and to select from the set of plans a selected plan to move the item.
Oka, however, does teach wherein the processor is further configured to generate for each item a set of plans to move the item from an associated source location to a corresponding destination location, each plan comprising a sequence of locations and poses, and to select from the set of plans a selected plan to move the item. ([0063] The grasp plan generator 54 calculates a grasping method and a grasping pose of the object OBJ at the initial position HP, and a moving route and via points along which the manipulator 20 or hand 22 is moved to the initial position HP. The grasp plan generator 54 also calculates a moving route and via points of the hand 22 to grasp a next intended object OBJ after releasing the object OBJ at the moving destination RP. In these cases, the object information acquired by the camera 32a is utilized in calculation of the moving route and via points to move the hand 22 without interfering with surrounding obstacles such as wall surfaces of the containers 14a and 14b or an object or objects other than the currently moved object OBJ.
[0086] To create the placement plan, for example, the route calculator 56 acquires, from the camera 32a and the laser range scanners 33a and 33b, input information including the grasping pose and the size of the object OBJ currently grasped by the hand 22 or suction pad 22a and to be moved and placed at the moving destination RP (container 14b) (S100). Subsequently, the route calculator 56 calculates the pose of the currently grasped object OBJ to place the object OBJ in the container 14b in accordance with the grasping pose and the size of the object OBJ (S102). The route calculator 56 calculates a placeable position and a pose of the grasped object OBJ, that is, a candidate for the moving destination RP on the basis of the status information of previously set objects OBJs in the container 14b (S104). The route calculator 56 calculates a plurality of patterns of position and pose candidates of a fingertip TCP of the hand 22 in placing the object OBJ, on the basis of information on the previously set object OBJs in the container 14b (S106), and selects an optimal position and pose candidate from the patterns (S108). The route calculator 56 also sets information including a target force value, a position of a pressing surface, and a pressing direction used in the force control by the manipulator 20 (hand 22) during movement or placement (S110). The route calculator 56 calculates via points (via position and pose RAP) on the moving route of the object OBJ from the size and the grasping pose of the object OBJ, a state of the previously set objects OBJs, the position and pose of the fingertip TCP at the time of placing the object OBJ (S112). The moving destination RP and the via position and pose RAP calculated as the candidates are associated with scores such as preset priority, and the route calculator 56 selects an optimum moving destination RP and via position and pose RAP according to the scores. After the route calculator 56 succeeds in generating the moving route not to interfere with obstacles such as the previously set objects OBJs or the container 14b, the robot controller 57 causes the manipulator 20 to operate.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Galluzzo in light of Oka such that for each item a set of plans is created including poses and locations and then one is selected because having the locations and poses for a trajectory would allow the system to understand if any collisions are possible, or if it will be effective. By choosing one, the system can select the option that is most likely to succeed and less likely to cause other issues.
For Claim 2, Galluzzo teaches The mobile logistics robot of claim 1, wherein the one or more source locations include one or more conveyors, chutes, shelves containers, pallets, or other receptacles. ([0084] While each of the aforementioned actions of the manipulator arm and end effector, and optional extension tool, are discussed with respect to picking an item from a shelf, the robotic devices, systems, and methods disclosed herein may also be useful for picking bins, totes, or cases from a shelf, or from another robot (i.e., an autonomous mobile robot, other manipulation robots, conveyance systems, human workers, etc.). Moreover, the picking of individual items is discussed with reference to FIGS. 5A and 5B, wherein the items are stored openly on shelving. The present systems and methods envision picking of items that may be stored as multiples of items, e.g., multipacks, and/or may be stored in any configuration within bins, i.e., individually, as multiples, mixed with other items in a bin, etc.)
For Claim 3, Galluzzo teaches The mobile logistics robot of claim 1, wherein the one or more destination locations includes one or more conveyors, chutes, shelves containers, pallets, or other receptacles. ([0085] After items are picked, they may be placed into the storage bed 140 for transportation. The bed may also carry a container 145, such as a box or tote, in which the items can be placed. This method enables multiple items to be picked for a given order or batch of orders. This method frees the robot manipulator arm 120 to pick additional items without needing to take multiple trips to and from an order transfer area 360 (See FIG. 3). Additionally, by carrying a packing box or container or transport tote 145 onboard, the manipulation robot 100 is able to aggregate order items together into a single container that can be easily swapped with a different container for additional order fulfillment.)
For Claim 4, Galluzzo teaches The mobile logistics robot of claim 1, wherein the set of destination locations includes a plurality pallets or other receptacles. ([0005] When goods need to be retrieved individually for order fulfillment or selection by a customer, they are typically stored individually and are not grouped into cases or pallets. The process of breaking the cases or pallets for individual product picking, that is, taking the individual pieces from the case or pallet and placing them in a specific storage location in a facility, is called put-away. The process of picking or selecting individual items from a specific storage location in a facility is known as piece picking or each-picking. Put-away and piece picking happens in both distribution warehouses and retail centers, whereas case-picking or pallet-picking typically only happens at a wholesale distribution center.
[0085] After items are picked, they may be placed into the storage bed 140 for transportation. The bed may also carry a container 145, such as a box or tote, in which the items can be placed. This method enables multiple items to be picked for a given order or batch of orders. This method frees the robot manipulator arm 120 to pick additional items without needing to take multiple trips to and from an order transfer area 360 (See FIG. 3). Additionally, by carrying a packing box or container or transport tote 145 onboard, the manipulation robot 100 is able to aggregate order items together into a single container that can be easily swapped with a different container for additional order fulfillment.)
For Claim 5, Galluzzo teaches The mobile logistics robot of claim 4, wherein the pallets or other receptacles are arranged in a manner that constrains movement of the mobile chassis on two or more sides. (items to another transport robot 867 for delivery to any of those locations.
[0128] In order to perform pick work, the manipulator robots 100 may move and navigate between pick locations in the work zone 330 and an order transfer area 360. During navigation the sensor data may be processed by the onboard computer processor 218 in a navigation software module 212 to extract two modalities of information. The first modality may be local mapping information that indicates which areas around the manipulation robot 100 are traversable and which areas contain obstacles. The ground facing sensors 150 on the manipulation robot 100 are primarily used to generate this mapping information and collision detection information. There may be two ground facing sensors 150, a front-facing one and a rear-facing one. This unique design allows the manipulation robot 100 to navigate while driving both forwards and backwards, which in certain picking scenarios, eliminates the need for the manipulation robot 100 to turn around, thus reducing travel time and increasing picking efficiency.
[0125] The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.
[0019] In certain embodiments of the system, the server memory may comprise computer program instructions executable by the one or more server processors to receive data from a warehouse management system and dispatch the at least one mobile manipulation robot. The server communication interface may connect with the remote communication interface to send and receive piece picking data which may include a unique identification for each piece to be picked, a location within the logistics facility of the pieces to be picked, and a route for the at least one mobile manipulation robot to take within the logistics facility. The unique identification for the piece to be picked may comprise a shape of the piece, a size of the piece, a weight of the piece, a color of the piece, a property of the construction material of the piece, such as roughness, porosity, and deformability, a visual marking on the piece, a barcode on the piece, or any combination thereof. Further, the connection between the server communication interface and the robot communication interface may be via one or more wired or wireless networks, or a combination thereof.
Figures 3-6 clearly show shelves and storage that would obstruct movement of the mobile chassis.)
For Claim 6, Galluzzo teaches The mobile logistics robot of claim 4, wherein the pallets or other receptacles are arranged in parallel rows that define a lane via which the mobile logistics robot is configured to transit to access pallets or other receptacles on either side of a lane. ([0109] FIG. 6 shows another exemplary floor plan for a section of a logistics facility 800 in which the manipulation robot may be deployed. According to certain aspects of the presently disclosed invention, various work zones may be defined within a logistics facility. For example, a logistics facility 800 may include zones that are robot specific work zones where human workers are excluded 830, zones where humans and robots may work side-by-side 820, and human-only work zones where robots are substantially or totally excluded 810. While shown in FIG. 6 to include entire rows of shelving units 840, these zones may be setup in any user defined manner, such that portions of shelving or storage rows or even individual units may include two or more work zones.
[0110] These various work zones may be mapped using granular information, such as 1D bar codes placed on ends of racks, or may be mapped in a more defined manner, such as using identifiers that define specific regions of interest (e.g., individual racks in a row of racks; described in more detail hereinbelow).
Figures 3-6 clearly show shelves and storage that would obstruct movement of the mobile chassis.)
For Claim 7, Galluzzo teaches The mobile logistics robot of claim 1, wherein the processor is configured to control the plurality of independently controllable drive elements to move the mobile chassis laterally. ([0071] As shown in FIGS. 1A, 1B and 2, in certain embodiments the individual manipulation robots 100 may have a wheeled mobile base 160, internal batteries 190, and an onboard computer processor 218 with memory storage 216. The robots may also have at least one temporary storage bed 140 for picked items and at least one robotic manipulator arm 120. The onboard computer processor 218 may be configured to run a set of programs with algorithms capable of performing navigation and picking. Further, the onboard computer processor 218 utilizes data from sensors (150, 110), to output control signals to the mobile base 160 and manipulator arm 120 for navigation and picking, respectively.
[0043] The present invention further relates to a method for autonomous robot navigation and region of interest localization, the method comprising: receiving data captured by a sensor coupled to a robotic device during navigation of the robotic device; analyzing the received data to detect at least one identifier corresponding to a region of interest; for each detected identifier: using the data to determine a current pose of the robotic device within a logistics facility; and generating a navigation instruction for navigation of the robotic device to a location of an item, the navigation instruction based on the current pose of the robotic device and a location of a region of interest at which the item is located.
[0161] The manipulator robots 100 have a mobile base 160 that is controlled by the onboard computer processor 218. The mobile base may have two main drive wheels 167, each driven by a servo motor. Each drive wheel 167 may have an encoder that provides motion feedback, which is used to precisely control the speed of each wheel in order to achieve the desired rotation and translation velocities of the robot 100. The feedback data is also used for odometry to estimate the motion of the robot 100 relative to the facility. The odometry is responsible for guiding the robot 100 navigation at times when visual markers 420 are out of sensor (150, 110) range. The mobile base 160 may also use passive wheels, such as casters 165, for stability and weight distribution.)
For Claim 8, Galluzzo teaches The mobile logistics robot of claim 1, wherein the processor is configured to control the plurality of independently controllable drive elements to rotate the mobile chassis about an arbitrary vertical axis. ([0161] The manipulator robots 100 have a mobile base 160 that is controlled by the onboard computer processor 218. The mobile base may have two main drive wheels 167, each driven by a servo motor. Each drive wheel 167 may have an encoder that provides motion feedback, which is used to precisely control the speed of each wheel in order to achieve the desired rotation and translation velocities of the robot 100. The feedback data is also used for odometry to estimate the motion of the robot 100 relative to the facility. The odometry is responsible for guiding the robot 100 navigation at times when visual markers 420 are out of sensor (150, 110) range. The mobile base 160 may also use passive wheels, such as casters 165, for stability and weight distribution.)
For Claim 12, Galluzzo teaches The mobile logistics robot of claim 1, further comprising a camera or other sensor and wherein the processor is configured to use image or other sensor data generated by the camera or other sensor to control the one or more robotic arms and the plurality of independently controllable drive elements as needed to pick items from a set of one or more source locations and place each item in a corresponding destination location included in a set of one or more destination locations. ([0018] Further, the plurality of sensors provide signals related to detection, identification, and location of the piece to be picked, and the one or more robot processors analyze the sensor information to generate articulated arm control signals to guide the end effector of the at least one articulated manipulator arm to pick the piece. The sensors may also provide signals related to a unique identification for the piece to be picked, an obstacle detected in the path of the at least one mobile manipulation robot, and a current location within the logistics facility of the at least one mobile manipulation robot.
[0066] As used herein, the terms “shelf tag” and “marker” may refer to an object used to identify a location. Most commonly a shelf tag or marker may be a fiducial marker placeable in the field of view of an imaging system. Exemplary fiducial markers include at least 1D and 2D bar codes and ArUco markers. Shelf tags or marker may also be understood to refer to an object that is not visually perceived, such as RFID, sound, or tactile markers that may identify or differentiate an identity.
[0020] In embodiments of the system, the at least one mobile manipulation robot may be able to autonomously navigate and position itself within the logistics facility by recognition of at least one landmark by at least one of the plurality of sensors. The landmark may be a vertically mounted marker placed at a specific location within the logistics facility or may be other identifiable visual or audible landmarks within the logistics facility. The sensors may be any 3D device capable of sensing the local environment such as, for example, 3D depth cameras, color cameras, grey scale cameras, laser ranging devices, sonar devices, radar devices, or combinations thereof.)
For Claim 13, Galluzzo teaches The mobile logistics robot of claim 1, further comprising a camera or other sensor and wherein the processor is configured to use image or other sensor data generated by the camera or other sensor to detect one or both of the presence and the approach of a human and to control the one or more robotic arms and the plurality of independently controllable drive elements as needed to ensure the mobile logistics does not endanger the human. ([0075] Furthermore, each manipulation robot 100 may be configured to receive signals from the central server 200, or directly from the WMS 201, which may indicate an emergency and may direct the robot 100 to stop and/or may further activate the one or more safety lights or strobes 155 and/or audible warning annunciator or horn. In the event that an unstable and/or unsafe diagnostic state for the manipulation robot 100 is detected by the one or more robot processors 218, the robot 100 may be stopped. The manipulation robot 100 may also be stopped if the sensors (150, 110) detect a human or obstacle in close proximity or detect unsafe operation of the robot 100. Such signals may be processes at the central server 200 which may then control the robot speed and or direction of operation.)
For Claim 14, Galluzzo teaches The mobile logistics robot of claim 13, wherein the processor uses the image or other sensor data generated by the camera or other sensor to steer the mobile chassis as needed to ensure the human remains at a safe distance. ([0075] Furthermore, each manipulation robot 100 may be configured to receive signals from the central server 200, or directly from the WMS 201, which may indicate an emergency and may direct the robot 100 to stop and/or may further activate the one or more safety lights or strobes 155 and/or audible warning annunciator or horn. In the event that an unstable and/or unsafe diagnostic state for the manipulation robot 100 is detected by the one or more robot processors 218, the robot 100 may be stopped. The manipulation robot 100 may also be stopped if the sensors (150, 110) detect a human or obstacle in close proximity or detect unsafe operation of the robot 100. Such signals may be processes at the central server 200 which may then control the robot speed and or direction of operation.
[0076] The safety features of the robots disclosed herein may include a health monitor module on the robot processor/memory that may receive signals from the various sensors and may communicate a fault or error state to a remote server. As example, the health monitor may register a power loss, or obstacle, or sensor failure and may communicate this information to the remote server. The robotic health monitor may cause the robot to stop, slow movement, signal an audible or visual error state, or change routes, or after receiving signals from the robot regarding an error or fault state, the remote server may cause any of these actions. Certain limits may be dynamically set for the robots depending on the logistics facility and/or specific job requirements of the robot. For example, in facilities where human workers may work side-by-side with the robots of the present invention, the distance limits at which an object is registered as an obstacle may be set to avoid accidental contact with a human, or the robot may be configured to slow when approaching a human worker. Additionally, should an error be registered at the remote server for a robot, a human worker may be dispatched to clear the error (e.g., move an obstacle).)
For Claim 16, Galluzzo teaches The mobile logistics robot of claim 1, wherein the processor is configured to drive the mobile chassis to a position beneath a conveyor or other conveyance structure and to use the one or more robotic arms to pick items from or place items to a surface of the conveyance structure. ([0093] The manipulation robot 600 may unload a tote 650 stored on the containment area 640 by first aligning the containment area 640 with the conveyor or another staging area. The manipulator arm 620 may then grasp/move the tote from the containment area 640 and move it to the conveyor or to a transport robot or to the platform. Alternatively, or in addition, the containment area 640 of the manipulation robot may include a conveyance means, such as a conveyor belt or roller bars.
[0094] According to certain aspects, the containment area 640 may include more than one level configured to hold a bin or tote 650, wherein at least that portion of the containment area 640 closest to the main body of the robot 600 is configured to hold the multiple totes 650 and to remain stationary with respect to the manipulator arm 620. An adjacent portion of the containment area 640 closest to/connected to the manipulator arm 620 may be vertically moveable with respect to the main body 615 of the manipulation robot 600.)
For Claim 17, Galluzzo teaches The mobile logistics robot of claim 16, wherein the processor is further configured to drive the mobile chassis fore and aft along a longitudinal access of the conveyor or other conveyance structure as needed to use the one or more robotic arms to pick items from or place items to a surface of the conveyance structure. ([0128] In order to perform pick work, the manipulator robots 100 may move and navigate between pick locations in the work zone 330 and an order transfer area 360. During navigation the sensor data may be processed by the onboard computer processor 218 in a navigation software module 212 to extract two modalities of information. The first modality may be local mapping information that indicates which areas around the manipulation robot 100 are traversable and which areas contain obstacles. The ground facing sensors 150 on the manipulation robot 100 are primarily used to generate this mapping information and collision detection information. There may be two ground facing sensors 150, a front-facing one and a rear-facing one. This unique design allows the manipulation robot 100 to navigate while driving both forwards and backwards, which in certain picking scenarios, eliminates the need for the manipulation robot 100 to turn around, thus reducing travel time and increasing picking efficiency.
[0022] Certain embodiments of the system may further comprise a conveyance device configured to accept items from the at least one mobile manipulation robot. The conveyance device may be a conveyor belt which transfers the accepted items from a transfer area to a receiving area, wherein the receiving area is a packing area, a shipping area, a holding area, or any combination thereof.)
For Claim 18, Galluzzo teaches A method to control a mobile logistic robot comprising a mobile chassis having a plurality of independently controllable drive elements ([0071] As shown in FIGS. 1A, 1B and 2, in certain embodiments the individual manipulation robots 100 may have a wheeled mobile base 160, internal batteries 190, and an onboard computer processor 218 with memory storage 216. The robots may also have at least one temporary storage bed 140 for picked items and at least one robotic manipulator arm 120. The onboard computer processor 218 may be configured to run a set of programs with algorithms capable of performing navigation and picking. Further, the onboard computer processor 218 utilizes data from sensors (150, 110), to output control signals to the mobile base 160 and manipulator arm 120 for navigation and picking, respectively.
[0161] The manipulator robots 100 have a mobile base 160 that is controlled by the onboard computer processor 218. The mobile base may have two main drive wheels 167, each driven by a servo motor. Each drive wheel 167 may have an encoder that provides motion feedback, which is used to precisely control the speed of each wheel in order to achieve the desired rotation and translation velocities of the robot 100. The feedback data is also used for odometry to estimate the motion of the robot 100 relative to the facility. The odometry is responsible for guiding the robot 100 navigation at times when visual markers 420 are out of sensor (150, 110) range. The mobile base 160 may also use passive wheels, such as casters 165, for stability and weight distribution.
Figure 1A and 1B)
and one or more robotic arms mounted on io the mobile chassis, ([0101] A robot manipulator arm 620 may be used in the presently disclosed manipulation robot 600 to pick totes 650 or bins within a logistics facility. As shown in FIGS. 7A and 7B, the manipulator arm 620 may be mounted to the containment area 640 at a position distal from the mobile base 660 of the manipulation robot 600. The vertical reach of the manipulator arm 620 may be extended, for example, by mounting a proximal end of the containment area 640 on a track that may provide vertical motion of the platform with respect to the mobile base 660.
[0102] The manipulator arm 620, which is at the distal end portion of the containment area 640, may be mounted on a vertical actuator stage which may raise and lower the manipulator arm 620 relative to the containment area 640. This may provide the clearance necessary to enable the manipulator arm 620 to transfer more than one tote 650 or bin onto the containment area 640, such as stacked one on another, or as detailed above (multiple levels on the transport platform). Further, the articulated segments of the manipulator arm 620 may provide the clearance necessary to enable the manipulator arm 620 to transfer more than one tote 650 or bin onto the containment area 640 without the need or use of a vertical actuator stage.
Figure 1A and 1B)
the method comprising using a processor to control the one or more robotic arms and the plurality of independently controllable drive elements as needed to pick items from a set of one or more source locations and place each item in a corresponding destination location included in a set of one or more destination locations, including by using the independently controllable drive elements to move the mobile chassis within a space bounded at least in part by is the one or more source locations and the one or more destination locations. ([0014] The presently disclosed invention overcomes many of the shortcomings of the prior art by providing systems, devices and methods for robotic piece picking or put-away directly from existing stock item locations in a logistics facility. The presently disclosed invention provides a mobile robotic system that includes sensors and manipulator arm(s) to perceive, localize, reach, grasp and transfer SKUs from a storage rack to a transport container for piece picking, or conversely, from a transport container to a storage rack for put-away. This system and method allow existing facility infrastructure to remain intact and further allows the facility to use both manual picking and robotic picking interchangeably.
[0016] According to its major aspects, and briefly stated, the presently disclosed invention includes a system for piece picking or put-away within a logistics facility comprising a central server and at least one mobile manipulation robot. The logistics facility may be a warehouse, distribution center, manufacturing facility, or retail facility. The central server comprises a server communication interface, one or more server processors, and a server memory. Each of the mobile manipulation robots comprise a mobile base, at least one articulated manipulator arm having an end effector, at least one piece containment area, a plurality of sensors, a remote communication interface, a robot memory configured to store robot specific information, and one or more robot processors coupled to the sensors, the robot memory, the mobile base, and the at least one articulated manipulator arm.
[0112] As shown in zone 830, manipulation robots 874 may pick items, totes, or bins from standard shelving 840 and may transport those picks to a different zone within the logistics facility, such as to a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). The manipulation robots 873 may pick items, totes, or bins and may transfer those items to a transport robot 868. The transport robot 868 may then travel to a different location to deliver the item. Exemplary locations include a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). Alternatively, the transport robot 868 may transfer the items to another transport robot 867 for delivery to any of those locations.
[0128] In order to perform pick work, the manipulator robots 100 may move and navigate between pick locations in the work zone 330 and an order transfer area 360. During navigation the sensor data may be processed by the onboard computer processor 218 in a navigation software module 212 to extract two modalities of information. The first modality may be local mapping information that indicates which areas around the manipulation robot 100 are traversable and which areas contain obstacles. The ground facing sensors 150 on the manipulation robot 100 are primarily used to generate this mapping information and collision detection information. There may be two ground facing sensors 150, a front-facing one and a rear-facing one. This unique design allows the manipulation robot 100 to navigate while driving both forwards and backwards, which in certain picking scenarios, eliminates the need for the manipulation robot 100 to turn around, thus reducing travel time and increasing picking efficiency.
[0125] The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.
[0019] In certain embodiments of the system, the server memory may comprise computer program instructions executable by the one or more server processors to receive data from a warehouse management system and dispatch the at least one mobile manipulation robot. The server communication interface may connect with the remote communication interface to send and receive piece picking data which may include a unique identification for each piece to be picked, a location within the logistics facility of the pieces to be picked, and a route for the at least one mobile manipulation robot to take within the logistics facility. The unique identification for the piece to be picked may comprise a shape of the piece, a size of the piece, a weight of the piece, a color of the piece, a property of the construction material of the piece, such as roughness, porosity, and deformability, a visual marking on the piece, a barcode on the piece, or any combination thereof. Further, the connection between the server communication interface and the robot communication interface may be via one or more wired or wireless networks, or a combination thereof.
Figures 3-6 clearly show shelves and storage that would obstruct movement of the mobile chassis.)
Galluzzo does not explicitly teach wherein the processor is further configured to generate for each item a set of plans to move the item from an associated source location to a corresponding destination location, each plan comprising a sequence of locations and poses, and to select from the set of plans a selected plan to move the item.
Oka, however, does teach wherein the processor is further configured to generate for each item a set of plans to move the item from an associated source location to a corresponding destination location, each plan comprising a sequence of locations and poses, and to select from the set of plans a selected plan to move the item. ([0063] The grasp plan generator 54 calculates a grasping method and a grasping pose of the object OBJ at the initial position HP, and a moving route and via points along which the manipulator 20 or hand 22 is moved to the initial position HP. The grasp plan generator 54 also calculates a moving route and via points of the hand 22 to grasp a next intended object OBJ after releasing the object OBJ at the moving destination RP. In these cases, the object information acquired by the camera 32a is utilized in calculation of the moving route and via points to move the hand 22 without interfering with surrounding obstacles such as wall surfaces of the containers 14a and 14b or an object or objects other than the currently moved object OBJ.
[0086] To create the placement plan, for example, the route calculator 56 acquires, from the camera 32a and the laser range scanners 33a and 33b, input information including the grasping pose and the size of the object OBJ currently grasped by the hand 22 or suction pad 22a and to be moved and placed at the moving destination RP (container 14b) (S100). Subsequently, the route calculator 56 calculates the pose of the currently grasped object OBJ to place the object OBJ in the container 14b in accordance with the grasping pose and the size of the object OBJ (S102). The route calculator 56 calculates a placeable position and a pose of the grasped object OBJ, that is, a candidate for the moving destination RP on the basis of the status information of previously set objects OBJs in the container 14b (S104). The route calculator 56 calculates a plurality of patterns of position and pose candidates of a fingertip TCP of the hand 22 in placing the object OBJ, on the basis of information on the previously set object OBJs in the container 14b (S106), and selects an optimal position and pose candidate from the patterns (S108). The route calculator 56 also sets information including a target force value, a position of a pressing surface, and a pressing direction used in the force control by the manipulator 20 (hand 22) during movement or placement (S110). The route calculator 56 calculates via points (via position and pose RAP) on the moving route of the object OBJ from the size and the grasping pose of the object OBJ, a state of the previously set objects OBJs, the position and pose of the fingertip TCP at the time of placing the object OBJ (S112). The moving destination RP and the via position and pose RAP calculated as the candidates are associated with scores such as preset priority, and the route calculator 56 selects an optimum moving destination RP and via position and pose RAP according to the scores. After the route calculator 56 succeeds in generating the moving route not to interfere with obstacles such as the previously set objects OBJs or the container 14b, the robot controller 57 causes the manipulator 20 to operate.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Galluzzo in light of Oka such that for each item a set of plans is created including poses and locations and then one is selected because having the locations and poses for a trajectory would allow the system to understand if any collisions are possible, or if it will be effective. By choosing one, the system can select the option that is most likely to succeed and less likely to cause other issues.
For Claim 19, Galluzzo teaches The method of claim 18, wherein one or both of the source locations and the destination locations comprise pallets or other receptacles arranged in a manner that constrains movement of the mobile chassis on two or more sides. ([0084] While each of the aforementioned actions of the manipulator arm and end effector, and optional extension tool, are discussed with respect to picking an item from a shelf, the robotic devices, systems, and methods disclosed herein may also be useful for picking bins, totes, or cases from a shelf, or from another robot (i.e., an autonomous mobile robot, other manipulation robots, conveyance systems, human workers, etc.). Moreover, the picking of individual items is discussed with reference to FIGS. 5A and 5B, wherein the items are stored openly on shelving. The present systems and methods envision picking of items that may be stored as multiples of items, e.g., multipacks, and/or may be stored in any configuration within bins, i.e., individually, as multiples, mixed with other items in a bin, etc.)
For Claim 20, Galluzzo teaches A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions to control a mobile logistic robot ([0019] In certain embodiments of the system, the server memory may comprise computer program instructions executable by the one or more server processors to receive data from a warehouse management system and dispatch the at least one mobile manipulation robot. The server communication interface may connect with the remote communication interface to send and receive piece picking data which may include a unique identification for each piece to be picked, a location within the logistics facility of the pieces to be picked, and a route for the at least one mobile manipulation robot to take within the logistics facility. The unique identification for the piece to be picked may comprise a shape of the piece, a size of the piece, a weight of the piece, a color of the piece, a property of the construction material of the piece, such as roughness, porosity, and deformability, a visual marking on the piece, a barcode on the piece, or any combination thereof. Further, the connection between the server communication interface and the robot communication interface may be via one or more wired or wireless networks, or a combination thereof.) comprising a mobile chassis having a plurality of independently controllable drive elements (([0071] As shown in FIGS. 1A, 1B and 2, in certain embodiments the individual manipulation robots 100 may have a wheeled mobile base 160, internal batteries 190, and an onboard computer processor 218 with memory storage 216. The robots may also have at least one temporary storage bed 140 for picked items and at least one robotic manipulator arm 120. The onboard computer processor 218 may be configured to run a set of programs with algorithms capable of performing navigation and picking. Further, the onboard computer processor 218 utilizes data from sensors (150, 110), to output control signals to the mobile base 160 and manipulator arm 120 for navigation and picking, respectively.
[0161] The manipulator robots 100 have a mobile base 160 that is controlled by the onboard computer processor 218. The mobile base may have two main drive wheels 167, each driven by a servo motor. Each drive wheel 167 may have an encoder that provides motion feedback, which is used to precisely control the speed of each wheel in order to achieve the desired rotation and translation velocities of the robot 100. The feedback data is also used for odometry to estimate the motion of the robot 100 relative to the facility. The odometry is responsible for guiding the robot 100 navigation at times when visual markers 420 are out of sensor (150, 110) range. The mobile base 160 may also use passive wheels, such as casters 165, for stability and weight distribution.
Figure 1A and 1B)) and one or more robotic arms mounted on the mobile chassis, ([0101] A robot manipulator arm 620 may be used in the presently disclosed manipulation robot 600 to pick totes 650 or bins within a logistics facility. As shown in FIGS. 7A and 7B, the manipulator arm 620 may be mounted to the containment area 640 at a position distal from the mobile base 660 of the manipulation robot 600. The vertical reach of the manipulator arm 620 may be extended, for example, by mounting a proximal end of the containment area 640 on a track that may provide vertical motion of the platform with respect to the mobile base 660.
[0102] The manipulator arm 620, which is at the distal end portion of the containment area 640, may be mounted on a vertical actuator stage which may raise and lower the manipulator arm 620 relative to the containment area 640. This may provide the clearance necessary to enable the manipulator arm 620 to transfer more than one tote 650 or bin onto the containment area 640, such as stacked one on another, or as detailed above (multiple levels on the transport platform). Further, the articulated segments of the manipulator arm 620 may provide the clearance necessary to enable the manipulator arm 620 to transfer more than one tote 650 or bin onto the containment area 640 without the need or use of a vertical actuator stage.
Figure 1A and 1B)
the computer instructions including computer instructions to control the one or more robotic arms and the plurality of independently controllable drive elements as needed to pick items from a set of one or more source locations and place each item 25 in a corresponding destination location included in a set of one or more destination locations, including by using the independently controllable drive elements to move the mobile chassis within a space bounded at least in part by the one or more source locations and the one or more destination locations; and comprising computer instructions to: ([0014] The presently disclosed invention overcomes many of the shortcomings of the prior art by providing systems, devices and methods for robotic piece picking or put-away directly from existing stock item locations in a logistics facility. The presently disclosed invention provides a mobile robotic system that includes sensors and manipulator arm(s) to perceive, localize, reach, grasp and transfer SKUs from a storage rack to a transport container for piece picking, or conversely, from a transport container to a storage rack for put-away. This system and method allow existing facility infrastructure to remain intact and further allows the facility to use both manual picking and robotic picking interchangeably.
[0016] According to its major aspects, and briefly stated, the presently disclosed invention includes a system for piece picking or put-away within a logistics facility comprising a central server and at least one mobile manipulation robot. The logistics facility may be a warehouse, distribution center, manufacturing facility, or retail facility. The central server comprises a server communication interface, one or more server processors, and a server memory. Each of the mobile manipulation robots comprise a mobile base, at least one articulated manipulator arm having an end effector, at least one piece containment area, a plurality of sensors, a remote communication interface, a robot memory configured to store robot specific information, and one or more robot processors coupled to the sensors, the robot memory, the mobile base, and the at least one articulated manipulator arm.
[0112] As shown in zone 830, manipulation robots 874 may pick items, totes, or bins from standard shelving 840 and may transport those picks to a different zone within the logistics facility, such as to a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). The manipulation robots 873 may pick items, totes, or bins and may transfer those items to a transport robot 868. The transport robot 868 may then travel to a different location to deliver the item. Exemplary locations include a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). Alternatively, the transport robot 868 may transfer the items to another transport robot 867 for delivery to any of those locations.
[0128] In order to perform pick work, the manipulator robots 100 may move and navigate between pick locations in the work zone 330 and an order transfer area 360. During navigation the sensor data may be processed by the onboard computer processor 218 in a navigation software module 212 to extract two modalities of information. The first modality may be local mapping information that indicates which areas around the manipulation robot 100 are traversable and which areas contain obstacles. The ground facing sensors 150 on the manipulation robot 100 are primarily used to generate this mapping information and collision detection information. There may be two ground facing sensors 150, a front-facing one and a rear-facing one. This unique design allows the manipulation robot 100 to navigate while driving both forwards and backwards, which in certain picking scenarios, eliminates the need for the manipulation robot 100 to turn around, thus reducing travel time and increasing picking efficiency.
[0125] The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.
[0019] In certain embodiments of the system, the server memory may comprise computer program instructions executable by the one or more server processors to receive data from a warehouse management system and dispatch the at least one mobile manipulation robot. The server communication interface may connect with the remote communication interface to send and receive piece picking data which may include a unique identification for each piece to be picked, a location within the logistics facility of the pieces to be picked, and a route for the at least one mobile manipulation robot to take within the logistics facility. The unique identification for the piece to be picked may comprise a shape of the piece, a size of the piece, a weight of the piece, a color of the piece, a property of the construction material of the piece, such as roughness, porosity, and deformability, a visual marking on the piece, a barcode on the piece, or any combination thereof. Further, the connection between the server communication interface and the robot communication interface may be via one or more wired or wireless networks, or a combination thereof.
Figures 3-6 clearly show shelves and storage that would obstruct movement of the mobile chassis.)
control the plurality of independently controllable drive elements to move the mobile logistics robot in any direction, orientation, and path within the space bounded at least in part by the one or more source locations and the one or more destination locations; and ([0043] The present invention further relates to a method for autonomous robot navigation and region of interest localization, the method comprising: receiving data captured by a sensor coupled to a robotic device during navigation of the robotic device; analyzing the received data to detect at least one identifier corresponding to a region of interest; for each detected identifier: using the data to determine a current pose of the robotic device within a logistics facility; and generating a navigation instruction for navigation of the robotic device to a location of an item, the navigation instruction based on the current pose of the robotic device and a location of a region of interest at which the item is located.
[0117] The system's central server 200 may be used to process order information that is transacted with a WMS 201 and may coordinate the fulfillment of orders with a plurality of manipulation robots 100. All computation on the server 200 may be executed by one or more internal processors 220. In certain embodiments, the server may have two software modules that enable this order fulfillment coordination. The first processor may be a task dispatch module 228, which analyzes orders received from a WMS 201, and determines which of the plurality of manipulation robots 100 is to be assigned to an order. After a manipulation robot 100 is selected for picking an order, the task dispatcher 228 instructs the robot 100 with high-level order picking information, such as, route navigation paths, SKU locations, and an order drop-off location. The task dispatcher 228 works closely with a system state monitor 230 to obtain key feedback information from the system. The system state monitor 230 may communicate with the manipulation robots 100 to keep track of their current physical location within the facility, along with status information, which may include but is not limited to: whether the robot 100 is currently assigned an order, any faults or error modes, health information, such as remaining battery power, or charging status.
[0161] The manipulator robots 100 have a mobile base 160 that is controlled by the onboard computer processor 218. The mobile base may have two main drive wheels 167, each driven by a servo motor. Each drive wheel 167 may have an encoder that provides motion feedback, which is used to precisely control the speed of each wheel in order to achieve the desired rotation and translation velocities of the robot 100. The feedback data is also used for odometry to estimate the motion of the robot 100 relative to the facility. The odometry is responsible for guiding the robot 100 navigation at times when visual markers 420 are out of sensor (150, 110) range. The mobile base 160 may also use passive wheels, such as casters 165, for stability and weight distribution.
Figure 1A and 1B, the robot can clearly be controlled to move in any direction or orientation.)
Galluzzo does not teach generate for each item a set of plans to move the item from an associated source location to a corresponding destination location, each plan comprising a sequence of locations and poses; and
select from the set of plans a selected plan to move the item.
Oka, however, does teach generate for each item a set of plans to move the item from an associated source location to a corresponding destination location, each plan comprising a sequence of locations and poses; and
select from the set of plans a selected plan to move the item.
([0063] The grasp plan generator 54 calculates a grasping method and a grasping pose of the object OBJ at the initial position HP, and a moving route and via points along which the manipulator 20 or hand 22 is moved to the initial position HP. The grasp plan generator 54 also calculates a moving route and via points of the hand 22 to grasp a next intended object OBJ after releasing the object OBJ at the moving destination RP. In these cases, the object information acquired by the camera 32a is utilized in calculation of the moving route and via points to move the hand 22 without interfering with surrounding obstacles such as wall surfaces of the containers 14a and 14b or an object or objects other than the currently moved object OBJ.
[0086] To create the placement plan, for example, the route calculator 56 acquires, from the camera 32a and the laser range scanners 33a and 33b, input information including the grasping pose and the size of the object OBJ currently grasped by the hand 22 or suction pad 22a and to be moved and placed at the moving destination RP (container 14b) (S100). Subsequently, the route calculator 56 calculates the pose of the currently grasped object OBJ to place the object OBJ in the container 14b in accordance with the grasping pose and the size of the object OBJ (S102). The route calculator 56 calculates a placeable position and a pose of the grasped object OBJ, that is, a candidate for the moving destination RP on the basis of the status information of previously set objects OBJs in the container 14b (S104). The route calculator 56 calculates a plurality of patterns of position and pose candidates of a fingertip TCP of the hand 22 in placing the object OBJ, on the basis of information on the previously set object OBJs in the container 14b (S106), and selects an optimal position and pose candidate from the patterns (S108). The route calculator 56 also sets information including a target force value, a position of a pressing surface, and a pressing direction used in the force control by the manipulator 20 (hand 22) during movement or placement (S110). The route calculator 56 calculates via points (via position and pose RAP) on the moving route of the object OBJ from the size and the grasping pose of the object OBJ, a state of the previously set objects OBJs, the position and pose of the fingertip TCP at the time of placing the object OBJ (S112). The moving destination RP and the via position and pose RAP calculated as the candidates are associated with scores such as preset priority, and the route calculator 56 selects an optimum moving destination RP and via position and pose RAP according to the scores. After the route calculator 56 succeeds in generating the moving route not to interfere with obstacles such as the previously set objects OBJs or the container 14b, the robot controller 57 causes the manipulator 20 to operate.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Galluzzo in light of Oka such that for each item a set of plans is created including poses and locations and then one is selected because having the locations and poses for a trajectory would allow the system to understand if any collisions are possible, or if it will be effective. By choosing one, the system can select the option that is most likely to succeed and less likely to cause other issues.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Galluzzo in light of Oka in light of Tang et al (US Pub 2021/0276188 A1), hereafter known as Tang.
For Claim 9, Galluzzo teaches The mobile logistics robot of claim 1,
Galluzzo does not explicitly teach wherein the processor is configured to control one of the one or more robotic arms and the plurality of independently controllable drive elements to move an item continuously and smoothly through a planned trajectory.
Tang, however, does teach wherein the processor is configured to control one of the one or more robotic arms and the plurality of independently controllable drive elements to move an item continuously and smoothly through a planned trajectory. [0037] In the present example, the waypoints of trajectory 325 cause robotic arm 305 to move in a slow, jerky manner that may lead to unstable suction or grip on box 310. Alternatively, trajectory 330 is the result of a trajectory optimization process in accordance with the present disclosure. Trajectory 330 was generated using one or more trajectory optimization DNNs responsible for minimizing time while adhering to a set of constraints. Trajectory 330 does not include any way points and follows a continuous, smooth path from bin 315 to 310. A continuous, smooth path such as trajectory 330 allows robotic arm 305 to move faster because it avoids jerky movements that could cause robotic arm 305 to drop box 310 or reduce suction stability below an accepted threshold. Jerky movements may also cause damage to objects that may be moved by robotic arm 305. In addition to being a geometrically and dynamically optimized path, trajectory 330 does not cause any collisions or self-collisions and maintains suction stability on box 310.)
Therefore it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Galluzzo in light of Tang such that the trajectory is smooth and continuous because a smooth non jerky motion would reduce the change that the robotic arm drop the object, and it would allow it to move faster.
For Claim 10, Galluzzo teaches The mobile logistics robot of claim 9, wherein the planned trajectory starts at a pick location and ends at a place location that was not within reach of the robotic arm when the item was picked at the pick location. ([0085] After items are picked, they may be placed into the storage bed 140 for transportation. The bed may also carry a container 145, such as a box or tote, in which the items can be placed. This method enables multiple items to be picked for a given order or batch of orders. This method frees the robot manipulator arm 120 to pick additional items without needing to take multiple trips to and from an order transfer area 360 (See FIG. 3). Additionally, by carrying a packing box or container or transport tote 145 onboard, the manipulation robot 100 is able to aggregate order items together into a single container that can be easily swapped with a different container for additional order fulfillment.
[0112] As shown in zone 830, manipulation robots 874 may pick items, totes, or bins from standard shelving 840 and may transport those picks to a different zone within the logistics facility, such as to a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). The manipulation robots 873 may pick items, totes, or bins and may transfer those items to a transport robot 868. The transport robot 868 may then travel to a different location to deliver the item. Exemplary locations include a conveyor or a pack/ship zone (see 320 and 360 of FIG. 3). Alternatively, the transport robot 868 may transfer the items to another transport robot 867 for delivery to any of those locations.)
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Galluzzo in light of Oka in light of Hoffman et al (US Pub 2023/0415934 A1), hereafter known as Hoffman.
For Claim 11, Galluzzo teaches The mobile logistics robot of claim 1,
Galluzzo does not explicitly teach wherein the processor is further configured to control the one or more robotic arms and the plurality of independently controllable drive elements to pick an item using one of the one or more robotic arms, move with the item in the grasp of the robotic arm used to pick it to a location associated with a label printer, and manipulate the item, using the robotic arm, to cause a label printed by the label printer to be affixed to the item.
Hoffman, however, does teach wherein the processor is further configured to control the one or more robotic arms and the plurality of independently controllable drive elements to pick an item using one of the one or more robotic arms, move with the item in the grasp of the robotic arm used to pick it to a location associated with a label printer, and manipulate the item, using the robotic arm, to cause a label printed by the label printer to be affixed to the item.
([0019] The labeler 16 of the system 10 is configured to apply a label (e.g., a patient specific label) to the pharmaceutical containers C. In one embodiment, the labeler 16 may print and then apply the label to the pharmaceutical container C. The labeler 16 applies the label after the pharmaceutical container C has been picked by the container selector 14. Labelers are generally known in the art, and thus a further description of labeler 16 is omitted herein. For example, the labeler 16 may be a pass through labeler that applies the label to the pharmaceutical container C as the container is moved through the labeler by another component, such as the picker 44 or label transporter 18. After the container selector 14 grabs a pharmaceutical container C, the container selector 14 moves the pharmaceutical container generally towards the labeler 16. This movement may be accomplished by the picker 44 moving and/or the carriage 48 moving. In one embodiment (not shown), the container selector 14 (e.g., picker 44) may move the pharmaceutical container C to (and through) the labeler 16 to apply the label to the pharmaceutical container. In the illustrated embodiment, the container selector 14 moves the picked pharmaceutical container C to the container holder 20 of the system 10. The container holder 20 is configured to receive and hold a pharmaceutical container C from the container selector 14. In this embodiment, after picking a pharmaceutical container C, the container selector 14 moves the container and deposits (e.g., places) the container with (e.g., on) the holder By placing the pharmaceutical container C on the holder 20, instead of moving it directly to the labeler 16, the cycle time for the container selector 14 to pick a pharmaceutical container is reduced, allowing the system 10 to process more pharmaceutical containers in a given time frame. In the illustrated embodiment, the holder 20 includes a support platform or plate 62 defining a support surface on which the container selector 14 places the pharmaceutical container C. Desirably, the holder 20 is configured to hold (e.g., grip) the pharmaceutical container C. In the illustrated embodiment, the support platform 62 includes (e.g., defines) one or more openings or apertures 64 (e.g., vacuum ports) in the support surface that are fluidly coupled to a negative pressure source 68, such as a vacuum. The negative pressure source, via the openings 64, applies suction to the pharmaceutical container C to hold the container on the support platform 62. This way, the holder 20 inhibits the pharmaceutical container C form moving after the holder receives the pharmaceutical container from the container selector 14. In one embodiment, the holder 20 may be used to temporarily hold and store a container C for pre-staging with the labeler 16 (e.g., the staging of a container while another container is being labeled), for accommodating product back log and/or for accommodating product flow issues.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Galluzzo in light of Hoffman such that the robotic arm can manipulate the item through a labeling area because labels are useful to have in a warehouse, and there must be some conveyance method to get an object into and out of a labeling space or device. This is frequently performed by humans, so it would also be practical to have a robotic manipulator perform this simple task that humans can perform.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Galluzzo in light of Oka in light of Theobald et al (US Pub 2020/0033883 A1), hereafter known as Theobald.
For Claim 15, Galluzzo teaches The mobile logistics robot of claim 13,
Galluzzo does not teach wherein the processor uses the image or other sensor data generated by the camera or other sensor to divert the mobile chassis to a stow location not in the path of the human.
Theobald, however, does teach wherein the processor uses the image or other sensor data generated by the camera or other sensor to divert the mobile chassis to a stow location not in the path of the human.
([0033] The sensor system 39 includes one or more sensors 46 such as, for example, location sensors. These sensors 46 may be operated to spatially locate (e.g., triangulate) the mobile robot 30 relative to, for example, its surrounding environment, its geographic location and/or one or more locators. Examples of a locator include, but are not limited to, an RFID tag and a physical landmark. The sensors 46 may also or alternatively be operated to locate and/or identify other objects within the operating environment 33. Examples of a sensor 46 which may be included with the sensor system 39 include, but are not limited to, a proximity sensor, a global position system (GPS), a radar system, an infrared system, a laser system, a radio transceiver, and a visual location system with at least one optical camera.
[0053] To avoid a known or unknown obstacle (e.g., human, object or any other type of other entity) along the path, the controller 42 may signal the drive system 41 to slightly or dramatically divert its course around the obstacle based on data received from the sensor system 39. The controller 42 may also or alternatively signal the obstacle (e.g., a remotely actuated doorway) to partially or completely move or open. Upon arriving at the pickup location 34A, the controller 42 may signal the drive system 41 to stop and “park” the mobile robot 30 such that the conveyor system 38 may gather or otherwise receive the item 32 as described below.)
Therefore, it would be obvious to one of ordinary skill in the art prior to the effective filing date to modify Galluzzo in light of Theobald such that the system stows itself out of the way of a human because collisions with humans may injure the human and damage the equipment. By pausing activity of the robot and stowing it somewhere out of the way, it reduces the chance that a collision occurs, and prevents accidents from occurring.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Vu et al (US Pub 2018/0222052 A1) relates to safety related actions related to humans.
Ringart et al (US Pub 2024/0059485 A1) relates to a robotic arm that can assist in labeling.
Murphy et al (US Pub 2022/0305641 A1) relates to maneuverable wheels that provide a robot with flexible mobility.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRISTAN J GREINER whose telephone number is (571)272-1382. The examiner can normally be reached Mon - Fri 7:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tran Khoi can be reached at Monday-Thursday. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.J.G./Examiner, Art Unit 3656 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights