Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office action is in response to the amendment filed on 05/11/2026. Claims 1-20 are currently pending with claims 1-3, 8-12, and 17-19 being amended.
Response to Amendment
The amendments to the claims submitted on 05/11/2026 overcome the claim objections set forth in the previous Office action except for those set forth in the claim objection section.
Response to Arguments
Applicant's arguments filed 05/11/2026 have been fully considered but they are not persuasive. Regarding the applicants assertion that the prior art does not suggest that the known information is sufficient for retrieval operations, the examiner disagrees.
The currently provided claim language does not require no portion of the retrieval to be possible/performed. Under BRI, the retrieval may be started or partially completed before a determination that the known information is insufficient for successful operation. Further, the currently provided claim language does not preclude the generation of a new strategy for grasping, only that the known information is determined to be insufficient. The Applicant has asserted that Diankov does not suggest “determining that the stored operational data is insufficient for retrieval operations”. The Examiner disagrees and would note paragraphs 0169-0170 and 0181 of Diankov. These demonstrate the process taught by Diankov to include identifying when “the confidence measure does not satisfy the sufficiency threshold” (Diankov 0181) triggering different actions and control methods accordingly. These include calculating new control sequences and performing additional/advanced scanning of the object for further information. Combining these teachings with the method of creating, modifying, and analyzing information representing the facility and environment (in map format) taught by Galluzzo and with the ability to monitor the age of information representing the environment and identifying when portions should be updated to maintain an accurate representation (in map format) of the environment Bailly would be obvious to one of ordinary skill in the art. This combination would result in a system which may determine that the currently known information does not meet a confidence measure/sufficiency threshold and the system is unable to perform the desired task/operation (Diankov) based on an age of the information (Bailly) and triggering further action, (Diankov 0128) including modifying/gathering additional information (Galluzzo 0125) to update the known information for a successful operation.
Examiner notes wherein Applicant argues the newly amended limitations, which have not been addressed by the prior art of record. As such, Examiner has augmented the below rejection(s) in view of the prior art of record to address the newly amended limitations.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 5-12, and 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Galluzzo et al. (US 20200242544 A1), hereinafter Galluzzo in view of Diankov et al. (US 20240075619 A1), hereinafter Diankov, Bailly et al. (US 20210004363 A1), hereinafter Bailly, and Saito et al. (US 20210170601 A1), hereinafter Saito.
Regarding claim 1, Galluzzo teaches:
1. (Currently Amended) A retrieval apparatus comprising:
…
at least one moveable arm configured to engage a surface of at least one of a plurality of objects disposed on shelves, wherein the at least one moveable arm …
at least one sensor configured to obtain sensor data; (Paragraph 0014, "The presently disclosed invention provides a mobile robotic system that includes sensors and manipulator arm(s) to perceive, localize, reach, grasp and transfer SKUs from a storage rack to a transport container for piece picking, or conversely, from a transport container to a storage rack for put-away.")
a memory that stores computer-executable instructions; and
at least one processor in electronic communication with the at least one moveable arm and the at least one sensor, wherein the at least one processor is configured to execute the computer-executable instructions (Paragraph 0017, "The robot specific information may include at least calibration data for the plurality of sensors. The robot memory may comprise computer program instructions executable by the one or more robot processors to receive data from and send data to the central server, process data received from each of the sensors, and output control signals to the mobile base and the at least one articulated manipulator arm.") to:
… modify the stored operational data based at least in part on the sensor data. (Paragraph 0125, "The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.")
Galluzzo does not specifically discuss a housing which the arm retracts into or determining that the known information is not sufficient for performing a retrieval or basing this determination on an age of the data.
… a housing; … is disposed in the housing when the at least one moveable arm is in a retracted position; (Paragraph 0030, “FIG. 1 shows an overview of an external appearance of a conveyance robot 1 according to a first embodiment. As shown in FIG. 1, the conveyance robot 1 according to the first embodiment is configured such that a housing, which will be described later, is covered by a safety cover 10. A robot arm 21 is attached to the housing covered by the safety cover 10. The robot arm 21 picks up an object stored in a shelf by the arm being extended from and retracted in an arm opening part 11 provided in a predetermined surface of the safety cover 10. The storage boxes 22 and 23 are placed on the housing inside the safety cover 10. The storage boxes 22 and 23 can be taken in and out by, for example, a worker or by a mechanism of the conveyance robot 1.”)
However, Diankov, in the same field of endeavor of robotics, teaches:
… in response to determining that facilitating retrieval operations (Paragraph 0122, "As an illustrative example, in a case where the operation object 112 is the operation object 302 placed in the first pose 312 of FIG. 3A (in this case, the top surface 322 of the operation object 302 generally faces upward and is exposed) and the confidence measure for the pose is high (i.e., a degree of a certainty exceeds the sufficiency threshold and the determined pose is more likely accurate), the robotic system 100 can create or derive a first control sequence 422 that includes a first approach location 432 and a first presentation location 442. At this time, for example, since there is a sufficient certainty that the top surface 322 of the operation object 302 faces upward (i.e., the bottom surface 324 with the identifier 332 of the operation object 302 of FIG. 3C faces downward), the robotic system 100 can calculate the first control sequence 422 that includes the first approach location 432 for placing the end effector directly over the top surface 322 of the operation object 302." Please also see Paragraphs 0123-0126) …
However, Bailly, in the same field of endeavor of autonomous systems, teaches:
… when the stored operational data is associated with a timestamp that is prior to at least one of a time stamp associated with the sensor data and a threshold time period, (Paragraph 0206, "Outdated map alerts comprise notifications to the map data collection module 460, such as from the map update module 420, which indicate that a portion of an HD map is outdated and requires updating with new information. It is desirable for HD map data to be up to date. This requires at least periodic updating of the HD map data. Not all HD map data is of the same age, with some data having been collected earlier than other data. The online HD map system 110 may track how old HD map data is. For any map data unit (for example, lane element) the online HD map system 110 may record the newest and oldest timestamps when the data unit was processed and/or produced. An outdated map alert may be sent requesting a refresh of the map data unit if either the oldest timestamp or newest timestamp of that data unit is older than a respective threshold age. For example, if the oldest data is more than four weeks old, or if the newest data is over a week old, an outdated map alert may be sent requesting additional data to update the HD map. As described herein, any response to a map discrepancy could similarly be applied to addressing an outdated map alert.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and operating methods as taught by Galluzzo with the ability to identify outdated data and determine that an operation may not be performed as taught by Bailly and Diankov respectively and further with the safety cover/housing structure as taught by Saito. This would ensure operation of the system runs smoothly and the system does not spend time attempting to perform a task which it is incapable of or risk damage to the target object, environment or system by attempting an operation in a location where the environment has changed and therefor operation needs to be adjusted. Providing a safety cover/housing for the arm would also increase safety of the system and the efficiency by allowing the arm to operate with greater speed when within the enclosed environment.
Regarding claim 2, where all the limitations of claim 1 are discussed above, Galluzzo further teaches:
2. (Currently Amended) The retrieval apparatus of claim 1, wherein the at least one processor is further configured to, in response to receiving a retrieval request, (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.") cause the at least one moveable arm to engage a surface of at least one of the plurality of objects (Paragraph 0158, "To perform individual item picks, onboard sensors (150, 110) may be used to detect and localize individual items along with the specific locations and orientations of the grasp points 501 on the surface of an item. FIGS. 5A and 5B illustrate top and front views, respectively, of items stored on a shelf in a logistics facility. As shown, the manipulation robot 100 may use its end effector 175 and optionally an extension tool 170 to grasp an item at a specific point and orientation 501. During a pick, the sensors 110 also locate the infrastructure around the items such as shelving levels 500, rack columns, shelf faces and signage. This geometry information for the infrastructure is required for pick manipulation algorithms in a manipulation software module 214 to determine a pick trajectory that is collision free, such that the manipulation robot 100 is able to grasp the item without colliding with surrounding objects." as well as Paragraph 0079, "The present robot system uses a grasping end effector 175 on the manipulator arm 120 to pick items from their stored location and transfer them to a temporary location, or vice-versa. In a preferred embodiment, the grasping end effector may be a suction cup 175, which may be connected to a vacuum pump through an onboard computer processor 218 controlled valve. The vacuum suction at the suction cup 175 may be engaged and disengaged by actuating the valve, thereby allowing the manipulation robot 100 to grasp the desired pick item on contact and then release it when necessary. The use of a suction cup 175 also allows the robot to grasp a target item at a single point and orientation, which reduces the computation time required for the system to determine how to grasp the pick item.") based at least in part on the modified operational data. (Paragraph 0125, "The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.")
Regarding claim 3, where all the limitations of claim 2 are discussed above, Galluzzo further teaches:
3. (Currently Amended) The retrieval apparatus of claim 2, wherein the retrieval apparatus comprises a receiving frame, (Paragraph 0086, "In certain embodiments, the storage bed 140 may comprise a calibration target which may be viewed by one or more sensors 110 placed at a central location on the manipulator arm 120 (see FIG. 1). These centrally located sensors 110 may be positioned to view the calibration target on the storage bed 140 when the manipulator arm is rotated. As such, information on the calibration target may be used to calibrate these sensors 110 to ensure that all parameters are within specifications, and if not, update the parameters to reflect the current configuration. The dual use of the storage bed 140—as a platform to hold picked items and as a calibration target—reduces the size profile of the presently disclosed manipulation robot and improves the accuracy of the system.") and wherein the at least one moveable arm is configured to retrieve at least one of the plurality of objects and place it on the receiving frame. (Paragraph 0095, "According to such a configuration, the manipulation robot 600 may hold a tote 650 on the mobile portion of the containment area 640 during picking operations, and may move that tote 650 to one of the stationary levels of the platform when the picking operation is complete by aligning the mobile portion of the containment area 640 with the stationary level holding the tote of interest. Transfer may be actuated by the manipulator arm 620 or by a conveyance system on either or both of the stationary/mobile platforms. The manipulation robot 600 may select another empty or partially filled tote 650 (i.e., incomplete pick order) from another of the stationary levels of the platform, align the mobile portion of the containment area 640 with the stationary level holding the tote of interest, transfer that tote/bin to the mobile portion of the containment area 640, and start/continue a different pick order. In this way, the manipulation robot 600 may complete several orders at one time without the need to unload/travel to the staging area or a conveyor.")
Regarding claim 5, where all the limitations of claim 1 are discussed above, Galluzzo further teaches:
5. (Original) The retrieval apparatus of claim 1, wherein the at least one sensor comprises one or more light detection and ranging (LiDAR) sensors, radio detection and ranging (RADAR) sensors, infrared (IR) cameras, 3D cameras, 360° cameras, photoelectric sensors or proximity sensors. (Paragraph 0020, "In embodiments of the system, the at least one mobile manipulation robot may be able to autonomously navigate and position itself within the logistics facility by recognition of at least one landmark by at least one of the plurality of sensors. The landmark may be a vertically mounted marker placed at a specific location within the logistics facility or may be other identifiable visual or audible landmarks within the logistics facility. The sensors may be any 3D device capable of sensing the local environment such as, for example, 3D depth cameras, color cameras, grey scale cameras, laser ranging devices, sonar devices, radar devices, or combinations thereof.")
Regarding claim 6, where all the limitations of claim 1 are discussed above, Galluzzo further teaches:
6. (Original) The retrieval apparatus of claim 1, wherein the at least one sensor is attached to at least a surface of the at least one moveable arm. (Paragraph 0087, "The central location of one or more sensors 110 on the manipulator arm 120 allows for improved item picking accuracy. First, these sensors 110 will have an unobstructed and enlarged view of the items to be picked and their storage locations, which is improved over the view that is provided by sensors placed at the end of a robotic arm or on the mobile base, as is the case in many prior art systems. Furthermore, sensors mounted at the end of a robotic arm may get in the way of, or reduce the range of positions available to, an end effector. Sensors mounted on the mobile base may have their view of the items to be picked and their storage locations impeded by movement of the robotic arm.")
Regarding claim 7, where all the limitations of claim 1 are discussed above, Galluzzo further teaches:
7. (Original) The retrieval apparatus of claim 1, wherein the at least one sensor comprises at least one location sensor. (Paragraph 0126, "The manipulation robots may also dynamically upload mapping data automatically collected by their various sensors. That is, the one or more robots may collect image data and utilize simultaneous localization and mapping (SLAM) to generate a map of the logistics facility using the location of various markers (e.g., see 420 in FIG. 4) dispersed throughout the warehouse. SLAM provides real-time construction/updating of a map of an unknown environment while navigating through that environment. Popular SLAM approximate solution methods include particle filter, extended Kalman filter, and GraphSLAM.")
Regarding claim 8, where all the limitations of claim 1 are discussed above, Galluzzo further teaches:
8. (Currently Amended) The retrieval apparatus of claim 1, wherein a retrieval request comprises the stored operational data, and wherein the retrieval request is provided by a computing entity in electronic communication with the retrieval apparatus. (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.")
Regarding claim 9, where all the limitations of claim 1 are discussed above, Galluzzo further teaches:
9. (Currently Amended) The retrieval apparatus of claim 1, wherein the stored operational data comprises object location information and object characteristics associated with the plurality of objects disposed on the shelves. (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.")
Regarding claim 10, where all the limitations of claim 9 are discussed above, Galluzzo further teaches:
10. (Currently Amended) The retrieval apparatus of claim 9, wherein the at least one processor is further configured to … and transmit an indication requesting additional operational data in an instance in which the stored operational data is deemed insufficient for facilitating the retrieval operations. (Paragraph 0123 "If the SKU at the location is close in characteristics to the expected SKU, or if the target location includes SKU's having similar characteristics stored side-by-side, the manipulation robot may request clarification from the central server 200 and/or WMS 201, or may request assistance from a human worker. For example, the request may be displayed on a display screen (130, 630) of the manipulation robot (100 and 600, respectively), and the human worker may indicate the correct pick for the robot. Alternatively, the request may be displayed on a remote display screen, and a human may indicate the correct pick for the robot.")
Galluzzo does not specifically teach determining if the retrieval may be performed based on the known data. However, Diankov, in the same field of endeavor of robotics, teaches:
… determine whether the stored operational data is sufficient for facilitating the retrieval operations, (Paragraph 0122, "As an illustrative example, in a case where the operation object 112 is the operation object 302 placed in the first pose 312 of FIG. 3A (in this case, the top surface 322 of the operation object 302 generally faces upward and is exposed) and the confidence measure for the pose is high (i.e., a degree of a certainty exceeds the sufficiency threshold and the determined pose is more likely accurate), the robotic system 100 can create or derive a first control sequence 422 that includes a first approach location 432 and a first presentation location 442. At this time, for example, since there is a sufficient certainty that the top surface 322 of the operation object 302 faces upward (i.e., the bottom surface 324 with the identifier 332 of the operation object 302 of FIG. 3C faces downward), the robotic system 100 can calculate the first control sequence 422 that includes the first approach location 432 for placing the end effector directly over the top surface 322 of the operation object 302." Please also see Paragraphs 0123-0126) …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and operating methods as taught by Galluzzo with the ability to determine that an operation may not be performed as taught by Diankov. This would ensure operation of the system runs smoothly and the system does not spend time attempting to perform a task which it is incapable of, risking damage to the target object, environment or system.
Regarding claim 11, where all the limitations of claim 1 are discussed above, Galluzzo further teaches:
11. (Currently Amended) The retrieval apparatus of claim 1, wherein the at least one processor is further configured to transmit at least a portion of the sensor data to a computing entity. (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.")
Regarding claim 12, Galluzzo further teaches:
12. (Currently Amended) A method for dynamically retrieving a plurality of objects by a retrieval apparatus, the retrieval apparatus comprising … at least one moveable arm, at least one sensor, a memory having computer-executable instructions, and at least one processor configured to execute the computer-executable instructions to control operations (Paragraph 0017, "The robot specific information may include at least calibration data for the plurality of sensors. The robot memory may comprise computer program instructions executable by the one or more robot processors to receive data from and send data to the central server, process data received from each of the sensors, and output control signals to the mobile base and the at least one articulated manipulator arm.") for dynamically retrieving the plurality of objects by the retrieval apparatus, the method comprising:
obtaining, by the at least one sensor, sensor data; (Paragraph 0014, "The presently disclosed invention provides a mobile robotic system that includes sensors and manipulator arm(s) to perceive, localize, reach, grasp and transfer SKUs from a storage rack to a transport container for piece picking, or conversely, from a transport container to a storage rack for put-away.")
…
modifying, by the at least one processor, the stored operational data based at least in part on the sensor data; (Paragraph 0125, "The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.") and
causing, by the at least one processor, in response to receiving a retrieval request, (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.") (Paragraph 0158, "To perform individual item picks, onboard sensors (150, 110) may be used to detect and localize individual items along with the specific locations and orientations of the grasp points 501 on the surface of an item. FIGS. 5A and 5B illustrate top and front views, respectively, of items stored on a shelf in a logistics facility. As shown, the manipulation robot 100 may use its end effector 175 and optionally an extension tool 170 to grasp an item at a specific point and orientation 501. During a pick, the sensors 110 also locate the infrastructure around the items such as shelving levels 500, rack columns, shelf faces and signage. This geometry information for the infrastructure is required for pick manipulation algorithms in a manipulation software module 214 to determine a pick trajectory that is collision free, such that the manipulation robot 100 is able to grasp the item without colliding with surrounding objects." as well as Paragraph 0079, "The present robot system uses a grasping end effector 175 on the manipulator arm 120 to pick items from their stored location and transfer them to a temporary location, or vice-versa. In a preferred embodiment, the grasping end effector may be a suction cup 175, which may be connected to a vacuum pump through an onboard computer processor 218 controlled valve. The vacuum suction at the suction cup 175 may be engaged and disengaged by actuating the valve, thereby allowing the manipulation robot 100 to grasp the desired pick item on contact and then release it when necessary. The use of a suction cup 175 also allows the robot to grasp a target item at a single point and orientation, which reduces the computation time required for the system to determine how to grasp the pick item.") based at least in part on , (Paragraph 0125, "The markers for each of these redundant storage locations or slots would not be the same. The central server 200 may store information about the infrastructure of the facility of operation in a map storage database 254. This can include information about the storage racks 310 such as shelving dimensions (width, depth and height), separate shelf level heights, shelf face widths, and rack column widths. The infrastructure information can be created, modified and analyzed through a map creation software module 224 on the central server 200. By using this module a human operator can manually create a facility map or may in some embodiments load the map data from a predefined file, such as a Computer Aided Drawing (CAD) file, or in other embodiments may load mapping data automatically collected by a robot 100, which can use its onboard sensors (150, 110), to observe the facility infrastructure and automatically generate a map.") …
Galluzzo does not specifically discuss a housing which the arm retracts into or determining that the known information is not sufficient for performing a retrieval or basing this determination on an age of the data. However, Saito, in the same field of endeavor of robotics, teaches:
… a housing, … wherein the at least one moveable arm is disposed in the housing when the at least one moveable arm is in a retracted position. (Paragraph 0030, “FIG. 1 shows an overview of an external appearance of a conveyance robot 1 according to a first embodiment. As shown in FIG. 1, the conveyance robot 1 according to the first embodiment is configured such that a housing, which will be described later, is covered by a safety cover 10. A robot arm 21 is attached to the housing covered by the safety cover 10. The robot arm 21 picks up an object stored in a shelf by the arm being extended from and retracted in an arm opening part 11 provided in a predetermined surface of the safety cover 10. The storage boxes 22 and 23 are placed on the housing inside the safety cover 10. The storage boxes 22 and 23 can be taken in and out by, for example, a worker or by a mechanism of the conveyance robot 1.”)
However, Diankov, in the same field of endeavor of robotics, teaches:
… determining, by the at least one processor, that the stored operational data is insufficient for facilitating retrieval operations (Paragraph 0122, "As an illustrative example, in a case where the operation object 112 is the operation object 302 placed in the first pose 312 of FIG. 3A (in this case, the top surface 322 of the operation object 302 generally faces upward and is exposed) and the confidence measure for the pose is high (i.e., a degree of a certainty exceeds the sufficiency threshold and the determined pose is more likely accurate), the robotic system 100 can create or derive a first control sequence 422 that includes a first approach location 432 and a first presentation location 442. At this time, for example, since there is a sufficient certainty that the top surface 322 of the operation object 302 faces upward (i.e., the bottom surface 324 with the identifier 332 of the operation object 302 of FIG. 3C faces downward), the robotic system 100 can calculate the first control sequence 422 that includes the first approach location 432 for placing the end effector directly over the top surface 322 of the operation object 302." Please also see Paragraphs 0123-0126) …
However, Bailly, in the same field of endeavor of autonomous systems, teaches:
… when the stored operational data is associated with a timestamp that is prior to at least one of a time stamp associated with the sensor data and a threshold time period; (Paragraph 0206, "Outdated map alerts comprise notifications to the map data collection module 460, such as from the map update module 420, which indicate that a portion of an HD map is outdated and requires updating with new information. It is desirable for HD map data to be up to date. This requires at least periodic updating of the HD map data. Not all HD map data is of the same age, with some data having been collected earlier than other data. The online HD map system 110 may track how old HD map data is. For any map data unit (for example, lane element) the online HD map system 110 may record the newest and oldest timestamps when the data unit was processed and/or produced. An outdated map alert may be sent requesting a refresh of the map data unit if either the oldest timestamp or newest timestamp of that data unit is older than a respective threshold age. For example, if the oldest data is more than four weeks old, or if the newest data is over a week old, an outdated map alert may be sent requesting additional data to update the HD map. As described herein, any response to a map discrepancy could similarly be applied to addressing an outdated map alert.") …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and operating methods as taught by Galluzzo with the ability to identify outdated data and determine that an operation may not be performed as taught by Bailly and Diankov respectively and further with the safety cover/housing structure as taught by Saito. This would ensure operation of the system runs smoothly and the system does not spend time attempting to perform a task which it is incapable of or risk damage to the target object, environment or system by attempting an operation in a location where the environment has changed and therefor operation needs to be adjusted. Providing a safety cover/housing for the arm would also increase safety of the system and the efficiency by allowing the arm to operate with greater speed when within the enclosed environment.
Regarding claim 14, where all the limitations of claim 12 are discussed above, Galluzzo further teaches:
14. (Original) The method of claim 12, wherein the at least one sensor comprises one or more light detection and ranging (LiDAR) sensors, radio detection and ranging (RADAR) sensors, infrared (IR) cameras, 3D cameras, 360° cameras, photoelectric sensors or proximity sensors. (Paragraph 0020, "In embodiments of the system, the at least one mobile manipulation robot may be able to autonomously navigate and position itself within the logistics facility by recognition of at least one landmark by at least one of the plurality of sensors. The landmark may be a vertically mounted marker placed at a specific location within the logistics facility or may be other identifiable visual or audible landmarks within the logistics facility. The sensors may be any 3D device capable of sensing the local environment such as, for example, 3D depth cameras, color cameras, grey scale cameras, laser ranging devices, sonar devices, radar devices, or combinations thereof.")
Regarding claim 15, where all the limitations of claim 12 are discussed above, Galluzzo further teaches:
15. (Original) The method of claim 12, wherein the at least one sensor is attached to at least a surface of the at least one moveable arm. (Paragraph 0087, "The central location of one or more sensors 110 on the manipulator arm 120 allows for improved item picking accuracy. First, these sensors 110 will have an unobstructed and enlarged view of the items to be picked and their storage locations, which is improved over the view that is provided by sensors placed at the end of a robotic arm or on the mobile base, as is the case in many prior art systems. Furthermore, sensors mounted at the end of a robotic arm may get in the way of, or reduce the range of positions available to, an end effector. Sensors mounted on the mobile base may have their view of the items to be picked and their storage locations impeded by movement of the robotic arm.")
Regarding claim 16, where all the limitations of claim 12 are discussed above, Galluzzo further teaches:
16. (Original) The method of claim 12, wherein the at least one sensor comprises at least one location sensor. (Paragraph 0126, "The manipulation robots may also dynamically upload mapping data automatically collected by their various sensors. That is, the one or more robots may collect image data and utilize simultaneous localization and mapping (SLAM) to generate a map of the logistics facility using the location of various markers (e.g., see 420 in FIG. 4) dispersed throughout the warehouse. SLAM provides real-time construction/updating of a map of an unknown environment while navigating through that environment. Popular SLAM approximate solution methods include particle filter, extended Kalman filter, and GraphSLAM.")
Regarding claim 17, where all the limitations of claim 12 are discussed above, Galluzzo further teaches:
17. (Currently Amended) The method of claim 12, wherein the retrieval request comprises the stored operational data, and wherein the retrieval request is provided by a computing entity in electronic communication with the retrieval apparatus. (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.")
Regarding claim 18, where all the limitations of claim 12 are discussed above, Galluzzo further teaches:
18. (Currently Amended) The method of claim 12, wherein the stored operational data comprises object location information and object characteristics associated with the plurality of objects. (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.")
Regarding claim 19, where all the limitations of claim 12 are discussed above, Galluzzo further teaches:
19. (Currently Amended) The method of claim 12, further comprising:
… and
transmitting, by the at least one processor, an indication requesting additional operational data in an instance in which the stored operational data is deemed insufficient for facilitating the retrieval operations. (Paragraph 0123 "If the SKU at the location is close in characteristics to the expected SKU, or if the target location includes SKU's having similar characteristics stored side-by-side, the manipulation robot may request clarification from the central server 200 and/or WMS 201, or may request assistance from a human worker. For example, the request may be displayed on a display screen (130, 630) of the manipulation robot (100 and 600, respectively), and the human worker may indicate the correct pick for the robot. Alternatively, the request may be displayed on a remote display screen, and a human may indicate the correct pick for the robot.")
Galluzzo does not specifically teach determining if the retrieval may be performed based on the known data. However, Diankov, in the same field of endeavor of robotics, teaches:
… determining, by the at least one processor, whether the stored operational data is sufficient for facilitating the retrieval operations; (Paragraph 0122, "As an illustrative example, in a case where the operation object 112 is the operation object 302 placed in the first pose 312 of FIG. 3A (in this case, the top surface 322 of the operation object 302 generally faces upward and is exposed) and the confidence measure for the pose is high (i.e., a degree of a certainty exceeds the sufficiency threshold and the determined pose is more likely accurate), the robotic system 100 can create or derive a first control sequence 422 that includes a first approach location 432 and a first presentation location 442. At this time, for example, since there is a sufficient certainty that the top surface 322 of the operation object 302 faces upward (i.e., the bottom surface 324 with the identifier 332 of the operation object 302 of FIG. 3C faces downward), the robotic system 100 can calculate the first control sequence 422 that includes the first approach location 432 for placing the end effector directly over the top surface 322 of the operation object 302." Please also see Paragraphs 0123-0126) …
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and operating methods as taught by Galluzzo with the ability to determine that an operation may not be performed as taught by Diankov. This would ensure operation of the system runs smoothly and the system does not spend time attempting to perform a task which it is incapable of, risking damage to the target object, environment or system.
Regarding claim 20, where all the limitations of claim 12 are discussed above, Galluzzo further teaches:
20. (Original) The method of claim 12, further comprising:
transmitting, by the at least one processor, at least a portion of the sensor data to a computing entity. (Paragraph 0039 "The present invention further relates to a method for executing an order fulfillment operation within a logistics facility, the method comprising: receiving at a remote server via a server communication interface at least one order including at least one item to be picked; generating at the remote server a task list based on the at least one order, wherein the task list includes a unique identification for the at least one item, and a first location within the logistics facility for the at least one item; sending, from the remote server via the server communication interface, the task list to a server of a manipulation robot; and moving the manipulation robot autonomously along a route within the logistics facility to a first location for the at least one item, and picking the at least one item from the first location using an arm of the manipulation robot, wherein at least one sensor on the manipulation robot provides signals related to detection, identification, and location of the at least one item to be picked, and one or more processors of the server of the manipulation robot analyze the signals to generate arm control signals to guide an end effector of the arm to pick the at least one item with a collision free path throughout a controlled motion of the arm.")
Claim(s) 4 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Galluzzo in view of Saito, Diankov, and Bailly and in further view of Elazary et al. (US 20180005173 A1), hereinafter Elazary.
Regarding claim 4, where all the limitations of claim 1 are discussed above, Galluzzo does not specifically teach determining the spacing between objects. However, Elazary, in the same field of endeavor of robotics, teaches:
4. (Original) The retrieval apparatus of claim 1, wherein the sensor data describes at least a first gap between at least two of the plurality of objects in a first direction, and at least a second gap between the at least two of the plurality of objects in a second direction that is parallel to the first direction. (Paragraph 0041, “In some embodiments, the robot 310 accounts for gaps between adjacent items in the row to more accurately compute the horizontally arranged item inventory. In some such embodiments, when the robot 310 processes the image to compute the total width and minimum distance between the repeating feature, the robot 310 also detects and measures any gaps between the items. The gaps are distinguishable because of a lack of any pattern appearing in those spaces. Therefore, the robot computes the total width between two items at either end of the row and subtracts any measured gaps from the total. The robot computes the minimum distance between the repeating feature and subtracts any gap between the two items. The robot then computes the number of items in the row using the gap modified total width and the gap modified minimum distance between the repeating feature.” Please also see Figure 9)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and methods of operation as taught by Galluzzo with the ability to detect the spacing between objects as taught by Elazary. This would allow the system to accurately control the end effector to grasp objects and remove them from stacks of objects without disrupting the objects other than the target.
Regarding claim 13, where all the limitations of claim 12 are discussed above, Galluzzo does not specifically teach determining the spacing between objects. However, Elazary, in the same field of endeavor of robotics, teaches:
13. (Original) The method of claim 12, wherein the sensor data describes at least a first gap between at least two of the plurality of objects in a first direction, and at least a second gap between the at least two of the plurality of objects in a second direction that is parallel to the first direction. (Paragraph 0041, “In some embodiments, the robot 310 accounts for gaps between adjacent items in the row to more accurately compute the horizontally arranged item inventory. In some such embodiments, when the robot 310 processes the image to compute the total width and minimum distance between the repeating feature, the robot 310 also detects and measures any gaps between the items. The gaps are distinguishable because of a lack of any pattern appearing in those spaces. Therefore, the robot computes the total width between two items at either end of the row and subtracts any measured gaps from the total. The robot computes the minimum distance between the repeating feature and subtracts any gap between the two items. The robot then computes the number of items in the row using the gap modified total width and the gap modified minimum distance between the repeating feature.” Please also see Figure 9)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the robotic system and methods of operation as taught by Galluzzo with the ability to detect the spacing between objects as taught by Elazary. This would allow the system to accurately control the end effector to grasp objects and remove them from stacks of objects without disrupting the objects other than the target.
Conclusion
The Examiner has cited particular paragraphs or columns and line numbers in the referencesapplied to the claims above for the convenience of the Applicant. Although the specified citations arerepresentative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested of the Applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP 2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole, including portions that would lead away from the claimed Invention. W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851 (1984). See also MPEP §2123.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
/H.J.K./Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights